Merit Review Frequently Asked Questions (FAQs)
Broader Impacts
- What happened to the document that contained examples illustrating activities likely to demonstrate broader impacts?
- Where can I find text that defines broader impacts?
- What are some elements of a well-written broader impacts section?
- What is the PIs responsibility for developing metrics to assess their broader impacts outcomes and provide those at project report time?
Merit Review Principles, Criteria, and Elements
- How are merit review principles to be used by Principal Investigators, reviewers, or Program Officers?
- What does "in the aggregate" mean in the second merit review principle?
- The third review element asks whether the plan is well-reasoned and incorporates a mechanism to assess success. Does this mean that PIs will want to lay out plans for assessment of their Intellectual Merit and Broader Impacts activities? Will reviewers be expected to comment on whether the proposal includes plans for assessment of these activities and whether they are sound?
- There seems to be a tension between the third merit review principle and the third review element. There is not a need for individual project assessment, but the PI must have a plan to assess success?
- In the new list of elements to consider in the review, there are 1(a), 1(b), 2, 3, 4, and 5. Are these intended to mean that in evaluation of the Intellectual Merit of the project, elements 1(a), 2, 3, 4, and 5 are to be used, and in evaluation of the Broader Impacts of a project, elements 1(b), 2, 3, 4, and 5 are to be used? Or does this mean that in evaluation of the Broader Impacts of a project, only 1(b) need be considered?
- Review element #2 asks: To what extent do the proposed activities suggest and explore creative, original, or potentially transformative concepts? How does this apply to Broader Impacts?
- The second merit review principle states that, The project activities may be based on previously established and/or innovative methods and approaches, but in either case must be well justified. How does this relate to the second review element regarding creative, original, or potentially transformative concepts?
- Are there weights assigned to the review criteria?
Project Description
Certification Regarding Organizational Support
Broader Impacts
1. What happened to the document that contained examples illustrating activities likely to demonstrate broader impacts
The list of examples illustrating activities likely to demonstrate broader impacts has been removed as of January 14, 2013. NSF does not want to provide undue influence to proposers regarding what their likely broader impacts activities might be or imply that the exemplary activities are in any way proscriptive.
2. Where can I find text that defines broader impacts?
The Grant Proposal Guide, Chapter II.C.2.d contains information regarding broader impacts; however the list of outcomes is not all inclusive nor definitive. Broader impacts may be accomplished through the research itself, through the activities that are directly related to specific research projects, or through activities that are supported by, yet are complementary to the project. The Foundations goal is to encourage thoughtful development of ideas in the community so that strong activities addressing broader impacts are brought forward in proposals. Hence, the PI is responsible for proposing what broader impacts may result from a project.
3. What are some elements of a well-written broader impacts section?
A well-written broader impacts section should include activities that are clearly described; have a well-justified rationale; and demonstrate creativity or originality, or have a basis in established approaches. The proposer should have a well-organized strategy for accomplishment of clearly stated goals; establish the qualifications of those responsible for the activities; and demonstrate sufficient resources for support. A plan should be in place to document the results.
4. What is the PIs responsibility for developing metrics to assess their broader impacts outcomes and provide those at project report time?
PIs are expected to be accountable for carrying out the activities described in the funded project. Individual projects should include clearly stated goals, specific descriptions of activities that the PI intends to do, and a plan in place to document the outputs of those activities. The annual and final project reports should address progress in all activities of the project, including any activities intended to address the Broader Impacts criterion that are not intrinsic to the research.
Merit Review Principles, Criteria, and Elements
5. How are merit review principles to be used by Principal Investigators, reviewers, or Program Officers?
The Merit Review Principles help explain NSFs overall mission and describe concepts that the Foundation considers when ascertaining progress toward its mission as outlined in the NSF Strategic Plan. Consequently, they provide context for the Merit Review Criteria (Intellectual Merit and Broader Impacts), to help proposers, reviewers and NSF staff fully understand their intent. As the community uses the criteria in the development and evaluation of NSF proposals, the principles should be used as a guide. PIs and organizations should understand what the principles are to ensure that their proposed activities align with them and help NSF achieve its goals. NSF program staff should consider the principles when determining whether or not to recommend proposals for funding and while overseeing awards.
6. What does "in the aggregate" mean in the second merit review principle?
The second merit review principle states, NSF projects, in the aggregate, should contribute more broadly to achieving societal goals. These Broader Impacts activities may be accomplished through the research itself, through activities that are directly related to specific research projects, or through activities that are supported by, but are complementary to, the project.
Both the America COMPETES Reauthorization Act of 2010 and the NSF Strategic Plan emphasize the value of broader impacts of scientific research, beyond the intrinsic importance of advancing scientific knowledge. NSF recognizes that broader impacts activities may vary from project to project. Such activities may be tied to scientific outcomes inherent to the research or societal outcomes that are complementary to the project. NSF also recognizes that individual projects by themselves are not likely to achieve societal goals. However, there is strength in numbers, and so in the aggregate, NSF projects that address similar goals should help make a difference in achieving those goals. Thus, NSF programs, divisions, and directorates/offices must ensure that their portfolios of funded projects fulfill this principle.
7. The third review element asks whether the plan is well-reasoned and incorporates a mechanism to assess success. Does this mean that PIs will want to lay out plans for assessment of their Intellectual Merit and Broader Impacts activities? Will reviewers be expected to comment on whether the proposal includes plans for assessment of these activities and whether they are sound?
NSF expects PIs to be accountable for carrying out the activities described in the funded project, i.e., there is an expectation that within individual projects, there are clearly stated goals, specific descriptions of the PIs intended activities, and a plan in place to document the results. Reviewers are asked to consider what the proposers want to do, how they plan to do it, how they will know if they succeed, and what benefits could accrue if the project is successful.
8. There seems to be a tension between the third merit review principle and the third review element. Although there is not a need for individual project assessment, the PI must have a plan to assess success. Can NSF explain this apparent discrepancy?
PIs are expected to be accountable for carrying out the activities described in the funded project. Individual projects should include clearly stated goals, specific descriptions of activities that the PI intends to do, and a plan in place to document the outputs of those activities. A distinction should be made between such an accounting of outputs (the third review element) and an assessment of outcomes (the third merit review principle). PIs may propose to include funds for evaluation of individual projects, although individual project evaluations are not required for every project. NSF is exploring ways in which it may support assessment and evaluation at a higher level, e.g., at a program or institutional level.
9. In the new list of elements to consider in the review, there are 1(a), 1(b), 2, 3, 4, and 5. Are these intended to mean that in evaluation of the Intellectual Merit of the project, elements 1(a), 2, 3, 4, and 5 are to be used, and in evaluation of the Broader Impacts of a project, elements 1(b), 2, 3, 4, and 5 are to be used? Or does this mean that in evaluation of the Broader Impacts of a project, only 1(b) need be considered?
Five elements are to be considered in the review of each of the two criteria. For Intellectual Merit, element 1(a) applies, and for Broader Impacts, element 1(b) applies. For both Intellectual Merit and Broader Impacts, elements 2-5 also apply.
10. Review element #2 asks: To what extent do the proposed activities suggest and explore creative, original, or potentially transformative concepts? How does this apply to Broader Impacts?
The PI should describe the broader impacts derived directly or indirectly from the proposed project (see #2 above), including any creative, original, or potentially transformative approaches and/or expected outcomes. For example, if a PI proposed to engage students in a massive open online course (MOOC) an approach and evaluate the resulting student learning an outcome, the broader impact could be better understanding how technology can be used to improve STEM education.
11. The second merit review principle states that, The project activities may be based on previously established and/or innovative methods and approaches, but in either case must be well justified. How does this relate to the second review element regarding creative, original, or potentially transformative concepts?
NSF seeks to support projects that have the potential to advance, if not transform, the frontiers of knowledge. Although a project may appear to have the potential to transform knowledge at the time it is being proposed, its not always possible to know which projects will have transformative outcomes. Transformative results may not be evident until after the project is completed. Regardless of whether or not a proposed project has the potential to be transformative, NSF projects should be of the highest quality, including methods and approaches that are well justified. Also note that the second review element says, creative, original, or potentially transformative concepts. Not all projects are expected to include all three characteristics.
12. Are there weights assigned to the review criteria?
No. Weights have not been assigned to the review criteria. Both criteria are important and should be given full consideration during the development of the proposal, its review, and the decision-making process.
Project Description
13. The Project Description now requires separate sections with a discussion of the broader impacts of the proposed activities and Results from Prior Support, if applicable. Is the lack of these explicit sections cause for return without review?
Yes. Proposals that are not in compliance with these requirements will be returned without review.
Certification Regarding Organizational Support
14. What does the new organizational support certification mean for our organization?
The organizational support certification addresses a requirement described in Section 526 of the America COMPETES Reauthorization Act of 2010, specifically that there must be evidence of institutional support for the proposals broader impacts activities. Because the merit review criteria are intertwined, NSF expanded this certification to include evidence of organizational support for intellectual merit activities. The certification does not imply an organizational commitment beyond what is already anticipated at the time of proposal submission, that is, if funded, an organization will provide the support necessary to ensure that the proposed activities will be implemented successfully.