NSF & Congress
Dr. Cora Marrett
National Science Foundation
Before the U.S. House of Representatives
Committee on Science, Space, Technology
Subcommittee on Research and Science Education
The Merit Review Process:
Ensuring Limited Federal Resources are Invested in the Best Science
July 26, 2011
Chairman Brooks, Ranking Member Lipinski, and distinguished Members of the Subcommittee, thank you for inviting me to participate in this hearing on "The Merit Review Process."
I am delighted to discuss the National Science Foundation's (NSF) Merit Review Process with you. As you well know, NSF is the primary Federal agency supporting research at the frontiers of knowledge, across all fields of science and engineering (S&E) and all levels of S&E education. Its mission, vision and goals are designed to maintain and strengthen the vitality of the U.S. science and engineering enterprise. As part of the overall national R&D enterprise, the basic research and education activities supported by NSF are vital to the economic advancement of the U.S. and provide the know-how that allows the U.S. to respond rapidly and effectively to a range of unexpected challenges. The NSF merit review process lies at the heart of the agency's strategy for accomplishing its overall mission. As such, NSF is continuously striving to maintain and improve the quality and transparency of the process.
Before I begin my discussion of the unique elements of the NSF merit review system, let me first describe the essential features of merit review writ large. In general, merit review refers to an independent assessment of a plan's worthiness. The Code of Federal Regulations (Section 600.13 of title 10) defines Merit Review as a "thorough, consistent and objective examination of applications based on pre-established criteria by persons who are independent of those individuals submitting the applications and who are knowledgeable in the field of endeavor for which support is requested."
I would also like to note here that although the terms "merit review" and "peer review" are often used interchangeably, they are not equivalent terms. NSF made this distinction clear back in 1986, based on a report from an external Advisory Committee on Merit Review, established by then-director Erich Bloch at the request of the National Science Board. As is described by Marc Rothenberg, the NSF historian, in his 2010 article “"Making Judgments about Grant Proposals: A Brief History of the Merit Review Criteria at the National Science Foundation:"
"According to the committee, the term 'peer review' was properly a restrictive term referring to the evaluation of the technical aspect of the proposal. However, for more and more federally funded research, 'technical excellence' was, in the words of the committee, 'a necessary but not fully sufficient criterion for research funding.' Acknowledging that the NSF (as well as other federal agencies) was using a wide range of nontechnical criteria as part of the decision-making process, the committee suggested that the term 'merit review' more accurately described the NSF selection process."
The committee's recommendation was accepted by Director Bloch, and since then NSF has used the term "merit review" to describe our process.
Since its founding, NSF has relied on the merit review process to allocate the vast majority of its funding. As in other agencies, this has involved the use of proposals from prospective researchers that are judged on their merits by knowledgeable persons. But there are several elements that give merit review at the NSF its distinct features. For one, right from the beginning, NSF utilized the project grant mechanism (as opposed to a contract mechanism) for providing funds. This was a rather radical concept back in 1951, when most government operations used contracts. Since that time, the use of the grant mechanism has been adopted by many federal extramural research funding organizations.
NSF's process for deciding which proposals to fund differs from the approach of a number of other funding agencies and organizations (such as philanthropic foundations) nationally and internationally. Perhaps the most distinctive differences are our reliance on expertise from both outside and within the Foundation, and the discretionary authority vested in the NSF program officer to make funding recommendations. Unlike many philanthropic foundations (and even some federal research funding programs), NSF policy requires that the program officers seek external expert advice before making most of their funding recommendations. However, in contrast to a number of other funding bodies, the external reviewers do not make binding recommendations that the program officer is obliged to follow, although program officers always pay close attention to all external reviews. Because of the responsibility we give our program officers, NSF sets a high standard for excellence in that position. Our program officers are subject matter experts in the scientific areas that they manage, and bring strong credentials with them, including advanced educational training (e.g., a Ph.D. or equivalent credentials) in science or engineering, and deep experience in research, education, and/or administration.
NSF has chosen to give the program officer the responsibility for making funding recommendations to enable a more strategic and long-term approach for building the award portfolio. As important as the input of the external scientific experts is, they have only a snapshot view of the current set of proposals they are evaluating. The NSF program officer is responsible for putting that snapshot view into the larger context of the entire award portfolio they are managing, which can lead to a more diverse and robust portfolio overall. Together with the division directors, who have the authority to review and act on the program officers' recommendations, program officer teams are poised to identify promising research that responds to national priorities identified by Congress and the Administration. In addition, program officers can incorporate agency or programmatic priorities, which are articulated in the annual agency budget, special solicitations, and standing program descriptions, all of which are available to the community via the NSF website.
The NSF merit review process is described in full detail on the NSF website (http://www.nsf.gov/bfa/dias/policy/meritreview/). There is also a summary of the major steps in the merit review process in the annual Report to the National Science Board on the Merit Review Process (the most recent report covering activities in FY 2010 can be found at http://www.nsf.gov/nsb/publications/2011/nsb1141.pdf). It is worth noting here that the key features of the NSF process have remained remarkably stable over time. Any changes that have been incorporated have sought primarily to clarify the process and make it more transparent. For example, initially only excerpts of the external reviews were shared with the proposal authors. Over time, NSF provided the verbatim reviews (but not the identities of the reviewers) to the applicant. Similarly, over time there have been modifications to the number and clarity of the review criteria. In the America COMPETES Reauthorization Act, the broader impacts criterion is specifically mentioned, and the National Science Board is in the process of analyzing the many comments received on this topic.
A flowchart that graphically depicts the major steps in the merit review process and a timeline is attached to this testimony as Appendix I. These steps include:
- Assignment to the appropriate program for review. Principal investigators initiate this process by selecting the program or programs to which they wish to submit their proposal. Once submitted, the cognizant program officers for those programs confirm that the assignment is appropriate. On occasion, a proposal may be reassigned to another program where there is a better fit. During this initial assignment process, it is not uncommon for proposals to be assigned to multiple programs for review, if the subject is interdisciplinary in nature, or if the question is of interest and relevance to more than one program.
- Administrative review of all proposals for compliance with NSF regulations. These regulations, which are intended to ensure fairness in the review process, are described in the Grant Proposal Guide, which is widely available to the NSF community on the NSF website (http://www.nsf.gov/pubs/policydocs/pappguide/nsf11001/nsf11_1.pdf). Proposals that do not comply with these regulations may be returned without review.
- Merit review of all proposals that pass the administrative review. As noted above, a critical feature of NSF's process is the use of both external review by experts in the field and internal review by NSF's corps of program officers. The program officers are responsible for administering the merit review process from beginning to end, starting with identifying and recruiting appropriate peer reviewers from the external community to serve either as individual reviewers for a particular proposal (referred to as "ad hoc" reviewers) or as members of a panel of reviewers who evaluate a larger set of proposals. To ensure that they receive substantive reviews from a variety of perspectives, the program officers reach out to a broad range of experts for input--in fiscal year 2010, over 46,000 external peer reviewers from academia, government, and occasionally industry provided authoritative advice to the Foundation. Selection of expert peer reviewers may be based on the program officer's knowledge, references listed in the proposal, individuals cited in recent publications or relevant journals, presentations at professional meetings, reviewer recommendations, bibliographic and citation databases, or suggestions from the proposal author (subject to the program officer's discretion). In making these selections, program officers pay very careful attention to avoiding conflicts of interest, both real and perceived.
NSF takes seriously its responsibility to ensure that the merit review process is fair and equitable. One of the ways in which we address this responsibility is through the briefings that are given to each review panel before it begins its work. In these briefings, panelists are instructed on NSF's review criteria (Intellectual Merit and Broader Impacts), and on maintaining confidentiality and avoiding conflicts of interest. In addition, review panel briefings typically include alerting the reviewers to the phenomenon of implicit bias, which may adversely impact new investigators, smaller institutions, and underrepresented groups. By guarding against the effects of implicit bias in the review process, NSF is working to ensure that there are equitable opportunities for all investigators.
I should note here that while the vast majority of the proposals received at NSF (~96%) are subject to both external and internal merit review, for some proposals the external review requirement is waived. This waiver provides necessary flexibility for handling proposals for which most of the external community would be conflicted (such as proposals for small conferences, workshops, or symposia), those for which there is a severe urgency (submitted through the Grants for Rapid Response Research, or RAPID, mechanism used, for example, on rapid-response research to the Deepwater Horizon oil spill), and those that request support for high-risk, potentially transformative exploratory work (submitted through the Early Grants for Exploratory Research, or EAGER, mechanism). These proposals are usually only reviewed internally by program officers with appropriate expertise.
- Development of funding recommendations. A central tenet of the NSF merit review process is that the reviewer input is advisory in nature. Funding recommendations are developed by the program officer, who is responsible for synthesizing the advice of the reviewers along with several other factors, with the goal of allocating funding to a diverse portfolio of projects that addresses a variety of considerations and objectives. In addition to their scientific expertise noted above, NSF program officers bring their own unique perspective born from their experience of working with hundreds, thousands, or--in some cases--tens of thousands of proposals. In developing recommendations within the larger context of their overall portfolio, program officers consider carefully the individual merits of each proposal with respect to both its intellectual merit and the potential broader impacts of the project, and how each proposal might help advance a variety of portfolio goals such as:
- Achieving special program objectives and initiatives;
- Fostering novel approaches to significant research and education questions;
- Building capacity in a new and promising research area;
- Supporting high-risk proposals with potential for transformative advances;
- Supporting NSF's core strategies of integration of research and education and integrating diversity into NSF's programs;
- Potential impact on human resources and infrastructure;
- Other available funding sources; and
- Geographic distribution.
NSF has set a goal for completing this process within six months, from the time the proposal is submitted to the point at which the proposal is either declined or recommended for funding and forwarded to the Division of Grants and Agreements for the final stages of review and processing. The proposal assignment and administrative review stage is typically complete within a few weeks. The bulk of the time is spent in the merit review stage, which can take three to four months to complete. Despite the volume of proposals that NSF receives annually (in FY 2010, over 55,000 proposals were submitted, an increase of 23% over the previous year), NSF routinely processes the majority of these proposals (>75%) in fewer than six months.
To ensure the integrity of the process, all program officer recommendations are reviewed by the division director (or other appropriate NSF official), who examines whether the process used to arrive at the decision has been executed in accordance with NSF’s policies and that the decision has been based on a thorough analysis of the merits of the proposal. Large awards may receive additional review, either by the Director's Review Board (DRB) or additionally by the National Science Board (NSB). The DRB examines award recommendations with an average annual award amount of 2.5 percent or more of the awarding division's prior year current plan. The NSB reviews recommended awards with an annual award amount of one percent or more of the awarding Directorate's or Office's prior year current plan, or less than one percent or more of the prior year total NSF budget at the enacted level. Once the funding recommendation is approved (at whatever level is appropriate), the Division of Grants and Agreements ensures that the award recommendation meets all of NSF's requirements before officially issuing the award.
In addition to having multiple layers of review of individual award recommendations, NSF requires that all programs undergo an external review by Committees of Visitors (COVs) every three years. COV reviews provide NSF with external expert assessments of the quality and integrity of program operations and program-level technical and managerial matters pertaining to the merit review and final proposal decisions. Finally, retrospective analysis of the process is periodically performed on a Foundation-wide basis, including the statistical reports submitted to the NSB every year and the Impact of Proposal and Award Management Mechanisms (IPAMM) report of 2007 (http://www.nsf.gov/pubs/2007/nsf0745/nsf0745.pdf).
At the request of Congress, in 2005 the NSB undertook an examination of NSF's Merit Review Process (http://www.nsf.gov/nsb/documents/2005/nsb05119.pdf). The report concludes that:
“The Board fully supports the current NSF system of merit review, which utilizes the peer review process as the principal driver in funding decisions. The Board also strongly endorses the role of NSF program officers' discretionary authority, in concurrence with division directors, for ensuring the implementation and goals of both Merit Review Criteria, along with achieving a balanced portfolio of research and education awards, both within directorates and across the suite of NSF programs. Unlike a system based solely on peer reviews' scores, NSF's merit review process incorporates peer review in a system that also considers those attributes of a proposal (risk, multidisciplinary nature, novelty) that are not readily accommodated by a numerical score, but essential to identifying the most innovative proposals.”
The National Academy of Sciences, in the 1994 report "Major Award Decisionmaking at the National Science Foundation," stated that, "The United States has built the most successful research system in the world. The use of peer review to identify the best ideas for support has been a major ingredient in this success. Peer review-based procedures such as those in use at NSF, the National Institutes of Health, and other federal research agencies remain the best procedures known for ensuring the technical excellence of research projects that receive public support." In November 2009, the Executive Director of the Transportation Research Board at the National Research Council, provided testimony before Congress on how to facilitate the implementation of research at the Department of Transportation. In that testimony, the Director endorsed strongly the fact that NSF's merit review process is well suited to the mission of the agency. His observation: "The more applied mitigation and adaptation research topics should be steered by the concerns and needs of policy makers and practitioners, while the fundamental research topics should be organized along the NSF model in which scholars and experts are guiding the decisions about which projects are likely to be most promising."
NSF's merit review process has served the agency, the scientific community, and indeed the country well for many years. Many Nobel Laureates, National Medal of Science and Technology winners, and MacArthur Foundation Fellows (popularly known as recipients of Genius Grants) have been supported by NSF at various stages in their careers. Through separate programs and in the course of funding specific scientific progress, over the past 25 years NSF has also supported the training of hundreds of thousands of graduate and post-graduate scholars in STEM fields. Discoveries stemming from NSF-funded projects have led to advances across all areas of science, engineering and education, with far-reaching impacts in the fields of nanotechnology, information technology, environmental science, genomics, STEM education, and many others.
The high quality of NSF's merit review process is recognized globally, as evidenced by the fact that it has been used as a model by countries around the world that are newly establishing their own funding agencies. The merit review system for L’Agence Nationale de la Recherche (ANR), the French counterpart to NSF, is explicitly modeled after NSF, as is that of the Foundation for Polish Science. NSF helped the European Research Council establish its merit review system some five years ago, and was instrumental in helping Ireland establish Science Foundation Ireland. Back in 1986, a Chinese official came to NSF for 6 months to learn about our merit review and decision making processes, and subsequently incorporated what he had learned in establishing the National Natural Science Foundation of China (NSF-C). These are just a few examples of international agencies where NSF has had an explicit role in helping develop their merit review systems, but there are literally dozens of others that have borrowed our approach over the years.
As the nature of research and the scientific enterprise continues to change--becoming more interdisciplinary, technological, international and collaborative--NSF continues to explore ideas and strategies that could strengthen the merit review process by enlarging the range of tools that can be used in proposal evaluation. These ideas have come from a variety of sources--internally, from the research community, from the practices of other funding agencies, and from the scientific literature on merit review. One idea that we are actively exploring is a greater use of technology-mediated virtual panels when and where it makes sense, with the hope that decreasing the travel burden will expand the potential pool of reviewers. Among the benefits that NSF would derive from an expanded pool of reviewers are the inclusion of more and varied perspectives, increased opportunities for participation by underrepresented groups, decreased review burden per individual reviewer, and decreased travel costs for the agency. We have established an internal working group to identify other viable candidates for pilot activities, and to develop plans for running and evaluating those pilot activities. We will be discussing these with an advisory committee over the next few months to get their help in refining the processes.
For over 60 years NSF has been forward looking in terms of how the agency manages its research and education portfolio. Merit review fosters the "process of discovery," the means by which researchers can identify emerging scientific challenges and innovative approaches for addressing them. NSF is dedicated to ensuring that the merit review process remains robust, rigorous, and beyond reproach, in support of our mission and enabling us to pursue our goal of funding the world’s best research in science, engineering and education.
I appreciate the opportunity to appear before the Subcommittee to speak to you on this important topic. I would be pleased to answer any questions that you may have.
APPENDIX I: NSF PROPOSAL AND AWARD PROCESS AND TIMELINE
Here is a larger version. Click back to return to the testimony.