Abstract

Merit review is the cornerstone of the National Science Foundation's (NSF's, the Foundation's) work. Each year NSF receives over 40,000 new proposals and subjects virtually all of them to an external merit review. Through the use of merit review, NSF seeks to maintain the high standards of excellence and accountability for which it is known around the world. The National Science Board (NSB, the Board) and NSF have diligently worked together to ensure that the Foundation's merit review system remains an international 'gold-standard' for review of science and engineering research proposals. In 1996, the NSB-NSF Task Force on Merit Review was formed to evaluate the merit review criteria in use at the time. In September 2004, Congress requested the Board conduct a structured review of the NSF's merit review process and report its findings to the House Committee on Appropriations. This report presents a summary of a review of NSF documents relevant to each of these issues, along with information collected by the Board during interviews with current and former NSF staff.

Executive Summary

Merit review is the cornerstone of the National Science Foundation's (NSF's, the Foundation's) work. Each year NSF receives over 40,000 new proposals and subjects virtually all of them to an external merit review. In FY 2004, some 58,000 outside experts provided over 250,000 separate reviews to assist NSF in the evaluation of proposals, submitted to the Foundation's nine directorates and offices, covering a wide variety of topics. Through the use of merit review, NSF seeks to maintain the high standards of excellence and accountability for which it is known around the world.

The National Science Board (NSB, the Board) and NSF have diligently worked together to ensure that the Foundation's merit review system remains an international 'gold-standard' for review of science and engineering research proposals. In 1996, the NSB-NSF Task Force on Merit Review was formed to evaluate the merit review criteria in use at the time. After completing their year-long evaluation, the Task Force proposed the two merit review criteria that the Board approved for the current merit review system. In addition, NSF prepares an annual comprehensive report with summary information about the levels of proposal and award activity for the fiscal year and the process by which proposals are reviewed and awarded. This annual report on the merit review system is required by Board policy, and has been provided annually to NSB since 1977.

Competition for NSF funding is tough - over 72 percent of the proposals reviewed in FY 2004 received at least an average score of good/very good. Unfortunately, budget limitations allowed for the funding of less than 25 percent of the proposals NSF received. In FY 2004, nearly $2.1 billion of declined proposals received a high average rating through the NSF merit review system. These highly rated, but declined, proposals comprise a rich portfolio of unfunded research and education opportunities. Because the Foundation receives so many highly competitive proposals, NSF program officers have discretion to use their expert judgment in order to select, support, and manage a well-balanced portfolio of research and educational activities in their area of responsibility. Program officers use: 1) the results of the peer review process; 2) the Board-approved policies and merit review criteria; and 3) their own expert judgment. A well-balanced research and education portfolio, as approved by the NSB, strives to include considerations for a variety of approaches to research and education problems, a consideration of geographic distribution, as well as gender, ethnicity, and institutional diversity. Factors such as portfolio balance and diversity are not readily accommodated either by a comparison of numerical proposal scores or by outside reviewers unfamiliar with the Foundation's portfolio as a whole. The merit review process also gives program officers flexibility in distinguishing the exceptionally high-risk, multidisciplinary, and innovative projects from other well-ranked proposals. Program officers combine external peer reviews with their professional judgment in recommending proposals for funding within budget constraints to produce a diverse portfolio that meets the Foundation's programmatic emphasis.

In September 2004, Congress requested the Board conduct a structured review of the NSF's merit review process and report its findings to the House Committee on Appropriations. Congress expressed support for the NSF merit review system, but the Committee believes evaluations at regular intervals may ensure that the Foundation's system of award allocation continues to fund the highest quality science. NSF staff draw upon their scientific expertise to administer the merit-based peer review system that awards federal funds to the extramural scientific community. The merit review process consists of external reviewers, selected by NSF program officers, who quantitatively rate proposals on a five-point scale and provide written critique under each of the Board-approved criteria. The program officers then use the ratings and text (as well as panel deliberations, if any) to create a portfolio of recommended awards that is then reviewed by senior officials (typically division directors) for funding. This process, characterized as consensus decision-making according to NSF respondents, is designed to ensure that each award has at least two co-signatures within NSF. NSF's merit review process is designed to foster research in the larger science community that is consistent with the national goals of the Foundation and is to be carried out in a fair and unbiased fashion.

The central focus of the Congressionally requested Board review of this system is to examine the following issues:

• the discretion permitted and exercised by NSF staff in choosing peer review panels;
• the scientific, geographic and institutional composition of peer review panels;
• the quantitative methodologies used to assess relative quality among projects;
• the discretion permitted and exercised by NSF staff in selecting specific projects for funding and;
• the ability of the existing process to identify the most innovative proposals.

This report presents a summary of a review of NSF documents relevant to each of these issues, along with information collected by the Board during interviews with current and former NSF staff. In addition to addressing each topic in turn, the report also provides an analysis of recommendations from the NSF Office of the Inspector General (OIG), recent Committees of Visitors (COVs), and the Foundation-wide Advisory Committee for GPRA Performance Assessment (AC/GPA).

Related Documents

Related Topics