National Science Foundation Merit Review FY 1996

The enclosed FY 1996 Report on the NSF Merit Review System is submitted in accordance with National Science Board policy endorsed in March, 1977, and amended in March, 1984, which requests that the Director provide to the Board an annual report on the Foundation's proposal review system.

I trust you will find the report informative.

Neal Lane
NSF Director

Award categories

During FY 1996, NSF, with the cooperation of external peer reviewers, competitively reviewed 29,953 research and education proposals. NSF funding was awarded to 8,796 of the proposals, producing a funding rate of 29 percent. The number of proposals reviewed annually by NSF has been reasonably stable at about 30,000 proposals since 1992. The number of awards however, has dropped by 13 percent, resulting in a declining funding rate as shown in Text Figure 1. Funding rates vary among directorates, ranging from 23 percent to 37 percent as shown in Appendix Table 1.

In addition to funding proposals that were competitively reviewed during FY 1996, NSF awarded 7,708 continuing grant increments (CGIs) based on proposals which had been competitively reviewed in earlier years. CGIs are funded in annual increments from current year appropriations. The CGI procedure complements the other major award instrument - standard grants - where all funds for a multiple year project are obligated out of a single year appropriation. NSF policy limits the amount of future year CGI commitments to 65 percent of a program's current fiscal year operating plan.

Characteristics of Principal Investigators

The number of proposals received from female and minority Principal Investigators (PIs) has increased slowly since 1990. Proposals, awards, funding rates and trends by PI characteristics are shown in Appendix Table 2. During FY 1996, about 17 percent of competitively reviewed proposals were from female PIs. Funding rates of proposals from female PIs have been higher than proposals from male PIs for five of the past seven years. Funding rate trends for female and minority PIs are graphed in Text Figure 2.

Five percent of competitively reviewed proposals during FY 1996 were from minority PIs. The funding rate for proposals from minority PIs were below the NSF rate from FY 1990 through FY 1995, but exceeded the NSF rate in FY 1996.

Forty-five percent of competitively reviewed proposals in FY 1996 were from PIs who had not received an NSF award in a previous fiscal year (new PIs); down from fifty-one percent in FY 1990. The funding rate for proposals from PIs who had received an NSF award (prior PIs) was higher than proposals from new PIs (36 percent and 21 percent, respectively in FY 1996). The difference in funding rates between proposals from new and prior PIs has been declining since 1991.

Award amounts

The median annual award amount (adjusted for multiple year projects) among competitive awards made during FY 1996 was $52,313. The average award amount among the same set of awards was $85,385. The difference between the median and average award amounts is due to a combination of numerous small awards pulling the median down, and large awards for centers and facilities pulling the average up. Award amounts have been consistent over the past decade, when adjusted to constant dollars as measured by the Consumer Price Index. There are considerable variations among directorates as shown in Appendix Table 3.

Peer Review and Merit Review

There is a tendency to use the terms "peer review" and "merit review" interchangeably. It is more accurate to refer to the NSF proposal review process as "merit review with peer evaluation." The involvement of knowledgeable peers from outside the Foundation in the review of proposals is the keystone of NSF's proposal review system. Their judgments of the extent to which proposals address established criteria are vital for informing NSF staff and influencing funding recommendations. However, development of a portfolio of program awards that address all of NSF's diverse objectives requires a broader perspective than can be achieved solely on the basis of a set of independent or panel-generated proposal reviews. Therefore, NSF relies on the judgment of qualified program officers to make funding recommendations which, on the whole, produce a portfolio of awards addressing NSF's strategic goals and related factors such as:

• Contributions to human resource and institutional infrastructure development,
• Support for "risky" proposals with potential for significant advances in a field,
• Encouragement of interdisciplinary activities, and
• Achievement of program-level objectives and initiatives.

Each program officer funding recommendation is subject to a programmatic review by a higher level reviewing official (usually the division director), and an administrative review by a grants officer. Awards in excess of a $3 million commitment during a project year, or $15 million over five years, require approval by the National Science Board.

Mail-only, Panel-only and Mail-plus-Panel

NSF programs obtain external peer review by two principal methods, mail and panel. In addition to mail and panel reviews, site visits by NSF staff and external peers are often used to review large facility and center proposals. NSF program officers are given discretion in the specific use of review methods, subject to supervisory approval.

In mail reviews, peers are sent proposals and asked to submit written comments to NSF by postal mail, electronic mail, or facsimile. These mail reviews may either be used by the NSF program officer directly to support a funding recommendation ("mail-only" review), or presented to a panel to inform discussion of the proposal.

Many programs use a combination of mail and panel methods to obtain peer reviews. Major variations of such mail-plus-panel reviews are:

• A peer is asked to submit a written mail review and also serve as a panelist, in effect contributing two reviews for each proposal;
• A peer is asked to submit only a mail review which is considered at a later panel meeting; and
• A peer is asked to participate only as a panelist without submitting a written review prior to the meeting.

The mail-plus-panel method was used for 59 percent of proposals reviewed during FY 1996; 23 percent of proposals were reviewed by mail-only, and 18 percent by panel-only. The percentage of NSF proposals reviewed by mail-plus-panel has increased during the past decade, with a corresponding drop in the percentage of proposals reviewed by mail-only. These trends are shown in Text Figure 3. [See Appendix Table 4 for details on NSF trends.]

Directorates vary widely in their use of proposal review methods as summarized in Text Figure 4, and detailed in Appendix Table 5.

Reviewers

Diversity of the reviewer pool is an important feature of the NSF merit review system. Reviewers from diverse backgrounds help ensure that a wide range of perspectives are taken into consideration in the process of funding research and education proposals. NSF emphasizes reviewer diversity through a variety of processes, including a large and expanding Foundation-wide reviewer database, explicit policy guidance, mandatory training for all program officers, and directorate-level initiatives. NSF maintains a central electronic database of over 220,000 reviewers from which over 50,000 reviewers are selected annually. This database is continuously being updated; over 40,000 new potential reviewers have been added since 1992. Potential reviewers are identified from a variety of sources including applicant suggestions, references attached to proposals and published papers, and input from visiting scientists.

NSF policy states that each program funding recommendation must be accompanied by at least three external reviews (exceptions are described in Section F below). On average, proposals during FY 1996 were considered by 8.4 reviewers. The average number of reviews for proposals reviewed by mail-only, panel-only, and mail-plus-panels (shown below in text figure 5). There is considerable variation among directorates (shown below in appendix table 6).

Participation in the peer review process is voluntarily. Panelists are reimbursed for expenses; mail reviewers receive no financial compensation. Nevertheless, seventy percent (70%) of mail-only requests in FY 1996 produced reviews. This rate has been steady since 1990.

Reviewer Proposal Ratings

The NSF merit review system emphasizes reviewer narratives over summary ratings. Summary ratings are but one indicator of reviewer judgment of the proposal quality; written narratives, panel discussions, and portfolio management considerations make important contributions to program officer funding recommendations. The distribution of average summary ratings of mail reviews for awarded and declined proposals is provided in Appendix Table 7.

Every proposer receives a description of the context in which the proposal was reviewed from the NSF program officer, along with a copy of each review considered in making the funding decision. A declined PI may ask for additional clarification of the decision. If the PI is not satisfied that the proposal was fairly handled and reasonably reviewed a formal reconsideration may be requested from the cognizant Assistant Director (AD). If the AD upholds the original action, the applicant's institution may request a second reconsideration from the Foundation's Deputy Director (O/DD). The objective of the reconsideration process is to ensure that NSF's review has been fair and reasonable, both substantively and procedurally.

On average, NSF annually declines over 20,000 proposals but receives only 35 requests for formal reconsideration. Most program-level decisions are upheld in the reconsideration process. The number of requests for formal reconsideration and resulting decisions at both the AD and O/DD levels from FY 1992 through FY 1996 are displayed in appendix table 8 below.

Since the beginning of FY 1990, the Small Grants for Exploratory Research (SGER) option has permitted program officers throughout the Foundation to make short-term (one to two years), small-scale (less than $50 K) grants without formal external review. Characteristics of activities which can be supported by an SGER award include:

• Preliminary work on untested and novel ideas;
• Application of new approaches to "old" topics;
• Ventures into emerging research areas; and,
• Narrow windows of opportunity for data collection, such as natural disasters and infrequent phenomena.

NSF received 220 SGER proposals in FY 1996 and made 164 awards; a funding rate of 75 percent. The SGER funding rate is much higher than for regular, competitively reviewed proposals in large part because potential SGER applicants are encouraged to contact an NSF program officer before submitting an SGER proposal to determine its appropriateness for the SGER funding option. As potential SGER applicants have become familiar with this practice, the SGER funding rate has increased from 55 percent in the first year to 75 percent in FY 1996, as shown in text figure 7. See appendix table 9 for SGER proposal and award trends by directorate.

Program officers may commit up to five percent of their operating budget to SGER awards, but the NSF-wide average is less than one percent. The average SGER award amount in FY 1996 was $33,869, well under the maximum authorized SGER award amount of $50,000. The average award amount has been relatively stable over the seven-year existence of the SGER funding option. SGER funding trends by directorate are provided in appendix table 10.

A Committee of Visitors (COV) is a panel of external experts convened to review the technical and managerial stewardship of a specific NSF program or cluster of programs. Each program that awards grants or cooperative agreements is reviewed on a three-year cycle. There are currently 178 such programs at NSF; 63 programs were reviewed during FY 1996. A list of all programs subject to review by a Committee of Visitors and the fiscal year of the most recent review is provided in Appendix Table 11.

Each COV must operate in accordance with the Federal Advisory Committee Act (FACA) of 1972. In compliance with FACA regulations, virtually all COVs are established as subcommittees of an existing chartered advisory committee, and the COV report is reviewed and approved by the parent advisory committee. The cognizant Assistant Director (AD) provides the parent advisory committee with a written response to each COV report. The COV's report and the AD's response are public documents; some have been publicized in the professional literature.

Authorized exemptions to the peer review process are listed in NSF Manual 10, Section 122 (below) and include routine award actions such as continuing grant increments and no-cost extensions. In special circumstances, the Director or designee may waive peer review requirements. Such waivers of peer review were granted twice during FY 1996; one for the Education and Human Resources (EHR) directorate and one for the Office of Policy Support (OPS).

No changes in the peer review system were implemented during FY 1996 which required Board approval. However, several significant activities involving the merit review system have taken place during the year, as described in the following section.

In 1994, an applicant for an NSF grant filed suit against NSF in Federal Court asserting that the law required the agency to disclose the names of reviewers who wrote reviews of the applicant's declined proposal. In May, 1996, the U.S. Court of Appeals for the District of Columbia unanimously ruled that NSF may withhold the names of reviewers in connection with their reviews of specific proposals.

During FY 1996, NSF continued work on a major examination of its merit review system. One of the drivers of this effort was a Fall, 1994, Government Accounting Office (GAO) report on peer review at three government agencies. Partly in response to the GAO report, NSF established a senior-level Peer Review Study Group (PRSG) to examine relevant issues associated with merit review. Subsequently, several task groups of NSF staff examined the efficacy and implications of specific options and made recommendations for action. Most of the task groups' recommendations have been acted upon; a few are still under consideration.

In the process of obtaining external comments about the merit review system, several stresses and strains on the merit review system were identified that deserved further attention. To address these concerns, an external Proposal Review Advisory Team (PRAT) advisory committee was chartered in late FY 1996 to:

• Inventory and evaluate current stresses on the system;
• Develop feasible options for addressing the most important issues;
• Present pros and cons of the options, from the perspective of proposers and reviewers; and,
• Provide analysis and advice on related issues.

The PRAT met for two days in December, 1996, and is expected to present their report to the Deputy Director by the Summer of 1997.

A joint NSF and NSB Task Force on Merit Review was charged at the October, 1996, Board meeting with examining the Board's generic review criteria and making recommendations on retaining or changing them, along with accompanying guidance on their use. The Task Force carried out an assessment of how the criteria have been applied in the review process, and recommended that the criteria be simplified and the language be harmonized with the NSF Strategic Plan. The Task Force released a Discussion Report in November, 1996 and asked NSF to publicize it widely and solicit comments from the science and engineering community. The Task Force will study the comments and make recommendations to the Board in the Spring of 1997.