National Science Foundation Merit Review FY 1998

The attached FY 1998 Report on the NSF Merit Review System is submitted in accordance with National Science Board policy endorsed in March, 1977, and amended in March, 1984, which requests that the Director provide to the Board an annual report on the Foundation's proposal review system.

I trust you will find the report informative.

Rita R. Colwell
Director

Overview

During FY 1998, NSF took action on 28,321 competitive, merit reviewed research and education proposals. The number of proposals reviewed annually by NSF had been reasonably stable at about 30,000 proposals since 1994. This drop to the 28,000 level has not been analyzed yet.

NSF funding was awarded to 9,280 of the proposals, resulting in an overall-funding rate of 33 percent. As shown in Text Figure 1, the number of awards made each year has varied between approximately 9,000 and 10,000. In FY 1998 the number of awards decreased by 5.9 percent relative to FY 1997 funding rate. Funding rates vary among directorates, ranging from 25 percent to 41 percent as shown in Appendix Table 1.

In addition to funding proposals that were competitively reviewed during FY 1998, NSF awarded 6,963 continuing grant increments (CGIs) based on proposals which had been competitively reviewed in earlier years. CGIs are funded in annual increments from current year appropriations. The CGI procedure complements the other major award instrument - standard grants - where all funds for a multiple year project are obligated out of a single year appropriation. NSF policy limits the amount of future year CGI commitments to 65 percent of a program's current fiscal year operating plan.

Characteristics of Principal Investigators

The number of proposals received from female Principal Investigators (PIs) has increased in most years since 1991. The same has been true for PIs from minority groups, though the increase has been slower than for female PIs (10 percent increase from FY 1991 to 1998 for minority PIs; 19 percent for female PIs.) During FY 1998, about 20 percent of competitively reviewed proposals were from female PIs (a one- percent increase from FY 1997, with a corresponding decrease in proposals from male PIs), and four percent were from minority PIs (level with FY 1997.)

Funding rates of proposals from female PIs have been higher than proposals from male PIs for the past five years. The funding rates for proposals from minority PIs have been above the overall NSF funding rate for only one of the past eight years.

Forty-three percent of the competitively reviewed proposals in FY 1998 were from PIs who had not received an NSF award in a previous fiscal year ('new PIs'). This percentage represents a decrease from 44 percent in FY 1997 and 51 percent in FY 1991. The funding rate for proposals from PIs who had received an NSF award (prior PIs) was higher than proposals from new PIs (39 percent and 25 percent, respectively in FY 1998). The difference in funding rates between proposals from new and prior PIs has remained approximately level since 1993.

Trends in funding rate for all PIs, female and minority PIs, and prior and new PIs are graphed in Text Figure 2. Proposals, awards, funding rates and trends by PI characteristics are presented in Appendix Table 2.

Award Amounts

The median annual award amount (adjusted for multiple year projects) among competitive awards made during FY 1998 was $61,666, a 5.5 percent increase from FY 1997. The average award amount in FY 1998 was $105,382, an increase of 9.6 percent from FY 1997. The difference between the median and average award amounts reflects the effect of numerous small awards on the median, and a few large awards for centers, facilities, and large systemic initiatives on the average award amount. Award amounts have been consistent over the past decade, when adjusted to constant dollars as measured by the Consumer Price Index. There are considerable variations among directorates, as shown in Text Figure 3. Data on median and average award amounts from FY 1994-1998 are presented by directorate in Appendix Table 3.

Peer Evaluation and Merit Review

The involvement of knowledgeable peers from outside the Foundation in the review of proposals is the keystone of NSF's proposal review system. Their judgements of the extent to which proposals address established criteria are vital for informing NSF staff and influencing funding recommendations. For this reason, NSF's system of proposal review can accurately be characterized as "merit review with peer evaluation."

Review Processes used at NSF

NSF programs obtain external peer review by two principal methods, mail and panel. In addition to mail and panel reviews, site visits by NSF staff and external peers are often used to review proposals for large facilities, centers, and systemic reform initiatives. NSF program officers are given discretion in the specific use of review methods, subject to supervisory approval. For example, some programs try to reduce proposal pressure by requiring submission of preliminary proposals. Review of preliminary proposals varies widely, ranging from non-binding advice form program officers to proposers, to binding decisions from external reviewers. Systematic data on the use of preliminary proposals are not available.

In "mail-only" reviews, peers are sent proposals and asked to submit written comments to NSF by postal mail, facsimile, electronic mail, or through FastLane, NSF's Web-based system for electronic proposal submission and review. These mail reviews are then used by the NSF program officer directly to support a recommendation for award or decline.

"Panel-only" review refers to the process of soliciting reviews only from those peers who meet in a panel review setting to discuss their reviews and provide advice directly to the program officer. Most programs that use this process mail proposals out to panelists and receive their reviews prior to the panel meeting. Other programs provide panelists with access to the proposals at the beginning of the panel meeting, allowing them a period of time during which they prepare their reviews at the meeting.

Most proposals submitted to NSF are reviewed using some combination of these two processes ("mail-plus-panel" review). Those programs that employ the mail-plus-panel review process have developed several different configurations, such as:

• A peer is asked to submit a written mail review and also serve as a panelist, in effect contributing two reviews for each proposal.
• A peer is asked to participate only as a panelist, with responsibility only for reviewing and discussing mail reviews written by others and providing verbal and/or written advice to the program officer.

The mail-plus-panel method was used for 63 percent of proposals reviewed during FY 1998; 21 percent of proposals were reviewed by mail-only, and 17 percent by panel-only. Directorates vary in their use of proposal review methods. Mail-plus-panel review was the most common review process used in the BIO, CSE, ENG, GEO, and SBE Directorates. Mail-only review was the predominant mode of review in MPS. Panel-only review was the most commonly used method in EHR. Directorate-level data on the use of different review processes during FY 1998 are presented in Appendix Table 4.

The use of various review processes has varied over time. The percentage of NSF proposals reviewed by mail-plus-panel has increased from 42 to 63 percent of all proposals since FY 1989.

There has been a steady decline in the use of mail-only review from 39 to 21 percent during the past decade. The use of panel-only review has varied less than other methods, increasing slowly for several years and then declining from 24 to 17 percent during the past four years. These trends are shown in Text Figure 4, and the corresponding data are presented in Appendix Table 5.

NSF policy states that each recommendation for final action on a proposal must be accompanied by at least three external reviews, unless the requirement has been waived (see section F, below). The total numbers of reviews and the average numbers of reviews peer proposal obtained by these different methods are presented in Text Figure 5. Directorate-level data for FY 1998 are presented in Appendix Table 6.

Reviewers

Diversity of the reviewer pool is an important feature of the NSF merit review system. Reviewers from diverse backgrounds help ensure that a wide range of perspectives are taken into consideration in the review process. NSF emphasizes reviewer diversity through a variety of processes, including use of a large and expanding Foundation-wide reviewer database, explicit policy guidance, mandatory training for all program officers, and directorate-level initiatives. NSF maintains a central electronic database of 241,000 reviewers. For proposal decisions in FY 1998, 47,000 of these reviewers were sent one or more proposals for mail review, 31,700 reviewed at least one proposal by mail, and 7,800 reviewers served as panelists. In all, 51,000 individuals either served on a panel, were sent a proposal for mail review, or served in both functions.

Potential reviewers are identified from a variety of sources including applicant suggestions, references attached to proposals and published papers, and input from mail reviewers, panelists, and visiting scientists. During FY 1998, approximately 28,500 of the 241,000 records now in the reviewer database were either added or updated.

Participation in the peer review process is voluntary. Panelists are reimbursed for expenses; mail reviewers receive no financial compensation. In FY 1998, 62 percent of requests for mail reviews produced responses, which represents a slight decrease from the 64 percent response rate that has been stable since 1991.

Reviewer Proposal Ratings

The NSF merit review system emphasizes reviewer narratives over summary ratings. Summary ratings are but one indicator of reviewer judgment of the proposal quality. The written narratives provided by reviewers, the deliberations by panel members, and the expert opinions provided by program officers are all important components of the merit review system. No one component is allowed to dominate over the others.

The distribution of average summary ratings of reviews for awarded and declined proposals is provided in Text Figure 6. Only those ratings provided for mail-only and mail-plus-panel reviewers have been included. Panel-only reviewers often submit comments without a summary rating. These data indicate considerable overlap among the average reviewer ratings of successful and unsuccessful proposals, most notably in the range of "very good" average ratings. The judgment of NFS staff is essential to making this difficult separation between awards and declines.

NSF Program Officers

The narrative comments and summary ratings provided by external reviewers are essential inputs in NSF's system of merit review. Once received, these inputs inform the judgment of the program officers who formulate award and decline recommendations to NSF's senior management. These program officers are scientists, engineers, and educators to whom NSF looks for expert judgment and program management. In making recommendations to award or decline proposals, these highly qualified individuals produce and manage a portfolio of awards addressing NSF's strategic goals and related factors such as;

• Contributions to human resource and institutional infrastructure development,
• Support for "risky" proposals with potential for significant advances in a field,
• Encouragement of interdisciplinary activities, and
• Achievement of program-level objectives and initiatives.

The number of program officers employed by NSF has remained stable at slightly over 400 for the past five years, despite increases in proposal pressure and general workload. Depending on their professional experience, program officers are classified as assistant program director, associate program director, or program director. They can be permanent NSF employees or temporary employees. Some temporary program officers are "on loan" as visiting scientists, engineers, and educators (VSEEs) for up to three years from their host institutions. Others are employed through grants to the home institutions under the terms of the Intergovernmental Personnel Act. The distribution of these program officers among these and other categories is presented in Text Figure 7.

Review of Program Officer Award Recommendations

Each program officer's recommendation to award or decline a proposal is subject to a programmatic review by a higher level reviewing official (usually the division director), and an administrative review by a grants officer in the Office of Budget, Finance, and Award Management (BFA). All award recommendations in excess of $1.5 million in any one project year or $6 million over five years must be reviewed by the Director's Review Board (DRB). Awards in excess of a $3 million commitment during a project year, or $15 million over five years, require approval by the National Science Board.

NSF policy includes several mechanisms that provide proposers with information on how the review process led to a decline recommendation, and on procedures for obtaining additional explanations for declinations. These policies help to ensure that NSF's review has been fair and reasonable, and that NSF maintains the openness, quality, and integrity of the merit review process.

Every proposer receives from the NSF program officer a description of the context in which the proposal was reviewed, along with a verbatim copy of each review that was considered in the review process. A declined PI may ask the cognizant program officer for additional clarification of the decision. If after considering this additional information a PI is not satisfied that the proposal was fairly handled and reasonably reviewed, he or she may request formal reconsideration from the cognizant assistant director (AD). This request can be based on the PI's perception of procedural errors or on disagreements over the substantive issues dealt with by reviewers. If the AD upholds the original action, the applicant's institution may request a second reconsideration from the Foundation's Deputy Director (O/DD).

On average, NSF annually declines over 20,000 proposals but receives, on average, only 50 requests for formal reconsideration. Most program-level decisions are upheld in the reconsideration process. The number of requests for formal reconsideration and resulting decisions at both the AD and O/DD levels from FY 1994 through FY 1998 are displayed in Appendix Table 7.

Since the beginning of FY 1990, the Small Grants for Exploratory Research (SGER) option has permitted program officers throughout the Foundation to make short-term (one to two years), small-scale (less than $50 K) grants without formal external review. Characteristics of activities which can be supported by an SGER award include:

• Preliminary work on untested and novel ideas;
• Application of new approaches to "old" topics;
• Ventures into emerging research areas; and
• Narrow windows of opportunity for data collection, such as natural disasters and infrequent phenomena.

The funding rate for SGERs in FY 1998 was 83 percent. The SGER funding rate is much higher than for regular, competitively reviewed proposals in large part because potential SGER applicants are encouraged to contact an NSF program officer before submitting an SGER proposal to determine its appropriateness for the SGER funding option. As potential SGER applicants have become familiar with this practice, the SGER funding rate has increased from 55 percent in its first year (FY 1990) to 83 percent in FY 1998. Additional details are shown in Text Figure 8.

NSF management has been concerned by the decrease in SGER proposal pressure since the activity's inception in 1990. In response, Staff Memorandum O/D 97-06 (dated June 5, 1997) announced a three-year experiment that increases the SGER award limit from $50,000 to $100,000. Program officers were also given permission to grant six-month extensions and supplements of up to $50,000 for extant SGER awards. (EHR and BIO elected not to offer these time extensions or supplements.)

Coincident with these policy changes, the downward trends in SGER proposal pressure and number of awards were reversed in FY 1997. NSF received 242 SGER proposals in FY 1997 and made 193 awards. NSF received 299 SGER proposals in FY 1998 and made 247 awards. Directorates vary in the degree to which their program officers discourage potential proposers from submitting inappropriate ideas as formal SGER proposals. As a result, FY 1998 funding rates for SGER proposals varied among directorates from 56 to 96 percent. Directorate-level data on SGER proposal pressure and funding rates are presented in Appendix Table 8.

The total amount awarded to SGERs also increased in FY 1998 to $12,320,200. This represents an all-time high for SGER support and a 46 percent increase from FY 1997. The average SGER award amount in FY 1998 was $49,879, a 14 percent increase relative to the FY 1997 average award amount of $43,591. Despite these increases, the total NSF investment in SGERs remains less than one percent of the operating budget for research and education, far below the five percent that program officers may commit to SGER awards. The history of SGER awards by directorate from FY 1996 to FY 1998 is presented in Appendix Table 9.

Committees of Visitors (COV) are panels of external experts convened to review the technical and managerial stewardship of NSF programs or clusters of programs. Each program that awards grants or cooperative agreements normally is reviewed on a three-year cycle. There are currently 179 such programs at NSF; 54 programs were reviewed during FY 1998. A list of all programs subject to review by a Committee of Visitors and the fiscal year of the most recent review is provided in Appendix Table 10. Due to reorganization of several directorates, COV review of a number of programs has been delayed. Where this has occurred, the directorates have been notified and have agreed to correct this situation by scheduling COVs during FY 1999 and FY 2000.

Each COV must operate in accordance with the Federal Advisory Committee Act (FACA) of 1972. In compliance with FACA regulations, virtually all COVs are established as subcommittees of an existing chartered directorate advisory committee, and the COV report is reviewed and approved by the parent advisory committee. The cognizant assistant director (AD) provides the parent advisory committee with a written response to each COV report. The COV's report and the AD's response are public documents; some have been publicized in the professional literature.

Authorized exemptions to the peer review process are listed in NSF Manual 10, Section 122 (below) and include routine award actions such as continuing grant increments and no-cost extensions. In special circumstances, the Director or designee may waive peer review requirements. Such waivers of peer review were granted 7 times during FY 1998; 5 for OIA, 1 for SBE and 1 for CISE.