Dr. Mary E. Clutter
Thank you, Mr. Chairman, for inviting me to testify before your committee on the merit review process used by the National Science Foundation (NSF) to aid us in investing in scientific research and education.
I will comment on the role and value of the merit review process, including the importance of the NSF program officer in making recommendations, and I will point out the measures we have taken over the years to insure that the system is as open and fair as possible. I will also say a few words about NSF's relationship with other Federal agencies.
While my comments are primarily centered on the biological sciences at NSF, most of what I have to say applies equally well to other parts of NSF.
Merit or peer review is the process of using experts (or peers) to evaluate and help determine the best science to support. It is the cornerstone of the NSF decision-making process since the Foundation was chartered in 1950. Through the use of merit review, NSF seeks to maintain the high standards of excellence and accountability for which it is known around the world.
Merit review has significant value for NSF and for our stakeholders. First, it ensures that the Foundation supports the best, leading-edge science. Second, it aids us in developing our priorities -- especially vital in this time of limited resources. Third, it has great value to the institutions that submit the proposals, as they are assured that those proposals have competed in the national arena. Fourth, it provides peer feedback to scientists to help improve their projects. And fifth, it provides accountability to the public, by assuring that our tax dollars are spent most wisely. As a result of these many attributes, I am convinced that merit review represents the very best system for determining the Nation's investment of public funds for fundamental research.
THE NSF PROGRAM OFFICER
Before describing NSF's merit review process, let me say a little about the NSF program officer. The "functional unit" at NSF is a well-trained, scientifically literate program officer who is an expert in a particular scientific area. Over the course of a proposal's life-time at NSF, the program officer is responsible for managing virtually every aspect of its evaluation -- overseeing the external review, evaluating the comments of the reviewers, and making a recommendation as to whether the proposal should be supported. At the same time, program officers must take into account other considerations, such as the potential contribution of a particular proposal to the advance of the field as a whole, and its relation to other pending proposals -- and, of course, the amount available in the program's budget.
Without the scientific expertise acquired by the program officer over years of training and participation in the process of science, NSF would find it difficult if not impossible to make the decisions necessary to support the science required to ensure continued world leadership.
The Foundation places a very high value on recruiting and using Visiting Scientists , commonly referred to as "rotators", because they have recent bench-level experience and are able to infuse our decision-making systems with their fresh viewpoints. At any given time, about a third of all NSF program officers are "rotators" on leave from their home institutions, most from U. S. colleges and universities, but some from industry and research institutions. Many of the permanent staff began as "rotators". In my own case, for example, I was an established investigator at Yale University who initially agreed to come to the Foundation for 1 year.
THE MERIT REVIEW PROCESS AT NSF
To give you a practical understanding of how merit review works at NSF, I would like to begin by describing the overall system for handling a proposal in the biological sciences from the time it is received through the time when an institution is formally notified of an award or declination. I will describe the central role of merit review in this process.
The research programs of the Foundation are organized along the lines of fields of science and engineering. Each proposal received by NSF is assigned to the appropriate NSF program, which is responsible for undertaking merit review of the proposal and for making a final recommendation regarding it. Every attempt is made to ensure the best fit for the science in the proposal, and proposals with a multidisciplinary or interdisciplinary focus are commonly reviewed simultaneously by several different programs, or by special multidisciplinary panels.
Across NSF, the specific review process employed depends on the tradition of the particular field of research. In every case, however, the process relies on volunteers from the scientific community, who are the expert "peers" at the core of merit review. These volunteers read and evaluate the proposal and provide a written review to NSF.
In some sciences, ad hoc mail review, using external experts, is the primary review mode. In biology, a panel of scientists reviews all competing proposals in addition to the ad hoc review. For large projects and some special competitions in all disciplines, written reviews are often supplemented with site visits by NSF staff and teams of outside experts.
In the ad-hoc mail review procedure, the program officer selects individuals identified as expert on the topic. Ensuring the quality, completeness and fairness of the external review is one of the principal responsibilities of the program officer. The reviewer receives standard instructions and review forms, and responds directly to the program officer. Where panels are used, groups of experts meet at some specified interval -- generally two times a year in the disciplinary programs -- to evaluate a group of proposals, taking into account the comments of the ad hoc reviewers. The first criterion in choosing panelists is expertise and experience in the field, at the same time ensuring the balance of the panels with regard to such factors as participation of women and underrepresented minority scientists and of scientists from the wide range of academic institutions and from industry.
In the biological sciences, we feel that the combination of ad hoc and merit review gives us the optimum information needed to make a funding decision. Ad hoc reviews provide an intensive, deep look at the science in the proposal, while the panel review gives an extensive, comparative analysis of all the proposals that are in competition at a particular time.
To give you some idea of the scale of operations at NSF, during FY 1996 NSF took final action on nearly 29,951 competitively reviewed proposals. Of these, 8,796 received awards. NSF received a total of 247,877 reviews to aid us in making these decisions. The individuals who provided these reviews came from a wide range of institutions -- not only academic institutions, large and small, but also from industry, government and even foreign research and educational organizations. The NSF merit review system asks reviewers to consider four criteria to evaluate proposals. The four criteria are: 1) research performance competence; 2) intrinsic merit of the research; 3) utility or relevance; and 4) the effect on the infrastructure of science and engineering. Reviewers are instructed to apply them "in a balanced and judicious manner."
These merit review criteria were established by the National Science Board in 1981, at a time when NSF funded very few education awards and when the Foundation had less emphasis than today on the integration of research and education. For these reasons, the Board is currently examining whether the criteria need to be clarified or altered.
Reviewers are asked to provide an overall rating for each proposal and, more importantly, narrative comments. These free-form narrative comments from experts are not just valuable to program officers. They add value to panel discussions and play a vital role in the deliberative process. A less known, but highly significant added value of the narratives is the advice they provide to the applicants. This results in a feedback loop that leads to constant improvement in proposals. One result of this feedback loop is that the quality of the average NSF proposal has improved dramatically over the years. NSF now receives world-class proposals from all kinds of academic institutions, not just the top-twenty research universities.
The program officer's job is only half finished once the reviews have been completed. It is very easy to make a final decision between highly rated proposals and very low rated ones. However, it has become increasingly difficult to make final recommendations as the number of very highly rated proposals has steadily increased over the years. Since there is rarely enough money to fund all the highly rated proposals, the program officer is called on to use his or her scientific judgment and experience in selecting among them.
The program officer's tentative recommendations are reviewed and discussed at one or more higher levels of NSF management. Most frequently these discussions take place at the division level during a post-panel discussion between the division director and the program officer. Prior to this point in the process, the program officer is primarily focusing on ensuring the best scientific evaluation for the project proposed. That information is brought into the division "post-panel discussions," at which time broader priorities are also discussed.
The post-panel discussion focuses on the overall balance in the award recommendations, the portfolio for each program. Some of the factors considered include the range of institutions recommended for support, broad representation of principal investigators and adequate coverage of the scientific topics included in the program's purview. Program officers must also pay attention to other NSF priorities such as integration of research and education, support for important research resources, including multi-user instrumentation, databases, and genetic stock centers, and impact of the proposals on national needs. To ensure an open and fair system, we have built in several safeguards. These include rules concerning potential conflicts of interest, or even the appearance of conflicts; return of verbatim but anonymous reviews to the applicant; and a system for formal reconsideration of proposal decisions, at the request of the applicant.
The merit review system is also extremely valuable in aiding NSF in setting priorities. As mentioned earlier, the system helps in setting priorities by identifying cutting edge science. It also points out trends and gaps, and surfaces topics for NSF staff to address at higher levels, such as the various committees of the National Science and Technology Council.
In summary, merit review is a very potent "bottoms up" process that assures that the best science is funded and provides a conduit for the scientific community to participate in priority setting.
RELATIONSHIP OF NSF TO MISSION AGENCIES
Finally, I wish to examine the relationship of NSF to the so-called mission agencies. While each Federal agency has its specific mission, NSF's being the overall health of science and engineering research and education across disciplines at the nation's colleges and universities, there are many areas, including many of the sub-areas of agriculture, where the nation benefits from partnerships among the agencies. One very successful partnership of this type is the "NSF/DOE/USDA Collaborative Research in Plant Biology" program, which supports the interdisciplinary training of plant biologists to meet the challenges posed by new scientific research and modes of education. This program also supports the networking of scientists working on all aspects of plant biology, and the development of "collaboratories", in which they work together on common problems. I am convinced that other interagency interactions of this type would be very productive and should be encouraged.
THE CONTINUUM FROM BASIC TO APPLIED RESEARCH
Most research supported by NSF is generally classified as fundamental, with the primary goal of furthering our understanding of the basic science involved. However, we find it increasingly difficult (and increasingly fruitless) to make any strong distinctions between basic and applied research. Interdisciplinary approaches to research are producing webs of interaction where research that might be termed "basic" in one sub-field has immediate application in another and may even generate a series of both "basic" and "applied" questions in a third.
An example that illustrates the evanescence of the basic-applied dichotomy is the project to sequence the entire genome of the weed Arabidopsis, a plant species that itself is of no commercial or agricultural importance. However, Arabidopsis has proven to be an excellent model plant for genetic studies. The project to fully sequence its genome is an international effort spearheaded by NSF here in the US, in cooperation with USDA and the Department of Energy (DOE). Already several important genes have been isolated and sequenced from this weed, including those that control flowering and that directly or indirectly control productivity in crops such as maize. Other genes have been transferred to lumber-producing trees, inducing them to flower when they are only six to nine months old, rather than waiting until they are ten or twenty years old. As a result, forest scientists will now be able to cross trees which are very young and then select for those that grow faster, produce fewer branches and have fewer knots, or have stronger wood, all in one-tenth to one-hundredth of the time that such crosses and selection processes could have been done in the past.
NSF's recently published strategic plan, NSF in a Changing World, establishes three broad goals for the agency:
- enabling the US to maintain world leadership in all aspects of science and engineering;
- promoting the discovery and use of new knowledge in service to society; and
- achieving excellence in science, math, engineering and technology education at all levels.
Two operational procedures are critical to achieving these goals -- the use of merit review to identify the best proposals, and the formation of partnerships with other public and private organizations to ensure that NSF-funded research is applied to important national problems. In this regard, we in the biological sciences at NSF look forward to continued fruitful interactions with the Department of Agriculture as we develop and implement synergistic programs of benefit to both agencies.
That completes my formal statement, Mr. Chairman. I shall be glad to answer any questions you or other members of the task force may have.