This report, made pursuant to the Government Performance and Results Act (1993), covers activities of the National Science Foundation during Fiscal Year 2000. It is substantially more comprehensive than its predecessor, and records an unprecedented level of effort and achievement.
In conducting the FY 2000 assessment, NSF undertook the most rigorous and challenging performance review in its history. Several goals and indicators were added or revised, and evaluation criteria were made much more stringent.
Previously used ratings such as “partially successful” or “minimally effective” were eliminated; all outcomes were judged either “successful” or “not successful.” As a result, some of NSF’s performance scores were actually lower than in FY 1999, despite the fact that outside evaluators cited progress in those particular areas since 1999.
In addition to Foundation staff, about 400 external evaluators participated in the performance assessment, and generated 64 reports covering 78 of the NSF’s 200 programs. NSF engaged an independent outside examiner, PricewaterhouseCoopers LLP, to verify data compilations.
The Foundation aimed extremely high. For example, NSF set itself the goal of processing 70 percent of proposals within six months of submission. That would have been unmatched in the Foundation’s history, and would far exceed the norm at comparable granting agencies. In fact, during FY 2000 well over half of all NSF proposals were fully processed within six months and 71 percent were fully processed within seven months. By most measures, that would be considered extraordinary; yet it was not successful by NSF’s high standards.
Similarly, NSF attempted to develop the technological capability that would permit the electronic review and processing of tens of thousands of competitive proposals each year --making it possible to do so without generating any paperwork within the Foundation. No other research and education funding organization in government has attempted such a feat. NSF did encounter significant technological challenges in trying to realize this goal, but robust progress was made and the agency will initiate pilot projects to demonstrate its electronic review process capability during FY 2001.
Other goals in which NSF was rated “not successful” simply may not be achievable within a short time, such as “improved achievement in mathematics and science skills needed by all Americans”. Performance assessment activities in FY 2000 clearly demonstrated that a goal of reaching all Americans was unrealistic. Consequently, NSF will revise its performance indicators in future years to focus on related aspects more directly within the agency's responsibility and control.
However, even with new and revised goals and more exacting definitions of success, the Foundation met two-thirds of its 28 goals, which are divided into three broad areas: outcome goals, management goals, and investment process goals.
Outcome goals concern the practical, concrete results of NSF grants and programs, as opposed to the procedures and methods whereby the Foundation carries out its work.
NSF achieved 75 percent (6 out of 8) of its goals in the Outcomes category, notably including:
Two ambitious goals were rated as “not fully successful” (that is, not successful) by reviewers. One was the objective of ensuring that NSF grants contributed to improved scores in “mathematics and science skills needed by all Americans,” as judged by independent external evaluators.
The other was the attempt to produce a “diverse, globally oriented workforce of scientists and engineers.” Both outcomes are extremely difficult to achieve, and neither is completely within the Foundation’s responsibility and control. In NSF's pursuit of both goals, however, evaluators noted progress since FY 1999.
Management performance goals concern the effectiveness and efficiency of the way NSF handles its workload. For FY 2000, the Foundation identified 6 specific management goals, and achieved 5 of them (83 percent).
Goals that were met included:
Only one management performance goal was not met. NSF had set itself the objective of having the technological capability to move competitive proposals submitted electronically through the entire review and award/decline process without generating any paperwork.
While this may have been extremely ambitious, significant progress was made. By the end of FY 2000, the only significant impediment to attaining a full electronic review capability was the development of a secure “electronic signature”. This issue will continue to be addressed in FY 2001 when the agency will pilot ten all-electronic review projects.
Investment process goals involve the specific procedures whereby NSF makes grants, funds and manages capital projects, and serves its customers in general.
For FY 2000, NSF identified 15 such goals. One was found to be inapplicable. Of the remaining 14, the Foundation clearly achieved 7, or 50 percent. Those included:
NSF did not meet 7 of its Investment Process Goals, even though efforts during FY 2000 often produced remarkable, measurable progress toward achieving those goals.
For example, NSF attempted to ensure that 95 percent of program announcements and solicitations would be available at least three months prior to proposal deadlines or target dates. In fact, 89 percent of announcements and solicitations met that standard - up from 75 percent in the preceding year - and 97 percent of program announcements and solicitations were available within 5 days of the three-month goal.
Another important area involves the goal of making at least 30 percent of competitive research grants to new investigators. In determining its performance on this goal, NSF counted only awards made to new Principal Investigators. During FY 2000, 28 percent of awards were made to new Principal Investigators - up from 27 percent in FY 1999. However, more than 33 percent of FY 2000 awards were made to teams of Investigators where at least one Investigator was new - up from 31 percent in FY 1999. So although the goal was not achieved in its strictest interpretation, the results reported clearly demonstrate an increasing "openness in the system" that the agency is committed to maintain.
Finally, NSF has committed to ensure that external merit reviewers take both NSF criteria fully into account when evaluating proposals. The two generic criteria are, simply put, scientific importance to the individual field, and broader significance to science and society as a whole. Although evaluators noted considerable improvement in this area, they rated NSF’s overall performance as “not fully successful", that is, not successful.
That outcome was not unexpected since the new merit review criteria were implemented only in early FY 1998. In assessing the agency's FY 2000 performance, external evaluators examined proposals considered for funding in FY 1997, FY 1998, FY 1999 and in a few cases, FY 2000. Therefore, only about two thirds of proposals examined actually could have been measured against the new merit review criteria. However, evaluators did note that for those proposals subject to the new review criteria (i.e. those submitted for funding consideration in FY 1998, FY 1999 and FY 2000), NSF staff had been generally successful in employing both criteria in making funding decisions. They also noted that NSF needs to increase its outreach efforts to the research and education community to ensure that proposers and reviewers alike adequately address both criteria in proposals and in the proposal review process.
Realization of this goal is increasingly likely over the next few years as NSF effectively communicates to proposers and reviewers the importance of addressing both criteria in proposals and reviews, and as evaluators examine a full complement of proposals subject to these criteria.
In four other goal areas, NSF did not successfully realize the high standards it set.
In one, NSF had attempted to process 70 percent of proposals within six months of receipt. Although only 54 percent of proposals were processed within that time frame, 71 percent of proposals were processed within seven months. This represents an accomplishment, considering the increasing complexity and multidisciplinarity of proposals and NSF's continuing commitment to external merit review. Nonetheless, the Foundation will take steps to improve its performance in this area in FY 2001 and will again set its sights on processing 70 percent of proposals within six months.
Another goal was to identify the best practices with which NSF staff could explain the merit review process, answer questions, and handle complaints.
During FY 2000, NSF conducted several customer-service surveys in an attempt to understand and improve its performance in these areas. Three concerns identified were: the quality and fairness of proposal review; ease of submitting proposals via FastLane; and trouble finding the right person to contact when questions arise. Models of best practice are still being developed, and staff training is underway.
A third goal involved keeping facilities' construction and upgrades within annual schedules and assuring that total time for completion of each phase did not exceed 110 percent of estimates. This goal is ambitious, although it is one the agency strives to realize. Nonetheless, in many scientific construction projects, unforeseen (and perhaps unforeseeable) delays occur as a consequence of rapidly changing technology and as a result of the fact that many such projects are unique. In those circumstances, it is very difficult to produce exact guidelines or timetables. Even so, of eleven construction and upgrade projects supported by NSF, seven (or 64 percent) of them met the goal.
Finally, NSF had set a goal of holding operating time lost due to unscheduled downtime at NSF-funded facilities to less than 10 percent of total scheduled operating time. Again, that target proved difficult to achieve because of the special problems that crop up in such projects. Nonetheless, of 26 reporting facilities, 22 (85 percent) met the downtime goal, and only four did not.
Based on what was learned from the FY 2000 performance assessment, NSF will pay particular attention to certain areas in FY 2001; these include improving customer service by increasing the percentage of proposals processed within six months and focusing on the effective implementation of the merit review criteria.
Using the experience gained in measuring and assessing performance in FY 2000, the agency will revise its FY 2002 performance indicators to focus more closely on achievements for which it has responsibility and control. The Foundation will continue to set the highest standards for itself and to employ the most stringent criteria for review of its accomplishments.