Bypass Navigation


FY 2000 Performance Results and Related Issues

The purpose of this chapter is to provide a more detailed explanation of the performance results presented in the Management’s Discussion and Analysis and to discuss performance-related topics. For a complete and comprehensive discussion of NSF’s performance goals, final results and related issues, see NSF’s FY 2000 GPRA Performance Report, available on NSF’s Web site (https://www.nsf.gov/od/gpra/).

This is the second year NSF is reporting performance results. NSF began implementing GPRA in 1997, by developing an agency GPRA Strategic Plan. In compliance with the Results Act, NSF updated this Strategic Plan last fall. NSF’s GPRA Strategic Plan provides the guiding framework for NSF’s FY 2000 Annual Performance Plan, which was developed in conjunction with the development of NSF’s FY 2000 budget. The concurrent development of the performance plan and the budget creates a direct link between programmatic activities and the achievement of NSF’s strategic goals. A more detailed description of how performance goals and program activities are linked to the budget structure follows.

GPRA implementation has been a particular challenge for agencies like NSF whose mission involves research activities. This is primarily due to: (1) the difficulty of linking research outcomes to annual investments and the agency’s annual budget and (2) the fact that assessing the results of research is inherently retrospective and requires qualitative judgments of expertise. NSF has developed an alternative format that has been approved by OMB, using external expert review panels to assess research results and reporting research outcome goals utilizing a qualitative scale. The use of external expert panels to review research results and outcomes is a common, long-standing practice used by the academic research community.


Performance Goals

NSF’s FY 2000 Annual Performance Plan includes three sets of goals:

  • Outcome Goals focus on the results of NSF’s grants for research and education in
    science and engineering and relate directly to the mission of the agency. These Outcome Goals are also NSF’s long term strategic goals from NSF’s Strategic Plan, FY 1997-2003.
    In FY 2000, a new goal addressing data quality measures for reporting Science Resource Studies (SRS) products was added.
  • Management Goals address the efficiency and effectiveness of administrative activities in support of the NSF mission. In FY 2000, two new goals addressing electronic proposal processing and staff diversity were added.
  • Investment Process Goals focus on the means and strategies NSF uses to achieve its outcome goals and sets performance targets for the investment processes by which NSF shapes its portfolio of awards. Several new goals were added in FY 2000 to address customer service, the integration of research and education, and diversity.

These three sets of goals are mutually supportive. The longer term desired results of NSF awards are reflected in the Outcome Goals. Achieving the desired Outcome Goals depends in part on the quality of the investment process, which is related to the efficiency and effectiveness of the agency’s administration and management. The Investment Process Goals and Management Goals are necessary to ensure that the longer term Outcome Goals will be achieved.

NSF’s key strategy for success is through use of a rigorous merit review process in making awards for activities that will influence research and education in math, science and engineering, both directly and indirectly.


How Performance Goals are Linked to Areas of Emphasis and to the Budget Structure

NSF’s five Outcome Goals address the results of NSF’s grants for research and education in science and engineering and relate directly to the mission of the agency. Outcome Goal 1 (Discoveries at and across the frontier of science and engineering) and Outcome Goal 2 (Connections between discoveries and their use in service to society) address NSF’s research grants. Outcome Goal 3 (A diverse, globally oriented workforce of scientists and engineers) and Outcome Goal 4 (Improved achievement in mathematics and science skills needed by all Americans) address NSF’s education grants. Outcome Goal 5 (Timely and relevant information on the national and international science and engineering enterprise) addresses NSF’s legislative mandate to collect, interpret and analyze data on scientific and engineering resources, and to provide a source of information for federal policy formulation. This goal applies to both research and education activities.

The following chart shows how NSF’s Outcome Goals are linked to the NSF budget structure. NSF receives five Congressional appropriations: Research and Related Activities (R&RA); Major Research Equipment (MRE); Education and Human Resources (EHR); and Salaries and Expenses (S&E). The fifth appropriation funds the Office of Inspector General. Outcome Goals 1,2 and 5 are funded through the R&RA and MRE appropriations and Outcome Goals 3 and 4 are funded through the EHR appropriation. Because the S&E appropriation funds the internal administration and management of the agency, S&E funding applies to all the Outcome Goals, and as indicated in the Statement of Net Cost, is proportionately prorated between research and education programs based on each program’s direct cost.

The Linkage Between NSF’s Outcome Goals and the Budget Structure

linkage graphic

 

Data Verification and Validation

In FY 2000, NSF engaged PricewaterhouseCoopers LLP (PwC) to document and assess the process NSF uses to collect, maintain and report data for selected performance goals. PwC was also tasked with re-calculating the measures and assessing the reliability of the supporting processes. PwC mapped NSF procedures against the U.S. General Accounting Office (GAO) criteria for supporting processes to be considered valid and verifiable. For the Outcome Goals, PwC verified and validated the results.

In their final reports of their reviews of the Investment Process Goals and the Management Goals, PwC concluded that NSF was reporting its GPRA measures with “sufficient accuracy such that any errors, should they exist, would not be significant enough to change the reader’s interpretation as to the Foundation’s success in meeting the supporting performance goal. . .” Furthermore, PwC concluded that NSF “relies on sound business processes, system and application controls, and manual checks of system queries to confirm the accuracy of reported data. We believe that these processes are valid and verifiable.”


Summary of Results

In FY 2000, NSF was considerably more rigorous in evaluating goal achievement. Options for grading were limited to either successful or not successful. Justification was required for successful grades that used qualitative measures. Finally, for the Outcome Goals, an external firm, PricewaterhouseCoopers LLP, was engaged to verify the achievement data tables for the Outcome Goals. While NSF was successful in achieving 64% of its goals in FY 2000 as compared with achieving 78% for FY 1999, the results of the second year are very similar to the first. Positive trends were evident in some of the goals, indicating movement in a desirable direction. The areas identified as needing improvement continue to be:

(1) the use of both merit review criteria by reviewers and applicants; and (2) customer service goals such as decreasing time to decision. Both these areas will be focal points for FY 2001.
Overall, NSF achieved 18 of 28 performance goals—six out of eight Outcome Goals, five out of six Management Goals, and seven out of 14 Investment Process Goals. As in FY 1999, one Investment Process Goal dealing with facilities management was not applicable in FY 2000.

Performance Goal
Number of Goals Achieved
grey line
Outcome Goals 6 out of 8 (75%)
grey line
Management Goals 5 out of 6 (83%)
grey line
Investment Process Goals 7 out of 14 (50%); one goal did not apply
grey line
Total 18 out of 28 (64%)


Part I. Outcome Goals and Results for FY 2000

NSF’s long-term Outcome Goals address how the investments made by programs have led to results important to the broad mission of the agency. These Outcome Goals do not lend themselves to quantitative reporting, therefore NSF has developed an alternative format -- a qualitative scale that allows NSF to report whether or not the agency has been successful in achieving its Outcome Goals. Also, because many research results appear long after an investment is made, in some cases ten years or more, this assessment report of NSF’s program performance is retrospective. That is, the outcome results reported in FY 2000 are from investments made prior to FY 2000. The results of the investments made in FY 2000 will not begin to be reported until beyond FY 2000.

In FY 2000, NSF’s Outcome Goals 1, 2, 3 and 4a are expressed in a non-quantitative, qualitative form, each critical to ensure the progress of science. The results reported for the year are collected, tabulated and summarized by aggregating many individual reports prepared by committees of external experts assessing individual programs or clusters of programs throughout the fiscal year. The assessment is retrospective, covering a subset of one-third of NSF’s programs that represent activities spanning the entire agency over a period of three years or more.

How Research Results Are Assessed: Committees of external experts are carefully selected to provide NSF with an objective, independent assessment of programs for process and results. These committees, known as Committees of Visitors (COVs) and Advisory Committees (ACs), assess approximately one-third of NSF’s programs each year. In FY 2000, they were asked to evaluate the progress made by the programs in achieving each of NSF’s Outcome Goals as well as the decision process leading to awards.

Programs are evaluated on a three-year cycle, thus for FY 2000, the years 1997, 1998 and 1999 were most likely to be the years reviewed by the COVs. This process means that each year a different subset of NSF’s programs is evaluated by a different group of experts. Hence, in FY 1999, evaluators assessed one-third of NSF's programs and in FY 2000, evaluators assessed a different one-third subset of NSF’s programs.

In addition to the programmatic assessments conducted by the COVs and ACs each year, there are program evaluations carried out by independent contractors to address specific issues. For example, in FY 2000, program evaluations undertaken include an assessment of the current status of chemical sciences including an evaluation of current trends and key opportunities in the field; a review of the merit of seafloor observatories; and an assessment of the challenges and opportunities in the nanotechnology field. These program evaluations provide important information that enables NSF program staff to make better decisions about how to best invest NSF resources. These programmatic assessments do not directly address NSF’s GPRA goals.

Summary of Results: In FY 2000, NSF used a stricter definition of success in analyzing results for the Outcome Goals. Six out of eight Outcome Goals were achieved. External evaluators consistently judged NSF’s programs to result in high quality outputs and outcomes. Overall, results are similar to those obtained in FY 1999. This is an important result, since a different subset of NSF’s portfolio is evaluated each year by a different group of external experts. Thus, this second year of reporting provides NSF with a good indication of areas needing attention. In this second year, trends are beginning to appear which has helped NSF to identify areas for future improvement.

Reports by external evaluators indicate that NSF has successfully achieved the first two outcome goals (Goal 1 and Goal 2), and has achieved with limited success the second two outcome goals (Goal 3 and Goal 4a). Evaluators identified the same areas as having limited success and in need of improvement as in FY 1999. In general, many programs are showing improvement over FY 1999 performance in the area of increasing diversity through increased participation of underrepresented groups, but reports indicate that the numbers are still lower than expected. The evaluators comment that increasing participation of underrepresented groups is an area needing more attention for NSF. Other areas needing further improvement include balance of portfolio by funding more high-risk proposals; and use of both of NSF’s merit review criteria by applicants and reviewers. Several reports note that there are clear indications that use of the merit review criteria is evident in making decisions to fund or not fund applications. Common issues identified in some reports that may result in negative impact on program performance in general, include workload and delays in processing proposals.

 

teal line

OUTCOME GOAL 1: Discoveries at and across the frontier of science and engineering.
grey line

Performance Goal: NSF is judged successful in meeting this goal when NSF awards lead to important discoveries; new knowledge and techniques, both expected and unexpected, within and across traditional disciplinary boundaries; and high-potential links across these boundaries, as judged by independent external experts.
grey line

Results: This goal was achieved. Reports by external experts indicate NSF is successful in achieving this goal in the aggregate.
grey line

NSF supports cutting edge research that yields new discoveries over time. These discoveries are essential for maintaining the nation’s capacity to excel in science and engineering and lead to new and innovative technologies that benefit society.

NSF’s key strategy for success is to support the most promising ideas in research and education, as identified through merit review of competitive proposals. Innovation and creativity, cooperative research through partnerships, and education and training are emphasized and encouraged.

 

teal line

OUTCOME GOAL 2: Connections between discoveries and their use in service to society.
grey line

Performance Goal: NSF is judged successful when the results of NSF awards are rapidly and readily available and feed, as appropriate, into education, policy development, or use by other federal agencies or the private sector, as judged by independent external experts.
grey line

Results: This goal was achieved. The results this year are similar to those reported for FY 1999. Overall, the majority of reports from external experts indicate that most NSF programs evaluated were successful in meeting this goal in FY 2000. However, some programs could show improvement, as was noted in FY 1999.
grey line

America’s national security, economic competitiveness, health, environment, quality of life, and understanding of the world around us depend on taking advantage of discoveries. Discoveries resulting from basic research and education lead to new knowledge, which often cannot be identified at the start of a project. Thus, the connections are not immediately apparent, and may only be realized decades later. The new knowledge frequently leads to applications, which can have a significant impact on society. NSF views the public accessibility of NSF generated results as well as partnerships among government, academia, and industry as critical components for the progress of science and technological innovation.

NSF’s key strategy for success in achieving this goal is through use of the merit review process to make awards for research and education activities that focus on discovery and that create or have the potential for connections with use in service to society. Potential for use in service to society is an element in the merit review criteria established by NSF and used in the decision process leading to funding.

 

teal line

OUTCOME GOAL 3: A diverse, globally oriented workforce of scientists and engineers.
grey line

Performance Goal: NSF is judged successful in meeting this goal when in the aggregate: (1) participants in NSF activities experience world-class professional practices in research and education, using modern technologies and incorporating international points of references; (2) academia, government, business, and industry recognize their quality; and (3) the science and engineering workforce shows increased participation of underrepresented groups, as judged by independent external experts.
grey line

Results: NSF’s performance toward this goal was judged successful in the aggregate by external experts in committee reports with respect to achieving a globally oriented workforce, but not fully successful with respect to achieving diversity or increased participation of underrepresented groups.
grey line

For FY 2001, this goal has been incorporated into a broader goal that focuses on achieving NSF’s desired outcome of a diverse, internationally competitive and globally engaged workforce of scientists, engineers and well-prepared citizens.

Although NSF provides only a relatively small portion of the overall U.S. investment in the development of the science and engineering workforce through its programs, this investment is particularly important to the development of the workforce of the future. The quality of the future workforce is dependent on the investment being made now to educate and train students. A diverse science and engineering workforce that is representative of the American public and able to respond effectively to a global economy is vitally important to America. As a nation, we need new technical knowledge and people trained to use that knowledge. The competence and capabilities of the nation’s science and engineering workforce keep America at the forefront of innovation and technological progress.

One of NSF’s key strategies for success in achieving this goal is by providing opportunities for participation in integrative research and education experiences. To influence the development of integrated approaches, NSF has developed a number of Foundation-wide programs intended to facilitate the integration of research and education. Each of these programs relies on NSF’s close interaction with the academic science and engineering communities to draw research and education together. NSF works to achieve this goal by making awards for research and education activities that are intended to influence the development of the science and engineering workforce, and increase the participation of underrepresented groups.

 

teal line

OUTCOME GOAL 4: Improved achievement in mathematics and science skills needed by all Americans.
grey line

Performance Goal 4a: NSF is judged to be successful in meeting this goal when, in the aggregate, the results of NSF awards lead to: (1) the development, adoption, adaptation, and implementation of effective models, products, and practices that address the needs of all students; (2) well-trained teachers who implement standards-based approaches in their classrooms; and (3) improved student performance in participating schools and districts.
grey line

Results: This goal was judged successful in a limited context in the aggregate by external experts. Activities important to achieving success toward this goal included systemic approaches, attention to teacher preparation and development, partnership with other agencies, digital libraries, graduate teaching fellows as content resources in K-12 schools, and developing a strong research base for use by practitioners. In the aggregate, when this goal was a clear objective of the programs being evaluated and when there was sufficient information available to carry out the evaluation, most reports indicated NSF programs were successful in achieving this goal. External evaluators were uncertain how to assess performance where programs did not have funds directed to these objectives, resulting in an assessment of less than successful or no assessment. In aggregating results and using reports with substantive comments and ratings which were clearly justified for each area, NSF’s performance toward this goal was judged as successful or successful in a limited context by a majority of external experts, and therefore, its result is successful in a limited context and reported as not fully achieved in FY 2000.
grey line

The results obtained in FY 1999 and FY 2000 has led NSF to refine this goal and identify ways to improve data and information collection to assess progress in achieving this goal. However, it is likely to take a few years to acquire the database necessary for full reporting of this goal.

 

Performance Goal 4b: NSF is successful in meeting this goal when over 80% of schools participating in a systemic initiative program will: (1) implement a standards-based curriculum in science and mathematics; (2) further professional development of the instructional workforce; (3) improve student achievement on a selected battery of tests, after 3 years of NSF support.
grey line

Results: This goal was achieved.
grey line

 

Performance Goal 4c: NSF is successful in meeting this goal when through systemic initiatives and related teacher enhancement programs, NSF will provide intensive professional development experiences annually for at least 65,000 pre-college teachers.
grey line

Results: This goal was achieved.
grey line

This goal addresses a need widely recognized by all Americans. Proficiency in essential skills and understanding of basic concepts in mathematics and science are critical to the earning power of individuals, to the nation’s economic competitiveness, and to the quality of life in the 21st century. NSF is the only agency that directly aims at developing such proficiencies at all levels of education.

NSF has established linkages with other agencies, and supports the development of prototypes for cooperative activities involving state and local educational agencies, and the private sector.

NSF supports a continuum of activities that enables improvement of mathematics and science skills for all Americans. These activities include educational reform at the K-12 levels and beyond; teacher education and professional development; research activities that use science and technology to inform better educational practice; and activities that bring science into the classroom and place students at the sites of exploration and discovery. Common themes that are emphasized across the Foundation include the implementation of high quality, standards-based instruction for all students; integration of research and education; and coordination of resources, policies, and practices to maximize the impact of educational investments. These activities benefit students, teachers, and the general public nationwide.

Investments in education are made to facilitate the development of essential skills in mathematics and science for all Americans through the promotion of broad-based or system-wide reforms in science, mathematics, engineering, and technology education that are based on national standards.

 

teal line

post-it note graphic

OUTCOME GOAL 5: Timely and relevant information on the national and international science and engineering enterprise.
grey line

Performance Goal 5a: Maintain FY 1999 gains in timeliness for an average of 486 days the time interval between reference period (the time to which the data refer) and reporting of data.
grey line

Results: This goal was achieved.
grey line

Performance Goal 5b: Establish a standard set of data quality measures for reporting of Science Resource Studies (SRS) products. Prepare reports on these measures for all SRS surveys and publish them in electronic formats to inform users of SRS data quality.
grey line

Results: This goal was achieved. A standard format for reporting data quality measures was developed. For each ongoing SRS survey, the information on data quality measures, critical for the user to know for proper use of the survey data, was organized into the standard reporting format. These data quality reports were placed on the SRS Web site and linked to the other information available for each SRS survey.
grey line

This goal addresses NSF’s legislative mandate to collect, interpret, and analyze data on scientific and engineering resources, and to provide a source of information for federal policy formulation. In a recent survey, a sample of the science and engineering policy community indicated that improving timeliness of data was high priority for them. Measures of data quality help users determine the reliability of the information and the extent of likely variance introduced by sampling processes.

 

Part II. Management Goals and Results in FY 2000

NSF’s six management goals for FY 2000 address three issues of high priority at the Foundation—incorporating advanced technology into NSF’s business operations; staff diversity; and Y2K compliance. In FY 2000, NSF achieved five out of six Management Goals.

Results of FY 2000 Management Goals

Number of Goals Achieved

5

Number of Goals Not Achieved

1

 

green line

Management Goal 1 – FastLane Proposals
grey line

In FY 2000, NSF will receive and process at least 60% of full proposal submissions electronically through FastLane.
grey line

Results: This goal was achieved. FastLane is a collection of system modules that allows all transactions and communications between NSF and its grantees to be facilitated via the Internet. Under development since 1994, FastLane plays a major role in NSF’s goal of achieving a paperless environment by the end of FY 2001.
grey line

In FY 2000, 81% of full proposal submissions were received and processed through FastLane. The success of this goal can be attributed to an aggressive outreach strategy combined with the efforts of an external Helpdesk to provide customer assistance.

For FY 2001, the goal is full implementation, which translates to a target goal of 95% in order to accommodate the fact that some universities do not have the technical capability to utilize FastLane, and some will experience significant difficulties in transmission.

 

green line

Management Goal 2 – Electronic Proposal Processing
grey line

By the end of FY 2000, NSF will have the technological capability of taking competitive proposals submitted electronically through the entire proposal and award/declination process without generating paper within NSF.
grey line

Results: This goal was not achieved. Historically, NSF has required paper submission once grant proposals were submitted electronically. Efforts to modernize this process have been underway for several years, and the goal is to move to electronic processing for the entire internal proposal and award process. At the start of the year, only four functions within the Peer Review Process were still paper-based, namely: Communications between NSF and the peer reviewer; Electronic panel review system; Letters to Principal Investigators (PIs) with declined proposals, and Release of review results to PIs. By the end of the year, the technological barriers to a completely paperless process had been removed within NSF, except for one remaining issue, the electronic equivalent of a signature for funding approval. Two electronic signature pilot projects were initiated during the year, and the results are being evaluated. Technological, financial, and legal issues still need to be resolved before electronic signatures can be adopted. NSF will continue to address these issues during the upcoming year. In addition, NSF will utilize the technological capabilities established this year and initiate ten pilot projects that demonstrate the paperless processing capability.
grey line

 

green line

Management Goal 3 – Staff Diversity
grey line

In FY 2000, NSF will show an increase over 1997 in the total number of hires to Science and Engineering positions from underrepresented groups.
grey line

Results: This goal was achieved. In order to ensure that the United States maintains its world leadership role in science and technology the nation must maintain a premiere cadre of scientists, mathematicians, and engineers from all segments of society. NSF is committed to diversifying its staff of scientists and engineers (S&E) both in permanent positions and in the important rotating scientist positions. Of the 113 S&E employees hired in FY 2000, 39 were female and 19 were minority. This compares to 16 female and 15 minority hires in 1997.
grey line

In FY 2001, NSF will continue to actively pursue this goal. In addition to increasing emphasis from the Director’s office, NSF will increase its recruitment presence at major program workshops and seminars, target recruitment material towards underrepresented groups, and create a registry for minorities interested in serving on NSF advisory committees and panels. These committees and panels serve as a major pipeline for recruiting rotators and visiting scientists for the Foundation.

 

green line

Management Goal 4 – FastLane Training
grey line

By the end of FY 2000, all staff will receive an orientation to FastLane, and at least 80% of program and program support staff will receive practice in using its key modules.
grey line

Results: This goal was achieved. In order for NSF to successfully implement the FastLane system it is essential that staff be oriented and properly trained. By the end of FY 2000, 100% of NSF staff had received an orientation to FastLane and 90% of program and program support staff had received practice in using its key modules.
grey line

With staff turnover, FastLane orientation will be an on-going process. Moreover, as existing modules are enhanced or new modules added, the curricula will be modified to ensure that staff stays current in the use of FastLane and other electronic systems. Since the existing staff has been fully trained and procedures have been put in place to ensure that new staff receives orientation and training, FastLane training will no longer be reported as a GPRA goal.

 

green line

Management Goal 5 – Y2K Compliance
grey line

NSF will completed all activities needed to address the Year 2000 problem for its information systems according to plan, on schedule and within budget.
grey line

Results: This goal was achieved. All activities needed to address the Year 2000 problem were completed according to plan, on schedule and within budget.
grey line

 

green line

Management Goal 6 – Project Reporting
grey line

In FY 2000, at least 85% of all eligible project reports will be submitted through the new Project Reporting System.
grey line

Results: This goal was achieved. The Project Reporting System is part of NSF’s effort to use advanced technology to create a more efficient, paperless work environment, in which information between the Foundation and its research and education customer community is done electronically via the Internet. In its first two years of use, the system has provided a wealth of information that was previously not available electronically, leading to significant changes in how NSF can respond to internal as well as external requests for information on the technical aspects of NSF awards.
grey line

During FY 2000, NSF received 92% of final project reports through the Project Reporting System. Recognizing that minor exceptions are allowed for older awards, this represents nearly full implementation. Since the Project Report System has been successfully implemented and is being fully utilized, project reporting will no longer be reported as a GPRA goal in the future, although NSF will continue to emphasize the importance of using the Project Report System with our external community.

 

 

 


Part III.
Investment Process Goals

NSF’s Investment Process Goals address various aspects of NSF’s awards process, such as the use of merit review and the need to keep the awards system open to new people and new ideas. They also help to establish customer service standards for the agency, such as the time it takes to process a proposal. In addition, the facilities oversight performance goals for all federal science, space and technology agencies are included in NSF’s set of Investment Process Goals.

In FY 2000, seven out of fourteen investment process goals were achieved. Because there were no construction projects completed in FY 2000, one of the facilities management goals did not apply.

Results of FY 2000 Investment Process Goals
Number of Goals Achieved 7
Number of Goals Not Achieved 7

 

lime green line

post-it note graphic

post-it note graphic

Investment Process Goal 1 - Use of Merit Review
grey line

At least 90% of NSF funds will be allocated to projects reviewed by appropriate peers external to NSF and selected through a merit-based competitive process.

Performance Indicator: Percent of NSF funds allocated to projects reviewed by appropriate peers external to NSF and selected through a merit-based competitive process.
grey line

Results: This goal was achieved. Based on NSF’s original goal, which included merit reviewed projects as a percentage of all NSF funding, the Foundation exceeded its goal of 90% for FY 2000. As in FY 1999, NSF allocated 95% of its funds to merit reviewed projects.
grey line

Merit review is a critical component of NSF’s decision making process for funding research and education projects. The Foundation strongly believes that award selections based on a competitive merit review process with peer evaluation ensure those ideas from the strongest researchers and educators will be identified.

During FY 2000, OMB revised the federal goal, stating that 70-90% of research and development funds should be awarded to merit reviewed projects. However, under the new definition, federally funded research and development centers (FFRDCs) and merit-reviewed scientific research with competitive selection and internal (program) evaluation will not be considered merit reviewed. Taking into account the new definition, NSF has revised its target for FY 2001 to 85%.

 

 

lime green line

Investment Process Goal 2 - Implementation of Merit Review Criteria
grey line

NSF performance in implementation of the new merit review criteria is successful when reviewers address the elements of both generic review criteria appropriate to the proposal at hand and when program officers take the information provided into account in their decisions on awards, as judged by external independent experts.

Performance Indicator: Use of merit review criteria by reviewers and program staff.
grey line

Results: This goal was not achieved. About one-third of the evaluation reports rated NSF programs as successful in their use of the new merit review criteria. In most cases where NSF was rated not fully successful, reviewers and applicants were not fully addressing the second criterion regarding the broader impacts of the proposed activity.
grey line

In FY 1999, NSF revised its merit review criteria in order to simplify and harmonize them with the NSF strategic plan. The two merit review criteria now in place, established by the National Science Board, are designed to weigh a proposal’s technical merit, creativity, educational impact, and potential benefits to society. For this goal, advisory committees for each NSF directorate use the GPRA alternative format to judge how well NSF is implementing the two merit review criteria.

Full implementation of this goal is a priority for NSF in FY 2001 and beyond. To do so requires information to be included in proposals, addressed by reviewers, and taken into account by program staff. A number of measures have been undertaken, e.g., program announcements have been modified to encourage applicants and reviewers to address these criteria in proposals and reviews and NSF has recently re-issued guidance to the applicants and reviewers, stressing the importance of using both criteria in the preparation and evaluation of proposals submitted to NSF. NSF is considering taking additional steps to ensure that applicants address these criteria when reporting project results. Also, for FY 2001, different on-screen pages have been provided in FastLane, NSF’s electronic data system, so reviewers can address each merit-review criterion separately. The performance data will be collected from the FastLane database.

 

lime green line

Investment Process Goal 3 – Customer Service/General
grey line

Identify possible reasons for customer dissatisfaction with NSF’s merit review system and with NSF’s complaint system.

Performance Indicator: Results of NSF applicant survey, awardee survey, and regional grants seminars.
grey line

Results: This goal was achieved. In FY 2000, NSF commissioned additional surveys including the ACSI* survey of awardees only and regional grants seminar surveys, designed to identify the reasons for Principal Investigator dissatisfaction with the timeliness and efficiency of the proposal process, the quality and fairness of the merit review process, and the handling of customer complaints.
grey line

The survey results indicate that NSF customers’ primary concern regarding the timeliness and efficiency of the proposal process is the time it takes NSF to reach a funding decision. NSF is striving to improve the time to decision (see Goal 7). Applicants who stated that they have a specific problem or concern with the quality or fairness of merit review identified two primary concerns: reviews were inappropriate (i.e., reviews did not seem to adequately address the proposed project, in the opinion of the applicant) and reviews were uneven (i.e., the range of review scores included both high and low scores). Finally, survey participants who stated that they had complained to NSF described the nature of their complaints primarily in three ways: (1) concern about overall quality or fairness of proposal merit review process; (2) problem submitting a proposal, review, or project via FastLane; and (3) problem making timely contact with appropriate person at NSF. This feedback is helping NSF to focus its efforts to improve customer service.

*For the past two years, NSF has participated along with about 30 other federal agencies in a national assessment of customer satisfaction. The mechanism used to assess customer satisfaction is the American Customer Satisfaction Index (ACSI), a cross-industry index of customer satisfaction. This index is generated by the University of Michigan based on customer surveys.

 

lime green line

Investment Process Goal 4 – Customer Service/General
grey line

Identify best practices and training necessary for NSF staff to conduct merit review and answer questions about the review criteria and process. Identify best practices and training necessary for NSF staff to answer questions from the community and to deal with complaints in a forthright manner.

Performance Indicator: Development of models of best practices and NSF staff training, where appropriate.
grey line

Results: This goal was not achieved. During FY 2000, NSF conducted customer service surveys and solicited other forms of feedback in an effort to pinpoint specific customer issues and to identify effective practices for handling customer complaints within NSF. Further, other federal agencies were examined to locate a model with similar customer interactions, but no appropriate model was identified. As a result of this input, some priorities for action have been identified. However, models of best practices and NSF staff training are still being developed in FY 2001. NSF continues to place great importance on these issues and will complete this effort in FY 2001. In addition, NSF will pilot the best of the models in NSF divisions and provide specific customer service training to NSF staff.
grey line

 

lime green line

post-it note graphic

Investment Process Goal 5 – Customer Service/General
grey line

Improve NSF’s overall ACSI index compared to the FY 1999 index of 57, on a scale of 0-100. (See Investment Process Goal 3.)

Performance Indicator: Results of the ACSI survey.
grey line

Results: This goal was achieved. NSF achieved an ACSI index of 58 in FY 2000.
grey line

The Foundation's 1999 ACSI results indicated that NSF grant applicants generally held NSF in high regard and gave it high marks for the accessibility and usefulness of its information. However, the Foundation received only mid-level evaluations for its merit review process and for its handling of customer complaints. NSF began to examine these issues through additional customer surveys in FY 2000, per Investment Process Goal 3 above.

The 2000 ACSI survey indicated that NSF improved slightly in two key areas: (1) timeliness and efficiency of the proposal process and (2) quality and fairness of merit review. These were two of the areas of greatest concern from the FY 1999 survey. NSF will continue to address customer concerns; see Investment Process Goals 6 and 7.

lime green line

post-it note graphic

Investment Process Goal 6 – Time to Prepare Proposals
grey line

95% of program announcements and solicitations will be available at least three months prior to proposal deadlines or target dates.

Customer service standard: To make program announcements and solicitations available to relevant individuals and organizations at least three months prior to the proposal deadline or target date.

Performance Indicator: Percent of program announcements and solicitations available at least three months prior to proposal deadlines or target dates.
grey line

Results: This goal was not achieved. This customer service standard was established in response to a survey of NSF applicants who indicated that having a minimum of three months between program announcements and proposal deadlines was valued highly. In FY 2000, 89% of program announcements and solicitations were made available at least three months prior to their deadline/target date. Approximately 8% of program announcements and solicitations missed the 90-day time limit by fewer than 5 days. This is a significant improvement over FY 1999, when 75% of announcements met the three-month standard.
grey line

In FY 2000, a Web-based system for creating program announcements was put into place; this system is expected to decrease the time required for an announcement to be posted on the NSF Web site, which should aid the agency in achieving this goal. However, since this is the first year of implementation, not all announcements are being prepared using this system. NSF expects that there will be increased usage of this system and additional progress toward meeting this goal next year.

 

lime green line

post-it note graphic

Investment Process Goal 7 – Time to Decision
grey line

Maintain the FY 1999 goal to process 70% of proposals within six months of receipt, improving upon the FY 1998 baseline of 59%.

Customer Service Standard: NSF’s long term goal continues to be processing 95% of proposals within six months of receipt. In other words, NSF should be able to tell applicants whether their proposals have been declined or recommended for funding within six months of receiving them.

Performance Indicator: Percent of proposals processed within six months of receipt.
grey line

Results: This goal was not achieved. This customer service standard was established in response to a survey of NSF applicants who indicated that processing proposals within six months of receipt was valued highly. In FY 1999, 58% of proposals were processed within six months of receipt, somewhat better than the 52% average rate over the last five years, but nevertheless short of the 70% goal. In FY 2000, 54% of proposals were processed within six months of receipt, while an additional 35% of proposals were processed between six and nine months of receipt.

grey line

NSF recognizes the validity of the community’s interest in this customer service standard and is striving to expedite the time between proposal submission and agency decision without jeopardizing the quality and integrity of the review process. One factor leading to delay in awards processing is that some programs at NSF prefer to conduct merit review through the mail rather than by a panel. Mail reviews often take longer to implement. Another factor is that programs tend to hold some highly rated proposals until the end of the fiscal year, or even into the next fiscal year, in anticipation that more funds might become available. In FY 2000, a few programs reported temporary staffing shortages, which slowed down their review process. This situation has been corrected.

In FY 2001, NSF staff will work towards shortening the award processing time by making more effective use of electronic mechanisms in conducting the review, working cooperatively to eliminate overloads and bottlenecks, and carefully tracking the stage of processing and age of all proposals. In addition, some directorates are reconsidering the practice of holding over proposals for potential funding until the next fiscal year, while some divisions have added “performance on prompt handling of proposals” to their performance evaluation criteria. Moreover, NSF is committed to increasing staffing in FY 2001, to accommodate the anticipated increase in proposals associated with the budget increase and the major initiatives. This goal will be maintained in FY 2001.

 

lime green line

post-it note graphic

Investment Process Goal 8 – Maintaining Openness in the System
grey line

The percentage of competitive research grants going to new investigators will be at least 30%, 3% over the FY 1998 baseline of 27%.

Performance Indicator: Percent of competitive research grants going to new investigators.
grey line

Results: This goal was not achieved. The percentage of competitive research grants to new investigators was 28% in FY 2000, one percent higher than in FY 1999.
grey line

NSF believes that it is important that the proposal and award process be open to new people and new ideas, to help ensure that NSF is supporting research at the frontier of science and engineering. NSF is committed to maintaining openness in the system and will strive to increase the percentage of awards to new investigators.

This goal will be maintained in FY 2001. This is a challenging goal for NSF. NSF will continue to seek creative and innovative proposals from new investigators. Program staff will attend scientific meetings, conferences, and conventions and will conduct site visits to promote awareness of the research opportunities at NSF and to encourage new investigators to submit proposals. NSF will examine trends, such as whether the pool of new investigators is smaller than in previous years or whether they are submitting fewer proposals, and if needed, use this information to modify targets in the future.

 

lime green line

Investment Process Goal 9 – Proposer Attention to Integration of Research and Education
grey line

NSF will develop a plan and system to request that Principal Investigators (PIs) address the integration of research and education in their proposals, and develop and implement a system to verify that PIs have done so.

Performance Indicator: Outreach to community; implementation of system to verify that PIs address the integration of research and education in proposals.
grey line

Results: This goal was achieved.
grey line

In FY 2000, NSF implemented an electronic program announcement template clearance process (PAT) that is used by NSF staff to generate announcements and solicitations. Use of the PAT ensures that PIs are asked to address the integration of research and education in all announcements and solicitations. In addition, the Foundation has included language in the Proposal and Award Manual, the Grant Proposal Guide, and the FY 2000 Guide to Programs regarding the integration of research and education.

In order to verify that PIs are addressing the integration of research and education, NSF asks Committees of Visitors (COVs) to assess whether the broader impacts of the proposed activity are being addressed in proposals. The COV reporting template has been modified to explicitly address the use of both merit review criteria.

This goal will not be continued in FY 2001, but will be replaced by goals addressing broader use of the merit review criteria by reviewers and staff, which encompasses this goal.

 

lime green line

Investment Process Goal 10 – Reviewer Attention to Integration of Research and Education
grey line

NSF will develop and implement a system/mechanism to request and track reviewer comments tied to the merit review criterion, “What are the broader impacts of the proposed activity?”

Performance Indicator: Outreach to community; implementation of system to track reviewer comments.
grey line

Results: This goal was achieved.
grey line

During FY 2000, screens were added in FastLane, NSF’s electronic proposal and review system, so reviewers can address each merit-review criterion separately. The performance data will be collected from the FastLane database.

NSF has modified program announcements to encourage applicants and reviewers to address these criteria in proposals and reviews. NSF has recently re-issued guidance to the applicants and reviewers, stressing the importance of addressing both merit review criteria in the preparation and evaluation of proposals submitted to NSF. NSF staff continue to stress the importance of reviewers addressing the “broader impacts” criterion whenever they attend NSF sponsored seminars, science meetings, site visits, conferences, and conventions. NSF is considering taking additional steps to ensure that applicants address these criteria when reporting project results.

 

lime green line

Investment Process Goal 11 – Diversity of NSF applicants
grey line

NSF will identify mechanisms to increase the number of women and underrepresented minorities in the proposal applicant pool, and will identify mechanisms to retain that pool. (Revised goal; no baseline.)

Performance Indicator: Mechanisms to attract proposals from members of underrepresented groups in order to increase the total applicant pool; mechanisms to retain the applicant pool.
grey line

Results: This goal was achieved.
grey line

NSF is committed to the principle of diversity and deems it central to the programs, projects, and activities it considers and supports. NSF continues to work toward increasing diversity in its applicant pool:

  • To place the issue on equal footing as the quality of research being supported, NSF issued Important Notice No. 125 to presidents of universities and colleges encouraging PIs to address the merit review criterion – what are the broader impacts of the proposed activity, which embraces integrating diversity into all NSF supported activities;
  • Developed and increased funding for specialized programs designed to promote diversity;
  • Recruited members of underrepresented groups for merit review panels, COVs, and NSF workshops and conferences; and
  • Strongly encouraged women, minorities, and persons with disabilities to compete fully in NSF programs.

 

lime green line

Investment Process Goals 12 to 15 – Facilities Oversight
grey line

The following goals are for federal science, space and technology agencies that support construction projects and have responsibility for managing facilities (NSF, NASA, DOE).

In FY 1999, NSF developed a general facilities reporting template for use in reporting on the facilities management goals. This reporting system was linked to the new Project Reporting System as a module of the existing FastLane system. Facility managers located at the facility site report data to NSF using this reporting system.

In FY 2000, NSF reviewed the data collection and reporting effort and made modifications to the system where appropriate. This included allowing for reporting on construction/upgrade activities at facilities funded through the Research and Related Activities account, refining the on-screen language to be more clear and to more accurately address the facilities goals, automating most of the output, and instituting a stage for collecting estimates.

 

Construction and Upgrade:
Performance Indicators: Comparison with planned annual cost, planned annual schedule, and planned total cost.
grey line

GOAL 12: Maintain the FY 1999 goal to keep construction and upgrades within annual expenditure plan, not to exceed 110% of estimates.
grey line

Results: This goal was achieved. Of the 11 construction and upgrade projects supported by NSF, all were within annual expenditure plans.
grey line

 

GOAL 13: Maintain the FY 1999 goal to keep construction and upgrades within annual schedule, total time required for major components of the project not to exceed 110% of estimates.
grey line

Results: This goal was not achieved. Of the 11 construction and upgrade projects supported by NSF, seven were within the annual schedule goal. In several cases, missed milestones were due to circumstances beyond the project manager’s control. For example, one construction project was dependent upon the research and development of new instrumentation, the results of which were delayed. In another project, the missed milestone was due to difficulty obtaining required parts, non-performance of a sub-contractor, and underestimation of the complexity of the work. In FY 2001, NSF program managers are working more closely with project managers to ensure all NSF-supported construction/upgrade projects achieve this goal.
grey line

 

GOAL 14: For all construction and upgrade projects initiated after 1996, when current planning processes were put in place, keep total cost within 110% of estimates made at the initiation of construction.
grey line

Results: This goal did not apply; there were no construction projects completed in FY 2000.
grey line

 

Operations:
Performance Indicator: Comparison to scheduled operating time.
grey line

GOAL 15: Maintain the FY 1999 goal to keep operating time lost due to unscheduled downtime to less than 10% of the total scheduled possible operating time.
grey line

Results: This goal was not achieved. Of the 26 reporting facilities, 22 met the goal of keeping unscheduled downtime to below 10% of the total scheduled operating time. NSF program staff will work more closely with project managers to ensure that all achieve this goal in FY 2001.
grey line