FY 2000 Performance Results and Related Issues
The purpose of this chapter is to provide a more detailed explanation of the performance results presented in the Managements Discussion and Analysis and to discuss performance-related topics. For a complete and comprehensive discussion of NSFs performance goals, final results and related issues, see NSFs FY 2000 GPRA Performance Report, available on NSFs Web site (http://www.nsf.gov/od/gpra/).
This is the second year NSF is reporting performance results. NSF began implementing GPRA in 1997, by developing an agency GPRA Strategic Plan. In compliance with the Results Act, NSF updated this Strategic Plan last fall. NSFs GPRA Strategic Plan provides the guiding framework for NSFs FY 2000 Annual Performance Plan, which was developed in conjunction with the development of NSFs FY 2000 budget. The concurrent development of the performance plan and the budget creates a direct link between programmatic activities and the achievement of NSFs strategic goals. A more detailed description of how performance goals and program activities are linked to the budget structure follows.
GPRA implementation has been a particular challenge for agencies like NSF whose mission involves research activities. This is primarily due to: (1) the difficulty of linking research outcomes to annual investments and the agencys annual budget and (2) the fact that assessing the results of research is inherently retrospective and requires qualitative judgments of expertise. NSF has developed an alternative format that has been approved by OMB, using external expert review panels to assess research results and reporting research outcome goals utilizing a qualitative scale. The use of external expert panels to review research results and outcomes is a common, long-standing practice used by the academic research community.
NSFs FY 2000 Annual Performance Plan includes three sets of goals:
These three sets of goals are mutually supportive. The longer term desired results of NSF awards are reflected in the Outcome Goals. Achieving the desired Outcome Goals depends in part on the quality of the investment process, which is related to the efficiency and effectiveness of the agencys administration and management. The Investment Process Goals and Management Goals are necessary to ensure that the longer term Outcome Goals will be achieved.
NSFs key strategy for success is through use of a rigorous merit review process in making awards for activities that will influence research and education in math, science and engineering, both directly and indirectly.
NSFs five Outcome Goals address the results of NSFs grants for research and education in science and engineering and relate directly to the mission of the agency. Outcome Goal 1 (Discoveries at and across the frontier of science and engineering) and Outcome Goal 2 (Connections between discoveries and their use in service to society) address NSFs research grants. Outcome Goal 3 (A diverse, globally oriented workforce of scientists and engineers) and Outcome Goal 4 (Improved achievement in mathematics and science skills needed by all Americans) address NSFs education grants. Outcome Goal 5 (Timely and relevant information on the national and international science and engineering enterprise) addresses NSFs legislative mandate to collect, interpret and analyze data on scientific and engineering resources, and to provide a source of information for federal policy formulation. This goal applies to both research and education activities.
The following chart shows how NSFs Outcome Goals are linked to the NSF budget structure. NSF receives five Congressional appropriations: Research and Related Activities (R&RA); Major Research Equipment (MRE); Education and Human Resources (EHR); and Salaries and Expenses (S&E). The fifth appropriation funds the Office of Inspector General. Outcome Goals 1,2 and 5 are funded through the R&RA and MRE appropriations and Outcome Goals 3 and 4 are funded through the EHR appropriation. Because the S&E appropriation funds the internal administration and management of the agency, S&E funding applies to all the Outcome Goals, and as indicated in the Statement of Net Cost, is proportionately prorated between research and education programs based on each programs direct cost.
The Linkage Between NSFs Outcome Goals and the Budget Structure
Data Verification and Validation
In FY 2000, NSF engaged PricewaterhouseCoopers LLP (PwC) to document and assess the process NSF uses to collect, maintain and report data for selected performance goals. PwC was also tasked with re-calculating the measures and assessing the reliability of the supporting processes. PwC mapped NSF procedures against the U.S. General Accounting Office (GAO) criteria for supporting processes to be considered valid and verifiable. For the Outcome Goals, PwC verified and validated the results.
In their final reports of their reviews of the Investment Process Goals and the Management Goals, PwC concluded that NSF was reporting its GPRA measures with sufficient accuracy such that any errors, should they exist, would not be significant enough to change the readers interpretation as to the Foundations success in meeting the supporting performance goal. . . Furthermore, PwC concluded that NSF relies on sound business processes, system and application controls, and manual checks of system queries to confirm the accuracy of reported data. We believe that these processes are valid and verifiable.
2000, NSF was considerably more rigorous in evaluating goal achievement.
Options for grading were limited to either successful or not successful.
Justification was required for successful grades that used qualitative
measures. Finally, for the Outcome Goals, an external firm, PricewaterhouseCoopers
LLP, was engaged to verify the achievement data tables for the Outcome
Goals. While NSF was successful in achieving 64% of its goals in FY 2000
as compared with achieving 78% for FY 1999, the results of the second
year are very similar to the first. Positive trends were evident in some
of the goals, indicating movement in a desirable direction. The areas
identified as needing improvement continue to be:
NSFs long-term Outcome Goals address how the investments made by programs have led to results important to the broad mission of the agency. These Outcome Goals do not lend themselves to quantitative reporting, therefore NSF has developed an alternative format -- a qualitative scale that allows NSF to report whether or not the agency has been successful in achieving its Outcome Goals. Also, because many research results appear long after an investment is made, in some cases ten years or more, this assessment report of NSFs program performance is retrospective. That is, the outcome results reported in FY 2000 are from investments made prior to FY 2000. The results of the investments made in FY 2000 will not begin to be reported until beyond FY 2000.
In FY 2000, NSFs Outcome Goals 1, 2, 3 and 4a are expressed in a non-quantitative, qualitative form, each critical to ensure the progress of science. The results reported for the year are collected, tabulated and summarized by aggregating many individual reports prepared by committees of external experts assessing individual programs or clusters of programs throughout the fiscal year. The assessment is retrospective, covering a subset of one-third of NSFs programs that represent activities spanning the entire agency over a period of three years or more.
How Research Results Are Assessed: Committees of external experts are carefully selected to provide NSF with an objective, independent assessment of programs for process and results. These committees, known as Committees of Visitors (COVs) and Advisory Committees (ACs), assess approximately one-third of NSFs programs each year. In FY 2000, they were asked to evaluate the progress made by the programs in achieving each of NSFs Outcome Goals as well as the decision process leading to awards.
Programs are evaluated on a three-year cycle, thus for FY 2000, the years 1997, 1998 and 1999 were most likely to be the years reviewed by the COVs. This process means that each year a different subset of NSFs programs is evaluated by a different group of experts. Hence, in FY 1999, evaluators assessed one-third of NSF's programs and in FY 2000, evaluators assessed a different one-third subset of NSFs programs.
In addition to the programmatic assessments conducted by the COVs and ACs each year, there are program evaluations carried out by independent contractors to address specific issues. For example, in FY 2000, program evaluations undertaken include an assessment of the current status of chemical sciences including an evaluation of current trends and key opportunities in the field; a review of the merit of seafloor observatories; and an assessment of the challenges and opportunities in the nanotechnology field. These program evaluations provide important information that enables NSF program staff to make better decisions about how to best invest NSF resources. These programmatic assessments do not directly address NSFs GPRA goals.
Summary of Results: In FY 2000, NSF used a stricter definition of success in analyzing results for the Outcome Goals. Six out of eight Outcome Goals were achieved. External evaluators consistently judged NSFs programs to result in high quality outputs and outcomes. Overall, results are similar to those obtained in FY 1999. This is an important result, since a different subset of NSFs portfolio is evaluated each year by a different group of external experts. Thus, this second year of reporting provides NSF with a good indication of areas needing attention. In this second year, trends are beginning to appear which has helped NSF to identify areas for future improvement.
Reports by external evaluators indicate that NSF has successfully achieved the first two outcome goals (Goal 1 and Goal 2), and has achieved with limited success the second two outcome goals (Goal 3 and Goal 4a). Evaluators identified the same areas as having limited success and in need of improvement as in FY 1999. In general, many programs are showing improvement over FY 1999 performance in the area of increasing diversity through increased participation of underrepresented groups, but reports indicate that the numbers are still lower than expected. The evaluators comment that increasing participation of underrepresented groups is an area needing more attention for NSF. Other areas needing further improvement include balance of portfolio by funding more high-risk proposals; and use of both of NSFs merit review criteria by applicants and reviewers. Several reports note that there are clear indications that use of the merit review criteria is evident in making decisions to fund or not fund applications. Common issues identified in some reports that may result in negative impact on program performance in general, include workload and delays in processing proposals.
GOAL 1: Discoveries at and across
the frontier of science and engineering.
Goal: NSF is judged successful in meeting this goal when NSF awards
lead to important discoveries; new knowledge and techniques, both expected
and unexpected, within and across traditional disciplinary boundaries;
and high-potential links across these boundaries, as judged by independent
This goal was achieved. Reports by external experts indicate NSF
is successful in achieving this goal in the aggregate.
NSF supports cutting edge research that yields new discoveries over time. These discoveries are essential for maintaining the nations capacity to excel in science and engineering and lead to new and innovative technologies that benefit society.
NSFs key strategy for success is to support the most promising ideas in research and education, as identified through merit review of competitive proposals. Innovation and creativity, cooperative research through partnerships, and education and training are emphasized and encouraged.
GOAL 2: Connections between
discoveries and their use in service to society.
Goal: NSF is judged successful when the results of NSF awards are
rapidly and readily available and feed, as appropriate, into education,
policy development, or use by other federal agencies or the private sector,
as judged by independent external experts.
This goal was achieved. The results this year are similar to those
reported for FY 1999. Overall, the majority of reports from external experts
indicate that most NSF programs evaluated were successful in meeting this
goal in FY 2000. However, some programs could show improvement, as was
noted in FY 1999.
Americas national security, economic competitiveness, health, environment, quality of life, and understanding of the world around us depend on taking advantage of discoveries. Discoveries resulting from basic research and education lead to new knowledge, which often cannot be identified at the start of a project. Thus, the connections are not immediately apparent, and may only be realized decades later. The new knowledge frequently leads to applications, which can have a significant impact on society. NSF views the public accessibility of NSF generated results as well as partnerships among government, academia, and industry as critical components for the progress of science and technological innovation.
NSFs key strategy for success in achieving this goal is through use of the merit review process to make awards for research and education activities that focus on discovery and that create or have the potential for connections with use in service to society. Potential for use in service to society is an element in the merit review criteria established by NSF and used in the decision process leading to funding.
GOAL 3: A diverse, globally oriented workforce of scientists and engineers.
Goal: NSF is
judged successful in meeting this goal when in the aggregate: (1) participants
in NSF activities experience world-class professional practices in research
and education, using modern technologies and incorporating international
points of references; (2) academia, government, business, and industry
recognize their quality; and (3) the science and engineering workforce
shows increased participation of underrepresented groups, as judged by
independent external experts.
NSFs performance toward this goal was judged successful in the aggregate
by external experts in committee reports with respect to achieving a globally
oriented workforce, but not fully successful with respect to achieving
diversity or increased participation of underrepresented groups.
For FY 2001, this goal has been incorporated into a broader goal that focuses on achieving NSFs desired outcome of a diverse, internationally competitive and globally engaged workforce of scientists, engineers and well-prepared citizens.
Although NSF provides
only a relatively small portion of the overall U.S. investment in the
development of the science and engineering workforce through its programs,
this investment is particularly important to the development of the workforce
of the future. The quality of the future workforce is dependent on the
investment being made now to educate and train students. A diverse science
and engineering workforce that is representative of the American public
and able to respond effectively to a global economy is vitally important
to America. As a nation, we need new technical knowledge and people trained
to use that knowledge. The competence and capabilities of the nations
science and engineering workforce keep America at the forefront of innovation
and technological progress.
One of NSFs key strategies for success in achieving this goal is by providing opportunities for participation in integrative research and education experiences. To influence the development of integrated approaches, NSF has developed a number of Foundation-wide programs intended to facilitate the integration of research and education. Each of these programs relies on NSFs close interaction with the academic science and engineering communities to draw research and education together. NSF works to achieve this goal by making awards for research and education activities that are intended to influence the development of the science and engineering workforce, and increase the participation of underrepresented groups.
GOAL 4: Improved achievement
in mathematics and science skills needed by all Americans.
Goal 4a: NSF is judged to be successful in meeting this goal
when, in the aggregate, the results of NSF awards lead to: (1) the development,
adoption, adaptation, and implementation of effective models, products,
and practices that address the needs of all students; (2) well-trained
teachers who implement standards-based approaches in their classrooms;
and (3) improved student performance in participating schools and districts.
This goal was judged successful in a limited context in the aggregate
by external experts. Activities important to achieving success toward
this goal included systemic approaches, attention to teacher preparation
and development, partnership with other agencies, digital libraries, graduate
teaching fellows as content resources in K-12 schools, and developing
a strong research base for use by practitioners. In the aggregate, when
this goal was a clear objective of the programs being evaluated and when
there was sufficient information available to carry out the evaluation,
most reports indicated NSF programs were successful in achieving this
goal. External evaluators were uncertain how to assess performance where
programs did not have funds directed to these objectives, resulting in
an assessment of less than successful or no assessment. In aggregating
results and using reports with substantive comments and ratings which
were clearly justified for each area, NSFs performance toward this
goal was judged as successful or successful in a limited context by a
majority of external experts, and therefore, its result is successful
in a limited context and reported as not fully achieved in FY 2000.
The results obtained in FY 1999 and FY 2000 has led NSF to refine this goal and identify ways to improve data and information collection to assess progress in achieving this goal. However, it is likely to take a few years to acquire the database necessary for full reporting of this goal.
Goal 4b: NSF is successful in meeting this goal when over 80%
of schools participating in a systemic initiative program will: (1) implement
a standards-based curriculum in science and mathematics; (2) further professional
development of the instructional workforce; (3) improve student achievement
on a selected battery of tests, after 3 years of NSF support.
This goal was achieved.
Goal 4c: NSF is successful in meeting this goal when through
systemic initiatives and related teacher enhancement programs, NSF will
provide intensive professional development experiences annually for at
least 65,000 pre-college teachers.
This goal was achieved.
This goal addresses a need widely recognized by all Americans. Proficiency in essential skills and understanding of basic concepts in mathematics and science are critical to the earning power of individuals, to the nations economic competitiveness, and to the quality of life in the 21st century. NSF is the only agency that directly aims at developing such proficiencies at all levels of education.
NSF has established linkages with other agencies, and supports the development of prototypes for cooperative activities involving state and local educational agencies, and the private sector.
NSF supports a continuum of activities that enables improvement of mathematics and science skills for all Americans. These activities include educational reform at the K-12 levels and beyond; teacher education and professional development; research activities that use science and technology to inform better educational practice; and activities that bring science into the classroom and place students at the sites of exploration and discovery. Common themes that are emphasized across the Foundation include the implementation of high quality, standards-based instruction for all students; integration of research and education; and coordination of resources, policies, and practices to maximize the impact of educational investments. These activities benefit students, teachers, and the general public nationwide.
Investments in education are made to facilitate the development of essential skills in mathematics and science for all Americans through the promotion of broad-based or system-wide reforms in science, mathematics, engineering, and technology education that are based on national standards.
NSFs six management goals for FY 2000 address three issues of high priority at the Foundationincorporating advanced technology into NSFs business operations; staff diversity; and Y2K compliance. In FY 2000, NSF achieved five out of six Management Goals.
Goal 1 FastLane Proposals
2000, NSF will receive and process at least 60% of full proposal submissions
electronically through FastLane.
This goal was achieved. FastLane is a collection of system modules
that allows all transactions and communications between NSF and its grantees
to be facilitated via the Internet. Under development since 1994, FastLane
plays a major role in NSFs goal of achieving a paperless environment
by the end of FY 2001.
In FY 2000, 81% of full proposal submissions were received and processed through FastLane. The success of this goal can be attributed to an aggressive outreach strategy combined with the efforts of an external Helpdesk to provide customer assistance.
For FY 2001, the goal is full implementation, which translates to a target goal of 95% in order to accommodate the fact that some universities do not have the technical capability to utilize FastLane, and some will experience significant difficulties in transmission.
Goal 2 Electronic Proposal Processing
By the end of FY
2000, NSF will have the technological capability of taking competitive
proposals submitted electronically through the entire proposal and award/declination
process without generating paper within NSF.
This goal was not achieved. Historically, NSF has required paper
submission once grant proposals were submitted electronically. Efforts
to modernize this process have been underway for several years, and the
goal is to move to electronic processing for the entire internal proposal
and award process. At the start of the year, only four functions within
the Peer Review Process were still paper-based, namely: Communications
between NSF and the peer reviewer; Electronic panel review system; Letters
to Principal Investigators (PIs) with declined proposals, and Release
of review results to PIs. By the end of the year, the technological barriers
to a completely paperless process had been removed within NSF, except
for one remaining issue, the electronic equivalent of a signature for
funding approval. Two electronic signature pilot projects were initiated
during the year, and the results are being evaluated. Technological, financial,
and legal issues still need to be resolved before electronic signatures
can be adopted. NSF will continue to address these issues during the upcoming
year. In addition, NSF will utilize the technological capabilities established
this year and initiate ten pilot projects that demonstrate the paperless
Goal 3 Staff Diversity
In FY 2000, NSF will
show an increase over 1997 in the total number of hires to Science and
Engineering positions from underrepresented groups.
This goal was achieved. In order to ensure that the United States
maintains its world leadership role in science and technology the nation
must maintain a premiere cadre of scientists, mathematicians, and engineers
from all segments of society. NSF is committed to diversifying its staff
of scientists and engineers (S&E) both in permanent positions and
in the important rotating scientist positions. Of the 113 S&E employees
hired in FY 2000, 39 were female and 19 were minority. This compares to
16 female and 15 minority hires in 1997.
In FY 2001, NSF will continue to actively pursue this goal. In addition to increasing emphasis from the Directors office, NSF will increase its recruitment presence at major program workshops and seminars, target recruitment material towards underrepresented groups, and create a registry for minorities interested in serving on NSF advisory committees and panels. These committees and panels serve as a major pipeline for recruiting rotators and visiting scientists for the Foundation.
Goal 4 FastLane Training
By the end of FY
2000, all staff will receive an orientation to FastLane, and at least
80% of program and program support staff will receive practice in using
its key modules.
This goal was achieved. In order for NSF to successfully implement
the FastLane system it is essential that staff be oriented and properly
trained. By the end of FY 2000, 100% of NSF staff had received an orientation
to FastLane and 90% of program and program support staff had received
practice in using its key modules.
With staff turnover, FastLane orientation will be an on-going process. Moreover, as existing modules are enhanced or new modules added, the curricula will be modified to ensure that staff stays current in the use of FastLane and other electronic systems. Since the existing staff has been fully trained and procedures have been put in place to ensure that new staff receives orientation and training, FastLane training will no longer be reported as a GPRA goal.
Goal 5 Y2K Compliance
NSF will completed
all activities needed to address the Year 2000 problem for its information
systems according to plan, on schedule and within budget.
This goal was achieved. All activities needed to address the Year
2000 problem were completed according to plan, on schedule and within
Goal 6 Project Reporting
In FY 2000, at least
85% of all eligible project reports will be submitted through the new
Project Reporting System.
This goal was achieved. The Project Reporting System is part of
NSFs effort to use advanced technology to create a more efficient,
paperless work environment, in which information between the Foundation
and its research and education customer community is done electronically
via the Internet. In its first two years of use, the system has provided
a wealth of information that was previously not available electronically,
leading to significant changes in how NSF can respond to internal as well
as external requests for information on the technical aspects of NSF awards.
During FY 2000, NSF received 92% of final project reports through the Project Reporting System. Recognizing that minor exceptions are allowed for older awards, this represents nearly full implementation. Since the Project Report System has been successfully implemented and is being fully utilized, project reporting will no longer be reported as a GPRA goal in the future, although NSF will continue to emphasize the importance of using the Project Report System with our external community.
NSFs Investment Process Goals address various aspects of NSFs awards process, such as the use of merit review and the need to keep the awards system open to new people and new ideas. They also help to establish customer service standards for the agency, such as the time it takes to process a proposal. In addition, the facilities oversight performance goals for all federal science, space and technology agencies are included in NSFs set of Investment Process Goals.
In FY 2000, seven out of fourteen investment process goals were achieved. Because there were no construction projects completed in FY 2000, one of the facilities management goals did not apply.
Process Goal 2 - Implementation of Merit
NSF performance in implementation of the new merit review criteria is successful when reviewers address the elements of both generic review criteria appropriate to the proposal at hand and when program officers take the information provided into account in their decisions on awards, as judged by external independent experts.
of merit review criteria by reviewers and program staff.
This goal was not achieved. About one-third of the evaluation reports
rated NSF programs as successful in their use of the new merit review
criteria. In most cases where NSF was rated not fully successful, reviewers
and applicants were not fully addressing the second criterion regarding
the broader impacts of the proposed activity.
In FY 1999, NSF revised its merit review criteria in order to simplify and harmonize them with the NSF strategic plan. The two merit review criteria now in place, established by the National Science Board, are designed to weigh a proposals technical merit, creativity, educational impact, and potential benefits to society. For this goal, advisory committees for each NSF directorate use the GPRA alternative format to judge how well NSF is implementing the two merit review criteria.
Full implementation of this goal is a priority for NSF in FY 2001 and beyond. To do so requires information to be included in proposals, addressed by reviewers, and taken into account by program staff. A number of measures have been undertaken, e.g., program announcements have been modified to encourage applicants and reviewers to address these criteria in proposals and reviews and NSF has recently re-issued guidance to the applicants and reviewers, stressing the importance of using both criteria in the preparation and evaluation of proposals submitted to NSF. NSF is considering taking additional steps to ensure that applicants address these criteria when reporting project results. Also, for FY 2001, different on-screen pages have been provided in FastLane, NSFs electronic data system, so reviewers can address each merit-review criterion separately. The performance data will be collected from the FastLane database.
Process Goal 3 Customer Service/General
Identify possible reasons for customer dissatisfaction with NSFs merit review system and with NSFs complaint system.
of NSF applicant survey, awardee survey, and regional grants seminars.
This goal was achieved. In FY 2000, NSF commissioned additional
surveys including the ACSI* survey of awardees only and regional grants
seminar surveys, designed to identify the reasons for Principal Investigator
dissatisfaction with the timeliness and efficiency of the proposal process,
the quality and fairness of the merit review process, and the handling
of customer complaints.
The survey results indicate that NSF customers primary concern regarding the timeliness and efficiency of the proposal process is the time it takes NSF to reach a funding decision. NSF is striving to improve the time to decision (see Goal 7). Applicants who stated that they have a specific problem or concern with the quality or fairness of merit review identified two primary concerns: reviews were inappropriate (i.e., reviews did not seem to adequately address the proposed project, in the opinion of the applicant) and reviews were uneven (i.e., the range of review scores included both high and low scores). Finally, survey participants who stated that they had complained to NSF described the nature of their complaints primarily in three ways: (1) concern about overall quality or fairness of proposal merit review process; (2) problem submitting a proposal, review, or project via FastLane; and (3) problem making timely contact with appropriate person at NSF. This feedback is helping NSF to focus its efforts to improve customer service.
*For the past two years, NSF has participated along with about 30 other federal agencies in a national assessment of customer satisfaction. The mechanism used to assess customer satisfaction is the American Customer Satisfaction Index (ACSI), a cross-industry index of customer satisfaction. This index is generated by the University of Michigan based on customer surveys.
Process Goal 4 Customer
Identify best practices and training necessary for NSF staff to conduct merit review and answer questions about the review criteria and process. Identify best practices and training necessary for NSF staff to answer questions from the community and to deal with complaints in a forthright manner.
of models of best practices and NSF staff training, where appropriate.
This goal was not achieved. During FY 2000, NSF conducted customer
service surveys and solicited other forms of feedback in an effort to
pinpoint specific customer issues and to identify effective practices
for handling customer complaints within NSF. Further, other federal agencies
were examined to locate a model with similar customer interactions, but
no appropriate model was identified. As a result of this input, some priorities
for action have been identified. However, models of best practices and
NSF staff training are still being developed in FY 2001. NSF continues
to place great importance on these issues and will complete this effort
in FY 2001. In addition, NSF will pilot the best of the models in NSF
divisions and provide specific customer service training to NSF staff.
Process Goal 9 Proposer Attention
to Integration of Research and Education
NSF will develop a plan and system to request that Principal Investigators (PIs) address the integration of research and education in their proposals, and develop and implement a system to verify that PIs have done so.
to community; implementation of system to verify that PIs address the
integration of research and education in proposals.
This goal was achieved.
In FY 2000, NSF implemented an electronic program announcement template clearance process (PAT) that is used by NSF staff to generate announcements and solicitations. Use of the PAT ensures that PIs are asked to address the integration of research and education in all announcements and solicitations. In addition, the Foundation has included language in the Proposal and Award Manual, the Grant Proposal Guide, and the FY 2000 Guide to Programs regarding the integration of research and education.
In order to verify that PIs are addressing the integration of research and education, NSF asks Committees of Visitors (COVs) to assess whether the broader impacts of the proposed activity are being addressed in proposals. The COV reporting template has been modified to explicitly address the use of both merit review criteria.
This goal will not be continued in FY 2001, but will be replaced by goals addressing broader use of the merit review criteria by reviewers and staff, which encompasses this goal.
Investment Process Goal 10 Reviewer
Attention to Integration of Research and Education
NSF will develop and implement a system/mechanism to request and track reviewer comments tied to the merit review criterion, What are the broader impacts of the proposed activity?
to community; implementation of system to track reviewer comments.
This goal was achieved.
During FY 2000, screens were added in FastLane, NSFs electronic proposal and review system, so reviewers can address each merit-review criterion separately. The performance data will be collected from the FastLane database.
NSF has modified program announcements to encourage applicants and reviewers to address these criteria in proposals and reviews. NSF has recently re-issued guidance to the applicants and reviewers, stressing the importance of addressing both merit review criteria in the preparation and evaluation of proposals submitted to NSF. NSF staff continue to stress the importance of reviewers addressing the broader impacts criterion whenever they attend NSF sponsored seminars, science meetings, site visits, conferences, and conventions. NSF is considering taking additional steps to ensure that applicants address these criteria when reporting project results.
Process Goal 11 Diversity of NSF
NSF will identify mechanisms to increase the number of women and underrepresented minorities in the proposal applicant pool, and will identify mechanisms to retain that pool. (Revised goal; no baseline.)
to attract proposals from members of underrepresented groups in order
to increase the total applicant pool; mechanisms to retain the applicant
This goal was achieved.
NSF is committed to the principle of diversity and deems it central to the programs, projects, and activities it considers and supports. NSF continues to work toward increasing diversity in its applicant pool:
Process Goals 12 to 15 Facilities
The following goals are for federal science, space and technology agencies that support construction projects and have responsibility for managing facilities (NSF, NASA, DOE).
In FY 1999, NSF developed a general facilities reporting template for use in reporting on the facilities management goals. This reporting system was linked to the new Project Reporting System as a module of the existing FastLane system. Facility managers located at the facility site report data to NSF using this reporting system.
In FY 2000, NSF reviewed the data collection and reporting effort and made modifications to the system where appropriate. This included allowing for reporting on construction/upgrade activities at facilities funded through the Research and Related Activities account, refining the on-screen language to be more clear and to more accurately address the facilities goals, automating most of the output, and instituting a stage for collecting estimates.
12: Maintain the FY 1999 goal
to keep construction and upgrades within annual expenditure plan, not
to exceed 110% of estimates.
This goal was achieved. Of the 11 construction and upgrade projects
supported by NSF, all were within annual expenditure plans.
13: Maintain the FY 1999 goal
to keep construction and upgrades within annual schedule, total time required
for major components of the project not to exceed 110% of estimates.
This goal was not achieved. Of the 11 construction and upgrade
projects supported by NSF, seven were within the annual schedule goal.
In several cases, missed milestones were due to circumstances beyond the
project managers control. For example, one construction project
was dependent upon the research and development of new instrumentation,
the results of which were delayed. In another project, the missed milestone
was due to difficulty obtaining required parts, non-performance of a sub-contractor,
and underestimation of the complexity of the work. In FY 2001, NSF program
managers are working more closely with project managers to ensure all
NSF-supported construction/upgrade projects achieve this goal.
14: For all construction and
upgrade projects initiated after 1996, when current planning processes
were put in place, keep total cost within 110% of estimates made at the
initiation of construction.
This goal did not apply; there were no construction projects completed
in FY 2000.
15: Maintain the FY 1999 goal
to keep operating time lost due to unscheduled downtime to less than 10%
of the total scheduled possible operating time.
This goal was not achieved. Of the 26 reporting facilities, 22 met
the goal of keeping unscheduled downtime to below 10% of the total scheduled
operating time. NSF program staff will work more closely with project
managers to ensure that all achieve this goal in FY 2001.