|Critical Factors for Success|
|Means and strategies|
|II. Summary Table of Performance Goals|
|Performance Goals for Results|
|Performance Goals for NSF's Investment Process|
|Performance Goals for Management|
This is NSF’s first GPRA performance plan. It is based on NSF’s GPRA strategic plan, submitted to Congress in September, 1997. The mission, outcome goals, and critical factors for success from that plan are outlined below.
NSF’s continuing mission is set out in the preamble to the National Science Foundation Act of 1950 (Public Law 810507):
To promote the progress of science; to advance the national health, prosperity, and welfare; to secure the national defense; and for other purposes.
The Act authorizes and directs NSF to initiate and support:
NSF carries out its mission primarily by making merit-based grants and cooperative agreements to individual researchers and groups, in partnership with colleges, universities, and other institutions -- public, private, state, local, and federal -- throughout the U.S.
The outcomes of NSF investments, the results stemming from the grants and cooperative agreements we make, provide the evidence for NSF’s success as an investment agent. NSF staff pursue the following outcome goals, the general goals of the strategic plan, as they develop the NSF award portfolio:
Excellence in managing the agency’s processes is an NSF goal on a par with our mission-oriented outcome goals. In the GPRA strategic plan, NSF articulated four critical factors in managing for excellence that provide the framework for annual performance goals.
NSF’s primary business is to make merit-based grants and cooperative agreements to individual researchers and groups, in partnership with colleges, universities, and other institutions -- public, private, state, local, and federal -- throughout the U.S. By providing these resources, NSF contributes to the health and vitality of the U.S. research and education system, which enables and enhances the nation’s capacity for sustained growth and prosperity. The individuals and organizations in which NSF invests conduct the work that ultimately determines the outcomes of the investment process that NSF manages.
NSF uses merit review with external peer evaluation to select about 10,000 new awards each year from about 30,000 competitive proposals submitted by the science and engineering community for its consideration. Work continues through another 10,000 awards made in competitions of previous years. NSF's role in the fabric of federal funding of science and engineering is defined by the fundamental nature of the problems our grantees propose and explore, the innovative nature of the research and education we support, and our integrative approach to research and education.
Proposals and awards are managed through eight programmatic organizations, seven directorates and the Office of Polar Programs1.
NSF directorates make investments in three functional categories: research projects, research facilities, and education and training. Approximately 95% of NSF’s budget goes directly to these investments. A fourth function, administration and management, provides support for the immediate activities of the agency.
The FY 1999 Budget Request leads to the following distribution of NSF budget resources across the key functions, with a total Request of $3.773 billion.
Investments in Research Project Support fund the cutting edge research that yields new discoveries. These investments help to maintain the nation’s capacity to excel in science and engineering, particularly in academic institutions. The store of knowledge produced by NSF-funded research projects also provides a rich foundation for broad and useful applications of knowledge and the development of new technologies. Research projects contribute to the education and training of the next generation of scientists and engineers by giving them the opportunity to participate in discovery-oriented projects. NSF centers provide a particularly rich environment for broad interdisciplinary research and education at all levels. The FY 1999 Budget Request for Research Project Support is $2.126 billion.
NSF provides support for large, multi-user Research Facilities that meet the need for access to state-of-the-art research facilities that would otherwise be unavailable to academic scientists and engineers. Support includes funding for construction, upgrade, operations, and maintenance of facilities, and for staff and support personnel to assist scientists and engineers in conducting research at the facilities. Support for these unique national facilities is essential to advancing U.S. research and education capabilities, and is driven predominantly by research opportunities and priorities. Planning for facilities involves selecting from among many exciting ideas those where facilities will be initiated or upgraded. It also involves phasing down or terminating those components of facilities that no longer support cutting edge research. The FY 1999 Budget Request for Research Facilities totals $735 million.
Investments in Education and Training help ensure an adequate, well-prepared workforce of scientists and engineers that can maintain leadership in science and technology, both now and in the future, and help all students to achieve the mathematics and science skills needed to thrive in an increasingly technological society. NSF’s programs produce scientists and engineers knowledgeable about the most recent scientific and technical advances. These highly educated people then reach every sector of society and actively disseminate and use that knowledge in the service of innumerable social goals. They also provide well-prepared teachers and instructional materials and technologies that influence mathematics and science education at all levels. The FY 1999 Budget Request for Education and Training totals $737 million.
The FY 1999 level for Administration and Management of $175 million provides support for salaries and benefits of persons employed at the NSF; general operating expenses, including key activities to advance the agency’s information systems technology and to enhance staff training; and audit and Inspector General activities.
NSF is moving to revise its budget account structure to track with this functional approach to use of funds. The following table describes a crosswalk for the directorates.
NSF’s GPRA strategic plan outlines key investment strategies and an action plan for achievement of each of the outcome goals. There are common themes running through these investment strategies, and this performance plan reflects the importance of emphasizing activities that influence achievement of multiple objectives. Common strategies include: (1) using competitive merit review with peer evaluation to identify the most promising ideas from the strongest researchers and educators; (2) integrating research and education to strengthen both; (3) working in partnership with the science and engineering communities and potential users of the results in order to identify and act upon areas of emerging opportunity; and (4) assuring that both NSF and the research and education communities reap optimal benefit from the revolution in information, communications, and computing technologies. In addition, NSF is committed to using committees and panels of external experts to assess on a regular basis its effectiveness and directions.
These strategies guide directorates in their allocations of funds and in implementation of proposal and award processes. At the NSF level, response to these common strategic elements have led to the identification of broad thematic areas for investment that cut across NSF’s programmatic structure. Three thematic areas are emphasized in the FY 1999 Budget Request: Knowledge and Distributed Intelligence, Life and Earth’s Environment, and Educating for the Future. These areas for enhanced investment stimulate coordination and synergy outside the usual budget lines, that is, across directorates and with other agencies.
3Numbers may not add due to rounding.
Just as several investment strategies support progress toward more than one outcome goal, so do the directorates and key functions. The table below provides a picture of the interactions between outcome goals and key functions. The most heavily shaded areas for each outcome goal indicate the key functions that most directly support it; the more lightly shaded areas show the key functions that influence progress toward the outcome goal less directly. By comparing this table with the crosswalk above, it is possible to determine how directorates exert their influence on attaining the outcome goals.
|Discoveries at and across the frontier of science and engineering|
|Connections between discoveries and their use in service to society|
Diverse, globally-oriented science and engineering workforce
|Improved achievement in mathematics and science skills needed by all Americans|
Timely and relevant information on the national and international science and engineering enterprise4
NSF’s management uses information on past performance (where available) and applies the strategies for enhancing outcomes to allocate available resources. NSF staff, advised by the merit review process, select the individual projects to be supported, managing toward the optimal mix of outcomes, given the available resources.
NSF’s FY 1999 Budget Request and this performance plan are based on the strategies described in the GPRA strategic plan for FY 1997-FY 2003. The performance plan covers those strategies and actions deemed most important for attention in FY 1999. This first GPRA performance plan was developed solely by NSF staff. It reflects discussions of general principles with elements of the research and education communities, the administration, and congressional staff. A joint task force of NSF staff and members of the National Science Board provided guidance throughout the GPRA planning process.
[Return to Table of Contents]
NSF’s annual performance goals for FY 1999 fall into three categories:
|Outcome Goal||Annual Performance Goal||FY 1999 Areas of Emphasis
|NSF is successful when||
NSF is minimally effective when
|Discoveries at and across the frontier of science and engineering||
NSF awards lead to important discoveries; new knowledge and techniques, both expected and unexpected, within and across traditional disciplinary boundaries; and high-potential links across these boundaries.
|there is a steady stream of outputs of good scientific quality.||
|Connections between discoveries and their use in service to society||the results of NSF awards are rapidly and readily available and feed, as appropriate, into education, policy development, or use by other federal agencies or the private sector.||results of NSF awards show the potential for use in service to society, and when activities designed to enhance connections between discoveries and their use in service to society meet the successful standard.||
|A diverse, globally-oriented workforce of scientists and engineers||participants in NSF activities experience world-class professional practices in research and education, using modern technologies and incorporating international points of reference; when academia, government, business, and industry recognize their quality; and when the science and engineering workforce shows increased participation of underrepresented groups.||opportunities and experiences of students in NSF-sponsored activities are comparable to those of most other students in their fields; and when the participation of underrepresented groups in NSF-sponsored science and engineering projects and programs increases.||
|Improved achievement in mathematics and science skills needed by all Americans||NSF awards lead to the development, adoption, adaptation, and implementation of effective models, products, and practices that address the needs of all students; well-trained teachers who implement standards-based approaches in their classrooms; and improved student performance in participating schools and districts.||NSF awards lead to the development and adaptation of effective educational models, products, and practices; train and further develop teachers in the standards-based approaches; and prevent further deterioration in student achievement in participating schools and districts.||
6Elements in italics are highlighted in the government-wide performance plan.
|Outcome Goal||Annual Performance Goal|
|Timely and relevant information on the national and international science and engineering enterprise.||Decrease by 15% from the current average of 540 days the time interval between reference period (the time to which the data refer) and reporting of data.|
|Achieve customer satisfaction ratings with the relevance of products offered of at least 45% "excellent" and at least 90% "excellent" or "good".|
|Proposal and Award Processes|
|Use of Merit Review||At least 90 percent of NSF funds will be allocated to projects reviewed by appropriate peers external to NSF and selected through a merit-based competitive process.|
|Implementation of Merit
|NSF performance in implementation of the new merit review criteria is successful when reviewers address the elements of both generic review criteria appropriate to the proposal at hand and when program officers take the information provided into account in their decisions on awards; minimally effective when reviewers consistently use only a few of the suggested elements of the generic review criteria although others might be applicable.|
|Customer service --
Time to prepare proposals
|95% of program announcements and solicitations will be available at least three months prior to proposal deadlines or target dates.|
|Customer service --
Time to decision
Process 70% of proposals within six months of receipt.
|Award Duration||Increase average duration of awards for research projects from a FY 1997 base of 2.3 years to at least 2.7 years.|
|Maintaining Openness in
|NSF will increase the percentage of competitive research grants going to new investigators to at least 30%, 2.6% over a baseline of 27.4 %.|
|All directorates within NSF will establish Web sites for the science and engineering community to provide suggestions for and comment upon emerging opportunities.|
|Integration of Research and Education|
|Encouraging integration of
research and education
|NSF will ensure that all of its new announcements of opportunities and proposal solicitations will contain an explicit statement encouraging proposers to integrate research activities with improving education or public understanding of science.|
|Encouraging attention to diversity
in all aspects of NSF programming
|NSF will ensure that all of its new announcements of opportunities and proposed solicitations will include a statement encouraging proposers to address improving the participation of underrepresented groups in science and engineering in the course of their research and education activities.|
|Construction and upgrade||Keep construction and upgrades within annual expenditure plan, not to exceed 110 percent of estimates.|
Keep construction and upgrades within annual schedule, total time required for major components of the project not to exceed 110 percent of estimates.
For all construction and upgrade project initiated after 1996, keep total cost within 110 percent of estimates made at the initiation of construction.
Keep operating time lost due to unscheduled downtime to less than 10 percent of the total scheduled possible operating time.
9Performance goals in italics are highlighted in the government-wise performance plan.
10This performance goal is stated in the alternative format provided for in GPRA legislation. See Appendix 2 for the complete statement of the performance goal, including the minimally effective standard.
|Critical Factor for Success||Performance Goal|
|New and emerging technologies|
|Electronic proposal processing||NSF will receive and process at least 15% of full proposal submissions electronically through FastLane.|
|Diversity||In FY 1999, as all appointments for scientists and engineers are considered, the recruiting organization will demonstrate efforts to attract applications from groups that are underrepresented in the science and engineering staff as compared to their representation among Ph.D. holders in their fields.|
|Capability in use of information technology||By FY 1999, all staff will receive an orientation to FastLane, and at least 95% of program and program support staff will receive practice in using its key modules.|
Implementation of management reforms
|Year 2000||NSF will complete all activities needed to address the Year 2000 problem for its information systems according to plan, on schedule and within budget, during FY 1999.|
|Project Reporting System||During FY 1999, at least 70% of all project reports will be submitted through the new electronic Project Reporting System.|
[Return to Table of Contents]
PERFORMANCE GOALS FOR RESULTS
Results from NSF awards appear over time. Thus, in assessing performance toward the outcome goals during a given year, NSF must look carefully at (1) the noteworthy achievements of the year based on NSF awards, (2) the ways in which projects collectively affect progress, and (3) expectations for future performance based on the current set of awards.
The challenge of performance assessment for NSF is that both the substance and the timing of outcomes from NSF's activities are unpredictable. Annual outputs of NSF awards, while important to the success of individual projects, consistently understate progress toward outcomes. Thus, NSF has stated performance goals for results of NSF's investments in research and education as descriptive standards, under the GPRA option to set performance goals in an alternative format that describes successful and minimally effective performance. The stream of data and information on the products of NSF's awards will provide the basis for assessing NSF's performance through self-assessment and the expert judgment of independent external panels.
As a basis for NSF's annual performance report, each directorate12 will develop an annual self-assessment that addresses the three elements described above, articulating achievements, collective progress, and expectations for the future in understandable terms. The first two of these elements will provide means of assessing the performance of past NSF investments; the last, a means of assessing the characteristics of the current investments, with a focus on the likelihood of strong performance in the future. Directorate advisory committees, panels composed of experts external to NSF, will take the self-assessment and other available sources of information into account in benchmarking the directorate's performance against the performance standards. The advisory committees will be expected to address both strengths and weaknesses of the directorates, including ways in which the performance is exceptionally strong. NSF management, working with the advice and guidance of the National Science Board, will integrate the advisory committee recommendations into the GPRA performance report13. All of the performance goals are addressed to aggregate activities, and both advisory committee and NSF-wide reports will assess progress toward them based on the performance of collections of awards within key program functions.
13Advisory committee assessments, and thus, NSF's performance report, will have a richer information base when there are sources of input for particular parts of the directorate that go beyond the directorate self-assessment. We would expect that at least every three years the advisory committees would cover all aspects of directorate activity in some depth. In FY 1999's performance report, we expect to have emphasis in the following areas: BIO - neurosciences, genetics, environmental biology, research resources and training; CISE - computer and computation research, institutional infrastructure, networking, and microelectronic systems; EHR - urban systemic initiatives, EPSCOR, teacher enhancement, education research, and learning technologies; ENG - bioengineering and environmental systems, hazard mitigation, electrical and communications systems, and aspects of design, manufacture, and industrial innovation; GEO - upper atmospheric research, geology and paleontology, hydrological sciences, and petrology and geochemistry; MPS - astronomical sciences; SBE - research in social, behavioral and economic sciences, ethics and values, science and technology studies, and methodology, measurement and statistics.
Outcome Goal 1: Discoveries at and across the frontier of science and engineering
Awards based in the Research Project Support and Research Facilities key functions are the principal contributors toward reaching this goal.
Performance Goal: NSF's performance toward this outcome goal is
Elements of exceptionally strong performance include outcomes where NSF awards lead to advances of major scientific or engineering importance. These might, for instance: create new paradigms, stimulate important new areas of inquiry, or otherwise excite national and international attention.
The performance report will open with a discussion of important discoveries of FY 1999 resulting from NSF investments. It would then address the contributions of past investments to advancing discovery, including areas such as: growth of the knowledge pool, techniques and tools for generating new knowledge, growth of new areas of knowledge, and linkages across areas of knowledge. Finally, it will address the base the current set of awards provides for future performance.
The government-wide performance plan contains a performance goal that is particularly relevant to the advisory committee assessments of directorate performance.
Performance Goal: An independent assessment will judge NSF research programs to have the highest scientific quality and an appropriate balance of projects characterized as high-risk, multidisciplinary, or innovative.
NSF will ask all directorate advisory committees to examine the directorate's entire FY 1999 portfolio of research project support activities, identify activities they would characterize as high risk, multidisciplinary, or innovative, and make an assessment of overall scientific quality and balance with respect to these specific characteristics. The focus of this review will be to ensure that NSF is positioned well to attain the first outcome goal: discovery at and across the frontier of science and engineering. Directorates will structure their self-assessments to facilitate this review.
All directorate advisory committees will also be asked to examine the extent and appropriateness of investment in new types of scientific databases and the tools to use them. This is a critical component of activity under Knowledge and Distributed Intelligence, one of NSF's areas of emphasis in FY 1999.
In NSF's FY 1999 Budget Request, a number of multidisciplinary areas of research are identified as ripe for progress. Relevant directorate advisory committees will be asked to pay particular attention to these areas in their assessments.
Life in Extreme Environments began as a focused investment theme in FY 1997. Preliminary results coming from awards made then may be available in FY 1999. Advisory committees will also be able to examine the active portfolio of awards for their potential influence on progress in this exciting area. Relevant directorates are BIO, GEO, MPS, and OPP.
Nanoscience and Engineering is just beginning to develop as an area for coordinated investment. Advisory committees in ENG and MPS will be asked to examine the set of awards in this area for their potential influence on progress.
Outcome Goal 2: Connections between discoveries and their use in service to society
The Research Project Support, Research Facilities, and Education and Training key functions all contribute to attaining this outcome goal.
Performance Goal: NSF's performance toward this outcome goal is
Exceptionally strong performance is characterized by NSF staff and grantees actively reaching out to potential users, and NSF-supported work playing critical roles in important innovation or problem solving for society.
The performance report will open with a discussion of new uses of discoveries tied to NSF investments in ways relevant to society and new uses of tools or technologies resulting from NSF investments. It will also address contributions of past investments to advancing connections to use, including: accessibility of results of NSF work to potential users; partnerships between government, academia, and industry; and how the NSF-generated knowledge base enables innovation. Finally, it will address the appropriateness of the current set of funded activities as a base for future performance, including issues such as: cooperation with other agencies; incentives for enhanced partnerships between government, academia, and industry; specific attention to emerging opportunities with the potential for service to society; and education and training opportunities linked to the private sector.
In NSF's FY 1999 Budget Request, a number of multidisciplinary areas of research are identified as ripe for progress and of particular importance for their potential connections to use in service to society. In implementing focused research activities in these areas, NSF works in partnership with other agencies. Relevant directorate advisory committees will be asked to pay particular attention to these areas in their assessments, examining results of past investments, where results are available, and the contents of the current portfolio for (1) appropriateness in meeting the goals of the interagency program; (2) quality of the NSF research and infrastructure activities; and (3) balance among related areas of activity within NSF.
Global Change has been an important area of focused research investment at NSF for several years, in conjunction with NSF's participation in the U.S. Global Change Research Program. Advisory committees in BIO, GEO, OPP, and SBE will be asked to address performance in global change, addressing both past results and the current portfolio.
The Next Generation Internet (NGI) is an interagency activity aimed at increasing hundred-fold the speed and capacity of today's Internet. NSF's role is to facilitate high-performance links among academic institutions and national centers and to conduct research on networking. NSF activities are built on past investments in similar areas in networking research and infrastructure. The CISE advisory committee will be asked to assess NSF's performance in NGI, with the focus on the current portfolio.
Research in Education and Training Technologies was given high priority in the report of the President's Committee of Advisors in Science and Technology on the Use of Technology to Strengthen K-12 Education in the United States (March 1997). NSF, in partnership with the Department of Education, will build on past investments in this area with a major initiative in FY 1999. While all directorates will participate in this effort, the EHR advisory committee will be asked to provide an assessment of the new investments for NSF as a whole.
Plant Genome Research received a major funding increase in FY 1998. The BIO directorate is implementing a program of support for research and infrastructure development, consistent with the recommendations of the 1998 report of the National Science and Technology Council entitled National Plant Genome Initiative. In FY 1999, the BIO advisory committee will review progress in developing a strong portfolio in this area, including interactions with other agencies, other nations, and the private sector. They will pay particular attention to sequencing of the model plant Arabidopsis and progress toward the new goal of completing the sequencing by 2000.
Outcome Goal 3: A diverse, globally-oriented workforce of scientists and engineers
The Research Project Support, Research Facilities, and Education and Training key functions all contribute to attaining this outcome goal.
Performance Goal: NSF's performance toward this outcome goal is
Exceptionally strong performance is characterized by recognition of scientists or engineers who received NSF support during their training; and when the production of degree recipients in science, mathematics, and engineering increases markedly for underrepresented groups.
NSF's performance report for this goal will begin with relevant information from the past year on recognition of those supported during the early stages of their careers by NSF funding. This will be followed by a discussion of contributions of past NSF investments to enhancing the science and engineering workforce, including the ability of students and new faculty to participate in research activities, the development of flexible graduate and undergraduate programs that addresses the broad range of possible opportunities for students, and levels of participation in science and engineering by underrepresented groups. Finally, the report will address the base for future performance with particular attention to opportunities for new investigators, integration of research and education at all levels, and increased participation by underrepresented groups on NSF awards. NSF will want to ensure that its investments in support of workforce development are based on good information on the future needs of the public and private sectors for the workforce and based, where appropriate, on valid research on teaching and learning.
Providing opportunities for participation in integrative research and education experiences is one of NSF's key investment strategies for addressing this outcome goal. In order to influence the development of integrated approaches, NSF has developed a number of programs intended to synergize the integration of research and education. The most prominent of these programs are the Faculty Early Career Development (CAREER) program, the Research Experiences for Undergraduates (REU) program, and the Integrative Graduate Education and Research Training (IGERT) program. Each of these programs is aimed at using NSF's close interaction with academic science and engineering to support on-going efforts within academe to draw research and education together. They are supported across the Foundation and in FY 1999 receive significant increases. All advisory committees will assess the activities of the directorates they advise in the CAREER and REU programs, examining results of past investments and the level and quality of current investments. IGERT is a newer program that was undertaken in response to a National Science Board recommendation that NSF experiment with alternative forms of graduate student support and training. It builds on earlier investments in the BIO and EHR directorates. All directorates will address the quality of the current set of awards. The BIO and EHR advisory committees will also assess the results of past related investments and provide guidance on NSF's implementation of IGERT.
The three programs addressed above, along with NSF-supported centers, are key elements of NSF's activities to increase the participation of underrepresented groups in integrative research and education experiences. Their program announcements ask proposers to address how the activity proposed will impact diversity in the science and engineering workforce, and participation by members of underrepresented groups is strong. All directorate advisory committees will be asked to include an assessment of the impact of past activities on participation of underrepresented groups and the extent to which the current portfolio can be expected to have an impact on increasing participation.
Outcome Goal 4: Improved achievement in mathematics and science skills needed by all Americans
This outcome goal is primarily supported through the Education and Training key function.
Performance Goal: NSF's performance toward this outcome goal is
Exceptionally strong performance is characterized by awards that lead to high impact development, adoption, adaptation, and implementation of effective models, products, and practices that extend beyond the period of NSF investment or beyond the scope of NSF award sites; generate novel approaches with high potential to improve and change the education process; or are associated with significant improvement in student performance
NSF's performance report in this area will open with evidence of improved achievement toward standards on the part of schools and districts involved with NSF programs. Then it will address the contributions of past NSF investments toward improving achievement, including providing for a well-trained teaching corps with opportunities for professional development, appropriate instructional materials and teaching methodologies, including learning technologies, a good research base for teaching and learning, and opportunities for learning outside the classroom. Assessment of the base for future performance will focus on building the research base for teaching and learning and using it in making investments, appropriate use of systematic approaches, cooperation with other agencies, development and use of new technologies and other tools, and development of models for broader adaptation.
The government-wide performance plan contains two performance goals that are particularly relevant to the advisory committee assessments of performance in this area. Both are related to NSF's systemic activities in K-12 education. At the start of the decade, NSF initiated major programs for the systemic reform of science, mathematics, engineering, and technology education. Based on the belief that all students can learn and achieve in science and mathematics at much higher levels than then obtained, systemic projects treat whole systems and build much-needed educational capacity at state, urban, rural, school district, and school levels. These projects are unique in their involvement of broad partnerships and development of comprehensive goals, solutions, and actions.
Performance Goal: Over 80 percent of schools participating in a systemic initiative program will (1) implement a standards-based curriculum in science and mathematics; (2) further professional development of the instructional workforce; and (3) improve student achievement on a selected battery of tests, after three years of NSF support.
Performance Goal: Through systemic initiatives and related teacher enhancement programs, NSF will provide intensive professional development experiences for at least 75,000 precollege teachers.
Both of these performance goals have quantitative forms that could be addressed without the use of the advisory committee process. However, they will be addressed through the EHR advisory committee so that there will be an assessment of quality as well.
Improving achievement in K-8 mathematics is the focus of a partnership with the Department of Education that began in FY 1998 and will expand significantly in FY 1999. Cooperative activities include public engagement activities and joint support for capacity-building grants to assist school districts in planning coherent use of federal and other funds to improve mathematics education, with a focus on teacher professional development and implementation of appropriate instructional materials and technology. The EHR advisory committee will also address performance of NSF in this partnership, focusing on the quality of the activities being undertaken and their potential for improving achievement for the set of awards made in FY 1998 and FY 1999.
Outcome Goal 5: Timely and relevant information on the national and international science and engineering enterprise
NSF's provision of information on the national and international science and engineering enterprise is a customer-oriented activity, centered in the Division of Science Resources Studies (SRS). Funding is provided through the Education and Training key function. The performance goals for this activity aim for improved quality through enhanced timeliness and continued levels of customer satisfaction with relevance and accuracy.
In a recent survey, a sample of the science and engineering policy community indicated that improving timeliness of data was high priority for them.
Performance Goal: Decrease by 10% from the current average of 540 days the time interval between reference period (the time to which the data refer) and reporting of data.
Performance is measured as a two-year moving average of the number of days between the end of the data reference period and the public availability of data (usually electronic dissemination) for surveys SRS supports. In FY 1999, performance for FY 1998-99 will be reported. The goal refers only to data reported through the eleven core surveys that SRS supports, but SRS will calculate and report comparison numbers for data it reports from surveys sponsored by other federal agencies and other organizations. The baseline is the record of surveys performed during FY 1995 and 1996. That average time interval was 540 days. This improvement in timeliness will be largely accomplished by taking advantage of advances in information and communications technologies.
The value of information on the science and engineering enterprise is highly dependent on its relevance to those who seek to use it in making policy decisions. Different users are interested in different aspects of the enterprise.
Performance Goals: Achieve customer satisfaction ratings with the relevance of products offered of at least 45% "excellent" and at least 90% "excellent" or "good".
Performance is measured as relevance ratings by users of SRS data. A baseline measure is found in a customer service survey done in 1996. On average, 38% of SRS data users who responded to this survey rated the relevance of SRS data on the topics of most interest to them as "excellent," and a further 50% as "good." In 1999, information from a follow-up survey will form the basis for establishing achievement of this performance goal. Other information from key customers (science and technology policy makers and other influential data users) will permit a more complete assessment of customer satisfaction with the relevance of SRS products and will form a basis for future performance goals. Information from the 1996 customer service survey provides a basis for determining where improvements in relevance are needed and for modifying existing surveys as needed to gain customer satisfaction.
Use of the alternative format performance goals for outcomes in research and education allows for human judgment to consider both quantitative and qualitative information on performance and weigh it in a balanced assessment. The descriptive performance goals are designed to be used as tools in NSF's management process through a combination of internal self-assessment and review by external panels through the directorate advisory committees. Members of these committees are highly credible experts in their fields. Because of the need for the judgments of potential users and international perspectives on quality, committee membership will evolve to provide enhanced independence of judgments.
In FY 1997 and early FY 1998, NSF has run a series of experiments with advisory committees and their subcommittees to determine how to adapt existing processes to obtain effective assessments of outcomes. By FY 1999, all advisory committees will give judgments of program effectiveness, describing strengths and weaknesses using the target levels of performance. The credibility of these reports will rest on the provision of qualitative detail about program results, to demonstrate which of the levels of performance are actually achieved. Advisory committees will have full access to the data sources used in the self-assessment.
Most of the information base that will underlie assessments of achievement on the descriptive performance standards comes from outside the agency, through two major grantee reporting systems: final project reports and their projected electronic successor, the Project Reporting System, and the Impact Data Base and project monitoring system, designed by the Directorate for Education and Human Resources and built and maintained by a contractor. Through these systems, output indicators such as the following will be available to program staff, third party evaluators, and advisory committees:
This information is largely provided to NSF from external sources. When the data are gathered through a contractor, NSF staff will include data verification capacity in its evaluation of competing bids. The Impact Data Base and project monitoring system of the Education and Human Resources directorate is an example of a contractor-supported system.
Project reports for NSF's research activities are currently submitted largely on paper. They are subject to general review by the relevant program officer, but would be cumbersome in a wide-ranging assessment process. Thus, NSF is in the testing phase of an electronic project reporting system that will permit organized reporting of aggregate information. We anticipate that the reliability of the information in the system will improve over time, as investigators and institutions become comfortable with its use. FY 1999 will be the first year of its full implementation.
The scientific data from the reporting system will be tested for plausibility as a natural part of the external assessment process. In addition, data from the reporting system will be used to address progress under prior support when investigators reapply to NSF. Thus, the investigators have a strong interest in providing accurate information that reviewers may rely upon. NSF will work with institutions in order to establish some basic guidelines for reporting information on the demographics of participants in a reliable way.
NSF has worked with the university community to ensure that the added reporting and assessment burden will be minimal. This is important to having a viable performance plan for NSF. More direct efforts to verify and validate information in the project reporting system would add significantly to the cost and to the burden on the grantee community.
For the outcome goal on policy information, the Division of Science Resources Studies will calculate the time between collection and reporting of policy information, and their data will be open for verification and validation. Customer satisfaction ratings will come via a customer survey, from a source outside the Foundation.
[Return to Table of Contents]
PERFORMANCE GOALS FOR THE NSF INVESTMENT PROCESS
The following sections provide information about the means and strategies NSF uses in support of its outcome goals and articulates performance goals for the investment process by which NSF shapes its portfolio of awards. In FY 1999, NSF will invest $3.6 billion in these areas.
Proposal and Award Processes
NSF's role in achieving its outcome goals for research and education is implemented through program officer selection of projects from among those submitted by the scientific community. This is a competitive process based on merit review by external peer reviewers, using criteria established by NSF. The scientists and engineers comprising NSF's program staff take NSF priorities and the advice of the external reviewers into account in developing their portfolio of awards.
Use of merit review.
Performance Goal: At least 90 percent of NSF funds will be allocated to projects reviewed by appropriate peers external to NSF and selected through a merit-based competitive process.
Use of competitive merit review is a fundamental investment strategy for NSF that applies to all outcome goals for research and education. The government-wide performance plan stresses the importance of competitive processes for awarding funds for scientific research. All NSF awards are made through merit-based processes. NSF makes a few exceptions to its general requirement for external merit review. These include situations in which objective external reviewers may be difficult to find, when natural phenomena such as volcanic eruptions or earthquakes make an external review process for proposals to study them too lengthy, or when researchers are proposing such new ideas that knowledgeable external reviewers may not exist.
Implementation of Merit Review Criteria
NSF's generic review criteria were recently reviewed and revised by the National Science Board. Implementation of those criteria is an important, immediate goal in the merit review/project selection process, in order to better support the outcome goals for research and education, as noted in NSF's GPRA strategic plan. The new criteria go into effect in FY 1998. NSF staff will monitor their use, and NSF's external Committees of Visitors will begin to check on their implementation in FY 1999.
Performance Goal: NSF performance in implementation of the new merit review criteria is
This performance goal reflects the multipurpose nature of NSF's investments. We expect our discovery-oriented investments to enable connections to use and workforce development. Likewise, we expect our education-oriented investments to bring the tools of the discovery process, the new knowledge generated through discovery, and the people involved in discovery to bear on educational challenges.
NSF will use the directorate14 advisory committees to monitor this performance goal as they will the performance goals for research and education outcomes. The advisory committees and their subcommittees will address questions on implementation of the new merit review criteria using the samples of reviews they routinely examine in their judgment of the effectiveness and fairness of the review process. As part of the request to these committees, NSF will ask that they describe in their reports the samples and evidence they used to come to their judgments.
In 1995, NSF adopted a set of customer service standards, primarily related to the proposal review process, treating grantees and potential grantees (applicants) as the primary customers for NSF's administrative processes. In a recent survey, applicants valued three standards most highly: (1) clear guidelines for proposal content and preparation, (2) a minimum of three months between program announcements and proposal deadlines, and (3) notification of proposal funding recommendation within six months of proposal submission. The survey measured baseline levels of customer satisfaction, with reference to FY 1995 experiences. The survey will be repeated in FY 1999, referring to FY 1998 experiences. Customer service has a potential impact on the number and quality of proposals received and thus on NSF's ability to meet all outcome goals.
For this performance plan, we focus on two of these standards, ones NSF staff have devoted special attention to since the standards were adopted.
Customer Service - Time to Prepare Proposals
Customer Service Standard: Make program announcements and solicitations available to relevant individuals and organizations at least three months prior to the proposal deadline or target date.
FY 1999 Performance Goal: 95% of program announcements and solicitations will be available at least three months prior to proposal deadlines or target dates.
In FY 1997, approximately 80 percent of specialized program announcements and solicitations were available to prospective applicants at least three months prior to proposal deadlines or target dates, with the bulk of proposals submitted in response to standing announcements. NSF staff work toward this performance goal by limiting the number of special competitions requiring individual program announcements and solicitations, planning for such competitions as far in advance as possible, and initiating clearance processes at least six months prior to the anticipated proposal deadlines.
NSF will maintain records of timing between announcement and deadline. Timing will begin when the announcement is placed on the Web for public information.
Customer Service - Time to Decision
Customer Service Standard: For 95 percent of proposals, be able to tell applicants whether their proposals have been declined or recommended for funding within six months of receipt.
For activities with deadline or target dates for receipt of proposals, the stated deadline or target date is set as the date of receipt for purposes of evaluating performance toward the standard.
FY 1999 Performance Goal: Process 70% of proposals within six months of receipt.
Processing proposals within six months of receipt has been a challenging goal for NSF, and we are therefore scaling annual performance targets to reach the customer service standard by FY 2002. In FY 1997, NSF processed 61 percent of proposals within six months. NSF staff work toward this performance goal by making effective use of electronic mechanisms in conducting the review, working cooperatively to eliminate overloads and bottlenecks, and keeping careful track of the stage of processing and age of all proposals.
This customer service standard is applicable to research project support and education and training. The proposal process for research facilities is more extended due to the nature of the projects.
Many scientists and engineers contribute to NSF's proposal and award system -- writing proposals, reviewing proposals, and conducting NSF-funded research. Their most productive time is the conduct of research. This would argue for providing needed resources for a sufficient duration to complete the work, thus requiring successful researchers to write fewer proposals.
Performance Goal: Increase average duration of awards for research projects from a FY 1997 base of 2.3 years to at least 2.7 years.
This performance goal is applicable only to competitive Research Project Support awards. Adequate award size and duration are important both to getting high quality proposals and to ensuring that proposed work can be accomplished as planned. They affect progress toward all outcome goals addressed through Research Project Support.
Maintaining openness in the system.
NSF must continually keep the proposal and award process open to new people and new ideas. Likewise, there must be opportunity to address particularly timely proposals that need more immediate action than the merit review process can provide. In FY 1999, NSF will concentrate in improving access to new investigators.
Performance Goal: NSF will increase the percentage of competitive research grants going to new investigators to at least 30 percent, a 2.6 percent rise over a baseline of 27.4 percent.
In the early 1990's, NSF had percentages approximating 30 percent of all competitive research grants15 going to new investigators. The percentage dropped over the mid-1990's and is now rising slightly. The baseline is a three year average, 1995-1997. NSF expects all program officers to be sensitive to the importance of bringing new individuals into their programs. By examining trends over time for both their own programs and the Foundation, they will be able to establish target levels for the program that are appropriate to the Foundation meeting its performance goal.
Identifying and acting upon emerging opportunities in science and engineering is another basic strategy that supports all of the outcome goals for research and education.
Identifying emerging opportunities.
NSF uses many mechanisms to assist in the identification of areas of emerging opportunity: advisory committees, special workshops, interaction with elements of the National Research Council, proposal review panels, and even the proposals themselves. In FY 1999, NSF will introduce a mechanism for this purpose that takes advantage of newly expanded computer networking capabilities. As NSF undertakes its annual planning processes, this information will help establish priorities for future funding. NSF provides quality control and maintains information on active Web sites that will be used to verify that this goal is met. The performance goal is applicable in all key functions.
Performance Goal: All directorates will establish Web sites for the science and engineering research and education community to provide suggestions for and comment upon emerging opportunities for significant advances in disciplinary and interdisciplinary fields of exploration; innovative research facilities; and new approaches to education and training.
Integration of Research and Education
Integrating research and education appears as part of the investment strategies supporting all of the outcome goals for education and research as described in NSF's GPRA strategic plan. NSF expects to see continuous improvement in the extent to which its research and education functions are accomplished jointly.
Encouraging integration in all NSF activities.
NSF's generic merit review criteria address the extent to which proposals will advance discovery and understanding while promoting teaching, training, and learning and vice versa. At present, many proposers do not include information that would permit reviewers to address this element. To embed integration of research and education in all aspects of NSF activities, NSF will encourage proposers to discuss this aspect of their activity in proposals. This performance goal is relevant to all key functions.
Performance Goal: NSF will ensure that all of its new announcements of opportunities and proposal solicitations will contain an explicit statement encouraging proposers to integrate research activities with improving education or public understanding of science.
In 1980, legislation gave NSF explicit responsibility for addressing issues of equal opportunity in science and engineering. Recognizing that progress toward all outcome goals for research and education requires maximal diversity of intellectual thought, NSF is emphasizing attention to enhancing the participation of groups currently underrepresented in science and engineering, including women, underrepresented minorities, and persons with disabilities, in all its programs. This performance goal applies to all key program functions.
Performance Goal: NSF will ensure that all of its new announcements of opportunities and proposed solicitations will include a statement encouraging proposers to address improving the participation of underrepresented groups in science and engineering in the course of their research and education activities.
The performance goals listed under this heading are applicable to the Research Facilities key program function.
Construction and Upgrade of Facilities
NSF puts a high premium on professional initial planning for construction and upgrade of facilities. But any planning for unique, state-of-the-art facilities must take into account the exploratory nature of the facilities themselves. Such facilities stretch the limits of technological capability.
Every year, in the President's Budget Request, NSF sets out a cost plan and schedule for major construction and upgrade projects currently underway or planned for initiation. Consistent with interagency norms, NSF will establish performance goals and measurements with respect to these plans. NSF program officers will work closely with the project directors to ensure that the performance goals can be met. Where potential problems are identified, the program officer will immediately inform the NSF-wide team assigned to that project so that all appropriate actions can be taken to keep the project within cost and schedule. NSF expects each construction and upgrade activity to meet these performance goals.
Construction and Upgrade of Facilities -- Annual Cost
Performance Goal: Keep construction and upgrades within annual expenditure plan, not to exceed 110 percent of estimates.
All active construction and upgrade projects included with the budget request will report on costs in comparison to their annual plan.
Construction and Upgrade of Facilities -- Annual Schedule
Performance Goal: Keep construction and upgrades within annual schedule, total time required for major components of the project not to exceed 110 percent of estimates.
Performance is measured by the number of estimated weeks ahead of or behind schedule at the end of FY1999, compared to the plan for the year, averaged over all facilities undergoing construction or upgrades in FY1999, as described in the President's Budget Request.
Construction and Upgrade of Facilities -- Total Cost
Performance Goal: For all construction and upgrade projects initiated after 1996, when current planning processes were put in place, keep total cost within 110 percent of estimates made at the initiation of construction.
Performance is measured by the average ratio of the cumulative actual cost to budgeted cost upon the completion of any discrete construction phase in FY 1999.
Operations and Management of Facilities
Facilities must operate efficiently and reliably and must offer appropriate opportunities, if they are to be valuable to those they serve. NSF program officers work closely with facilities' directors to ensure that facilities have appropriate resources to conduct operations and to provide maintenance that ensures reliable operations.
Performance Goal: Keep operating time lost due to unscheduled downtime to less than 10 percent of the total scheduled possible operating time.
Performance is measured as the average percentage among all facilities of full capacity "user units" lost during the year to breakdowns or other circumstances considered within the control of the facilities. The average across facilities is used in this instance because, although there should be latitude for some facilities to be run at greater failure rates with good reason, those facilities should be balanced by others operating more reliably. User units are defined separately for each facility, and are typically user-hours or something similar. Preliminary baseline performance for FY96, based on data from 26 facilities, is that an average of 6% of possible user units were lost. Facilities such as computer and communications networks where the concept of "full capacity user units" does not make sense are excluded from the calculation.
Data in support of performance goals under the heading Proposal and Award Processes are maintained in NSF's proposal and award systems. These systems are subject to regular checks for accuracy and reliability. One exception is the performance goal on time to prepare proposals. The budget division will maintain records for this performance goal that will be open to public scrutiny. Another exception is the performance goal on use of merit review criteria. Advisory committees will be provided with summary information developed from random samples of review records as they make their assessments. Background information to validate the accuracy of the summaries will be available upon request.
The performance goals in the categories of Emerging Opportunities, Integration of Research and Education, and Diversity will be supported by examination of materials available to the public.
Data in support of the performance goals under the heading Facilities Oversight are maintained by the directorates supporting the facilities, based on input from the facilities' leadership. NSF verifies the accuracy and completeness of the information through constant interaction between NSF staff and the management of the facilities. During FY 1998 and FY 1999, NSF staff will work to establish a central NSF database for facilities information.
[Return to Table of Contents]
PERFORMANCE GOALS FOR MANAGEMENT
Excellence in managing the agency's processes is an NSF goal on a par with our mission-oriented outcome goals. In the GPRA strategic plan, NSF articulated four critical factors in managing for excellence that provide the framework for annual performance goals.
Performance goals related to the merit review system, given their role in NSF investment processes, have been addressed in Appendix 2. The following performance goals for FY 1999 represent key indicators that NSF is managing its centrally funded and coordinated administrative activities efficiently and effectively in support of its mission. NSF has chosen to emphasize performance goals related to implementation of information technologies and human resources development in FY 1999. These performance goals are largely accomplished through the Administration and Management key function.
New and emerging technologies
A state-of-the-art communications and technology infrastructure has been essential to NSF's success in managing an increasing workload with approximately level resources. This investment also provides incentives in recruitment and retention of high quality employees.
Electronic proposal processing
Among federal agencies, NSF has also been among the leaders in doing its external business electronically. The research and education communities have worked with NSF staff to build our Web-based interface with grantee institutions, called FastLane. Each FastLane module has gone through a phase of expanding use.
Performance goal: In FY 1999, NSF will receive and process at least 10% of full proposal submissions electronically through FastLane.
The most complex use of FastLane is for the submission of full technical proposals. The proposal module was used approximately 800 times through June 1, 1997. NSF is the only federal research agency currently receiving proposals electronically on a production basis. In fact, among those proposing entities capable of it, electronic submission is the preferred mode at NSF. NSF's performance goal for FY 1999 will focus attention of staff on achieving full use of this capability. It will be met through a combination of increasing user friendliness of the system and programs requiring submission through FastLane. Enhanced staff training will enable NSF staff to use the FastLane modules effectively. NSF has requested a significant increase in funding for FY 1999 within Administration and Management to fully operationalize FastLane. This is important to realizing our long-term goal of using FastLane in the processing of 80-90% of all NSF proposals by 2007. Approximately 3000 proposals need to be handled through this route to meet the FY 1999 goal.
In order to increase the diversity of the U.S. science and engineering workforce, it is particularly important that the program officers at NSF exemplify that diversity. Yet this is the segment of the staff at NSF that shows the highest levels of under-representation of women, those minority groups under-represented in the science and engineering careers, and persons with disabilities. Realistic goals for closing that gap vary from one area to another across research and education. The most important link in the recruitment chain may be finding appropriate candidates and attracting them to apply for NSF positions.
Performance goal: In FY 1999, as all appointments for scientists and engineers are considered, the recruiting organization will demonstrate efforts to attract applications from groups that are underrepresented in the science and engineering staff as compared to their representation among Ph.D. holders in their fields.
NSF will stimulate members of under-represented groups to apply for NSF's science and engineering positions through increased outreach efforts including targeted advertising, attendance at job fairs, and site visits to minority institutions and organizations. The recruiting organization will document their recruiting efforts or the presence of members of underrepresented groups on their lists of finalists under consideration to the Division of Human Resources Management, which will maintain the data in support of this performance goal under the guidance of the Chief Information Officer.
Capability to work with information technology
Electronic communication is changing the character of work for support, administrative, and science and engineering staff. Everyone at NSF must have good computer skills and be able to master new ones on a continuing basis. Since so much of the Foundation's business will be done through FastLane in the future, our goal for FY 1999 focuses on that system.
Performance goal: By the end of FY 1999, all staff will receive an orientation to FastLane, and at least 95% of program and program support staff will receive practice in using its key modules.
A range of formal and individual orientation opportunities will be developed to support achievement of this goal.
Implementation of management reforms
In order to fully support its mission, NSF's information systems must be able to withstand the problems predicted for many systems at the turn of the century. Based on guidance from OMB, NSF has developed and submitted a plan (May, 1997) for evaluating, correcting, and testing its systems.
Performance goal: NSF will complete all activities needed to address the Year 2000 problem for its information systems according to plan, on schedule and within budget, during FY 1999.
NSF will comply with OMB guidelines and milestones for assessment, renovation, validation and implementation.
Assessing results for NSF's outcome goals requires a more accessible database of project results than NSF has previously maintained. A new project reporting system has begun a testing phase which will continue through FY 1998. During FY 1999, NSF will monitor the use of the system and the quality of the information gathered, and take appropriate steps to address problems.
Performance goal: During FY 1999, at least 70% of all project reports will be submitted through the new Project Reporting System.
At present there is no baseline from which to develop realistic target levels of performance. The level provided is an estimate of where we need to be to have a reasonably good basis for performance assessment using this system. The level of performance will be revisited based on the FY 1998 testing phase and provided at the time the FY 1999 Current Plan is developed. NSF must move quickly to full implementation of this system (at least 95 percent of all project reports submitted electronically -- and the remainder entered into the electronic database at NSF -- with high degrees of accuracy and completeness) so that the database represents faithfully the outputs and outcomes of NSF-funded activities.
The project reporting system and data on its use will be jointly monitored by the Division of Information Systems and the Budget Division. The information reported in the system will be used in reporting on the results of prior support when investigators reapply to NSF for further funding. This will help address issues of validity in terms of scientific elements, as will program officer perusal of annual and final progress reports and their use in annual assessment processes. We will work with universities and investigators to ensure consistent reporting of demographic information on participants.
All of these performance goals are supported by internal NSF management systems.
The Division of Information Services (DIS/IRM) maintains an extensive database concerned with use of FastLane. They will continue to maintain statistics on submission of full proposals through FastLane and on submission of project reports through the still-developing project reporting system. DIS will also monitor performance on the Year 2000 performance goal. All these data are subject to regular checks for accuracy and reliability.
The Division of Human Resources Management (HRM/IRM) maintains information related to staff recruitment and staff training, under the guidance of the Chief Information Officer. All these data are subject to regular checks for accuracy and reliability.
[Return to Table of Contents]