previous sectionprevious section  |  Table of Contents  |  next sectionnext section

Part I • Outcome Goals and Results for FY 1999

This section describes how the taxpayer benefits from investments made by NSF’s programs. NSF’s long-term outcome goals address how the investments made by programs have led to results important to the broad mission of the agency. These outcome goals do not lend themselves to quantitative reporting, therefore NSF has developed an "alternative format"—a qualitative scale that allows NSF to report whether the agency has been "successful" or "minimally effective" in achieving its outcome goals. Also, because many research results appear long after an investment is made, this assessment report of NSF’s program performance is mostly retrospective. That is, the outcome results reported in FY 1999 are from investments made prior to FY 1999. The results of the investments made in FY 1999 will not begin to be reported until beyond FY 2000.

How Research Results Are Assessed: Each NSF program is reviewed every three years by independent external panels of experts—Committees of Visitors (COVs) and Advisory Committees (ACs). In FY 1999, the Foundation sponsored more than 190 programs. Of these, 82 programs were reviewed by COVs, producing a total of 43 COV reports. Each COV is a subcommittee of an Advisory Committee, which reviews the products of the COVs. Both COVs and ACs are asked to judge whether NSF is successful in meeting its goals. Their judgment is important for the management of programs and in advising NSF. The COV and AC meetings are subject to Federal Advisory Committee Act rules.

Examples of Research Results: The following discussion of NSF performance includes only a sample of the many noteworthy achievements reported by NSF programs in FY 1999. The examples presented here are illustrations of the impact and success of NSF programs in achieving its outcome goals and are selected to demonstrate the range of activities supported by NSF. In many instances, they reflect connections with other agencies or other countries, or areas of special emphasis identified in the FY 1999 NSF Performance Plan. Many have been identified as significant examples in COV reports. While these examples may appear to be anecdotal, in each case a grant number issued by NSF can be used to identify the example for purposes of verification.

Summary of Results: Overall, based on FY 1999 evaluations, NSF is largely successful in meeting its performance goals. NSF’s programs were consistently judged by external evaluators to have high quality scientific outputs and outcomes. Two areas identified as needing improvement include increasing participation of underrepresented groups, and implementing use of the new merit review criteria that were introduced in 1998. In cases where COVs or ACs did not rate the goals, the committee reports indicated that the goals did not apply to the programs under review, or indicated that they did not have adequate information to provide a rating.


OUTCOME GOAL 1: Discoveries at and across the frontier of science and engineering.
1999 Performance Goal: NSF is successful in meeting this goal when, in the aggregate, NSF grantees (1) make important discoveries; uncover new knowledge and techniques, both expected and unexpected, within and across traditional boundaries; and (2) forge new high-potential links across those boundaries.
1999 Results: Successful. In FY 1999, 35 COV and 8 AC reports rated NSF for this goal. All reports rated NSF successful in achieving this goal.

NSF supports cutting edge research that yields new discoveries over time. These discoveries are essential for maintaining the nation’s capacity to excel in science and engineering and lead to new and innovative technologies that benefit society.

NSF’s key strategy for success is to support the most promising ideas in research and education, as identified through merit review of competitive proposals. Innovation and creativity, cooperative research through partnerships, and education and training are emphasized and encouraged.

In addition, the FY 1999 government-wide performance plan contains a performance goal that is particularly relevant to NSF research and education programs:

1999 Performance Goal: An independent assessment will judge NSF research programs to have the highest scientific quality and an appropriate balance of projects characterized as high-risk, multidisciplinary, or innovative.

1999 Results: This goal was achieved. In FY 1999, all COV reports indicated that NSF programs were of high scientific quality. Of the 30 COV reports that gave an opinion on balance of projects in the programs under review, 24 reports indicated that the balance was appropriate. Six reports indicated that a few programs under review did not have appropriate balance, with lack of investments in high-risk projects being the most frequent concern. To address these concerns, NSF staff will be asked to make investments that address this goal, and will strive to identify high-risk projects for reviewers and evaluators. NSF is looking at how to improve the indicators for this goal, and how to provide better data to evaluators of this goal for FY 2000.

The following examples illustrate the impact and success of NSF’s programs in achieving important discoveries. The examples were chosen to address the FY1999 areas of emphasis across NSF’s programs and to demonstrate linkages across traditional disciplinary boundaries through collaborations.

Advances in Biology

  • NSF-funded scientists, using material obtained from a plant virus, determined the three-dimensional structure of part of the molecular machinery responsible for the virus' ability to control the genetic material of a host cell for its own purposes. This structure provides important clues about ways to modify or control replication of viruses in plants and animals.

  • Scientists funded by NSF are studying how insects feeding on plants are controlled by a combination of their natural enemies and limited plant resources. The interactions go beyond a simple predator-prey relationship because of the occurrence of high-order interactions among the various natural enemies. Not only is there a complex of predators acting, but the predators may attack each other. The ability to predict the combined effect of multiple predators is critical both to the refinement of predatory-prey theory and the improved management of severe agricultural pests such as planthoppers.

The Nobel Prize in Chemistry, awarded annually by the Royal Swedish Academy, recognizes major discoveries over the lifetime of the recipient’s career and is an indication of the influence that the individual has had on chemistry. The 1999 Prize was awarded to a researcher whose NSF support over a 20-year period has resulted in an understanding of the details of the birth and death of molecules during chemical reactions. Just as the motion of dancers can be frozen in time using the pulsed light from a strobe, the motion of molecules during chemical reactions can be followed in time using pulsed lasers, much like a rapid strobe. The work for which the Nobel Prize was awarded used rapid laser pulses only a few femtoseconds in duration. A femtosecond is to a second what a second is to 32 million years. This area of chemical research has been named femtochemistry. This research has begun to uncover the details of reactions that form the basis for important processes ranging from catalysis to vision.

NSF Research in the Antarctic and Arctic

  • Fossil bones of hadrosaur and mosasaur dinosaurs were discovered on the Antarctic Peninsula. This finding was awarded "Discovery of the Year" by the Royal Geographic Society of London. The findings are important because current knowledge of these dinosaurs is based mostly on North American fossil sites, and includes relatively rare juvenile mosasaurs. Finding the remains of the hadrosaur, a large terrestrial herbivore, is important because the presence of this animal implies a robust and productive vegetation component of the ecosystem.

  • An interdisciplinary team found evidence of climate warming in the Arctic, where the ice cover in the Arctic is rapidly diminishing, due to an encroachment of warmer deep water, melting from a warmer atmosphere, and a positive feedback in the air-snow-ice energy exchange.

Discoveries in How the Young Learn

  • Infant Development: NSF findings in infant cognition have radically altered our picture of early development. To probe the infant’s mind, researchers have used innovative methods that rely on a simple and reliable behavior: infants will look longer at unexpected events. Using this principle, researchers have examined infants’ concepts of the "object," and their concept of everyday things (such as a cat, dog, or chair). The research shows that infants know a great deal. They can track objects through space and time, and even as the objects move behind a screen and then become visible again. They can enumerate small numbers of objects, suggesting they develop some basic knowledge of numbers at an early age.

  • Young Children: NSF-supported research is making discoveries in the foundations of algebraic reasoning among young children. Findings suggest that youngsters are capable of mathematics and science learning that greatly exceeds traditional expectations.

  • Students: NSF researchers have developed teaching materials and approaches that help youngsters to acquire deeper understanding of the causes of scientific phenomena. Students in the program faced with initial challenges were better able to understand complex phenomena, such as those with multiple causes, resulting in learning that exceeded that of their peers.


OUTCOME GOAL 2: Connections between discoveries and their use in service to society.
1999 Performance Goal: NSF is successful in meeting this goal when in the aggregate, the results of NSF awards: (1) are rapidly and readily available; (2) feed, as appropriate, into education, policy development; or (3) are used by other federal agencies or the private sector.
1999 Results: Successful. In FY 1999, 35 COV reports and 8 AC reports rated NSF for this goal. Of these, 34 COV reports and all AC reports rated NSF successful in achieving this goal.
America’s national security, economic competitiveness, health, environment, quality of life, and understanding of the world around us depend on taking advantage of discoveries. Discoveries resulting from basic research and education lead to new knowledge, which often cannot be identified at the start of a project. Thus, the connections are not immediately apparent, and may only be realized decades later. The new knowledge frequently leads to applications, which can have a significant impact on society. NSF views the public accessibility of NSF generated results as well as partnerships among government, academia, and industry as critical components for the progress of science and technological innovation.

NSF’s key strategy for success in achieving this goal is through use of the merit review process to make awards for research and education activities that focus on discovery and that create or have the potential for connections with use in service to society. Potential for use in service to society is an element in the merit review criteria established by NSF and used in the decision process leading to funding.

The following examples illustrate the impact and success of NSF’s programs in achieving important connections between discoveries in FY 1999:

Predicting Storms: Local high-impact weather causes economic losses in the United States that average $300 million per week, impacting over 10 percent of the U.S. economy each year. The mission of one NSF-funded center is to demonstrate the practicability of numerical weather prediction of storms and to develop, test, and validate a regional forecast system appropriate for operational, commercial, and research applications. The May 3, 1999 tornado outbreak in Central Oklahoma was used to test the storm model, and included networking, collecting, and processing data from several National Weather Service Doppler weather radars. While the storm-scale forecast was not perfect, improvement over current operational models was substantial—showing increased precision in predicting. It clearly demonstrated the value of using mesoscale radar data in storm prediction that can provide a two-hour forecast down to the scale of a county. The project was able to generate short-range high-resolution forecasts that dramatically out-performed the National Weather Service forecast during the tornado outbreak. As this forecasting capability is further developed, it will become a critical tool in determining which areas will be most severely hit by storms, thereby allowing sufficient and timely warnings to be issued to persons in affected areas. Who stands to benefit? The commercial airlines industry, power and communications industries, surface transportation, agriculture, defense and space flight, construction, insurance and recreation industries will clearly benefit, as well as the National Weather Service and the general public.

Oceanographic Research in Service of Fisheries Management: Over the past decade, there have been dramatic decreases in fish populations in the major fisheries of the United States and Canada including the Cod and Haddock fisheries of the eastern seaboard from the Grand Banks to Georges Bank. These decreases have had disastrous economic and societal consequences. Within the U.S., NSF spearheaded a research program whose goal is to predict changes in the distribution and abundance of marine species as a result of changes in their physical and biotic environments as well as to anticipate how their populations might respond to climate change. The program will provide information to fisheries organizations so they may better preserve stocks.

Practical Application of Digital Library: Research in digital libraries led to practical technology exploited in many different areas. The FBI applied digital library (DL) technology to establish an "electronic reading room" to comply with the Freedom of Information Act. The California Department of Transportation applied DL to roadside vegetation, quasi-real-time tracking of road conditions, and flood-related emergency services.

Global Change Research and Life in Extreme Environments—Sustainability Project: Results from a large, long-term interdisciplinary study involving eight natural and social science disciplines have provided a combined assessment of the effects of a predicted global warming, oil development, tourism, and government cutbacks on the sustainability of Arctic villages in the range of the porcupine caribou herd. The effects of global change on the tundra food sources for caribou on the Alaskan north slope and elsewhere are critically important to local native villages where a subsistence lifestyle is practiced either as a necessity for survival or as a cultural choice.

Bill Nye the Science Guy: This NSF-supported television series was awarded several Emmys in 1998-1999, including Outstanding Children’s Series, Outstanding Directing in a Children’s Series, Outstanding Achievement in Single-Camera Editing, and Outstanding Achievement in Sound Editing. The series resulted in increased comprehension and application of science facts and concepts among its child viewers, who were able to provide more complete and complex explanations of scientific concepts. "Bill Nye the Science Guy" is one of the most popular projects in NSF’s informal science education programming.


OUTCOME GOAL 3: A diverse, globally-oriented workforce of scientists and engineers.
1999 Performance Goal: NSF is successful in meeting this goal when in the aggregate: (1) participants in NSF activities experience world-class professional practices in research and education, using modern technologies and incorporating international points of references; (2) academia, government, business, and industry recognize their quality; and (3) the science and engineering workforce shows increased participation of underrepresented groups.
1999 Results: Successful. In FY 1999 36 COV reports and 8 AC reports rated this goal. Of these, 33 COV reports and 5 AC reports rated NSF successful in achieving all or most areas of the goal. Four COV reports and 2 AC reports indicated that NSF should do more in the area of showing increased participation of underrepresented groups. A few reports noted that the data necessary to measure this goal were not always available, and more outreach by NSF is necessary. These issues are being addressed by NSF staff in the FY 2000 and FY 2001 performance goals and plans.

Although NSF provides only a relatively small portion of the overall U.S. investment in the development of the science and engineering workforce through its programs, this investment is particularly important to the development of the workforce of the future. The quality of the future workforce is dependent on the investment being made now to educate and train students. A diverse science and engineering workforce that is representative of the American public and able to respond effectively to a global economy is vitally important to America. As a nation, we need new technical knowledge and people trained to use that knowledge. The competence and capabilities of the nation’s science and engineering workforce keep America at the forefront of innovation and technological progress.

One of NSF’s key strategies for success in achieving this goal is by providing opportunities for participation in integrative research and education experiences. To influence the development of integrated approaches, NSF has developed a number of Foundation-wide programs intended to facilitate the integration of research and education. Each of these programs relies on NSF’s close interaction with the academic science and engineering communities to draw research and education together. NSF works to achieve this goal by making awards for research and education activities that are intended to influence the development of the science and engineering workforce, and increase the participation of underrepresented groups.

The following examples illustrate the impact and success of NSF’s programs in achieving Goal 3 in FY 1999.

  • NSF programs contribute to this goal by enabling outstanding education and training opportunities in all areas of science, engineering, education, and technology. In the past year, 900 of the nation’s top graduate students were awarded NSF Graduate Research Fellowships. Of these, 49% were to women and 8% were to minorities. NSF has supported 1,355 graduate research trainees since 1993, of which 37% were females and 8% were African Americans, both proportionately higher percentages than national enrollment statistics indicate. During this same period, more than 1,300 NSF graduate research trainees authored or co-authored 1,466 scientific articles, 171 book chapters, 31 books, and applied for 15 patents.

  • State-of-the-art training opportunities at 42 new technical training sites reached over 700,000 students. Research Experiences for Undergraduates, in nearly all of the fields that the Foundation supports, were available to students at more than 300 sites in 46 states.

  • In FY 1999, several NSF programs supported K–12 science and engineering outreach activities, with one program alone impacting more than 300 teachers and 11,000 students from states across the nation. Another program to increase participation of girls in science and technology careers documented approximately 1,000 teachers indirectly trained by the program through knowledge transfer. The total number of girl participants impacted by this program is estimated to be approximately 10,000.

  • Programs sponsored by NSF are bringing high quality science education to more than 70,000 future teachers, and enhancing the professional development of more than 150,000 teachers who are already in the classroom. These activities will help to address a projected need for more than 200,000 new secondary mathematics and science teachers over the next decade. This estimate is based on Department of Education projections of aggregate numbers of teachers needed and the current proportion of teachers who are teaching secondary mathematics and science.


OUTCOME GOAL 4: Improved achievement in mathematics and science skills needed by all Americans.
1999 Performance Goal: NSF is judged to be successful in meeting this goal when, in the aggregate, the results of NSF awards lead to: (1) the development, adoption, adaptation, and implementation of effective models, products, and practices that address the needs of all students; (2) well-trained teachers who implement standards-based approaches in their classrooms; and (3) improved student performance in participating schools and districts.
1999 Results: Successful. In FY 1999, 19 COV reports and 3 AC reports rated NSF for this goal. Of these, 15 COV reports and 3 AC reports rated NSF successful in achieving this goal in all or most areas of the goal. Three COV reports indicated NSF was not fully successful in some areas of this goal. This goal is viewed by many COVs and ACs as being more applicable to NSF’s educational activities, and therefore rated it only when applied to those programs.

This goal addresses a need widely recognized by all Americans. Proficiency in essential skills and understanding of basic concepts in mathematics and science is critical to the earning power of individuals, to the nation’s economic competitiveness, and to the quality of life in the 21st century. NSF is the only agency that directly aims at developing such proficiencies at all levels of education.

NSF has established linkages with other agencies, and supports the development of prototypes for cooperative activities involving state and local educational agencies, and the private sector.

NSF supports a continuum of activities that enables improvement of mathematics and science skills for all Americans. These activities include educational reform at the K-12 levels and beyond; teacher education and professional development; research activities that use science and technology to inform better educational practice; and activities that bring science into the classroom and place students at the sites of exploration and discovery. Common themes that are emphasized across the Foundation include the implementation of high quality, standards-based instruction for all students; integration of research and education; and coordination of resources, policies, and practices to maximize the impact of educational investments. These activities benefit students, teachers, and the general public nationwide.

Investments in education are made to facilitate the development of essential skills in mathematics and science for all Americans through the promotion of broad-based or system-wide reforms in science, mathematics, engineering, and technology education that are based on national standards. Important for FY 1999 are projects that include systemic approaches, attention to teacher preparation and development, partnership with other agencies, and developing a strong research base for use by practitioners.

The following examples serve to illustrate the impact and success of some of NSF’s programs in achieving improvement in math and science skills, including areas of emphasis in FY 1999.

  • NSF-supported activities benefit students, teachers, and the general public nationwide. For academic year 1998, NSF education reform efforts in states, urban centers, and rural areas were underway in 38 states, and the Commonwealth of Puerto Rico. An estimated total of 19,000 schools, more than 156,000 K–12 mathematics and science teachers, and more than 11 million K–12 students benefit from these programs. Informal education through exhibits, radio, television, film, and youth and community-based activities are estimated by grantees to reach over 150 million people – more than half of all Americans – every year.
  • In the area of mathematics at the elementary and middle school levels, the enhanced quality of professional development activities, coupled with increased course-taking requirements in areas such as algebra, geometry, and measurement, resulted in significant improvement in student achievement. For example,

    • For participating Dallas students, mathematics gains exceeded expectations in seven of eight grades. For grades 3-8, the schools participating over a five-year period outperformed their state-wide counterparts on a state mathematics test at five of the six grade levels tested.

    • In Philadelphia, student scores in math and science increased for the three grades being tested (4,8, and 11), according to results of a national assessment tool used by major school systems to cover core academic subjects. For example, at grade 4 the percentage of students demonstrating science proficiency increased from 40% in 1996, to 49% in 1998.

  • Bringing Kids to Research through the Internet:

    • Chickscope is the result of an interdisciplinary program that puts magnetic resonance imaging (MRI) technology into the K–12 classrooms through the Internet. Chickscope enables teachers and students to access and operate an MRI system on a website, to peer inside a chick embryo, and to observe its development over the 21 days it takes for the egg to mature.

    • Bugscope is the result of a major research instrumentation award that allows students to use a remote controlled microscope that magnifies more than 5,000 times, to see what makes bugs tick. Bugscope is a nationwide science program developed by researchers as an innovative way for students to see that the study of science can be fun and exciting.


OUTCOME GOAL 5: Timely and relevant information on the national and international science and engineering enterprise.
1999 Performance Goals: NSF is successful in meeting this goal when: (1) the average time interval between the reference period (the time to which the data refer) and the reporting of data decreases by 10%; and (2) customer satisfaction ratings are achieved with the relevance of products offered of at least 45% "excellent" and at least 90% "excellent" or "good."
1999 Results: Successful. Both the timeliness and the relevance goals were achieved.

This goal addresses NSF’s legislative mandate to collect, interpret, and analyze data on scientific and engineering resources, and to provide a source of information for federal policy formulation. In a recent survey, a sample of the science and engineering policy community indicated that improving timeliness of data was high priority for them. The value of information on the science and engineering enterprise is highly dependent on its relevance to those who seek to use it in making policy decisions. Different users are interested in different aspects of the enterprise.

previous sectionprevious section  |  Table of Contents  |  next sectionnext section