FY 2000 Report on the NSF Merit Review System
Bypass Navigation  
Introductory Letter
I. Introduction
II. National Policy Concerns and Needs
III. Major Findings and Issues
IV. Conclusions and Recommendations


Decision makers in the Executive and Legislative branches of government are concerned about the management of Federal investments in research, which in the most recent budget had reached over $90 billion as measured by R&D. Articulating this concern, former OMB Director Franklin Raines in a 1998 Science article raised the following questions: How large a scientific enterprise does the United States need? How can we set priorities in the Nation's R&D enterprise? How can we measure the success of our Nation's research programs? How can we strengthen the government-university partnership? How do we engage the American people in the excitement and wonder of science?

Likewise, Chairman of the House Science Committee, Sherwood Boehlert, responded to the proposal to double Federal funding for research by questioning: "What are we going to get for that money? How will we know if we are under- or over-spending in any field?" He went on to warn: "I want the Committee, early on, to take a serious look at the balance within the federal research portfolio...You can ... count on me to ask tough and uncomfortable questions to ensure that the scientific community is acting in its and the nation's long term interest…We really need to push for more data". (1/21/2001).

"A primary resource that would provide immediate benefits to decision makers is a broad-based, continuous capability for expert advice to both OMB and Congress during the budget allocation process."

How should the scientific community respond to these questions and expectations? How should it raise public awareness that the quality of life in the future will depend in large measure on the generation of new wealth, on safeguarding human health and the health of our planet, and on opportunities for enlightenment and individual development made possible by science and engineering discoveries? Will the response of the scientific community be effective against competing claims on the Federal budget?

These issues have prompted a vigorous policy debate over the last decade involving the Executive and the Legislature, the National Academies, and professional societies. Nonetheless, this debate has to this point generated no widely accepted process for the Federal government, with systematic input from the scientific community, to make priority decisions about the allocations in and across fields of research in support of Federal goals. The National Science Board has participated in this debate, issuing a series of policy statements, including an NSB working paper on Government Funding of Scientific Research in 1997. The NSB concluded in that paper that within the Federal budget there should be an overall strategy for research, with areas of increased and decreased emphasis and a level of funding adequate to both serve national priorities and to foster a world-class scientific and technical enterprise. To this end Congress and the Administration need to establish a process that examines the Federal research budget before the total Federal budget is disaggregated for consideration by Congressional committees. The Board further concluded in its 1998 Strategic Plan that a prerequisite for a coherent and comprehensive Federal allocation process for research is the development of an intellectually well founded and broadly accepted methodology for setting priorities across fields of science and engineering.

As followup to its earlier work, the Board undertook, beginning in March 1999, a focused examination of Federal priority setting for research in the U.S. at three levels: 1) in setting Federal goals, 2) in allocation decisions by Congress and the Administration that produce the Federal portfolio for research and 3) in Federal agencies and departments in achieving their missions in alignment with stated Federal priorities. The Board determined that the appropriate level for its focus is the second level, that is, the White House and Congressional processes that in the aggregate produce the Federal portfolio of investments in research.

Context for Federally Funded Research

The Federal role has always encompassed the missions of Federal agencies and departments and, beyond those missions, has helped to sustain a healthy national infrastructure for S&T. The Federal role today is especially critical for research that is high risk, requires long-term investment in the expectation of future high payoffs to society or that is unlikely to be funded by the private sector; for unique, costly, cutting edge research facilities and instrumentation; and for academic research that, as a primary purpose, supports the education of the future science and engineering workforce.

Box 2.

"The nation must reach a common ground and define a more realistic, pragmatic framework for allocating federal R&D resources. Only an inclusive national dialogue that brings together both the executive and legislative branches of government with the private sector and the U.S. university community will produce the needed consensus." --Council on Competitiveness, Endless Frontier, Limited Resources, 1996.

The national science and technology enterprise has grown and become more pervasive in both the private sector and in government, even as the Federal share of support to the enterprise has declined. Now, more than ever, achieving Federal goals for sustaining U.S. leadership in S&T demands partnerships and cooperation with other sectors. Understanding where Federal funding can be best employed and the level of investment required to assure the health of U.S. science and technology are critical to prudent management of the Federal portfolio. Commitment to an intellectually well founded, long-term strategy for Federal research must be an integral aspect of a sound fiscal policy, regardless of year-to-year fluctuations in available funds. The Federal budget process for research must assure sustained and sufficient support for a diverse, flexible, opportunistic portfolio of investments, emphasizing the long-term health of the knowledge base and infrastructure for research-including human resources.

Need for A Different Approach to Budget Coordination and Priority Setting

NSB discussions with spokespersons from Executive and Legislative branches and with experts on the budget, data and analytic methods, as well as reviews of the literature on budget coordination and priority setting identified the following needs.

Methodologies for:

  • Determining the appropriate size of the enterprise;

  • Determining the appropriate level of support to individual fields;

  • Achieving balance in the portfolio;

  • Setting priorities for the Nation's R&D enterprise;

  • Achieving effective communication on scientific matters with the American people;

  • Strengthening government-university partnerships.

Improved data, expert analyses, and scientific advice are needed to address these issues, including:

  • A continuing mechanism for expert advice representing a broad cross-section of the science and engineering research and education community to support difficult decisions on research investments-especially in major infrastructure projects;

  • Better quantitative data and methods of analysis adequate to measure the benefits of research;

  • A mechanism to identify and track the relevant Federal funds for S&T through the budget process in the Administration and Congress.

The Current Federal System

"The Federal role today is especially critical for research that is high risk, requires long-term investment in the expectation of future high payoffs to society or that is unlikely to be funded by the private sector; for unique, costly, cutting edge research facilities and instrumentation; and for academic research that, as a primary purpose, supports the education of the future science and engineering workforce."

The current Federal system for allocating funds for research is an incremental process that results in final allocation decisions based on input from a range of stakeholders, including the science and engineering communities. Ultimately, the Federal budget for research rests on aggregated political decisions in thirteen congressional appropriations subcommittees. There has been a host of critiques and suggestions for improving the process, many focused on the goals for research, but some suggesting changes to the process itself. The most frequent critique addresses a perceived lack of a clear methodology for priority setting and coordination. Several possible remedies have been suggested: structural changes to the process, alternative interpretations of the appropriate goals for Federal research, and new mechanisms for funding allocations and better management of the Federal research portfolio.

Since the late 1980s, and under both Republican and Democratic Administrations, there has been substantial attention devoted to developing better mechanisms for coordinating the Federal budget for research through the Office of Management and Budget (OMB) and the Office of Science and Technology Policy (OSTP) (Box 3).


Office of Science and Technology Policy (OSTP): The legislation that established OSTP declares that the United States shall adhere to a national policy for science and technology which includes the following principles: (1) the continuing development and implementation of a national strategy for determining and achieving the appropriate scope, level, direction, and extent of scientific and technological efforts based upon a continuous appraisal of the role of science and technology in achieving goals and formulating policies of the United States; (2) the enlistment of science and technology to foster a healthy economy in which the directions of growth and innovation are compatible with the prudent and frugal use of resources and with the preservation of a benign environment; and (3) the development and maintenance of a solid base for science and technology in the United States. It states the declaration of Congress that the Federal Government should maintain central policy-planning elements in the executive branch in mobilizing resources for essential science and technology programs, in securing appropriate funding for those programs, and to review systematically Federal science policy and programs and to recommend legislative amendments when needed. The functions of the Office include: (1) advise the President of scientific and technological considerations involved in areas of national concern; (2) evaluate the scale, quality, and effectiveness of the Federal effort in science and technology and advise on appropriate actions; (3) advise the President on scientific and technological considerations with regard to Federal budgets; and (4) assist the President in providing general leadership and coordination of the research and development programs of the Federal Government. (Excerpted from Public Law 94-292)

National Science And Technology Council (NSTC): The NSTC functions were to: 1) coordinate the science and technology policy-making process; 2) ensure science and technology policy decisions and programs are consistent with the President's stated goals; 3) help integrate the President's S&T policy agenda across the Federal Government; 4) ensure S&T are considered in development and implementation of Federal policies and programs; and 5) further international cooperation in science and technology (Executive Order 12881, November 23, 1993, Section 4).

Federal Coordinating Council for Science, Engineering and Techology (FCCSET): Established in 1976 under OSTP in the National Science and Technology Policy, Organization, and Priorities Act, FCCSET was "to consider problems and developments in fields of science, engineering, and technology and related activities affecting more than one Federal agency, and to recommend policies designed to provide more effective planning and administration of Federal scientific, engineering, and technological programs." (Title IV, Public Law 94-292).

The cabinet level National Science and Technology Council (NSTC) (Box 3) in the previous Administration and the earlier Federal Coordinating Council for Science, Engineering and Technology (FCCSET) (Box 3) provided mechanisms in OSTP for identifying major national initiatives that cut across agencies in designated priority areas (e.g., nanotechnology, global climate change, and information technology). Under the last Administration, the NSTC was established by Executive order as part of the OSTP science and technology policy apparatus. However, unlike FCCSET, OSTP and the Director of OSTP, which were established through legislation, the NSTC had no permanent status. Likewise, the President's Committee of Advisors on Science and Technology (PCAST), whose purpose was to provide "critical links to industry and academia," was established by Executive order.

Furthermore, in neither the Executive nor the Legislative branches is there a mechanism for evaluation that takes into account the breadth of Federal investments within the context of Federal goals for research. The Executive Branch, through OMB, OSTP and PCAST, made an effort to treat Federal funding of research as a portfolio, recently taking into account the issue of balance among fields of science in Federal support across all agencies and departments.

These steps have been in the right direction, but are only a preliminary effort. Congress (Box 4) also has directed attention to what might be done to improve its process but has not yet taken any action to implement formal Congressional mechanisms comparable to OSTP to coordinate functions across budget lines, agencies and departments, and committees.

Box 4. The Congressional Budget Process

At no time in the Congressional authorization or appropriations process is the R&D portfolio examined as a whole, across the Federal government. Impediments to coordination and informed priority setting exist in the form of budget "firewalls" between defense and civilian programs, and the reduction in advisory bodies for congressional R&D policy, planning and funding, including the Office of Assessment (OTA), and separate science units within Congressional Research Service and General Accounting Office. The House Science Policy Study, Unlocking Our Future, argues that "at a minimum Congress and the Executive Branch should improve their internal coordination processes to more effectively manage, execute, and integrate oversight...While the Office of Management and Budget can fill this role in the Executive Branch, no such mechanism exists in the Congress. In those cases where two or more Congressional committees have joint jurisdiction over or significant interest in large, complex technical programs, the affected committees should take steps to better coordinate their efforts. Wherever possible, the affected committees should consider holding joint hearings and perhaps even writing joint authorization bills."

To enhance the effectiveness of Federal investments in achieving long-term goals for research, a regular, credible process that relies in part on expert input from the science and engineering communities is essential for priority setting among competing investment choices. The Federal portfolio for research is an accounting device that aggregates the research portfolios of the individual departments and agencies funding S&T. It has not been managed as a portfolio. As a precondition for priority setting across the Federal research budget, coordination must be achieved among its diverse components. While efforts at better coordination through OSTP mechanisms have been useful in managing cross agency initiatives, coordinating mechanisms are also necessary for evaluating the performance of Federal research investments as a portfolio and for identifying gaps, overlaps, areas for decreased emphasis, and the top priorities for additional investments. Coordination and priority setting therefore must be intertwined in the Federal research budget process.

The Need for More and Credible Data and Analyses

Box 5.

"I doubt that anyone would sign on to a research project as poorly designed as our current national experiment in science and technology policy...our scientific enterprise remains adrift, without a connection to the broader society." -Representative George E. Brown, Jr., 1998.

No mechanism exists to provide strong quantitative input to justify a particular level of investment in Federal research based on expected benefits to society, due in part to the lack of data and methods to measure research benefits. Data on Federal research funding, especially at the field level, are often unavailable on a timely basis to inform budget allocation decisions, use outdated research field definitions, fail to capture important characteristics of research activities, and suffer from inconsistent applications of definitions across reporting units.

In spite of the need for more and better data on the Federal research enterprise, collecting such data requires the cooperation of a large number of Federal agencies and departments, many of which see no benefit to themselves from this effort. There are few resources available to address the major undertaking that would be required to generate reliable data tailored to the needs of budget decisions and outcomes for research funding allocations. It would require a concerted effort to define and obtain agreement among the many Federal units that would be involved, and would require support from OMB and Congress to assure collection of high quality, timely data.

Identifying the Composition of the Federal Research Portfolio

OMB requires agencies to report research and development (R&D) activities that they are funding for the annual budget process. Even if reliably measured across funding units, since "D" at about 55% of the total is larger than "R", reporting the sum of the two as the measure of Federal research investment results in an indicator that fails to reflect accurately the Federal funding to discovery and innovation. Also, significant fluctuations in support for "research" tend to be obscured when combined with the larger "development" category funds.

There have been several attempts to provide a better measure for the federally funded activities that contribute to national innovation. The National Academies proposed a coordinated "Federal Science and Technology Budget" (NRC, 1995), a subset of Federal R&D that constitutes "federal support for a national science and technology base." The FS&T budget would provide Congress with a tool for tracking the aggregated pool of Federal departmental and agency funds that support the science and technology base. OMB has employed over the last few budget cycles a similar mechanism for tracking the President's research priorities through the budget process. This mechanism comprised a collection of program budgets that are primarily research programs, but also includes non-research elements, such as the education and human resources component of the budget for NSF. OMB found this mechanism useful in highlighting Federal research investments and effective in supporting the President's priorities for research through the budget cycle.

The Board, for the purposes of this study, has focused on "S&T", in accord with the argument put forward in the NRC report and efforts by OMB to identify basic and applied research activities for tracking through the budget process. At the same time the Board recognizes the lack of consensus on federally funded activities that should comprise Federal "S&T". Criteria for inclusion of activities in a Federal budget for research for the purpose of monitoring and evaluating Federal activities as a portfolio will require further discussion and analysis. The important subset of research funding devoted to the long- term, high-risk basic research is especially vulnerable to becoming invisible in the larger budget for S&T. It is critical that this component-- which provides the long-term investment to produce as yet unforeseen major breakthroughs in knowledge--receive sustained public support.

Capturing the Character of Activities Supported

Within research, the character of research fields and activities has changed over time, resulting in definitions that no longer capture important distinctions in federally funded research activities. Special areas of weakness include multidisciplinary and cross disciplinary workgroups and teams, emerging areas, differences in interpretation across agencies' reporting units, and the evolving content of traditional research fields themselves. In addition, educational contributions of research--particularly in academic institutions for graduate education--are not captured in most agencies' databases.

Reliability and Timeliness

Differences in interpretation have resulted in wide discrepancies in research funding reported by performing and funding units--or even within the Federal government across agencies and programs-even though they ostensibly describe the same activities. In addition, timeliness, in most cases essential to budget allocation decisions, is not possible with Federal databases based on surveys. Much of the data measuring the Federal research portfolio with respect to programs funded, support for fields of science and engineering, and performing institutions are several years old at best. Timeliness will become increasingly more problematic as rapid changes in science and technology increase the need for current data to monitor Federal investments.

Assessing World Leadership of U.S. Science and Engineering

National capabilities in science and technology and the government role in enhancing these assets are a growing emphasis for governments around the world. As science and technology capabilities have become more broadly distributed, there is a need for the U.S. to monitor the U.S. enterprise against an international backdrop to detect declines in national capabilities in science and technology relative to other nations or to identify new opportunities for research investment that merit public support. The National Academies have urged regular international benchmarking at the field level to assess the health of individual fields of research in the U.S. in their 1993 report, Science, Technology and the Federal Government/National Goals for a New Era and their 1995 report, Allocating Federal Funds for Science and Technology. The use of international comparisons of the productivity of research fields and international expert participation in assessments of research programs are common in other countries.

The Board has noted the need for monitoring the relative health of U.S. science and technology as part of a continuing evaluation of the Federal portfolio, drawing on existing data and expert analyses, and continually improving data and methods for international comparisons that inform priority setting.

Understanding the Role of Federal Research in Producing Economic and Other Benefits

Box 6.

"In the long run-in good budget years as in bad-it is essential that policymakers...recognize the fragility of (the U.S. S&T) enterprise and the critical federal role in sustaining it. It is up to the members of the science and engineering community to carry this message to them" - A.H. Teich, AAAS, 1999.

A large number of studies have attempted to elucidate, and in many cases measure quantitatively, the relationship between research and innovation and the benefits of research for society. Organizations like the Council on Competitiveness, the Science and Technology Policy Institute, RAND; OSTP, and NSF have explored issues and methods for analyzing the role of a range of factors in innovation--including Federally funded research--and resulting economic and social benefits. Academic programs for addressing these questions are inadequately funded. This is an area where additional research investment could improve both qualitative and quantitative data to inform budget allocation decisions, communicate the benefits of research to the public, and contribute to the effectiveness of Federal research investments.

Back to NSB Publications