This document has been archived.

Report on the Evaluation of NSF's EPSCoR Program
Preface | Executive Summary | Table of Contents
Part A: EPSCoR Program and Its Evaluation | Part B: Evaluation Findings | Part C: Policy and Program Implications
References

SECTION A:
The EPSCoR Program and Its Evaluation

1. What Is the EPSCoR Program?

Since its creation in 1950, the National Science Foundation (NSF) has sought to advance research and education in science, mathematics, engineering, and technology throughout the United States. Explicit in its original mandate was a cautionary warning to avoid concentrating federal funding of academic research and development (R&D)1 on a geographic basis. Thus, NSF’s authorizing legislation, as amended (42 U.S.C. 1862, Sec. 3e), stated:

In exercising the authority and discharging the functions referred to in the foregoing subsections, it shall be an objective of the Foundation to strengthen research and education in the sciences and engineering, including independent research by individuals, throughout the United States, and to avoid undue concentration of such research and education. [Emphasis added.]

In 1978, congressional concern about the geographical concentration of federal funds for academic research led Congress to authorize the NSF to conduct the Experimental Program to Stimulate Competitive Research (EPSCoR) (Greenberg, 1967; U.S. Congress, Office of Technology Assessment, 1991; and Martino, 1992). Eligibility for EPSCoR participation was restricted to states that historically had received relatively low levels of federal R&D funding. Additionally, such states were required to demonstrate a commitment to improving the quality of university-based research in science, mathematics, engineering, and technology.

EPSCoR’s Mission

...enhancing the capability of eligible states to compete for research funds.

EPSCoR, as the program’s name reflects, was designed to stimulate competitive research--and not to be a special set-aside program (EPSCoR Program Solicitation, 1989b). Through EPSCoR, NSF would become a partner with participant state governments and universities. The role of state government was seen as especially important in achieving the EPSCoR program’s objectives because of the overwhelming importance of public universities within the higher education systems of EPSCoR states. In 1995, for example, 53 of the 56 universities participating in the EPSCoR program were publicly supported universities.

Over time, NSF’s goals for the EPSCoR program broadened to include enhancement of educational and human resource opportunities for underrepresented faculty and student populations in science, mathematics, engineering, and technology, and the transfer of academic research to the private sector.

EPSCoR’s Program Strategy

EPSCoR: A multifaceted program strategy

EPSCoR’s program strategy is designed to increase the competitiveness of merit-reviewed proposals from investigators in states that have historically received low percentages of federal R&D support. The assumption is that increased competitiveness will subsequently lead to increased R&D funding in these states. Further, on the premise that university science and engineering departments can positively influence a state’s economic and human resource development, EPSCoR pursues long-term partnerships with state leaders in government, business, and higher education (Malecki, 1991; and Luger and Goldstein, 1991). In pursuit of these goals, EPSCoR applies the following strategies:

  1. Support research investigators in states with the lowest share of R&D funds per scientist and population (18 states and Puerto Rico are eligible, according to the criteria defined by the program in 1991);
  2. Exclude, even in these states, proposals rated “excellent” and thus already considered competitive according to NSF’s traditional peer review rating system;
  3. Support research projects and explicit capacity-building strategies aimed at increasing the research competitiveness of the whole state (the capacity-uilding aspect was critical because of the long-term expectation of increased research funding, going beyond support from the EPSCoR program itself); and
  4. Support groups of related research projects within and across universities (research clusters)--rather than single investigators--as a means of promoting long-term, sustainable increases in institutional research competitiveness.
...systemic and sustainable change...

Although the strategies appear to focus on individual or groups of research projects, EPSCoR’s vision from the onset was systemic. The program always has sought to strengthen the entire science and technology (S&T) infrastructure in an eligible state--thereby increasing the ability of EPSCoR researchers to compete for federal and private sector R&D funding and accelerating the movement of EPSCoR researchers and institutions into the mainstream of federal and private sector R&D support.

Non-Federal Cost Sharing

The EPSCoR partnership between NSF and eligible states requires investment by both partners. A one-to-one dollar match was required for the awards made from 1980 to 1994.

Infrastructure Improvement

In preparing to submit a proposal, an EPSCoR steering committee within each eligible state was expected to have undertaken a comprehensive analysis of the strengths and opportunities for developing its research institutions in support of the state’s overall R&D objectives. Examples of infrastructure activities EPSCoR has funded include the following:

  • Start-up funding for new faculty, including seed funding of faculty research leading to the submission of competitive grant proposals;
  • Faculty exchange programs with major research centers;
  • Acquisition of state-of-the-art research instrumentation and development of nationally competitive high-performance computing and networking capabilities;
  • Partnerships between the state’s research universities and the private sector;
  • Innovations in graduate education that will expand student career options and facilitate the entry of individuals from traditionally underrepresented groups (that is, African-Americans, Hispanics, Native Americans, Pacific Islanders, women, and the physically disabled) into high-demand S&T fields;
  • Funding of senior faculty to 1) work with newly developing S&T businesses; 2) serve as policy advisors for state legislatures and S&T agencies; 3) serve as senior postdoctoral associates in established department- or institution-wide research programs; 4) develop new educational technologies and delivery systems; or 5) create new career alternatives for young scientists; and
  • Creation of graduate research training groups, or similar appropriate mechanisms, that integrate education and research, encourage multi-disciplinary educational experiences, or establish links with industry and national laboratories.
“...18 states and Puerto Rico...”

The NSF initiated the EPSCoR program in Fiscal Year (FY) 1978 with seven planning grants. From that time through FY1996, the program awarded $182.2 million in grants and cooperative agreements. Participating in the program by the end of the period were 18 states and the Commonwealth of Puerto Rico. The states were Alabama, Arkansas, Idaho, Kansas, Kentucky, Louisiana, Maine, Mississippi, Montana, Nebraska, Nevada, North Dakota, Oklahoma, South Carolina, South Dakota, Vermont, West Virginia, and Wyoming.

By 1994, EPSCoR had been operational for over 15 years and the NSF decided that, as part of its plan to evaluate the programs of the Directorate for Education and Human Resources, the time had come to look at whether the EPSCoR program was achieving its intended outcomes. This evaluation of the EPSCoR program covers the period 1980-1994, with 1994 as the last year for which R&D expenditures were available at the time of the evaluation. The evaluation thus does not cover changes or developments in the EPSCoR program after 1995.

____________________
1 The term “academic R&D” is used throughout to distinguish research funding at universities--the main concern of NSF’s original congressional mandate--from a state’s entire R&D funding, which includes industry, government laboratories, and other activities well beyond NSF’s mandate or reach. Wherever “R&D” is used, the intention is to limit its coverage to academic R&D.

2. What Is the EPSCoR Evaluation?

The EPSCoR evaluation--conducted by COSMOS 5 Corporation--was designed with two objectives in mind:

  1. to determine whether EPSCoR had an influence on reducing the pre-existing geographical concentration of federal R&D funds--that is, whether participating states and universities had increased their share of federal academic research funds over a 14-year period (1980-1994) and, if so,
  2. to identify the EPSCoR program strategies responsible for improving state government and university competitiveness in acquiring the additional research support.

The Evaluation Team

The design and conduct of the evaluation were carried out by a COSMOS research team in consultation with a variety of experts. First, the team convened an EPSCoR Workgroup, consisting of EPSCoR leaders, to provide advice to the evaluation. Next, the team met (twice) with an expert group of advisors, who discussed the design and preliminary findings. Finally, the team commissioned two special papers by two additional experts (Gumport,1996; and Hackett, 1996) to serve as background to the entire evaluation design. 2

The Evaluation Design

EPSCoR had deliberately selected the states with the lowest R&D activity in the United States. Consequently, a control group design could not be implemented by the evaluation. An alternative design was to test a causal model of EPSCoR using a theory-based model. The theory-based approach could not yield the same degree of certainty as a control group design. However, the extent to which the model could be supported by empirical evidence and alternative explanations ruled out, the more confidence the NSF, Congress, and other stakeholders would have in the model’s specifications regarding claims for EPSCoR’s effects on R&D funding outcomes.

The EPSCoR model investigated by COSMOS was specified as a series of hypothesized causal links depicted in Exhibit 1, which presupposes the following critical steps:

  • The EPSCoR-funded research projects should be at a level verging on national competitiveness but not yet nationally competitive (left part of Exhibit 1);
  • EPSCoR’s funds also should lead to improvements in university research infrastructures and university-state government relationships in support of R&D (middle part of Exhibit 1);
  • The funded research projects, together with improvements in university research infrastructures and state capabilities, should lead to more scientifically competitive research (right part of Exhibit 1); and
  • The increased research competitiveness should lead to an increased share of R&D funding for the state (right part of Exhibit 1).
Four evaluation questions

The evaluation sought the answers to four main questions, reflecting the presumed conditions in Exhibit 1:

  1. Did EPSCoR target research investigators and groups that were nearly, but not yet, nationally competitive? Projects funded by EPSCoR should neither have been already nationally competitive (and thus able to compete without EPSCoR’s “stimulation”) nor significantly below national peer review norms for acceptable research. Therefore, the evaluation reviewed funded proposals to confirm that these proposals fell within the “good” to “very good” range of NSF’s standard scoring system for peer review--but neither above nor below this range.

  2. Did EPSCoR’s funded projects influence changes in university research infrastructures and university-state government relationships? In site visits to five EPSCoR states, the evaluation examined the organizational structures and policies of EPSCoR-supported universities for systemic changes that could be attributed to EPSCoR. Fourteen of 16 universities in the five states were visited in 1995 and 1996, during which time the evaluation team collected data from interviews and observations and reviewed documentary and archival evidence.

    The evaluation also examined the EPSCoR states’ financial and administrative policies toward university-sponsored research. They tried to determine, in particular, 1) the role of a state’s EPSCoR steering committee in policy changes (if any); 2) the effect of NSF’s matching requirement on participation; and 3) the influence of inter-institutional relationships among participating universities. The site visits, therefore, included collecting data (through interviews and reviews of documentary evidence) about the state university, state government, and industry in the five sampled states.

  3. Did the quality of academic research performed by EPSCoR recipients become more scientifically competitive? To gauge whether the quality of academic research had improved in EPSCoR states, thus becoming more competitive, the evaluation used two measures:
    • Acceptance of papers on the EPSCoR-funded research by academic publications; and
    • The award of follow-on external funding to EPSCoR-funded researchers.
    The evaluation reviewed NSF monitoring report data on research productivity for 86 EPSCoR research clusters funded between 1992 and 1996. To the extent that EPSCoR-supported research projects could be associated with subsequent academic publications and external funding, it might plausibly be argued that EPSCoR was producing more competitive scientists--who would be better able to compete for merit-based federal research dollars.

  4. Was there an increase in the EPSCoR states’ share of funded research? The evaluation charted changes in the proportion of federal research funds received by EPSCoR versus non-EPSCoR states over four points in time: 1980, 1985, 1990, and 1994. These changes were taken to be the main indicator of the EPSCoR program’s effects. The evaluation also looked at the EPSCoR states’ share of NSF funding for the same period to track any concomitant changes.

To address these questions, the evaluation team collected data by using a variety of procedures. First, evidence about R&D funding, derived from NSF’s ongoing science resource studies, was analyzed. Second, information about EPSCoR’s funding practices was based on an analysis of NSF’s records of its individual research awards and interviews with former and current NSF program officers. Third, the evaluation team made multiple site visits, over a nine-month period, to five representative EPSCoR states (and 14 campuses in those states), to collect data about EPSCoR funded research projects, state S&T developments, and university research policies and practices. Many individuals were interviewed and numerous documents collected during these site visits. Finally, information about EPSCoR-related scientific publications and subsequent funding was derived from COSMOS’s monitoring work for the EPSCoR program conducted from 1994 to 1996, in which all 19 states were the subjects of site visits and in which principal investigators were asked to submit copies of their publications for the team’s review.

The evaluation’s findings and conclusions responding to the four evaluation questions are presented in Section B. The results are organized around three major themes:

  1. EPSCoR funding in practice (Question 1);
  2. Changes in R&D funding (Question 4); and
  3. EPSCoR’s influence on university infrastructure, university-government-industry relationships, and university competitiveness (Questions 2 and 3).

In Section C, the evaluation presents the implications of the findings for EPSCoR policy and program operations. First, however, the main evaluation conclusions (based on the data in Section B) are previewed below.

____________________
2 Copies of these papers are available from COSMOS Corporation, 3 Bethesda Metro Center, Suite 950, Bethesda, MD 20814, phone 301-215-9100.

 

3. What Were the Evaluation’s Main Conclusions? (A Preview) The EPSCoR program’s objective was to reduce the geographic concentration of federal R&D funds in the United States and its territories. The most direct measure of this desired impact was a comparison of changes in the share of federal R&D expenditures (and NSF obligations) across states between 1980 and 1994. For EPSCoR to demonstrate a positive impact, 1) the EPSCoR states’ share of R&D funding would have had to increase relative to the non-EPSCoR states’ share, and 2) EPSCoR’s program strategies would have had to show plausible influence in producing the observed increase in R&D funding share.

EPSCoR States Increased Their Share of Federal R&D Funding

EPSCoR states’ share did increase.

From 1980 to 1994, EPSCoR states’ share of federal R&D funding increased from .25 percent to .40 percent per state, or from $10.1 million to $50.5 million, per state. (The “per state” assessment is given because the program continually added newly eligible states during this period of time; overall, the EPSCoR states’ share of federal R&D funding represented 7.65 percent or $960 million by 1994). EPSCoR states’ shares of NSF funding showed a similar pattern of increase. Although the increases were small in absolute terms, the increases represent a successful outcome for the EPSCoR program (during the same period, the non-EPSCoR states’ shares decreased).3

Some of EPSCoR’s Program Strategies Appear to Have Been Responsible for Increasing Federal R&D Funding in EPSCoR States

EPSCoR’s program strategies did influence some of the results.

Within each EPSCoR state, the program required the formation of a statewide steering committee, to represent the R&D interests of key organizations and sectors of the state (EPSCoR Program Solicitation, 1989a). The committee assumed a pre-review function over the component research projects that formed a state’s overall proposal to NSF. Thus, the steering committee engaged in a peer review process that involved out-of-state experts. In many states, such innovation in itself created a new S&T environment and a foundation for increased R&D competitiveness.

EPSCoR Influenced the States’ S&T Environments through the Steering Committees and the State Funding Match Requirement

The EPSCoR steering committees and the state funding match requirement promoted dialogues, planning, and new S&T initiatives. In particular, the steering committees opened communications between state university research officials and their counterparts in business, industry, and state government. This led to the identification of state research priorities that factored in the combined resources of a state’s university system with the unique opportunities offered by a state’s natural or institutional environment. EPSCoR’s one-to-one dollar match requirement helped to focus the states’ R&D priorities and to foster collaboration among universities and between universities and state government and industry. State governments’ willingness or ability to provide the required matching funds, however, has been a recurring problem in some states.

The EPSCoR steering committees usually served as the initial opportunity for research collaboration among a state’s universities and between the universities and state government agencies and industry. These collaborations were institutionalized in many instances through the development of formal consortium relationships, state S&T plans, new S&T agencies, and state science advisory councils.

EPSCoR Enhanced Some Aspects of Universities’ Orientation to Research

Compared to its effects on the states’ S&T environments, EPSCoR’s influence on university policies or resources devoted to research was not significant. However, the EPSCoR awards did stimulate inter-university collaboration (in some cases overcoming strong, traditional rivalries), heighten university-industry research collaboration, install more rigorous standards of peer review, and create new interdisciplinary research facilities. EPSCoR also enabled state universities to offer larger and competitive startup packages to attract talented new faculty, which tended to increase the research orientation of state university faculties in the sciences.

EPSCoR-supported research also showed evidence of scientific productivity and, hence, competitiveness. This was evident from the number of subsequent academic publications and external funding awards associated with the EPSCoR-supported research projects.

____________________
3During the entire period from 1978 to 1996, NSF’s cumulative investment in the
EPSCoR program amounted to $182.2 million, or less than 1 percent of the NSF budget during this period; other non-NSF EPSCoR programs had just started around 1994 and had not made significant investments during the 1978-1994 period.