Overview  Survey Design  Survey Quality Measures  Trend Data  Availability of Data

1. Overview (2012 survey cycle) Top of Page.

a. Purpose

The 2012 Higher Education Research and Development (HERD) Survey collected information from 907 research-performing academic institutions. The survey is the primary source of information on R&D expenditures at higher education institutions in the United States and outlying areas. The survey’s predecessor, the Survey of R&D Expenditures at Universities and Colleges, was conducted annually from FY 1972 through FY 2009 and collected information on R&D expenditures by academic field as well as by source of funds. The survey was revised in 2010 and renamed the Higher Education Research and Development Survey. Although the HERD Survey continues to capture core information on R&D expenditures by source of funding and field to retain the historical time series, it has been expanded to include more detail. The survey was revised to include R&D within non–science and engineering (non-S&E) fields in the totals reported throughout the survey. Previously, non-S&E R&D was reported but not included in the overall totals. The scope of R&D was also broadened to include expenditures on clinical trials and research training grants.

b. Respondents

The HERD Survey is a voluntary establishment survey completed by designated contacts at U.S. universities and colleges. The majority of respondents for higher education institutions work in one of the following institutional offices: accounting, grants and contracts, controller, financial, institutional research, or sponsored programs.

c. Key variables

The HERD Survey requests data from institutions on their R&D in the following categories. R&D is defined as creative work conducted systematically to increase the stock of knowledge (research) and to use this knowledge to devise new applications (development).

2. Survey Design Top of Page.

a. Target population and sample frame

Since FY 2010, the target population for the HERD Survey has included nonprofit postsecondary institutions with bachelor’s or higher degree programs in any field and R&D expenditures of at least $150,000 in any field.

Before FY 2010, the population included only institutions with R&D spending and degree programs in S&E fields. Institutions that performed R&D in only non-S&E fields were excluded from the population. Several other minor changes to the target population and sampling frame were made prior to FY 2004. A complete listing of these changes can be found within the methodology report, available on request from the project officer (see section 5c, below).

Also beginning with FY 2010, each campus headed by a campus-level president, chancellor, or equivalent now completes a separate survey rather than combining its response with other campuses in a university system. As a result of this step, the overall number of academic institutions in the population increased from 711 in FY 2009 to 742 in FY 2010.

The starting frame for the FY 2012 HERD Survey was the list of institutions considered in scope for the FY 2011 survey. Institutions that granted a bachelor’s degree or higher and reported greater than $0 for research on the Integrated Postsecondary Education Data System 2010 Finance Survey were included in a population review to determine eligibility. All U.S. service institutions that granted a bachelor’s degree or higher and were not already part of the HERD Survey population were also included in the population review. Institutions in the population review were sent a brief questionnaire asking if the institution had R&D expenditures during FY 2011 and if those expenditures were less than $150,000, $150,000 to $999,999, or $1,000,000 or more.

The FY 2012 survey population consisted of 938 institutions. In order to reduce burden for institutions with minimal amounts of R&D expenditures, the National Science Foundation (NSF) introduced a shorter version of the HERD Survey for the FY 2012 collection. The short form included only a few core questions and was sent to the 282 institutions that reported R&D expenditures below $1 million during FY 2011. The remainder of the institutions (656) continued to receive the full version of the survey. At the conclusion of data collection, 907 institutions remained eligible for inclusion in the FY 2012 survey population.

b. Sample design

The FY 1997 survey was the last one conducted as a sample survey. Since FY 1998, the survey has been a census of all known eligible universities and colleges.

c. Data collection techniques

The FY 2012 survey was conducted by ICF International under contract to the National Center for Science and Engineering Statistics (NCSES). Respondents could choose to respond to the survey through printing an Adobe PDF questionnaire from the Web and submitting a paper survey, uploading an Excel version of the questionnaire, or using the Web-based data collection system.

Respondents were sent survey information and a user ID and password via e-mail. The Web-based system contained instructions and built-in help designed to replicate the printed forms. Each institution’s Web survey included comparable data from the 2 preceding years for arithmetical and trend check purposes. Large increases or decreases in expenditures compared with the preceding years were flagged, and respondents were asked to explain and reconcile these differences.

Data collection began in November 2012, with the official survey due date announced as 31 January 2013. Because the desired response rate had not been reached by the survey deadline, a series of reminder e-mail messages and phone calls were used to encourage response from all institutions between February and May of 2013, with priority placed on eliciting responses from the previous year’s top 100 institutions in total R&D expenditures. The survey was closed for new submissions by 14 June 2013, and data verification and follow-up took place in May and June of 2013.

d. Estimation techniques

This survey is a census. Imputation was performed for nonresponding institutions in order to make population estimates (see section 3c for more information). Estimates are computed by summing reported and imputed values for institutions with the appropriate characteristics.

3. Survey Quality Measures Top of Page.

a. Sampling variability

Because the FY 2012 survey was a survey distributed to all institutions in the universe, there was no sampling error.

b. Coverage

Coverage of the large research institutions is excellent, because they are easily identified using the NSF Survey of Federal Science and Engineering Support to Universities, Colleges, and Nonprofit Institutions. However, institutions with smaller amounts of R&D expenditures have been more difficult to find as they often do not receive federal funding for science and engineering R&D.

As part of the expanded HERD Survey collection, a special effort was undertaken in 2010 to screen 1,715 4-year and higher postsecondary institutions not currently included in the survey to identify additional institutions meeting the $150,000 threshold. These institutions were given detailed instructions regarding what to include as R&D expenditures and asked to respond to Question 1 of the HERD Survey, total R&D expenditures by source of funding. This HERD Short Form achieved a 64.4% response rate and identified 187 additional institutions over the $150,000 threshold. The total R&D expenditures reported by these institutions in FY 2010 was $249 million, or 0.42% of the $61.2 billion reported by the FY 2010 HERD Survey respondents. More details on this effort are included in the methodology report and technical notes for the FY 2010 survey.

c. Nonresponse

(1) Unit nonresponse—A total of 42 universities and colleges did not respond in FY 2012 out of a total of 907 eligible institutions, for a nonresponse rate of 4.6%.

Information for nonresponding institutions was imputed. Imputation was performed using prior years’ figures derived from the data of respondent institutions with similar characteristics, including highest degree granted and type of institutional control (public or private). Imputed values account for 0.2% of the total R&D expenditures included in the survey. When an institution failed to report in a given year but reported in the next year, the data for the missing year were re-imputed to reflect the new information if it appeared likely that the new information would have a significant impact on the original estimate.

(2) Item nonresponse—The item nonresponse rates varied from 0% for Question 1, total R&D expenditures by source of funds, to 18.3% for Question 6, R&D expenditures by character of work (basic research, applied research, and development). The other question with a nonresponse rate over 10% was Question 17, headcount of R&D postdocs (11.8%).

Missing information for nonresponse was imputed for all questions except expenditures funded by the American Recovery and Reinvestment Act of 2009 (Question 11). Imputations were based on average increases, decreases, or proportions derived from data of respondent institutions with similar characteristics, including highest degree granted and type of institutional control (public or private), and total R&D expenditures.

d. Measurement

The most likely source of measurement error is institutional records containing categories different from those requested on the survey. For example, institutions are asked to report all R&D expenditures by field. The NSF-designed fields do not always translate to an institution’s departmental structure, and adjustments must be made by the institution in order to complete the survey. A crosswalk between the NSF fields of R&D and the National Center for Education Statistics Classification of Instructional Programs (CIP) codes is provided with the survey in order to mitigate this source of measurement error.

Another example is NSF’s category of institutionally financed research. The survey requests that the institutions report discretionary internal funds used for research. NSF discovered through debriefings conducted at the conclusion of the FY 2010 survey that there are varying definitions of what is included as institutionally funded research on the HERD Survey. Some institutions include all expenditures from separate accounts designated for research, whereas others include only internal R&D projects that are competitively awarded and have detailed budgets. A workshop was held in the summer of 2012 in order to discuss these differences in definitional interpretations. Based on the findings from the workshop, the FY 2012 survey questionnaire was modified to clarify that all expenditures designated for research can be included within this category. A checklist question was also added to encourage inclusion of all eligible expenditures and to determine the full extent of the variation in reporting across institutions.

The reporting of unrecovered indirect costs is another known source of measurement error. The survey requests that the total amount of indirect costs associated with a research grant or contract be calculated and reported, including costs that were not reimbursed by the external funding source. Calculating the unrecovered indirect cost is done by multiplying the institution’s negotiated indirect cost rate by the corresponding base and then subtracting the actual indirect cost recovery, preferably on a project-by-project basis. In FY 2012, 8.8% of respondents reported that these data were unavailable.

4. Trend Data Top of Page.

Annual data are available for FY 1972–2012, with the exception of FY 1978. That year’s survey covered a different population and used different questions than preceding or subsequent surveys and is therefore not comparable to other years. When the review for consistency between each year’s data and submissions in prior years reveals discrepancies, it is sometimes necessary to modify prior years’ data. This is especially likely to affect trends for certain institutions that fail to report every year, because current-year data are used to impute prior-year data. Individuals wishing to analyze trends other than those published in NCSES’s most recent publication are encouraged to contact the project officer listed in section 5c for more information about comparability of data over time.

5. Availability of Data Top of Page.

a. Publications

The data from this survey are published annually, both in a summary InfoBrief and in full as detailed statistical tables in the series Higher Education Research and Development. Data are available by fiscal year on the NCSES website at http://www.nsf.gov/statistics/herd/. Data for major data elements are available starting in 1972. Information from the survey is also included in Science and Engineering Indicators, National Patterns of Research and Development Resources, Science and Engineering State Profiles, and Academic Institution Profiles.

b. Electronic access

All publications related to this survey are available on the NCSES website and in the NCSES Integrated Science and Engineering Resources Data System, WebCASPAR. Selected aggregate data are provided in public use data files upon request.

c. Contact for more information

Additional information about this survey may be obtained by contacting:

Ronda Britt
Research and Development Statistics Program
National Center for Science and Engineering Statistics
National Science Foundation
4201 Wilson Boulevard, Suite 965
Arlington, VA 22230
Phone: (703) 292-7765
E-mail: rbritt@nsf.gov