Section D.
Technical Notes

This section discusses the study methodology as well as various other technical aspects that the reader should consider when interpreting the data presented in this report. Where relevant, the discussion includes references to the three previous cycles of this survey. The following aspects are covered:


Definition of Terms Used in this Report

Department—a degree-granting academic unit
Facility—a non-degree granting academic unit
Faculty—includes tenured, non-tenured, teaching, and visiting faculty and researchers of faculty-equivalent rank; it does not include postdoctorates.
Maintenance/repair costs—includes maintenance agreements, service contract costs, salaries of department-provided or institution-provided maintenance/repair personnel, and costs of supplies, equipment, and facilities for servicing research instruments
Operation costs—includes salaries for technicians or other personnel paid to operate research equipment, and costs of supplies and materials used in operating the instruments
Research instrument (or research equipment)—any item (or interrelated collection of items comprising a system) of nonexpendable tangible property or software, having a useful life of more than 2 years and a cost of $500 or more, which is used wholly or in part for research
"System"—an interrelated collection of instrument items that effectively comprise one single item
Unit—denotes either a department or a facility

Universe and Samples

Institutions.—With slight additions to coverage in 1986-87, the instrumentation survey has been collected from the same panel of institutions since 1983. The first cycle of the survey was conducted in 1983-84. This baseline survey had a panel of 67 institutions: 43 colleges and universities and 24 medical schools. In the second cycle, conducted in 1986-87, the sample of colleges and universities was expanded to 55 schools, for a total of 79 in the panel of institutions. This same panel of 79 was used in both cycle III, conducted in 1989-90, and cycle IV, conducted in 1993 and 1994. The results of the 1993 survey, conducted in 1994, are presented in this report.[15]

This panel of 79 institutions was originally selected from the population of all institutions that each annually perform a minimum of $3 million in research and development. For each survey the results from the panel were generalized to the known universe of institutions that performed a minimum of $3 million in R&D in that year. For each survey this population in total accounted for more than 90 percent of the expenditures for academic R&D in science and engineering in the United States.

At the time of the 1993 survey, the latest year for which total R&D data were available was 1991; in that year there were 318 institutions that performed more than $3 million in R&D. Over the years in which the survey has been collected there has been a gradual increase in the number of institutions that perform more than $3 million in R&D. In cycle III the survey's panel of 79 institutions represented the 287 institutions that annually conducted more than $3 million in R&D. In cycle II it represented 174 institutions, and in cycle I, 155 institutions.

The panel of 79 institutions is divided into two samples:

The first sample (55 colleges and universities, excluding their medical components, if any) represents the 214 institutions that had R&D expenditures of more than $3 million in FY 1991. The probability of selection for institutions in this sample was proportionate to the total expenditures for R&D for those S&E fields included in the survey.

The second sample (24 medical schools, including medical components of colleges and universities) represents the universe of 104 medical schools that received at least $3 million in extramural awards for research from NIH in FY 1991. The probability of selection for elements in this second sample was approximately proportionate to the total amount of dollars for extramural awards given to medical institutions by NIH.

These two samples were selected independently. There is some overlap in institutional affiliation between the two samples, but no overlap in units or research instruments covered. For example, 15 of the 55 institutions selected to be in the sample of R&D colleges and universities are affiliated with a medical school that was independently chosen to be in the sample of 24 medical schools. Conversely, if an institution in the sample of 55 R&D colleges and universities had a medical school that was not independently selected to be in the sample of 24 medical schools, data for that medical school were not collected for the survey. The institutions in both of these samples are listed in appendix B.

Departments and Facilities ("Units")—Within the sampled institutions, departments and facilities were considered in-scope for the survey if they:

  1. had at least one instrument used for scientific research that had a minimum purchase price of $20,000; and

  2. were in the science and engineering fields of agriculture, biology, computer science, environmental sciences, chemistry, astronomy/physics, and engineering. (A list of the subfields included under these major fields is included in appendix C.)

The sampled institutions contained a total of 1,541 in-scope departments and facilities. From these, a sample of 996 were selected to be surveyed. (In four fields that had large numbers of departments or facilities—engineering and the agricultural, biological, and environmental sciences—a sample was selected. In the remaining fields—chemistry, computer science, and physics/astronomy—all of the eligible departments/facilities were selected.)

The survey excluded as out of scope any of the 18 university-administered federally funded research and development centers (FFRDCs), as well as any units that might be housed on a university campus but not administered by the university.

The detailed sampling plan followed for selecting the departments and facilities is available in a separate methodology report, National Survey of Academic Research Instruments and Instrumentation Needs, 1993: Methodology Report.[16]

Questionnaires

In each cycle of the survey, two types of data have been collected from two different sets of respondents:

The heads of academic departments and research facilities complete a Department/Facility Questionnaire in which they provide data for their entire units regarding expenditures for purchasing research instruments, the sources of these funds, their provisions for maintaining and repairing the instruments, and an evaluation of all their research instruments in terms of adequacy, capabilities, and needs. This report describes the findings based on this questionnaire. A copy of the questionnaire is included in appendix D.

Principal investigators complete an Instrument Data Sheet in which they provide detailed data about individual pieces of research instruments (e.g., its adequacy for research, pattern of usage, and technical capabilities). Data based on this questionnaire will be available in a forthcoming companion report, Characteristics of Science and Engineering Instrumentation in Academic Settings: 1993.

Changes in Data Collection Procedures for Cycle IV[17]

The data collection procedures used in the cycle IV 1993 survey differ from those used in earlier cycles of the instrumentation survey in several ways.
  1. Minimum Instrument Purchase Price Criterion.—To be eligible for inclusion in the three previous survey cycles, a department or facility must have had at least one research instrument with a purchase price of $10,000 or more. Similarly, only those research instruments with a purchase price of $10,000 or more were eligible for inclusion in the instrument sample in the survey. In cycle IV, the $10,000 minimum purchase price criterion was increased to $20,000 to reduce respondent burden.

    In this report, trend data were adjusted to accommodate this change in information collected. Data from the 1982-83, 1985-86, and 1988-89 surveys were standardized using the same minimum purchase price criterion of $20,000 in constant 1993 dollars, according to the GDP implicit price deflator.

  2. Survey Data Reference Periods.—Data for the three previous survey cycles were all collected over a two-year period. Half of the included fields were collected in each year, and two different types of information were collected:

    a. Data concerning expenditures were collected with the date referring to the fiscal year that preceded the period of survey collection. Engineering, chemistry, physics/astronomy and computer science expenditure data were collected to cover the years 1982, 1985, and 1988. Expenditure data for agriculture, biology, environmental sciences, and multidisciplinary areas were collected to cover the years 1983, 1986, and 1989.

    b. Data concerning equipment adequacy, needs, and priorities were collected with the date referring to the year in which the survey was collected (i.e., 1983, 1986, and 1989 for engineering, chemistry, physics/astronomy and computer science, and 1984, 1987, and 1990 for agriculture, biology, environmental sciences and multidisciplinary fields.)

    In cycle IV, the collection method was changed and data for all fields were collected during a single year, 1994. For this report, therefore, expenditure data for all fields refer to 1993, and adequacy and needs data refer to the collection year of 1994.

  3. Change in Criterion for In-Scope Departments and Facilities.—For the first two cycles of this survey, data were collected only for instruments with an original purchase price of $10,000 to $999,999. Beginning in cycle III, data were also collected for instruments with a purchase price of $1 million or more. To preserve the richness of trend data available since 1982-83 for instruments costing less than $1 million, the data in all trend tables in this report have been separated. This allows a display of the longitudinal data series since 1982 for the under $1 million instruments, and shows data on the over $1 million instruments beginning only in 1988-89. Where appropriate for analysis, selected other tables have also been similarly separated between the two price categories.

Response Rates, Estimates, and Sampling Errors

Response Rates.—Data were received from 54 of the 55 institutions in the sample of colleges and universities and from all 24 institutions in the sample of medical schools. Of the 996 science and engineering departments and facilities in the sample, 796 in-scope units responded to the survey (84.0 percent). The response rate for the questionnaire items ranged from 90.0 to 100.0 percent.

Estimates.—The findings are presented as national estimates calculated using department and facility data statistically weighted to represent all research departments and facilities in agriculture, biology, environmental sciences, chemistry, computer science, physics/astronomy, and engineering. These results from the departments and facilities at the panel of 79 institutions were generalized for the 1993 survey to the universe of 318 institutions that performed a minimum of $3 million in R&D in 1991 (the latest year for which data were available at the time of calculation). This population in total accounted for more than 90 percent of the expenditures for academic R&D in science and engineering in the United States. Over the years that the survey has been conducted there has been a gradual increase in the number of institutions that perform more than $3 million in R&D. In cycle III the panel of 79 institutions represented 287 institutions that annually conducted over $3 million in R&D. In cycle II it represented 174 institutions, and in cycle I, 155 institutions.

To ensure that the reported estimates fully represent all intended institutions and department/facilities, the final weights for these estimates are the product of the institution sampling weight (for each stratum), the department sampling weight, and the nonresponse adjustment factors for both the institution and the department or facility.

The findings from the 1993 survey were compared with those from the previous three cycles. All data are presented in current dollars, as are the majority of the percentage changes included in the text. In certain analyses, it was relevant to add constant dollar comparisons of changes. In those few instances, which are specifically noted in the text, the dollar amounts were adjusted for inflation using the GDP implicit price deflator and a base year of 1987.

Sampling Errors.—The estimates presented in this report are based on samples and are subject to variability due to sampling error. Most overall estimates (not broken down by field) have sampling errors (coefficients of variation) that range from 4 to 10 percent. This implies a 95-percent confidence interval of twice that magnitude, i.e., that the true value would be found within plus or minus 8 to 20 percent of the reported estimate. Estimates for the detail data (i.e., estimates by field of science) have sampling errors two to three times larger than those for all fields combined.[18]


Footnotes

[15] A major change was made to the methodology of the 1992 survey conducted in 1993. To relieve respondent burden, only the Department/Facility Questionnaire was used to gather instrumentation expenditures and needs, and the Instrument Data Sheet was not fielded in that year. Because of the major differences between that survey and all the others in the survey series, the 1992 survey results are not included in the trend data in this report. Other changes to the methodology in the cycle IV survey are described below in these technical notes.
[16] To obtain a copy of this report, contact Carolyn Arena, National Science Foundation (703-306-1774 or via e-mail at carena@nsf.gov).
[17] A detailed analysis of these changes and their effects on data in the survey is also included in the same separate methodology report, National Survey of Academic Research Instruments and Instrumentation Needs, 1993: Methodology Report.
[18] For example, the estimated total annual expenditures for the purchase of academic scientific research instrumentation in the biological sciences were $283 million in 1993. Assuming a sampling error of 10 percent, there is a 95-percent chance that the true amount of expenditures for research instrumentation will be found within the interval of $226 million to $340 million.