State Agency Research and Development Expenditures: Fiscal Year 2007
Appendix A. Technical Notes
The National Science Foundation (NSF) Survey of State R&D Expenditures measures the extent of research and development activity performed and funded by each of the nation's 50 states, the District of Columbia (DC), and Puerto Rico. (The District of Columbia and Puerto Rico are referred to as states in this report and are reported as "state equivalents" in the data tables.) Results of the survey are useful to a variety of data users interested in R&D performance, including the National Science Board; the Office of Management and Budget; the Office of Science and Technology Policy and other science policy makers; institutional researchers; and private organizations.
Scope of Survey
The survey covered state government departments, agencies, independent commissions, and other entities determined to be state-run. It excluded state-run colleges and universities, which are canvassed as part of the NSF Survey of R&D Expenditures at Colleges and Universities. It also excluded entities determined to be nonprofit or private, as defined by Census Bureau government classification. Although the universe of interest covered all state-run entities except state-run colleges and universities, evidence showed that some state agencies were not involved in R&D activity and did not need to be surveyed.
Using input from potential state coordinators, agency respondents, and R&D experts, survey staff developed a Web-based survey instrument for both the central state contacts and the state agency respondents. The survey was launched in May 2008 with a letter to governors asking for each state's participation.
Working with Census staff, governors' offices identified a central coordinator in each state to assist in collecting data from state agencies. For the FY 2006 state agency survey, a list of 330 preselected agencies was compiled, which included state agencies that were to be surveyed with certainty. The list was developed based on feedback during exploratory interviews, a review of responses from previous state R&D surveys, and extensive reviews of state government Web sites. State coordinators were then encouraged to add agencies not on the preselected list that they believed were also involved in R&D. State coordinators added 103 agencies, for a total FY 2006 agency population of 433.
For the FY 2007 survey, it was decided that all 433 agencies would be included in the survey population; however two duplicate agencies were removed from the population, resulting in a total FY 2007 preselected agency count of 431. Further, because the state coordinator for Maine opted to submit aggregate totals for Maine's 10 preselected agencies (as it had done for the FY 2006 survey), those agencies were removed from the list, leaving 421 state agencies preselected to receive the FY 2007 survey. State coordinators subsequently added 33 agencies to the respondent universe, for a total FY 2007 agency respondent universe of 454.
Before officially submitting the data for their state, coordinators performed a final verification of aggregated agency data. All responses, including the initial agency data submissions and final state coordinator verifications, were received via the Web form or e-mail. Basic logical edit checks, review of respondent comments, and comparisons of data from previous surveys allowed staff to detect and work with state respondents to correct data errors.
The respondent questionnaire consisted of one screening and four R&D expenditures data questions. The four R&D expenditures questions contained a total of 10 items.
Screening Question: The purpose of this question was to reduce the burden on agency respondents who did not have qualifying R&D expenditures during FY 2007 and to further clarify the scope of the survey for those agencies with R&D expenditures data to report. Respondents who answered "No" to the screening question were not required to complete the remaining questions.
Question 1: The first seven data items were grouped as one question on the survey form. The purpose of this question was to allow the respondent to report all R&D expenditures for their agency, with the exception of R&D plant expenditures. The first three items consisted of "Internal Performer" expenditures, by source. The second three items asked the respondent to report expenditures according to three types of "External Performers." The final data item, the agency's "Total R&D Expenditures," was calculated automatically as the respondent entered data.
Question 2: This question asked respondents to provide the total amount of basic research expenditures for their agency. Respondents were provided with the "Total R&D Expenditures" amount calculated in Question 1.
Question 3: The next data item was also a subset of the "Total R&D Expenditures" amount calculated in Question 1. This question requested total R&D expenditures from federal sources, regardless of whether the R&D was performed internally or externally.
Question 4: The final data item requested that respondents report R&D plant expenditures for FY 2007. This item was requested separately to avoid skewing the "Total R&D Expenditures" amount with large, one-time agency expenditures for construction projects or the purchase of land or buildings.
The instrument also allowed respondents to provide comments for each question.
The following definitions were provided to survey respondents:
Research and development is creative work conducted systematically to (1) extend scientific knowledge, or (2) devise new or improved applications, including materials, products, devices, processes, systems, or services.
Basic research is conducted primarily to acquire new knowledge without any specific product or process in mind.
Construction and acquisition of facilities used primarily for R&D includes major costs for new construction, major renovations, and purchase of land or buildings to be primarily used as R&D facilities.Sources and examples of R&D funding include the following:
Internal performers performers are the department/agency's own employees who perform R&D and includes R&D performed by those employees and services performed by others in support of an internal R&D project (e.g., lab testing).
External performers are those outside the department/agency who perform R&D under the administrative oversight or control of that department/agency. This may include projects for the department/agency, as well as its extramural research programs. External performers include the following:
Two response rates were calculated for the FY 2007 Survey of State R&D Expenditures, one for agency level responders and one for official data verification at the state level. All states participated in the survey; however, only 46 out of 52 (88.5%) officially verified final state data. The six states that did not officially verify data had some or all agencies submit data. One state, Maine, did not submit individual agency data, but submitted state totals via the state coordinator. For this reason, the 10 preselected agencies in Maine were not counted in the final response rates. The final agency response was 444 out of 454 (97.8%). Table A-1 displays final agency response rates, as well as counts of agencies that identified themselves as having qualifying R&D expenditures for FY 2007.
National Totals and Imputations
Statistical methods were not used to account for nonresponding agencies. All state and national totals are aggregates of reported agency data or amounts revised at the state level by state coordinators.
State coordinators in Arizona, California, Iowa, Louisiana, Puerto Rico, and Tennessee did not verify the aggregated agency data. At least one agency in each of the following states did not respond to the survey; California (1), Indiana (2), New Mexico (2), Puerto Rico (3), Washington (1), and West Virginia (1).
Although every effort was made to exclude out-of-scope amounts, it is likely that some reported data include expenditures for non-R&D activities, such as commercialization, environmental testing, or routine survey work. Some state data may also exclude minor R&D expenditure amounts from agencies not surveyed.
The current NSF Survey of State R&D Expenditures has been conducted for the past 2 years and collected FY 2006 and FY 2007 data. The previous NSF Survey of State R&D Expenditures collected FY 1995 data. Prior to that effort, surveys were conducted for FY 1987 and FY 1988, as well as two efforts in the 1970s. Because of differences in the survey populations, definition of covered R&D activities, and collection methods, the results of those surveys are not comparable with the statistics collected on the FY 2006 and FY 2007 Survey of State R&D Expenditures.
State R&D totals display considerable volatility between FY 2006 and FY 2007. Although total state agency R&D expenditures increased by 20% between these 2 years, 28 states (and D.C. and Puerto Rico) reported overall increases, and 22 states reported overall declines. Eight states (Florida, Indiana, New Jersey, North Carolina, Ohio, Rhode Island, West Virginia, and Wyoming) reported combined agency R&D increases of more than 100% from FY 2006 to FY 2007; five states (Georgia, Iowa, Minnesota, New Mexico, and North Dakota) reported combined agency R&D decreases of 50% or more.
State agency expenditures are influenced by a number of national and state-specific factors, and large changes (positive or negative) are not unusual, especially for discretionary spending items, such as R&D. However, given that states have comparatively little experience in tracking and measuring R&D, there is a high likelihood that some portion of the reported changes reflects measurement and coverage errors. State coordinators may not know which agencies fund R&D, and state agency respondents may not consistently report R&D activities. Coverage error would seem to be one of the more important factors accounting for several of the large year-to-year differences. Of the eight states reporting at least twice the amount of R&D in FY 2007 than in FY 2006, four of them (Florida, Indiana, New Jersey and Rhode Island) included agencies with large R&D expenditures in FY 2007 that had not been surveyed in FY 2006.
The data reported here focus exclusively on R&D expenditures of state departments, agencies, commissions, and dependent entities, with the exception of state-run colleges and universities. Universities, colleges, or other higher education entities surveyed under the NSF Survey of R&D Expenditures at Universities and Colleges were out of scope, as respondents, for this effort. State-run laboratories or experiment stations controlled by state universities were excluded. Any entities determined to be nonprofit or private, as defined by Census Bureau government classification, were also excluded from the respondent universe. Several industry-specific state commissions, which are generally chartered by state legislatures but are administered independently, were considered state agencies and included in the survey.
The data exclude states' R&D expenditures that did not flow through state agencies' budgets. The state totals do not include direct appropriations from state legislatures to colleges and universities. In FY 2007 universities and colleges reported expending $3.1 billion on separately budgeted R&D activities that were funded from all sources of state and local government support. State agencies reported $0.5 billion in expenditures used to support R&D performance by academic institutions. A major factor for the difference between totals reported in the Academic and the State R&D surveys is direct appropriations/grants to state-run universities that are included in the former, but not the latter. Another likely factor is the exclusion of R&D at agricultural experiment stations from the state survey totals.
Data from this and other NSF R&D surveys are available at http://www.nsf.gov/statistics/.