Survey of State Government Research and Development: FYs 2010 and 2011
Appendix A. Technical Notes
The National Science Foundation (NSF) Survey of State Government Research and Development measures the extent of R&D activity performed and funded by each of the nation's 50 state governments, the District of Columbia, and Puerto Rico. State governments play an important role in advancing knowledge—notably, through their support of academic and other extramural researchers and also through their own internal research activities.
State R&D totals can display considerable volatility between the surveys. For example, state agency expenditures are influenced by several national and state-specific factors, and large changes (either positive or negative) are not unusual, especially for discretionary spending items such as R&D. States often will create special funds to support specific research activities for a limited time. These funds often will have a one-time appropriation from the legislature and expire within 2–5 fiscal years; state agencies obligate those funds for specific R&D projects, depending on availability and expiration of funding authority as well as other program-specific and administrative considerations.
Scope of the Survey
The survey covered state government departments, agencies, commissions, public authorities, and other entities operating separately or somewhat autonomously from the central state government but over whose activities the state government maintains administrative or fiscal control. Several industry-specific state commissions, which are generally chartered by state legislatures but administered independently, were considered state agencies and are included in the survey's population of interest.
The survey asked about state agencies' expenditures for R&D as of the end of FY 2010 and of FY 2011. Most states have a fiscal-year period that begins 1 July and ends the following 30 June. For example, FY 2011 is the state fiscal period beginning on 1 July 2010 and ending on 30 June 2011. There are, however, four exceptions to the 30 June fiscal-year end: New York (ends 31 March), Texas (ends 31 August), Alabama (ends 30 September), and Michigan (ends 30 September). For comparability, these four states are surveyed with the other 46 states that end on 30 June. The District of Columbia follows the federal government fiscal year, which ends 30 September, while Puerto Rico's fiscal year begins 1 July and ends on 30 June.
The survey excluded state-run colleges and universities, which are included as part of NSF's Higher Education Research and Development (HERD) Survey, as well as state-run laboratories or experiment stations controlled by state universities. It also excluded entities determined to be nonprofit or private, as defined by the U.S. Census Bureau. Although the population of interest covered all state-run entities except state-run colleges and universities, evidence showed that not all state agencies were involved in R&D activity and so did not need to be surveyed.
Using input from potential state coordinators, agency respondents, and R&D experts, survey staff developed a Web-based survey instrument for both the central state coordinators and the state agency respondents. The survey was launched in July 2012 with a letter from the Director of the U.S. Census Bureau to governors asking for each state's participation (see appendix B).
Working with Census Bureau staff, governors' offices identified a state coordinator in each state to assist in collecting data from state agencies. Each coordinator was given a list of agencies in his or her state and asked to add any other state agencies that the coordinator believed could have conducted R&D. The survey was then sent to all agencies with potential R&D, as identified by the coordinators. The final list of agencies surveyed consisted of agencies that reported R&D on previous survey cycles, agencies added by the state coordinator, and agencies surveyed in previous survey cycles that may not have had any R&D but continued to have the potential to perform or fund R&D. State coordinators also were able to remove an agency that was preselected if the agency's mission had changed, the agency had been reorganized, or the agency no longer performed or funded any R&D and would not in the future. Through this process, 25 agencies were removed from the survey population of interest prior to the mailout of the FY 2010 and FY 2011 survey. Of the final count of 472 agencies identified for the FY 2010 and FY 2011 survey, 413 agencies were surveyed in the FY 2009 survey cycle, and 59 agencies were added to the FY 2010 and FY 2011 survey's population of interest by the state coordinators.
Before officially submitting the data for their state, coordinators were asked to perform a final verification of aggregated agency data. All responses, including the initial agency data submissions and final state coordinator verifications, were received via the Web-based survey form or e-mail. Basic logical edit checks, review of respondent comments, and comparisons of data from previous surveys allowed staff to detect data errors and work with state respondents to correct them.
The questionnaire consisted of one screening question and five R&D expenditure data questions. The five R&D expenditure questions contained a total of 17 items for each fiscal year surveyed for a total of 34 data items on the survey.The purpose of the screening question was twofold: (1) to reduce the burden on agency respondents who did not have qualifying R&D expenditures during FY 2010 or FY 2011, and (2) to clarify the scope of the survey for those agencies with R&D expenditure data to report. Respondents who answered "No" to the screening question were not required to complete the remaining questions.
Agencies that answered "Yes" to the screening question were then asked for information about the type of performer—namely how much was expended for internal and external R&D. In the case of internal performers, respondents were asked to provide expenditures by source of funding (e.g., state funds, federal funds, other), whereas in the case of external performers, respondents were asked to identify how much money was provided to academic institutions, companies and individuals, and all other performers to conduct R&D on behalf of the agency for all types of funds (e.g., state funds, federal funds, other).
Additional questions asked respondents to identify how much of the total R&D spending (for both internal and external performers) was considered basic research and how much of their total R&D spending came from federal funds. Further questions asked respondents to identify how much was spent on several different types of research—namely, agriculture, energy, environment and natural resources, health, transportation, or other activities. Finally, state departments and agencies were asked to provide expenditures for construction and acquisition of facilities used primarily for R&D.
The instrument also allowed respondents to provide comments on each question.
Data Limitations and Comparability
State coordinators in Arizona, Indiana, Louisiana, Michigan, Mississippi, Nevada, New Mexico, Ohio, Oklahoma, Tennessee, Texas, Utah, and the District of Columbia did not verify the aggregated agency data. One agency in Louisiana and one agency in New Mexico did not respond to the survey.
Although every effort was made to exclude out-of-scope amounts, it is likely that some reported data included expenditures for non-R&D activities, such as commercialization, environmental testing, or routine monitoring work.
Given that states have comparatively little experience in tracking and measuring R&D specifically, it is likely that some portion of the reported changes reflects measurement and coverage errors. In the case of R&D funds for external performers, some agencies were able to report only obligations made in a given fiscal year and were not able to provide detailed expenditures for each fiscal year.
The current NSF Survey of State Government R&D has been conducted for FY 2006, FY 2007, FY 2009, and FY 2010 and FY 2011. Data presented in trend tables in this report are from the most recently completed survey cycle, which canvassed 2 fiscal years (FY 2010 and FY 2011) as part of the same survey cycle. References to prior-year data should be restricted to those published in this report for two reasons: (1) when completing the current-year survey, survey respondents may revise their prior-year data, and (2) NSF reviews data for prior years for consistency with current-year responses and, if necessary, may revise these data in consultation with respondents.
The previous NSF Survey of State Government R&D collected data for FY 1995 data. Prior to that effort, surveys were conducted for FY 1987 and FY 1988, as well as for FYs 1977, 1973, 1972, 1967, 1968, 1965, and 1964. As a result of differences in the survey populations, definitions of covered R&D activities, and collection methods over time, the results of these historical surveys are not comparable with the statistics collected on the FY 2006, FY 2007, FY 2009, and FY 2010 and FY 2011 Survey of State Government R&D.
Total state agency R&D expenditures increased by more than 11% between FY 2010 and FY 2011: 33 states and the District of Columbia reported overall increases, and 17 states plus Puerto Rico reported overall declines. Two states (Iowa and Tennessee) reported combined agency R&D increases of more than 100% from FY 2010 to FY 2011, while New Jersey showed the only combined agency R&D decrease of more than 30%.
Direct comparison of state agency expenditures should also be viewed with caution because states will often reorganize departments and agencies such that some divisions and offices that were part of one agency may be moved to another agency. In other instances, entire departments may be reorganized into newly created departments. Although the FY 2010 and FY 2011 Survey of State Government R&D encountered several instances of these organizational changes in several states, the survey itself is not designed to measure specific changes in state government organization. To account for these and other changes in the data, Census Bureau and NSF staff conducted follow-up calls for agencies with data changes of plus or minus 50% and at least $1 million between FY 2009 and FY 2011 to ensure the accuracy of the FY 2010 and FY 2011 survey data.
Data specific to state government agencies were first released with the FY 2009 survey results and are also included in the FY 2010 and FY 2011 detailed statistical tables (DSTs). Since the FY 2010 and FY 2011 Survey of State Government R&D collected 2 fiscal years of data at once, some agencies only had qualifying R&D activities in 1 of those 2 years. In these cases, the agency is presented only on the DSTs where the agency had R&D for that specific fiscal year. Specific agency-level data for FY 2006 and FY 2007 are not available.
The data exclude R&D expenditures of states that did not flow through state agencies' budgets. The state totals do not include direct appropriations from state legislatures to colleges and universities. In FY 2011, universities and colleges reported expending $3.8 billion on separately budgeted R&D activities that were funded from all sources of state and local government support (NSF/NCSES 2012). In FY 2011, state agencies reported $0.5 billion in expenditures used to support R&D performance by academic institutions. A major factor for the difference between totals reported in NSF's Higher Education and the State R&D surveys is direct appropriations or grants to state-run universities that are included in the former but not in the latter. Another likely factor is the exclusion of R&D at agricultural experiment stations from the state survey totals because they are generally associated with land-grant colleges and universities and are canvassed on the HERD Survey.
Two response rates were calculated for the FY 2010 and FY 2011 Survey of State Government R&D: one for official data verification at the state level, and one for agency-level responders. All 50 state governments, the District of Columbia, and Puerto Rico participated in the survey; however, 39 out of 52 (75%) state coordinators officially verified the final aggregate state data. The 12 states and the District of Columbia, which did not verify data officially, had some or all agencies submit data. The final agency response was 470 out of 472 agencies (99.6%); of those agencies that responded, 256 agencies (54.5%) reported having some R&D activity. Table A-1 presents final agency response rates and also counts of agencies that identified themselves as having qualifying R&D expenditures on the FY 2010 and FY 2011 survey.
Given the high agency response rate, no statistical methods were used to account for nonresponding agencies. All state and national totals are aggregates of reported agency data.
Basic research is conducted primarily to acquire new knowledge. For example, it can be research without a specific product or process in mind or research to produce a broad base of knowledge for future research.
Construction and acquisition of facilities used primarily for R&D includes major costs for construction and purchase of buildings to be used primarily as R&D facilities; also includes new construction, major renovations, and purchase of land or buildings.
Performers—external are those outside the department or agency who perform R&D under the administrative oversight or control of that department or agency. This may include projects for the department or agency as well as the department's or agency's extramural research programs. External performers include the following:
Performers—internal are the department's or agency's own employees who perform R&D, which includes R&D performed by those employees and services performed by others in support of an internal R&D project (e.g., lab testing).
Research and development is creative work conducted systematically to (1) extend scientific knowledge, or (2) devise new or improved applications, including materials, products, devices, processes, systems, or services.
Sources and examples of R&D funding include the following:
Data from this and other NSF R&D surveys are available at http://www.nsf.gov/statistics/.
 The District of Columbia and Puerto Rico are treated as state equivalents and referred to as states in this report.
 See chapter 1 of the U.S. Census Bureau's Government Finance and Employment Classification Manual: http://www2.census.gov/govs/pubs/classification/2006_classification_manual.pdf.
 No survey was conducted for state government in FY 2008.
 Agricultural experiment stations in Connecticut are in the population of interest for the Survey of State Government R&D because they are organized as state agencies and not affiliated with any university system.
National Science Foundation, National Center for Science and Engineering Statistics (NSF/NCSES). 2012. Universities Report Highest-Ever R&D Spending of $65 Billion in FY 2011. Detailed Statistical Tables NSF 13-325. Arlington VA. Available at http://www.nsf.gov/statistics/nsf13325/.