Survey Overview (FY 2012 and FY 2013 Survey of State Government Research and Development)

Purpose. The Survey of State Government Research and Development is the only source for comprehensive uniform statistics regarding the extent of R&D activity performed and funded by departments and agencies in each of the nation's 50 state governments, the government of the District of Columbia, and the government of Puerto Rico.

Data collection authority. The information is solicited under the authority of the National Science Foundation Act of 1950, as amended.

Survey contractor. U.S. Census Bureau, under National Science Foundation (NSF) interagency agreement number NCSE-1347875, collected, processed, and tabulated the statistics in this report.

Survey sponsor. NSF, National Center for Science and Engineering Statistics.

Top of page. Back to Top

Key Survey Information

Frequency. Biennial.

Initial survey year. 2006.

Reference period. State governments' fiscal years ending in 2012 and 2013.

Response unit. State government departments, agencies, commissions, public authorities, institutions, and other entities that operate separately or somewhat autonomously from the central state government with the capacity or authority to perform or fund R&D.

Sample or census. Census.

Population size. 366 agencies from the 50 state governments, the District of Columbia, and Puerto Rico with the capacity or authority to perform or fund R&D during fiscal years 2012 and 2013.

Sample size. Not applicable.

Top of page. Back to Top

Survey Design

Target population. State government departments, agencies, commissions, public authorities, institutions, and other entities that operate separately or somewhat autonomously from the central state government but where the state government maintains administrative or fiscal control over their activities, as defined by the U.S. Census Bureau Government Finance and Employment Classification Manual (see chapter 1). Several industry-specific state commissions, which are generally chartered by state legislatures but are administered independently, are considered state agencies and included in the survey's population. State-run colleges and universities, which are canvassed as part of NSF's Higher Education Research and Development (HERD) Survey, are excluded from the survey frame. State-run laboratories or experiment stations controlled by state universities are also excluded from the respondent universe, as are any entities determined to be nonprofit or private, as defined by the U.S. Census Bureau government classification criteria. However, because agricultural experiment stations in Connecticut are legally organized as a state government–dependent agency and are not affiliated with any university system, they are included in the survey's population.

Sample frame. All state government–dependent units, including those for the District of Columbia and Puerto Rico, with the capacity or authority to perform and fund R&D, as identified by a state survey coordinator who is appointed by the governor of each state.

Sample design. For the FY 2012 and FY 2013 survey, state coordinators were provided with the list of agencies that remained in the survey after the FY 2010 and FY 2011 survey, and coordinators were encouraged to add agencies that they believed were involved with R&D and were not already on the list of preselected agencies. A total of 309 agencies were preselected for the FY 2012 and FY 2013 survey from the previous survey universe by the state coordinators. State coordinators added 57 agencies to the respondent universe for a total agency respondent universe of 366. State coordinators also made adjustments to the agency universe to remove agencies that have never had any qualifying R&D to report to NSF and to address organizational changes within their respective states since the previous survey.

Top of page. Back to Top

Data Collection and Processing Methods

Data collection. On January 13, 2014, a letter was sent to the office of each state governor, as well as to the governor of Puerto Rico and the mayor of the District of Columbia, asking them to appoint a state coordinator who would provide updates to the universe of state government-dependent agencies that have the capacity to perform or fund R&D during FY 2012 and FY 2013. State coordinators logged into a secure website to add state-dependent agencies that might have some R&D and remove any agencies from the survey universe that no longer perform or fund R&D or have been reorganized. Once the state coordinators completed updates to the list of active agencies to be surveyed, they then initiated introductory e-mails to the agency respondents to log in and complete the survey. State government agency respondents would log in to a secure, Web-based survey instrument and complete the questionnaire. Upon completion by all agencies, the state coordinators were prompted to review the agency responses and the state-aggregated responses and officially lock the survey results before they were provided to NSF for final analysis and dissemination.

Mode. Self-administered online questionnaire (Web) for both the central state coordinator and the state agency respondents.

Response rates. Two response rates were calculated for the FY 2012 and FY 2013 Survey of State Government R&D: one for official data verification at the state level, and one for agency-level respondents. Table A-1 presents final agency response rates and also counts of agencies that identified themselves as having qualifying R&D expenditures on the FY 2012 and FY 2013 survey.

TABLE A-1. Final response rates and agencies reporting R&D to the survey: FY 2012 and FY 2013
SOURCE: National Science Foundation, National Center for Science and Engineering Statistics, Survey of State Government Research and Development, FYs 2012 and FY 2013.
Agencies Responses R&D reported
Agencies selected surveyed Agencies Rate (%) Agencies Responses (%)
All agencies 366 365 99.7 287 78.6
Preselected 309 308 99.7 250 81.2
Added 57 57 100.0 35 61.4

Data editing. Before officially submitting the data for their state, coordinators were asked to perform a final verification of aggregated agency data. All responses, including the initial agency data submissions and final state coordinator verifications, were received via the Web-based survey. Basic logical edit checks, review of respondent comments, and comparisons of data from previous surveys allowed Census Bureau and NSF staff to detect data errors and work with state respondents to correct them. Census Bureau and NSF staff also conducted follow-up calls to agencies with data changes of plus or minus 50% and at least $1 million in R&D between FY 2012 and FY 2013 to ensure the accuracy of the FY 2012 and FY 2013 survey data.

Imputation. Given the high agency response rate, no statistical methods were used to account for nonresponding agencies. All state and national totals are aggregates of reported agency data. Each state government's organizational structure, laws, and delegation of powers within its purview are unique. Therefore, there are no formal methods of imputation to account for these structural differences that are consistent with basic statistical methods.

Weighting. Not applicable.

Variance estimation. Not applicable.

Top of page. Back to Top

Survey Quality Measures

Sampling error. Not applicable.

Coverage error. NSF relies on the expertise of an appointed state coordinator to assist in identifying state government agencies that have the capacity or authority to perform or fund R&D. In cases where the state coordinator refuses to cooperate, it is possible that there is an undercount of state government R&D activities, despite efforts by NSF and Census Bureau staffs to conduct additional queries and conduct outreach with state agencies in lieu of an appointed state coordinator. In other instances, the appointed state coordinator could misinterpret the NSF definition and examples of qualifying R&D activities and thus fail to identify all state government–dependent units with the capacity to perform or fund R&D.

Nonresponse error. Of the 366 agencies in the survey universe, 365 (99.7%) responded to the survey. Of the 365 respondents, 287 (78.6%) reported having R&D activities in FY 2012 or FY 2013. Of the 309 agencies preselected for the survey who responded, 250 (80.9%) reported having R&D. Of the 57 respondent agencies added to the survey by state coordinators, 35 (61.4%) reported having R&D. No statistical methods were used to account for nonresponding agencies.

Measurement error. The most common form of nonsampling error may be found in the respondents' interpretation of the NSF definition of qualifying R&D activities. To mitigate any potential misinterpretations, NSF has provided a series of examples specific to the types of activities performed or funded by state government agencies in the survey questionnaire's definitions and examples. Before officially submitting data, state coordinators performed a final verification of aggregated agency data. All responses, including the initial agency data submissions and final state coordinator verifications, were received via the Web form or e-mail. Census staff performed basic logical edit checks and reviewed respondent comments, allowing staff to detect errors and work with state respondents to correct them. Despite these efforts, some of the data reported could include expenditures for non-R&D activities, such as commercialization, environmental testing, or routine survey or monitoring work.

Top of page. Back to Top

Data Comparability (Changes)

State government R&D totals can display considerable volatility between survey cycles. For example, state agency expenditures are influenced by several national and state-specific factors, and large changes (either positive or negative) are not unusual, especially for discretionary spending items such as R&D. States often will create special funds to support specific research activities for a limited time. These funds may have a one-time appropriation from the legislature and expire within 2–5 fiscal years; state agencies obligate those funds for specific R&D projects, depending on availability and expiration of funding authority as well as other program-specific and administrative considerations. Data reported are agency direct expenditures for R&D in a given fiscal year, not obligations; as such, in the case of multiyear grants to extramural performers, an agency's expenditures for that fiscal year may be greater than its obligations because expenditures may include spending from the previous year's appropriations, depending on the specific budget authority granted by the legislature. Given that state governments have comparatively little experience in tracking and measuring R&D specifically, it is likely that some portion of the reported changes reflects measurement and coverage errors. In the case of R&D funds for extramural performers, some agencies were able to report only obligations made in a given fiscal year and were not able to provide detailed expenditures for each fiscal year.

The survey asked about state agencies' expenditures for R&D as of the end of FY 2012 and of FY 2013. Most states have a fiscal-year period that begins 1 July and ends the following 30 June. For example, FY 2013 is the state fiscal period beginning on 1 July 2012 and ending on 30 June 2013. There are, however, four exceptions to the 30 June fiscal-year end: New York (ends 31 March), Texas (ends 31 August), Alabama (ends 30 September), and Michigan (ends 30 September). For comparability, these four states are surveyed with the other 46 states that end on 30 June. The District of Columbia follows the federal government fiscal year, which ends 30 September, while Puerto Rico's fiscal year begins 1 July and ends on 30 June.

A state's R&D priorities may be shaped by the state's unique legislative and budgeting processes. State budget practices vary considerably due to both political and historical reasons. As of 2010, 19 states enacted biennial budgets. Of these states, Montana, Nevada, North Dakota, and Texas have both biennial legislative sessions and biennial budgets. The remaining 15 states of Connecticut, Hawaii, Indiana, Kentucky, Maine, Minnesota, Nebraska, New Hampshire, North Carolina, Ohio, Oregon, Virginia, Washington, Wisconsin, and Wyoming hold annual legislative sessions but maintain biennial budgeting. Only North Dakota and Wyoming enact consolidated 2-year budgets; other biennial budget states enact two annual budgets at one time. As such, the nature of a state's budget priorities for R&D may be determined on a biennial basis in some states; in others, however, it may be determined on an annual basis. In states with biennial budgets, the legislatures will often make supplemental appropriations to the second-year budget, which may result in further changes to the initial funding priorities.

The data exclude R&D expenditures by state governments that did not flow through state agencies' budgets. The state totals do not include direct appropriations from state legislatures to colleges and universities. In FY 2012, universities and colleges reported expending $3.7 billion on separately budgeted R&D activities that were funded from all sources of state and local government support (see the HERD Survey). In FY 2012, state agencies reported $0.8 billion in expenditures used to support R&D performance by academic institutions. A major factor for the difference between totals reported in NSF's HERD Survey and Survey of State Government R&D is direct appropriations or grants to state-run universities that are included in the former but not in the latter. Another likely factor is the exclusion of R&D at agricultural experiment stations from the state survey totals because they are generally associated with land-grant colleges and universities and are canvassed on the HERD Survey.

Direct comparison of state agency expenditures should also be viewed with caution because state governments often reorganize departments and agencies such that some divisions and offices that were part of one agency may be moved to another agency. In other instances, entire departments may be reorganized into newly created departments. Although the FY 2012 and FY 2013 Survey of State Government R&D encountered several instances of these organizational changes in several states, the survey itself is not designed to measure specific changes in state government organization. To account for these and other changes in the data, Census Bureau and NSF staff conducted follow-up calls for agencies with data changes of plus or minus 50% and at least $1 million between FY 2012 and FY 2013 to ensure the accuracy of the FY 2012 and FY 2013 survey data.

Data specific to state government agencies were first released with the FY 2009 survey results and are also included in the FY 2012 and FY 2013 detailed statistical tables (DSTs). Since the FY 2012 and FY 2013 Survey of State Government R&D collected two fiscal years of data at once, some agencies only had qualifying R&D activities in one of those two years. In these cases, the agency is presented only in the DSTs where the agency had R&D for that specific fiscal year. Specific agency-level data for FY 2006 and FY 2007 are not available.

The current NSF Survey of State Government R&D has been conducted for FY 2006, FY 2007, FY 2009, FY 2010 and FY 2011, and FY 2012 and FY 2013. Data presented in trend tables in this report are from the most recently completed survey cycle, which canvassed two fiscal years (FY 2012 and FY 2013) as part of the same survey cycle. References to prior-year data should be restricted to those published in this report for two reasons: (1) when completing the current year's survey, survey respondents may revise their prior year's data, and (2) NSF reviews data for prior years for consistency with current-year responses and, if necessary, may revise these data in consultation with respondents.

Prior to the FY 2006 survey, NSF data collections of state government R&D were conducted by nonfederal organizations that were supported by NSF grants for state governments' FY 1995, FY 1988, and FY 1987. Prior to those efforts, NSF collected data for state governments' FY 1977, FY 1973, FY 1972, FY 1967, FY 1968, FY 1965, and FY 1964 in collaboration with the U.S. Census Bureau's Census of Governments and related programs. As a result of differences in the survey populations, definitions of covered R&D activities, and collection methods over time, the results of these historical surveys are not comparable with the statistics collected for the FY 2006 and subsequent Surveys of State Government R&D. (No survey was conducted for state governments' FY 2008.)

Changes in survey coverage and/or population. Each year, state coordinators update the universe of agencies most likely to have funded or performed R&D based on changes in funding authority, organization changes within the government, or other initiatives by the legislature. No survey was conducted for state governments' FY 2008. Beginning with the FY 2009 survey cycle, state coordinators were no longer able to overwrite the aggregate R&D data reported by state agencies to correct or modify the state total. Any changes or revisions were now required to be made at the state government agency level.

Changes in questionnaire.

Changes in reporting procedures or classification. None.

Top of page. Back to Top

Definitions

Basic research. Research conducted primarily to acquire new knowledge. For example, it can be research without a specific product or process in mind or research to produce a broad base of knowledge for future research.

Construction and acquisition of facilities used primarily for R&D. Includes major costs for construction and purchase of buildings to be used primarily as R&D facilities; also includes new construction, major renovations, and purchase of land or buildings.

Performers, extramural. Those outside the department or agency who perform R&D under the administrative oversight or control of that department or agency. This may include projects for the department or agency as well as the department's or agency's extramural research programs. Extramural performers include the following:

Performers, intramural. Department's or agency's own employees who perform R&D, which includes R&D performed by those employees and services performed by others in support of an internal R&D project (e.g., lab testing).

Research and development. Creative work conducted systematically to (1) extend scientific knowledge, or (2) devise new or improved applications, including materials, products, devices, processes, systems, or services. Sources and examples of R&D funding include the following:

Top of page. Back to Top