Debriefing Survey

To better understand the large increase in NFR counts from 2009 to 2010 and to confirm the reliability of the NFR data, the NFR Debriefing Survey was conducted following the 2012 data collection to obtain information about how schools define and identify NFRs. This Web survey was e-mailed to a purposive sample of schools based on school's reporting of NFR data. A school was selected if: NFR count increased by 15% or more between 2009 and 2010; NFR count is higher than postdoc count in 2012; or they reported no NFRs in 2012 (see Appendix B).The NFR Debriefing Survey was sent to 103 GSS schools, and 91 schools responded.

Schools indicated that they had definitional and other data issues (e.g., school or postdoc coordinator did not have access to these data, NFRs are not classified uniformly, school considered NFRs and postdocs to be faculty, school combined postdocs and NFRs for reporting). An analysis of the responses found the following:

Based on NFR Debriefing Survey responses, the schools that had specific reporting issues will be contacted to determine whether they will be able to report the NFR data correctly in the future.

Quality Measures

To assess the quality of the GSS NFR data further, the institution-level NFR counts were compared with funding and personnel data from the NSF/NCSES's Higher Education Research and Development Survey (HERD). The HERD is an annual census of U.S. colleges and universities with at least $150,000 in research and development (R&D) expenditures during the prior fiscal year. It collects information on the R&D expenditures by field of research as well as counts of R&D personnel supported by those funds. Given that the majority of R&D funds in academia are expended in science, engineering, and selected health (SEH) fields, substantial overlap exists between the HERD and GSS-eligible institutions. Therefore, the number of NFRs reported in the GSS is expected to be highly correlated with the level of R&D expenditures and counts of R&D personnel in the HERD at a given institution.

Table 5 presents the results of the institution-level match for the 2012 GSS and 2012 HERD data. A total of 454 institutions were in both the GSS and the HERD, 111 institutions were in the GSS only, and 182 institutions were in the HERD only. Institutions that are in the GSS only had less than $1,000,000 in R&D expenditures (i.e., not eligible for the full version of the HERD that collects data on R&D personnel and postdocs). Although the nonmatching institutions represent potential undercoverage for the surveys, the magnitude of this undercoverage is small, as 99% of R&D expenditures reported in the HERD were reported in institutions found in both the GSS and the HERD. Similarly, 99% of all postdocs and NFRs reported in the GSS were in the institutions found in both the GSS and the HERD.

TABLE 5. Results of GSS-HERD data match: 2012

NA = not available.

GSS = Survey of Graduate Students and Postdoctorates in Science and Engineering; HERD = Higher Education Research and Development Survey; NFR = nonfaculty researcher.

a a In 2011, the GSS began collecting data from potentially eligible new frame institutions from the Integrated Postsecondary Education Data System and other data sources. These new frame data are not included in the published GSS data while the final eligibility determination is made. These institutions are included for this comparison with the HERD data to maximize the institutional match.

SOURCES: National Science Foundation, National Center for Science and Engineering Statistics, Higher Education Research and Development Survey, and Survey of Graduate Students and Postdoctorates in Science and Engineering.

Table 5 Source Data: Excel file

GSS-HERD match In both GSS and HERD In GSS onlya In HERD only
Number of institutions 454 111 182
Total R&D funds (HERD Q1) (US$) 64,456,987 NA 1,317,537
Salaries, wages, benefits (HERD Q13a) (US$) 27,822,699 NA 518,325
HERD R&D personnel (HERD Q16) 890,117 NA 26,705
HERD postdocs (HERD Q17) 65,213 NA 1,459
GSS postdocs 62,377 474 261
GSS NFRs 21,773 135 51

Research expenditures are expected to track closely with the number of NFRs in the GSS; as an institution's funding for R&D increases, the number of NFRs in the GSS is expected to increase. A plot of research expenditures (from the HERD) and number of NFRs (from the GSS) for each institution in both the HERD and GSS confirms this expectation (figure 2). An ordinary least squares regression analysis found that 61% of the variance in NFR counts is explained by the amount of research funding; the correlation between the number of NFRs and funding is .78 (p < .0001). These findings provide substantial support for the quality of GSS NFR data.

Potential NFR reporting problems could be identified by an institution with a large R&D expenditures (in the HERD data) but few or no NFRs (in the GSS data) or, conversely, small R&D expenditures and a large number of NFRs. Further follow-up will be made in such cases in the next GSS cycle.

Data Review Results and Imputation of Missing NFR Data

Table 6 summarizes the results of the post-submission review of the NFR data from 2010 to 2012. The most common issue identified was a substantial change in total NFR counts; 404 units were reviewed in 2010, and about half (204) of these were followed up with the respondent. For the majority of these large changes, the respondent verified that the reported count was accurate. Combined with the increasing proportions of issues accepted in review or verified as correct by the respondent in 2011 and 2012, the decline in large count changes across years provides further evidence that the NFR data quality is improving and becoming more stable.

TABLE 6. Review of NFR data reported by the institution: 2010–12

NFR = nonfaculty researcher.

SOURCES: National Science Foundation, National Center for Science and Engineering Statistics, Higher Education Research and Development Survey, and Survey of Graduate Students and Postdoctorates in Science and Engineering.

Table 6 Source Data: Excel file

Type of check Data reviewed Data accepted in review Data updated in review Followed up with institution Data verified Data revised Data unresolved
2010
Total NFR count 404 170 30 204 141 37 26
NFR gender distribution 13 9 0 4 2 0 2
NFR medical degree distribution 32 19 0 13 8 0 5
Subtotal 449 198 30 221 151 37 33
Percent of all 100.0 44.1 6.7 49.2 33.6 8.2 7.3
Percent of retrieved - - - 100.0 68.3 16.7 14.9
2011
Total NFR count 301 207 0 94 51 35 8
NFR gender distribution 12 12 0 0 0 0 0
NFR degree type distribution 51 44 0 7 3 4 0
Units with same (nonzero) NFR data as prior year 83 61 0 22 18 1 3
Units with same (nonzero) NFR data in the current year 106 83 0 23 15 3 5
Other NFR comment 115 107 1 7 2 4 1
Subtotal 668 514 1 153 89 47 17
Percent of all 100.0 76.9 0.1 22.9 13.3 7.0 2.5
Percent of retrieved - - - 100.0 58.2 30.7 11.1
2012
Total NFR count 243 195 0 48 33 13 2
NFR gender distribution 2 0 0 2 1 1 0
NFR degree type distribution 26 23 0 3 1 1 1
Units with same (nonzero) NFR data as prior year 88 62 0 26 21 4 1
Units with same (nonzero) NFR data in the current year 66 44 0 22 17 3 2
Other NFR comment 151 143 3 5 4 1 0
Subtotal 576 467 3 106 77 23 6
Percent of all 100.0 81.1 0.5 18.4 13.4 4.0 1.0
Percent of retrieved - - - 100.0 72.6 21.7 5.7

Despite these improvements in reporting, respondents still had a harder time providing NFR counts than providing graduate student and postdoc counts in 2012. The imputation rates for each NFR item, shown in table 7, ranged from 7.3% to 7.6%. The NFR imputation rates are higher than the rates for other GSS data items (which are typically around 4.6%) and reflect the relative difficulty in gathering and reporting the NFR data.

TABLE 7. NFR item imputation rates: 2012

NFR = nonfaculty researcher.

SOURCES: National Science Foundation, National Center for Science and Engineering Statistics, Higher Education Research and Development Survey, and Survey of Graduate Students and Postdoctorates in Science and Engineering.

Table 7 Source Data: Excel file

Item Number imputed Imputation rate (%)
NFRs: Total 1,018 7.3
NFRs: Men 1,025 7.3
NFRs: Women 1,025 7.3
NFRs: With medical degrees (MD, DO, DDS, DVM) 1,057 7.6
NFRs: With medical degrees (MD, DO, DDS, DVM), men 1,054 7.6
NFRs: With medical degrees (MD, DO, DDS, DVM), women 1,045 7.5
NFRs: With nonmedical degree 1,057 7.6
NFRs: With nonmedical degree, men 1,059 7.6
NFRs: With nonmedical degree, women 1,050 7.5
NFRs: With dual medical/research doctoral degree 1,057 7.6
NFRs: With dual medical/research doctoral degree, men 1,054 7.6
NFRs: With dual medical/research doctoral degree, women 1,045 7.5
NFRs: With unknown degrees 1,057 7.6
NFRs: With unknown degrees, men 1,054 7.6
NFRs: With unknown degrees, women 1,045 7.5

In 2012, the total NFR count was 19,639 before imputation and 21,908 after imputation. Table 8 shows that the majority of the count changes were due to changes in reporting rather than from imputation. Moreover, the increased correlation and decreased standard deviation of the 2010 to 2012 counts by field show that imputation tends to stabilize trends by field. Table 8 also provides the reported values and the imputed values of total NFR counts by field for 2009 to 2012.

TABLE 8. Reported and imputed NFR counts, by field: 2009–12

GSS = Survey of Graduate Students and Postdoctorates in Science and Engineering; nec = not elsewhere classified; NFR = nonfaculty researcher.

SOURCE: National Science Foundation, National Center for Science and Engineering Statistics, Survey of Graduate Students and Postdoctorates in Science and Engineering.

Table 8 Source Data: Excel file

GSS code and description Reported totals Imputed totals
2009 2010 2011 2012 2009 2010 2011 2012
Total NFRs 13,807 19,095 19,372 19,639 14,059 21,345 21,498 21,908
Aerospace engineering, GSS CODE 101 40 53 32 49 40 58 35 49
Agricultural engineering, GSS CODE 102 52 64 62 63 52 70 62 65
Biomedical engineering, GSS CODE 103 150 211 198 258 153 250 247 295
Chemical engineering, GSS CODE 104 215 248 184 179 224 265 204 211
Civil engineering, GSS CODE 105 181 213 252 251 181 256 278 298
Electrical engineering, GSS CODE 106 292 366 339 362 296 395 406 405
Engineering science, GSS CODE 107 124 114 97 159 124 114 119 170
Industrial engineering, GSS CODE 108 76 87 70 56 76 108 87 70
Mechanical engineering, GSS CODE 109 245 335 282 298 246 355 318 389
Metallurgical and materials engineering, GSS CODE 110 172 214 223 236 180 224 233 245
Mining engineering, GSS CODE 111 1 7 4 10 1 7 4 10
Nuclear engineering, GSS CODE 112 28 37 40 29 28 39 44 30
Petroleum engineering, GSS CODE 113 17 23 36 40 17 23 36 40
Engineering nec, GSS CODE 114 113 160 222 196 113 227 228 209
Astronomy, GSS CODE 201 352 400 381 380 352 400 381 380
Chemistry, GSS CODE 202 753 899 898 814 766 944 997 911
Physics, GSS CODE 203 601 759 754 831 611 825 857 916
Physical sciences nec, GSS CODE 204 44 72 74 89 44 82 87 89
Atmospheric sciences, GSS CODE 301 190 250 299 306 190 250 310 319
Geosciences, GSS CODE 302 312 465 474 420 312 502 529 523
Oceanography, GSS CODE 303 178 282 390 319 178 300 405 334
Earth, atmospheric, and ocean sciences nec, GSS CODE 304 94 306 180 159 94 310 381 337
Computer sciences, GSS CODE 401 314 305 284 313 331 318 326 349
Mathematics and applied mathematics, GSS CODE 402 151 146 135 172 153 159 162 190
Statistics, GSS CODE 403 7 14 12 19 7 14 12 19
Agricultural sciences, GSS CODE 501 424 533 574 533 431 572 581 567
Anatomy, GSS CODE 601 75 92 103 95 75 96 106 101
Biochemistry, GSS CODE 602 503 573 609 670 523 619 652 708
Biology, GSS CODE 603 401 688 622 566 434 790 639 598
Biometry and epidemiology, GSS CODE 604 177 227 312 348 177 345 322 348
Biophysics, GSS CODE 605 11 44 30 32 15 53 30 32
Botany, GSS CODE 606 154 173 205 236 154 187 216 260
Cell biology, GSS CODE 607 443 539 527 598 448 611 607 627
Ecology, GSS CODE 608 66 61 67 72 66 62 67 72
Entomology and parasitology, GSS CODE 609 50 74 78 73 50 83 83 80
Genetics, GSS CODE 610 245 268 295 306 254 329 312 320
Microbiology, immunology, and virology, GSS CODE 611 555 602 703 728 562 779 800 822
Nutrition, GSS CODE 612 66 111 148 145 66 149 160 153
Pathology, GSS CODE 613 309 449 360 437 309 484 438 445
Pharmacology, GSS CODE 614 353 445 445 399 374 470 515 431
Physiology, GSS CODE 615 337 445 496 408 339 507 532 479
Zoology, GSS CODE 616 17 29 18 12 17 29 18 18
Biological sciences nec, GSS CODE 617 347 585 665 705 350 678 727 755
Anesthesiology, GSS CODE 701 69 114 115 122 69 137 140 138
Cardiology, GSS CODE 702 139 143 133 176 139 180 167 202
Oncology and cancer research, GSS CODE 703 418 429 625 631 437 465 631 665
Endocrinology, GSS CODE 704 94 98 81 99 94 106 87 110
Gastroenterology, GSS CODE 705 59 58 55 74 59 73 60 77
Hematology, GSS CODE 706 89 96 99 108 89 121 117 110
Neurology, GSS CODE 707 179 272 173 180 179 302 205 221
Obstetrics and gynecology, GSS CODE 708 48 62 64 65 48 80 80 72
Ophthalmology, GSS CODE 709 127 200 180 222 127 210 206 246
Otorhinolaryngology, GSS CODE 710 29 107 51 77 29 110 56 78
Pediatrics, GSS CODE 711 208 521 474 307 211 573 533 485
Preventive medicine and community health, GSS CODE 712 178 363 542 546 185 562 563 601
Psychiatry, GSS CODE 713 122 213 180 218 140 252 218 234
Pulmonary disease, GSS CODE 714 50 80 63 71 50 92 96 107
Radiology, GSS CODE 715 188 247 221 295 188 273 293 347
Surgery, GSS CODE 716 305 435 315 395 305 465 355 429
Clinical medicine nec, GSS CODE 717 633 903 931 921 634 1,010 1,023 952
Dental sciences, GSS CODE 718 52 113 92 66 55 116 94 85
Nursing, GSS CODE 719 27 113 79 89 31 113 79 92
Pharmaceutical sciences, GSS CODE 720 200 296 250 277 205 305 270 307
Veterinary sciences, GSS CODE 721 183 283 254 239 183 300 254 250
Other health nec, GSS CODE 722 114 256 253 290 114 307 258 302
Speech pathology and audiology, GSS CODE 723 53 36 38 33 53 36 38 37
Psychology, general, GSS CODE 801 104 205 200 211 105 220 219 230
Psychology, GSS CODE 802 154 212 174 164 154 216 194 184
Clinical psychology, GSS CODE 803 32 31 21 17 32 31 21 17
Agricultural economics, GSS CODE 901 31 34 20 19 31 34 22 21
Anthropology (cultural and social), GSS CODE 902 71 67 75 88 75 67 81 101
Economics (except agricultural), GSS CODE 903 30 38 58 59 30 44 61 62
Geography, GSS CODE 904 35 37 39 48 35 37 49 61
History and philosophy of science, GSS CODE 905 8 9 15 18 8 9 15 18
Linguistics, GSS CODE 906 10 29 11 12 10 29 13 13
Political science, GSS CODE 907 37 108 183 200 37 143 221 229
Sociology, GSS CODE 908 15 46 65 68 16 48 66 69
Sociology and anthropology, GSS CODE 909 3 0 0 1 3 0 0 1
Social sciences nec, GSS CODE 910 124 186 144 159 132 206 144 165
Family and consumer sciences and human sciences, GSS CODE 920 31 36 101 33 31 38 101 43
Communication, GSS CODE 930 9 19 15 12 9 24 17 14
Architecture, GSS CODE 940 6 12 0 6 6 15 11 11
Neuroscience, GSS CODE 950 77 180 325 322 77 191 378 356
Multidisciplinary and interdisciplinary studies, GSS CODE 980 231 440 479 330 231 467 509 497
Mean standard deviation of 2009–12 field counts - - - 42 - - - 51
Mean standard deviation of 2010–12 field counts - - - 28 - - - 22
Correlation of current and prior year field counts - 0.958 0.970 0.982 - 0.948 0.981 0.992