This working paper examines the consistency of doctorate-holding nonfaculty researcher (NFR) reporting in the Survey of Graduate Students and Postdoctorates in Science and Engineering (GSS) since 2010, analyzes key reporting patterns, and attempts to validate the GSS data to better understand the overall quality of the NFR data. The accurate reporting of NFRs is of critical interest to the National Science Foundation and other stakeholders because it is the first step in understanding the use and impact of this increasingly significant part of the academic science and engineering workforce.

After growing by an annual average of 7% for the prior 2 decades, the count of NFRs reported in the GSS jumped by 28% in 2008 and by 52% in 2010. These increases were likely driven by methodological changes to the GSS focused on improving the reporting of postdoctoral appointees (postdocs) and NFRs (see http://www.nsf.gov/statistics/infbrief/nsf13334/ for a summary of the impact of these changes on the postdoc data).

The methodological changes in 2010 included: a continued focus from 2008 to include centers and other non-degree-granting units; survey redesign to expand and separate the NFR items from the postdoc items; the designation of separate respondents for the graduate student and postdoc or NFR sections; and, a reminder to respondents of the importance of the postdoc and NFR data.

After implementing these changes, the number of schools and units reporting NFRs increased dramatically. Between 2009 and 2010, the percentage of GSS organizational units (academic departments, programs, research centers, or health care facilities) reporting NFRs increased from 19% to 26%, and the number of schools reporting NFRs increased from 38% to 48%. These large increases validated long-standing concerns that NFRs were being undercounted due to the lack of a common definition across institutions and limited access to these data among GSS respondents. Within the GSS, NFRs are defined as doctorate-holding researchers who are neither postdocs nor faculty members. The magnitude of the increases, however, raised the concern that some of the GSS institutions might have overreported their NFR counts.

From 2010 to 2012, NFR counts for most units (93%) have stabilized; 95% of all units had similar NFR counts in 2010 and 2011, and 96% had similar counts in 2011 and 2012. However, it is clear that some respondents have difficulty reporting NFRs. Of the units that reported having at least one NFR from 2010–12, 5% were unable to provide any data about them, and only half were able to provide complete responses to the NFR questions (see Appendix A for the NFR questions). Similarly, 122 of the 684 schools in the 2012 GSS reported having postdocs but no NFRs; this is fairly unlikely, and further follow-up with these schools is warranted to identify the cause of this discrepancy, the availability of relevant data at these institutions, and determine whether there are issues with the NFR definition.

In a short debriefing survey conducted following the 2012 data collection, 26% of respondents indicated that their institution had a common definition for NFRs. In addition, among the 32 responding schools that reported zero NFRs in 2012, the majority were unable to provide NFR counts because their institutional data management systems did not have the necessary information to identify the NFRs.

Finally, GSS counts of postdocs and NFRs were compared to the Higher Education Research and Development Survey data on funding, salaries, and postdocs. This comparison showed that research expenditures tracked closely with the number of NFRs reported in the GSS, supporting the data collected in both surveys, and identified several institutions needing follow-up for the potential over- and underreporting of NFRs in the GSS.

Based on these analyses, the 2010–12 NFR counts are much more reliable and accurate in gauging the size and distribution of this population across the GSS academic institutions than prior estimates. However, NCSES plans to continue working with those schools identified as having potential NFR data issues to improve their reporting practices and to ensure continuous quality improvement in the reporting of NFR data.