Consistency of NFR Reporting across Time

Given improvements to the data collection protocol in 2010, considerable increase in NFR counts was expected between 2009 and 2010 followed by stability after 2010. Although this pattern was observed at the aggregate level, the NFR data from 2010 to 2012 showed some inconsistencies in reporting NFR counts at the unit level.

Tables 1 and 2 show the level of consistency of NFR counts across all schools and all units for three time periods: between 2010 and 2011, between 2011 and 2012, and between 2010 and 2012. At the school level, 91% of schools reported similar NFR counts in 2010 and 2011, 94% in 2011 and 2012, and 89% in 2010 and 2012 (table 1). A school's NFR count was considered to be similar if the difference in the number of NFRs was within one standard deviation of the mean difference observed from 2010 to 2012, or plus or minus 22 NFRs. The difference in counts columns show the aggregate change in NFR counts across all schools included in that category.

TABLE 1. Distribution of schools, by difference in NFR counts between 2010 and 2012

NFR = nonfaculty researcher.

SOURCE: National Science Foundation, National Center for Science and Engineering Statistics, Survey of Graduate Students and Postdoctorates in Science and Engineering.

Table 1 Source Data: Excel file

School type 2010–11 2011–12 2010–12
Number of schools Difference in counts Number of schools Difference in counts Number of schools Difference in counts
Total schools 698 153 698 410 698 563
Schools with 200 or more NFRs over previous year 2 479 1 224 3 708
Schools with 100 to 199 more NFRs over previous year 4 574 1 170 6 812
Schools with 50 to 99 more NFRs over previous year 6 447 7 435 10 655
Schools with 23 to 49 more NFRs over previous year 14 465 21 607 20 643
Schools with the same number or within 22 NFRs compared with previous year 638 -132 655 -136 622 -156
Schools with 23 to 49 fewer NFRs over previous year 25 -813 6 -211 20 -631
Schools with 50 to 99 fewer NFRs over previous year 6 -341 4 -319 13 -783
Schools with 100 to 199 fewer NFRs over previous year 2 -317 3 -360 2 -270
Schools with 200 or fewer NFRs over previous year 1 -209 0 0 2 -415

Table 2 provides similar information on the consistency of NFR reporting at the unit level and shows even greater stability over time: 95% of all units reported similar NFR counts in 2010 and 2011, 96% in 2011 and 2012, and 93% in 2010 and 2012. At the unit level, the NFR count was considered to be similar between 2 years if the number of NFRs was within plus or minus 3 NFRs (i.e., within one standard deviation of the mean difference observed from 2010 to 2012).

TABLE 2. Distribution of units, by difference in NFR counts between 2010 and 2012

NFR = nonfaculty researcher.

NOTE: Unit consists of departments, degree-granting programs, research centers, and health facilities at the institutions.

SOURCE: National Science Foundation, National Center for Science and Engineering Statistics, Survey of Graduate Students and Postdoctorates in Science and Engineering.

Table 2 Source Data: Excel file

Unit type 2010–11 2011–12 2010–12
Number
of units
Difference
in counts
Number
of units
Difference
in counts
Number
of units
Difference
in counts
Total units 15,294 153 15,294 410 15,294 563
Units with 20 or more NFRs over previous year 32 1,169 13 355 35 1,185
Units with 10 to 19 more NFRs over previous year 64 827 62 832 107 1,422
Units with 4 to 9 more NFRs over previous year 264 1,455 269 1,452 413 2,332
Units with the same number or within 3 NFRs compared with previous year 14,556 -63 14,677 26 14,238 -31
Units with 4 to 9 fewer NFRs over previous year 284 -1,581 216 -1,159 376 -2,062
Units with 10 to 19 fewer NFRs over previous year 72 -924 40 -519 91 -1,212
Units with 20 or fewer NFRs over previous year 22 -730 17 -577 34 -1,071

Overall, the reported counts of NFRs from year to year are reasonably stable, with a few exceptions (as noted at the extremes in these tables). In addition, the counts appear to be getting more stable as schools gain experience in collecting and reporting these counts and with the appointment of separate respondents who are better able to provide more accurate postdoc and NFR data.

Despite the relative stability of overall counts, a recurring problem for some respondents has been the inability to gather and report all requested NFR data. The school coordinator or unit respondent may know that the unit has NFRs but may be unable to provide any information (i.e., all data are unknown) or may be able to provide only partial information (e.g., total counts but not counts by sex or doctoral degree type). Table 3 provides a unit-level analysis of the extent of missing data within the NFR items from 2010 to 2012.

TABLE 3. Unit reporting status of NFR data: 2010–12

NFR = nonfaculty researcher.

a Includes units reporting valid zero NFRs.

NOTE: Unit consists of departments, degree-granting programs, research centers, and health facilities at the institutions.

SOURCE: National Science Foundation, National Center for Science and Engineering Statistics, Survey of Graduate Students and Postdoctorates in Science and Engineering.

Table 3 Source Data: Excel file

Reporting status Including units with no NFRs Excluding units with no NFRs
Number Percent Number Percent
All units 15,294 100.0 6,478 100.0
NFR data complete in all yearsa 11,858 77.5 3,042 47.0
NFR totals provided all years, but some data missing in 1 or more years 1,182 7.7 1,182 18.2
NFR total count missing in 1 or more years 1,908 12.5 1,908 29.5
NFR total count missing in all years 346 2.3 346 5.3

Including units that reported having zero NFRs for three years from 2010 to 2012, 78% of the units in the 2010–12 GSS were able to provide complete NFR data across all years; however, only 47% of the 6,478 units that reported having NFRs at least once from 2010 to 2012 were able to provide complete NFR data across all years. An additional 18% of units with NFRs were able to provide total counts in all three years but were unable to provide one or more detailed counts from 2010 to 2012, and 29% were missing the total count in at least 1 year. Finally, 5% (346 units) indicated that they had NFRs but were unable to provide any data from 2010 to 2012.

Comparison of Postdoc and NFR Counts

Table 4 provides a comparison of postdoc and NFR reporting at the school level from 2007 to 2012. Despite a reduction in the overall number of GSS eligible schools, the number and proportion of schools reporting NFRs increased substantially as expected. Although there was only a relatively modest increase in schools reporting NFRs in 2008, there was a much larger spike in 2010, coinciding with the expansion of the GSS postdoc and NFR items and the addition of the postdoc coordinators at many schools. Although the number of schools reporting NFRs declined in 2011, the 2011 and 2012 reporting levels remain substantially higher than those in 2009 and prior years.

TABLE 4. School reporting status of postdocs and NFRs: 2007–12

- = data not collected prior to 2010; na = not applicable.

NFR = nonfaculty researcher.

SOURCE: National Science Foundation, National Center for Science and Engineering Statistics, Survey of Graduate Students and Postdoctorates in Science and Engineering.

Table 4 Source Data: Excel file

School reporting status Number of schools Reported 2012 counts
2007 2008 2009 2010 2011 2012 NFRs Postdocs
Total 700 708 701 690 686 684 19,639 60,452
Reported NFR and postdoc counts 240 258 255 296 279 281 19,484 44,892
Number of postdocs > number of NFRs 212 217 217 236 225 231 14,950 42,031
Number of postdocs = number of NFRs 6 24 13 17 6 11 211 211
Number of postdocs < number of NFRs 22 17 25 43 48 39 4,323 2,650
Reported NFR counts only 7 6 3 19 7 6 155 0
Reported postdoc counts only 122 113 134 107 121 122 0 15,560
Reported that they had postdocs and/or NFRs, but could not provide counts - - - 16 15 12 0 0
Reported no postdocs and no NFRs 315 323 305 247 257 261 0 0
Nonresponding school 16 8 4 5 7 2 0 0
Percent reporting NFRs 35.3 37.3 36.8 45.7 41.7 42.0 na na

Some respondents conveyed their difficulty in distinguishing differences between postdocs and NFRs. This difficulty resulted in schools not reporting counts for one or the other position or in their reporting the same individuals in both counts. Although the majority of these issues were usually resolved during data collection (see Data Review discussion below), some could not be resolved due to an inability to gather the requested data. As shown in table 4, more than 10 schools in each of the past 3 years did not report any counts on postdocs or NFRs, but indicated that they had postdocs or NFRs. (A check box to indicate the unit had postdocs and NFRs but could not report any data about them was added in 2010.) Similarly, more than 100 schools (15%–19% of schools) reported postdocs but no NFRs in each of the past 6 years. In 2012, 27 of these schools reported more than 100 postdocs, and 15,560 postdocs were reported in the 122 schools with postdocs and no NFRs. These schools are likely to have NFRs but are either not reporting them or are including them in their postdoc counts.

The opposite—reporting NFRs and no postdocs—is also potentially problematic, but the associated error would be quite limited. Only 6 schools reported having NFRs and no postdocs in 2012, and only 155 NFRs were reported from these schools. Schools reporting identical counts of NFRs and postdocs are also of concern, but inaccuracies are similarly limited by the total numbers of NFRs and postdocs in these schools. Of the 11 schools reporting equal numbers of each position in 2012, 8 schools have very low counts (6 or fewer) and the remaining 3 schools reporting equal numbers of postdocs and NFRs of 25 or more.

Lastly, there were 39 schools reporting more NFRs than postdocs in 2012. Although some schools might legitimately have more NFRs than postdocs, this is the exception rather than the rule. All these schools will be followed up with in the next GSS cycle to verify the postdoc and NFR counts.