nsf.gov - NCSES U.S. Academic Scientific Publishing - US National Science Foundation (NSF)
text-only page produced automatically by LIFT Text
Transcoder Skip all navigation and go to page contentSkip top navigation and go to directorate navigationSkip top navigation and go to page navigation
National Science Foundation National Center for Science and Engineering Statistics
U.S. Academic Scientific Publishing

1.0 Executive Summary

 

1.1 Introduction

The publication of research results in peer-reviewed scientific journals is a key output of scientific research. In an unexpected development in the early 1990s, the absolute number of science and engineering (S&E) articles published by U. S. based scientists in the world's major peer reviewed journals plateaued while resource inputs—funds and personnel—kept increasing (figure 1Figure.).[1]

The unprecedented plateau in the number of U.S. S&E articles should not be confused with a decades-long and familiar decline in the U.S. share of the world's S&E articles. As other states built up their S&E capabilities, the U.S. share of the world's articles in natural sciences and engineering dropped from 38% in 1973 to 28% in 2003. In an earlier report, "Changing U.S. Output of Scientific Articles: 1988–2003," NSF described the dimensions of this "plateauing" and compared trends in the United States with those in other countries and regions that are major producers of scientific articles.

Within the U.S. scientific community, the academic sector is critical to the overall health of the nation's research system. University-based scientists generate the most publications and, arguably, conduct much of the most important and innovative research (figure 2Figure.). In the course of this work, they train generations of new researchers, and they help to attract and retain talented scientists and engineers from around the world.

The implications of trends in article production and citation frequency are open to debate. Although publications and citations are indicators of productivity and influence, they are far from perfect indicators. The trends are worthy of attention because they indicate a marked shift from a historical pattern. Although the data might be taken to suggest that U.S. productivity and influence are declining relative to those of other major research-producing countries and regions, other interpretations are also reasonable.

However, the purpose of the analysis in this report is to examine the patterns and trends in the U.S. university sector in greater depth, not to make a judgment about its interpretation or significance. Specifically, this report addresses scientific publication trends in the top 200 U.S. academic R&D institutions, as measured by their 1988–2001 research expenditures, since these institutions produce most article output from the academic sector (figure 2Figure.). Such concentration of publications is not surprising because research is central to the overall mission of the top 200 R&D performing academic institutions. Many of these institutions achieve or aspire to worldwide recognition as research leaders.

Top of page. Back to Top

1.2 Research Objectives

The present report examines quantifiable relationships among publications, resource inputs and institutional characteristics in the top 200 R&D performing academic institutions. It addresses four principal research questions.

  • How do various resource inputs (e.g., R&D funding, science and engineering graduate students, Ph.D. recipients, and postdocs) and institutional characteristics (e.g., quality, institutional control, and patenting activity) relate to article production?
  • How do the relationships among resource inputs, institutional characteristics, and article production vary over time and across different parts of the academic sector?
  • Within institutions, how do changes over time in these resource inputs and institutional characteristics relate to changes in article production?
  • How do the variables related to an institution's article production differ for different fields of science?

In addressing these questions, our fundamental framework is that R&D funds, R&D personnel, and research infrastructure are instrumental in generating publications in S&E fields. In developing this framework, we are limited by the quantified measures that are available in our database. Additional unavailable measures, such as reliable institution-by-institution counts of S&E research faculty,[2] would allow for substantially different analyses, especially regarding the association between article production (see Exhibit 1) and particular kinds of resource inputs or individual institutional characteristics. 

At the end of this executive summary, we provide a list of limitations that should be borne in mind in interpreting the results of this study.

Exhibit 1. Article Counts

To understand the analyses we report, it is important to know how we count the number of science and engineering (S&E) articles that an institution produces.[3]

Comparisons over time are best made by examining articles in the population of influential journals.[4] The journals in this group change over time as new journals may emerge and attain influence, while a few older journals may decline or cease to exist. The overall direction of change is towards more articles and journals, as the worldwide research community continues to grow. Hence, analyses in this report were primarily conducted using an expanding set of journals covered by the Thomson ISI Science and Social Science Citation Indexes at any given point in time.

However, changes over time in journal coverage can inflate article counts and alter the national or field coverage of the journal set for reasons that have little or nothing to do with scientific influence. To examine whether there were changes that may have occurred for such extraneous reasons, parallel analyses were conducted using a fixed set of those journals that Thomson ISI had indexed throughout the study period.

Institutional and national patterns of change using data from the fixed and expanding journal sets are very similar.

In this study, credit for publications is assigned to institutions, not to individual authors, using two methods: whole and fractional counting. In whole counting, each institution that appears in the author list receives one credit for an article. In fractional counting, when more than one (institutional) author is involved, credit for the article is divided equally among the listed institutions. 

Publications as measured by whole counts are useful indicators of how often an institution is involved in producing articles. Publications as measured by fractional counts are useful in highlighting patterns and trends in the shares of credit attributable to different institutions. While neither method adequately captures the many factors that affect how the research community allocates credit for articles, taken together, they provide different perspectives on recent trends in the production of science and engineering articles in the top 200 R&D performing academic institutions.



Top of page. Back to Top

1.3 Resource Inputs, Institutional Characteristics, and Article Production

Variations among universities in research resource inputs (for which quantified data are available) largely account for differences in these universities' publication outputs. The primary measured resource inputs associated with publication counts across institutions are total academic R&D expenditures, the number of S&E postdoctorates, and (to a lesser extent) the number of S&E doctoral recipients (which may serve as a surrogate measure for the number of doctoral candidates available to assist in research). These three resource inputs account for over 90% of the variability in publication outputs, leaving little residual variability to be explained by other potential factors.

Analyses of the four different publication measures, defined by fractional or whole counting and a fixed or expanding journal set, yield very similar results (table 1Excel table.). Likewise, analyses of the corresponding four different measures of citations yield results that corresponded closely to those of the publication measures. In short, essentially the same explanatory variables turned out to be important for each of these measures.

A difference between institutions of $1M a year in total academic R&D funding is associated with a difference of 4.8 whole count publications in the expanding journal set (table 1Excel table.).  A difference of one postdoctoral student is associated with 1.7 additional publications, and a difference of one S&E doctoral recipient is associated with 1.5 additional publications.[5]  (Since the number of S&E doctoral recipients is highly correlated with faculty counts, these publications may reflect the efforts of faculty who taught these doctoral recipients). 

Within the category of total academic R&D expenditures, funds obtained from the federal government and other institutions such as foundations appear to be somewhat more influential than funds from state and local governments or private enterprise.[6]

Examination of a number of other factors that might influence publication counts failed to markedly improve the results of the analysis: 

  • Faculty counts were not found to be influential in improving the fit of regressions modeling publication or citation outcomes. The reason for this may be the deficiencies in the various measures available.
  • Accounting for differences in the degree of collaboration among institutions did not substantially affect the results.   
  • Some researchers have suggested that leading private research universities follow research and publication patterns that differ from those found elsewhere in the U.S. university system. However, these institutions behave much like the others with respect to the variables that were examined.
  • Institutions with relatively more citations to their published articles tend to have slightly more publications than expected on the basis of resource inputs, but the relationship was weak.
  • Once research funds and personnel are taken into account, higher National Research Council (NRC) quality ratings are not associated with a higher number of publications. This may reflect the presence of many different journals that accept publications at varying quality levels. Higher NRC ratings are strongly correlated with a higher ratio of citations to publications, which may be a measure of the relative influence of publications.
  • There is no convincing evidence that patents are substituting for publications. In fact, moderate amounts of patenting may be slightly associated with increased publication counts. While the three institutions with the largest amounts of patenting do appear to have reduced publication counts, there are too few institutions to draw a conclusion about the role of patenting in this result.[7]

These results suggest that once we take into account total academic expenditures, the number of S&E postdoctorates, and the number of S&E doctoral recipients, other factors not captured by these variables do not seem to matter much for article production.[8]  These intangibles include university prestige, unmeasured infrastructure, and unmeasured student or faculty quality. There are a number of possible explanations for this finding:

  • Institutional prestige and quality may affect publication output largely by enabling a university to attract S&E postdoctorates and funding necessary to conduct research.
  • Prestige and reputation may enable a university to attract a "better" faculty member (in the sense that a committee of experts in the individual's area of expertise would conclude that the quality of the individual's research is superior), without necessarily translating into more publication counts (since there are many different journals that accept publications at varying quality levels).
  • Finally, university prestige and quality may enable recruitment of "better" faculty who publish more often and are also able to generate more quantified inputs (e.g., successful grant applications that yield R&D funds). Since we have no direct measures of faculty quality, publication counts may instead appear to be linked to things we do measure (e.g. academic R&D, number of S&E doctoral recipients) that may in turn reflect such quality differences.

Top of page. Back to Top

1.4 Variation Over Time and Among Kinds of Institutions

The primary inputs for producing publications, R&D expenditures and personnel, have generally increased faster than the number of publications. For example, the amount of inflation adjusted R&D expenditures per fractional count publication increased by 29% between 1990 and 2001[9] (figure 3Figure.).

This pattern of increasing amounts of inputs required to yield the same publication outputs occurred across the entire U.S. academic system. It did not appear to vary meaningfully across different kinds of institutions, and it cannot be explained by factors specific to certain regions, quality levels, or other segments of the system. Possible reasons for the increasing inputs per article include a rise in complexity of research required for publication; costs for faculty, S&E postdoctorates, S&E doctoral recipients, and research materials and equipment that are increasing faster than the GDP implicit price deflator; increased communication costs for integrative collaborations,[10] etc.[11]

A previous SRS report, "Perceptions of Academic Researchers and Administrators," considers these and other factors that might bear on changing article output. The report, based on qualitative data from interviews and focus groups, summarizes the views of experienced observers and practitioners in research universities about how the worlds of academic science and engineering research and publication have been changing over the past 15 years.

Top of page. Back to Top

1.5 Relationship Between Changes in Institutional Characteristics and Changes in Article Production

Changes over time in resource levels available to a given institution appear to affect publication output by a smaller amount than would have been expected based upon the relationship between publication output and resource inputs across institutions (table 2Excel table.).

  • If university A has 10% more academic R&D expenditures, S&E postdoctorates, and S&E doctoral recipients than the average institution, it will have about 9% more whole count and fractional count publications than the average institution.
  • However, if university A starts with average resources (i.e. average academic R&D expenditures, S&E postdoctorate counts and S&E doctoral recipients) and then these resources are increased by 10%, its publications will increase much less: about 5% (by whole counts) and 2% (by fractional counts).

These smaller than expected increases in publications output may reflect that the three key inputs, academic R&D expenditures, S&E postdoctorates and S&E doctoral recipients, are themselves in part surrogates for other, slower or unchanging university characteristics that in turn are related to publications volume. Alternatively, it may suggest that publications exhibit some degree of inertia to "short term" changes (which may span years). 

While the relative level of publications of two or more institutions can be fairly reliably estimated based on the levels of these three inputs, there is considerable uncertainty concerning the change in fractional count publications that results from a change in resources for individual universities. This implies that a substantial portion of year-to-year changes in institutional outputs may be affected by unmeasured variations in inputs.

The analysis was nevertheless able to account for a substantial portion (about two-thirds) of the within-institution variability in publications over time, as measured by whole counts in the expanding journal set. Results suggest that changes in federally financed academic R&D expenditures have almost three times the impact on publications measured in this way as non-federally financed academic R&D expenditures. 

In addition, the type of postdoctoral student funded has a differential impact on changes in such publication counts, with S&E postdoctorates without M.D.s increasing publication counts, S&E postdoctorates with M.D.s funded by federal traineeships (which often emphasize clinical training) decreasing publication counts (presumably by redirecting resources from research to non-research oriented activities), and S&E postdoctorates supported by federal research grants  having little effect on changes in publication counts.

Top of page. Back to Top

1.6 Variations in Relationships For Different Fields of Science

The three publication input variables that best explained institutional-level publications output—R&D expenditures, S&E postdoctorates, and Ph.D. recipients—also account for about 90% of the explanatory ability within each of five different fields of science (medical sciences; computer sciences; psychology and social sciences; biology, life and agricultural sciences; and engineering, math and physical sciences, including chemistry, geosciences and astronomy).

A few additional variables slightly improved this model's fit in the five individual fields of science. Those variables included Carnegie R-1 classification[12] (which was associated with increased publication counts), basic research expenditures by institution (which may reflect the orientation of the institution and was associated with increased publication counts), and the number of S&E postdoctorates without M.D. degrees (which may reflect lesser number of publications per S&E postdoctorate with an M.D. degree, with the possible exception of postdoctorates with M.D. degrees who are supported by federal research grants). The precise effects of the explanatory variables differed by field; for example, S&E postdoctorates play a more prominent role in research in some fields than in others and tend to be clustered in some fields. 

However, dividing article outputs by field of science, using field-specific input data, and then summing across fields did little to improve this model's fit. It also does not appear that institutional variations in output are substantially related to gross features of disciplinary concentration.

The trends in inflation-adjusted R&D expenditures per publication are not the same across the fields of science (figure 4Figure.). Increases in R&D expenditures per publication were greatest for medical sciences and the field consisting of biology, life and agricultural sciences (greater than 30%) and intermediate for the field consisting of psychology and social sciences (24%). R&D expenditures per publication in the field encompassing engineering, math and physical sciences increased 15% from 1990 to 1997 and then reversed itself, resulting in a net increase of 9% from 1990 to 2001. R&D expenditures used per publication in computer sciences followed an erratic pathway, resulting in an increase of 12% over this time period.

Top of page. Back to Top

1.7 Study Limitations

Results from this study should be interpreted in light of the following limitations:

  1. Because the study is observational rather than experimental, any relationships that are observed are associations and do not imply causal links of inputs and observed outputs.[13]
  2. Many of the potential explanatory variables are highly correlated. Therefore, even if one of a collection of potentially explanatory variables were a true cause of publications output, it is difficult to determine which of these variables it is.
  3. The classification of resources (personnel and R&D funding) and publications into fields may result in some inconsistencies, since different sources of data classify resources and publications into fields using different schema and criteria. In some instances, the personnel, funding, and publication counts associated with a particular article might be allocated to different fields.
  4. Many potentially useful variables were not available. For example, we do not have counts of all faculty with S&E degrees by field, or even by institution, nor do we have estimates of the proportion of time spent by faculty in research or teaching duties.
  5. There is limited variability in resource trends over time. Most resources and outputs increased over time in a fairly linear fashion. If resources and publications had varied cyclically instead of trending in one direction, the reliability of our inferences would be higher.
  6. All journals in the fixed or expanding journal set are treated as being of equal importance. Similarly, all publications are treated as equal in the sense that a "breakthrough" article would contribute no more to publication counts than any other article.
  7. The dependent variable throughout much of the analysis is publications; manuscript submissions may also be relevant, but were unavailable. U.S. researchers may be submitting as many or more manuscripts, but finding that their acceptance rates, in the aggregate, have been falling. Variations between submission and acceptance rates may vary across institutions and/or fields.
  8. The analysis is restricted to factors within the U.S. academic sector. An analysis based on a larger framework (i.e., international competitive markets) may have yielded additional insights.
  9. Information about patents was limited to the patent count by institution and year. We were not able to classify patents according to the context of research production (such as whether the patent was speculative patenting of what seemed like an exploitable idea, was obtained as the basis of a spin-off launch, or was obtained as a result of a strategic research collaboration between a university and private sector company).

Top of page. Back to Top



Footnotes

[1] During this same period, publication output in the rest of the world increased, but a comparison of U.S. and non-U.S. publication trends is outside the scope of this report.

[2] National estimates of S&E academic researchers are based on a sample survey of S&E doctorate holders that cannot reliably be aggregated to produce institutional counts, lacks information about those with non-U.S. degrees, and provides no information about researchers who do not have a Ph.D. A reliable count of total faculty at each institution was available and was used in our analyses, although that measure also had many limitations.

[3] S&E fields consist of the life sciences, natural and physical sciences, engineering and technology, mathematics, psychology and social sciences and related fields.

[4] All publications and citations data derive from the Science and Social Science Citation Indexes maintained by Thomson ISI, with special tabulations provided to NSF by ipIQ, Inc.

[5] Our model associates an additional $1M in academic R&D expenditures with an additional 2.5 S&E postdoctorates and 1.4 S&E Ph.D. recipients. These additional personnel are associated with 6.2 more whole count publications. Consequently, $1M in additional R&D expenditures results in 11.0 additional whole count publications if combining the direct (+4.8 publications) and indirect effect (+6.2 publications).

[6] Funds from state and local government and industry might support research with a more applied component and a lower expectation of publication.

[7] These three include the entire University of California system, which does not report patents for individual campuses, thus further weakening the evidence.

[8] But the link between NRC quality ratings and citations relative to publications suggests an additional dimension of quality or influence of publications outputs.

[9] A regression modeling the relationship between the number of fractional count publications (based on inflation adjusted academic R&D funding, number of S&E postdoctorates, and number of S&E doctoral recipients) assuming a constant relationship between resources and publication production will underestimate the observed number of publications produced in 1990 and will overestimate the observed number of publications in 2001. From the amount of under- or overestimation, it can be calculated that the same resources that produced 100 publications in 2001 would have produced 129 publications in 1990.

[10] The number of institutional authors per U.S. S&E article increased from 1.78 in 1988 to 2.44 in 2003. During this time period the number of author (person) names per U.S. S&E article increased from 2.98 to 4.42.

[11] The increase in resources used per publication does not appear to be related to a change in the proportion of articles appearing in higher quality journals. The NSF study classifies a journal as being among the top 5% in terms of citations in 1992 if the articles that appeared in that journal in 1988 through 1990 had more citations in 1992 than articles published in 95% of other journals that appear in the expanding journal set. For the top 5% cited journals in 1992 NSF has calculated that 61.0% of the articles written in those journals in 1988 through 1990 have U.S. authors. In contrast, the U.S. share of the world's articles in the top 5% cited journals in 2003 decreased to 50.5%. The proportions in 1992 and 2003 are almost identical using the fixed journal set. During these years the U.S. share of all S&E publications decreased from 37.8% to 31.1%. Thus, the U.S. share of both total articles and articles appearing in the most highly cited journals decreased by approximately the same percentage (i.e., 17%–18%).

[12] Based on the 1994 Carnegie Classification of Institutions, these universities have a full range of baccalaureate programs and a commitment to graduate education through the doctorate, and they award at least 50 doctoral degrees a year and annually receive $40 million or more in Federal research funding.

[13] For example, suppose that there is an unmeasured underlying factor that causes a university to be both effective in generating publications and obtaining academic R&D funding. In this case, the academic R&D funding itself does not cause the publications to increase, and an experiment that doubles R&D funding without changing this underlying factor might be relatively unsuccessful in increasing publications.


 
U.S. Academic Scientific Publishing
Working Paper | SRS 11-201 | November 2010