Contents

---

1.0 Executive Summary

2.0 Overview of Report Structure

3.0 Introduction: Study Background, Historical Trends in Article Production, Methods for Counting Publications and Citations, and Attributing Counts to Institutions and Fields

4.0 Research Approach: Scope, Data Sources, and Analysis Methods

5.0 Trends in Publications, Citations, and Resources

6.0 Variable Transformations: Inflation Adjustment and Lagging

7.0 Decomposition of Variability: Between and Within Institution Components

8.0 Key Variables Associated with Institution-Level Publication Counts

9.0 Adequacy of Model Fit: Time Trends in Resource Utilization per Publication Count and Model Fit by Institutional Characteristic

10.0 Observed and Expected Publication Trends Within Institutions

11.0 Key Factors Associated with Publication Counts at the Field Group Level

Appendix A. U.S. Top 200 R&D Performing Academic Institutions

Appendix B. Study Methodology

Appendix C. Variable Descriptions and Database References

Appendix D. Analysis of Citation Counts

Appendix E. Crosswalk of WebCaspar and ipIQ Classifications

Appendix F. Analysis of the Relative Citation Index

Appendix G. Analysis of Publication Counts in the Fixed Journal Set

Appendix H. Regression Output for Chapter 9

Appendix I. Regression Output for Chapter 10

Appendix J. Regresson Output for Chapter 11

Suggested Citation

1.0 Executive Summary

1.1 Introduction

The publication of research results in peer-reviewed scientific journals is a key output of scientific research. In an unexpected development in the early 1990s, the absolute number of science and engineering (S&E) articles published by U. S. based scientists in the world's major peer reviewed journals plateaued while resource inputs—funds and personnel—kept increasing (figure 1).[1]

The unprecedented plateau in the number of U.S. S&E articles should not be confused with a decades-long and familiar decline in the U.S. share of the world's S&E articles. As other states built up their S&E capabilities, the U.S. share of the world's articles in natural sciences and engineering dropped from 38% in 1973 to 28% in 2003. In an earlier report, "Changing U.S. Output of Scientific Articles: 1988–2003," NSF described the dimensions of this "plateauing" and compared trends in the United States with those in other countries and regions that are major producers of scientific articles.

Within the U.S. scientific community, the academic sector is critical to the overall health of the nation's research system. University-based scientists generate the most publications and, arguably, conduct much of the most important and innovative research (figure 2). In the course of this work, they train generations of new researchers, and they help to attract and retain talented scientists and engineers from around the world.

The implications of trends in article production and citation frequency are open to debate. Although publications and citations are indicators of productivity and influence, they are far from perfect indicators. The trends are worthy of attention because they indicate a marked shift from a historical pattern. Although the data might be taken to suggest that U.S. productivity and influence are declining relative to those of other major research-producing countries and regions, other interpretations are also reasonable.

However, the purpose of the analysis in this report is to examine the patterns and trends in the U.S. university sector in greater depth, not to make a judgment about its interpretation or significance. Specifically, this report addresses scientific publication trends in the top 200 U.S. academic R&D institutions, as measured by their 1988–2001 research expenditures, since these institutions produce most article output from the academic sector (figure 2). Such concentration of publications is not surprising because research is central to the overall mission of the top 200 R&D performing academic institutions. Many of these institutions achieve or aspire to worldwide recognition as research leaders.

 Back to Top

1.2 Research Objectives

The present report examines quantifiable relationships among publications, resource inputs and institutional characteristics in the top 200 R&D performing academic institutions. It addresses four principal research questions.

• How do various resource inputs (e.g., R&D funding, science and engineering graduate students, Ph.D. recipients, and postdocs) and institutional characteristics (e.g., quality, institutional control, and patenting activity) relate to article production?
• How do the relationships among resource inputs, institutional characteristics, and article production vary over time and across different parts of the academic sector?
• Within institutions, how do changes over time in these resource inputs and institutional characteristics relate to changes in article production?
• How do the variables related to an institution's article production differ for different fields of science?

In addressing these questions, our fundamental framework is that R&D funds, R&D personnel, and research infrastructure are instrumental in generating publications in S&E fields. In developing this framework, we are limited by the quantified measures that are available in our database. Additional unavailable measures, such as reliable institution-by-institution counts of S&E research faculty,[2] would allow for substantially different analyses, especially regarding the association between article production (see Exhibit 1) and particular kinds of resource inputs or individual institutional characteristics. 

At the end of this executive summary, we provide a list of limitations that should be borne in mind in interpreting the results of this study.

 Back to Top

1.3 Resource Inputs, Institutional Characteristics, and Article Production

Variations among universities in research resource inputs (for which quantified data are available) largely account for differences in these universities' publication outputs. The primary measured resource inputs associated with publication counts across institutions are total academic R&D expenditures, the number of S&E postdoctorates, and (to a lesser extent) the number of S&E doctoral recipients (which may serve as a surrogate measure for the number of doctoral candidates available to assist in research). These three resource inputs account for over 90% of the variability in publication outputs, leaving little residual variability to be explained by other potential factors.

Analyses of the four different publication measures, defined by fractional or whole counting and a fixed or expanding journal set, yield very similar results (table 1). Likewise, analyses of the corresponding four different measures of citations yield results that corresponded closely to those of the publication measures. In short, essentially the same explanatory variables turned out to be important for each of these measures.

A difference between institutions of $1M a year in total academic R&D funding is associated with a difference of 4.8 whole count publications in the expanding journal set (table 1).  A difference of one postdoctoral student is associated with 1.7 additional publications, and a difference of one S&E doctoral recipient is associated with 1.5 additional publications.[5]  (Since the number of S&E doctoral recipients is highly correlated with faculty counts, these publications may reflect the efforts of faculty who taught these doctoral recipients). 

Within the category of total academic R&D expenditures, funds obtained from the federal government and other institutions such as foundations appear to be somewhat more influential than funds from state and local governments or private enterprise.[6]

Examination of a number of other factors that might influence publication counts failed to markedly improve the results of the analysis: 

• Faculty counts were not found to be influential in improving the fit of regressions modeling publication or citation outcomes. The reason for this may be the deficiencies in the various measures available.
• Accounting for differences in the degree of collaboration among institutions did not substantially affect the results.   
• Some researchers have suggested that leading private research universities follow research and publication patterns that differ from those found elsewhere in the U.S. university system. However, these institutions behave much like the others with respect to the variables that were examined.
• Institutions with relatively more citations to their published articles tend to have slightly more publications than expected on the basis of resource inputs, but the relationship was weak.
• Once research funds and personnel are taken into account, higher National Research Council (NRC) quality ratings are not associated with a higher number of publications. This may reflect the presence of many different journals that accept publications at varying quality levels. Higher NRC ratings are strongly correlated with a higher ratio of citations to publications, which may be a measure of the relative influence of publications.
• There is no convincing evidence that patents are substituting for publications. In fact, moderate amounts of patenting may be slightly associated with increased publication counts. While the three institutions with the largest amounts of patenting do appear to have reduced publication counts, there are too few institutions to draw a conclusion about the role of patenting in this result.[7]

These results suggest that once we take into account total academic expenditures, the number of S&E postdoctorates, and the number of S&E doctoral recipients, other factors not captured by these variables do not seem to matter much for article production.[8]  These intangibles include university prestige, unmeasured infrastructure, and unmeasured student or faculty quality. There are a number of possible explanations for this finding:

• Institutional prestige and quality may affect publication output largely by enabling a university to attract S&E postdoctorates and funding necessary to conduct research.
• Prestige and reputation may enable a university to attract a "better" faculty member (in the sense that a committee of experts in the individual's area of expertise would conclude that the quality of the individual's research is superior), without necessarily translating into more publication counts (since there are many different journals that accept publications at varying quality levels).
• Finally, university prestige and quality may enable recruitment of "better" faculty who publish more often and are also able to generate more quantified inputs (e.g., successful grant applications that yield R&D funds). Since we have no direct measures of faculty quality, publication counts may instead appear to be linked to things we do measure (e.g. academic R&D, number of S&E doctoral recipients) that may in turn reflect such quality differences.

 Back to Top

1.4 Variation Over Time and Among Kinds of Institutions

The primary inputs for producing publications, R&D expenditures and personnel, have generally increased faster than the number of publications. For example, the amount of inflation adjusted R&D expenditures per fractional count publication increased by 29% between 1990 and 2001[9] (figure 3).

This pattern of increasing amounts of inputs required to yield the same publication outputs occurred across the entire U.S. academic system. It did not appear to vary meaningfully across different kinds of institutions, and it cannot be explained by factors specific to certain regions, quality levels, or other segments of the system. Possible reasons for the increasing inputs per article include a rise in complexity of research required for publication; costs for faculty, S&E postdoctorates, S&E doctoral recipients, and research materials and equipment that are increasing faster than the GDP implicit price deflator; increased communication costs for integrative collaborations,[10] etc.[11]

A previous SRS report, "Perceptions of Academic Researchers and Administrators," considers these and other factors that might bear on changing article output. The report, based on qualitative data from interviews and focus groups, summarizes the views of experienced observers and practitioners in research universities about how the worlds of academic science and engineering research and publication have been changing over the past 15 years.

 Back to Top

1.5 Relationship Between Changes in Institutional Characteristics and Changes in Article Production

Changes over time in resource levels available to a given institution appear to affect publication output by a smaller amount than would have been expected based upon the relationship between publication output and resource inputs across institutions (table 2).

• If university A has 10% more academic R&D expenditures, S&E postdoctorates, and S&E doctoral recipients than the average institution, it will have about 9% more whole count and fractional count publications than the average institution.
• However, if university A starts with average resources (i.e. average academic R&D expenditures, S&E postdoctorate counts and S&E doctoral recipients) and then these resources are increased by 10%, its publications will increase much less: about 5% (by whole counts) and 2% (by fractional counts).

These smaller than expected increases in publications output may reflect that the three key inputs, academic R&D expenditures, S&E postdoctorates and S&E doctoral recipients, are themselves in part surrogates for other, slower or unchanging university characteristics that in turn are related to publications volume. Alternatively, it may suggest that publications exhibit some degree of inertia to "short term" changes (which may span years). 

While the relative level of publications of two or more institutions can be fairly reliably estimated based on the levels of these three inputs, there is considerable uncertainty concerning the change in fractional count publications that results from a change in resources for individual universities. This implies that a substantial portion of year-to-year changes in institutional outputs may be affected by unmeasured variations in inputs.

The analysis was nevertheless able to account for a substantial portion (about two-thirds) of the within-institution variability in publications over time, as measured by whole counts in the expanding journal set. Results suggest that changes in federally financed academic R&D expenditures have almost three times the impact on publications measured in this way as non-federally financed academic R&D expenditures. 

In addition, the type of postdoctoral student funded has a differential impact on changes in such publication counts, with S&E postdoctorates without M.D.s increasing publication counts, S&E postdoctorates with M.D.s funded by federal traineeships (which often emphasize clinical training) decreasing publication counts (presumably by redirecting resources from research to non-research oriented activities), and S&E postdoctorates supported by federal research grants  having little effect on changes in publication counts.

 Back to Top

1.6 Variations in Relationships For Different Fields of Science

The three publication input variables that best explained institutional-level publications output—R&D expenditures, S&E postdoctorates, and Ph.D. recipients—also account for about 90% of the explanatory ability within each of five different fields of science (medical sciences; computer sciences; psychology and social sciences; biology, life and agricultural sciences; and engineering, math and physical sciences, including chemistry, geosciences and astronomy).

A few additional variables slightly improved this model's fit in the five individual fields of science. Those variables included Carnegie R-1 classification[12] (which was associated with increased publication counts), basic research expenditures by institution (which may reflect the orientation of the institution and was associated with increased publication counts), and the number of S&E postdoctorates without M.D. degrees (which may reflect lesser number of publications per S&E postdoctorate with an M.D. degree, with the possible exception of postdoctorates with M.D. degrees who are supported by federal research grants). The precise effects of the explanatory variables differed by field; for example, S&E postdoctorates play a more prominent role in research in some fields than in others and tend to be clustered in some fields. 

However, dividing article outputs by field of science, using field-specific input data, and then summing across fields did little to improve this model's fit. It also does not appear that institutional variations in output are substantially related to gross features of disciplinary concentration.

The trends in inflation-adjusted R&D expenditures per publication are not the same across the fields of science (figure 4). Increases in R&D expenditures per publication were greatest for medical sciences and the field consisting of biology, life and agricultural sciences (greater than 30%) and intermediate for the field consisting of psychology and social sciences (24%). R&D expenditures per publication in the field encompassing engineering, math and physical sciences increased 15% from 1990 to 1997 and then reversed itself, resulting in a net increase of 9% from 1990 to 2001. R&D expenditures used per publication in computer sciences followed an erratic pathway, resulting in an increase of 12% over this time period.

 Back to Top

1.7 Study Limitations

Results from this study should be interpreted in light of the following limitations:

1. Because the study is observational rather than experimental, any relationships that are observed are associations and do not imply causal links of inputs and observed outputs.[13]
2. Many of the potential explanatory variables are highly correlated. Therefore, even if one of a collection of potentially explanatory variables were a true cause of publications output, it is difficult to determine which of these variables it is.
3. The classification of resources (personnel and R&D funding) and publications into fields may result in some inconsistencies, since different sources of data classify resources and publications into fields using different schema and criteria. In some instances, the personnel, funding, and publication counts associated with a particular article might be allocated to different fields.
   
4. Many potentially useful variables were not available. For example, we do not have counts of all faculty with S&E degrees by field, or even by institution, nor do we have estimates of the proportion of time spent by faculty in research or teaching duties.
   
5. There is limited variability in resource trends over time. Most resources and outputs increased over time in a fairly linear fashion. If resources and publications had varied cyclically instead of trending in one direction, the reliability of our inferences would be higher.
   
6. All journals in the fixed or expanding journal set are treated as being of equal importance. Similarly, all publications are treated as equal in the sense that a "breakthrough" article would contribute no more to publication counts than any other article.
   
7. The dependent variable throughout much of the analysis is publications; manuscript submissions may also be relevant, but were unavailable. U.S. researchers may be submitting as many or more manuscripts, but finding that their acceptance rates, in the aggregate, have been falling. Variations between submission and acceptance rates may vary across institutions and/or fields.
   
8. The analysis is restricted to factors within the U.S. academic sector. An analysis based on a larger framework (i.e., international competitive markets) may have yielded additional insights.
   
9. Information about patents was limited to the patent count by institution and year. We were not able to classify patents according to the context of research production (such as whether the patent was speculative patenting of what seemed like an exploitable idea, was obtained as the basis of a spin-off launch, or was obtained as a result of a strategic research collaboration between a university and private sector company).

 Back to Top

---