Ruts in "The Royal Road"

“The current size of NSF grants and their duration might be resulting in an inefficient process at U.S. academic institutions. Researchers might be spending too much time writing grant proposals instead of doing actual research. NSF has increased grant size and duration in previous years, particularly through its priority research areas; however there is little documentation that this is having a positive impact on research output. With the assistance of U.S. academic research institutions, NSF will develop efficiency measures of the research process and determine what is the right size for the myriad types of research the agency funds."

In other words, has increasing the size and lengthening the duration of some NSF grant awards, as NSF has begun to do, had any impacts on research? And do NSF- sponsored researchers nonetheless spend “too much time writing grant proposals instead of doing actual research?" Is there a “right size for the myriad types of research the agency funds"? In effect, are there ways in which the NSF grants program, which totalled $2.1 billion in FY2001 inefficient?

The agency was to investigate these matters in time for formulation of its FY 2004 budget. It contracted with Mathematica Policy Research Inc. to survey all the principal investigators who received competitively reviewed grant awards in FY 2001. This total, 6,180, was reduced to 5,793 when those who were awarded more than one grant were assigned a single grant to discuss in the survey. The questionnaire was returned by 91 percent of those surveyed, with 4989 questionnaires complete enough for analysis, an unusually high response.

Mathematica’s quantitative analysis of the PI survey and its survey of academic institutions, also made in response to the OMB query, appears as “National Science Foundation Report on Efficiency of Grant Size and Duration: Principal Investigator FY 2001 Grant Award Survey and Institutional Survey."¹

This report analyzes the written responses to three open-ended questions the PIs were asked in Mathematica’s survey. These were Questions 2.5, 3.8 and 5.14 in the questionnaire. For simplicity, this report refers to them as the first, second, and third written questions. The first asked what the impact was of changes NSF made in the PI’s’ grant proposals. The second asked what would be the impact if NSF gave them “what you need for what you want to accomplish." The third asked for “any other comments" about their experience with the NSF grant process that was "important."

Evidently, these questions touched a nerve – or several. The PIs’ answers ranged from a few words to several pages.

-- The first question, Question 2.5, got 1,401 responses out of 2,708 PIs eligible to answer. These print out at 106 pages.

-- The second question, Question 3.8, got 2,202 answers out of 4,510 PIs eligible to answer. These print out at 198 pages.

-- The third question, Question 5.14, got 2,548 responses out of 5038 PIs eligible to answer, printing out at 208 pages.

The total of 6,160 written responses print out at 512 pages.

“In surveys of this type, respondents normally don’t take the time to write answers to open-ended questions after they have answered the pre-coded choices," notes Mathematica Vice President Janice Ballou. She considers 10 -15% to be a good response rate to open-ended questions. In this survey 48% of eligible PIs wrote answers to the first question, 47% wrote answers to the second, and 50% wrote answers to the third question. Respondents’ guarantee of anonymity prevents this report from tying individual comments to specific NSF offices or actions. But even though anonymous, these comments are a new, rich lode of perspectives by the nation’s leading scientists and engineers about NSF and the conduct of research in the United States today.

This was the first major survey of NSF principal investigators since one conducted by Abt Associates for the agency in January 1987. ² That survey sought investigators’ views about NSF's merit review process. It was sent to 14,282 individuals whose proposals had been awarded or declined by NSF in FY1985. A total of 9,204 or 64.4% responded. The 1985 survey went to both awardees and declines, whereas the current survey went only to awardees. The subject of the current survey is different. It is focussed on the impacts of award size and duration, though the PIs also write about many aspects of the NSF grants program and the agency.

This report summarizes what is relevant for policymakers about:

- how the PIs see issues related to NSF’s award size and duration,

- the efficiency of the NSF funding process,

- how science and engineering is done,

- the role of NSF in the nation’s research and education processes,

- both human and research outcomes of the NSF investment for the nation,

- the realities of the research and educational enterprise in our universities.

It is a qualitative summary of the written responses, to complement Mathematica’s quantitative analysis of the multiple choice questions. This report is impressionistic; it represents one science policy analyst’s culling of this large archive for the principal and most interesting insights.

Structure of Report

The three parts of this report correspond to the three written questions. Because the PIs discussed the same issue in answer to different questions, comments have been grouped under issue headings. The Table of Contents is a guide to where specific issues are discussed.

Permissions

All verbatim quotes from Principal Investigators’ written responses are reprinted with their express permission.

Excerpts

Most excerpts represent the complete quote, omitting repetition or detail. A three-dot ellipsis (…) means a phrase has been deleted. A four- dot ellipsis (….) means the deleted bit starts or ends a sentence. Spelling errors have been corrected. Where a change has been made to correct grammar, such as to start a sentence with a capital letter, it appears in brackets [ ]. Double quotation marks setting off quotes within the written comments have been changed to single quotation marks.

Four notable and representative comments are reprinted in full in Appendix A. Appendix B reprints all the comments PIs made in answer to the first question that referred to the National Institutes of Health for which permissions were given.

PI References

The respondents are all Principal Investigators, or PIs. Unless self-identified, their gender was unknown, so they are referred to as “their" in follow-on references to avoid using “his" or “her." Of the 4,989 respondents whose complete questionnaires were used, 17% were female.

Author

The contractor writing this report is a journalist and author who has written extensively for “Science" magazine, for “Nature" and many other publications. She is author of two books and co-author of “Lost at the Frontier: U.S. Science and Technology Policy Adrift," published by the Institute for Scientific Information.

Footnotes

1 “National Science Foundation Report on Efficiency of Grant Size and Duration: Principal Investigator FY 2001 Grant Award Survey and Institutional Survey, " NSF, 2002.

2."National Science Foundation (U.S.). Program Evaluation Staff. Proposal Review at NSF: Perceptions of Principal Investigators." Washington, D.C., National Science Foundation, 1988. NSF 88-4

Ruts in "The Royal Road"

Executive Summary

In June 2001 the Office of Management and Budget asked in a letter to Dr. Rita R. Colwell, Director of the National Science Foundation, whether recent increases in the size of some grants and their duration, as NSF had done in some areas, had had any impacts on research. What impacts could be documented? And despite increases in some awards’ size and duration, do Principal Investigators on NSF grants spend “too much time writing grant proposals instead of doing actual research?" Is there a “right size for the myriad types of research the agency funds"?

The agency was charged with investigating these matters in time for formulation of its FY 2004 budget. It contracted with Mathematica Policy Research Inc. to survey the total population of Principal Investigators who received competitively reviewed grant awards in FY 2001, or 6,180 PIs who were involved in the $2.1 billion grants program that year. An exceptionally high proportion of 91% responded. Fully 4,989 questionnaires were complete enough for analysis.

This report complements Mathematica’s report on the survey questionnaire, and an accompanying survey of US academic institutions, titled “National Science Foundation Report on Efficiency of Grant Size and Duration: Principal Investigator FY 2001 Grant Award Survey and Institutional Survey."

The survey questionnaire had three open-ended questions to which PIs could write responses of any length. This report summarizes their answers: Part I summarizes responses to the first written question, Question 2.5; Part II summarizes responses to the second written question, Question 3.8; Part III summarizes responses to third written question, Question 5.14. Because the PIs discussed the same issues in answer to different questions, issues have been grouped under subject headings. The Table of Contents is a guide to where specific issues are discussed.

These comments amount to a new, rich lode of perspectives by the nation’s leading scientists and engineers about NSF’s grants program and the conduct of basic research in the United States today. They wrote about what the impacts were from changes NSF made in their grant awards (first question) and what they needed to accomplish their overarching goals (second question). The third open-ended question asked them to “write in any other comments you have about your experience with the NSF grant process that you think are important." Responses here covered a wide range of problems and opportunities, often including praise for NSF’s mission and execution. One wrote of the agency:

“It supports pure research and does not demand that there be a payoff in medical or economic terms. For the ordinary scientist, pure research is always the royal road to understanding and NSF is often the only place to go if one wants funding to travel that royal road."

What do these comments say about NSF’s grant process? Many PIs are concerned that grant durations of three years or less are too short [these were 85% of the total in FY 2001]. Many see too few opportunities for large awards [grants of 4 and 5 years were 16% of the total in FY 2001]. NSF’s average award of approximately to $100,000 per year supports about one student or postdoc after overhead and other costs, they said. Many say they spend too much time writing proposals to keep multiple short term grants flowing and this hinders efficiency. The PIs say there is too much churn in NSF’s process for them to attract the most promising students and keep them the four years or more needed through the PhD. Many say these factors impede optimal research results and make it harder to develop the next generation of world-class US scientists and engineers. Their take-home message is that the grant process has discontinuities that hinder NSF’s research and education missions. Many say larger awards and more awards would smooth the way. Many argue longer term awards would be more efficient. One could conclude from these comments that the venue of NSF grant research seems a rutted road, not in need of tearing up or of new direction, but in need of repair.

First question – Question 2.5

Here PIs were asked to comment on "impacts" from changes in the NSF grant awards. In FY2001 47% of competitively reviewed grants were funded up to 5% of what their PIs proposed; another 51% were reduced 5% or more from the amount they proposed. Most of the written comments to this question appear to be by PIs whose proposed grant amounts were cut.

Many PIs said that shorter award terms negatively impacted their efficiency; so did lower funding amounts, by reducing the numbers of students and post-docs on their projects.

The PIs listed cost-saving moves they make when NSF cuts their grant award amount. They do less undergraduate teaching, including less training of minorities who could become scientists and engineers. Some cut their own pay and use the funds to pay to their graduate students and postdocs. When NSF cuts a month or more of their proposed summer salary, their research suffers, they say, because these academics count on summer to concentrate more on research. Some donate summer time and do the work anyway. Or they take other paying work and defer the NSF project. When funds for collaboration and travel are cut the PIs feel less productive; the frequency with which they expressed such concerns reflects the growing collaborative and international character of science.

No matter what their experience with changes to NSF awards, many said the present award terms [average 2.9 years] and funding amounts [annual average $113,000] make it hard to attract the best students. Many say they propose less risky projects to NSF in the first place because they cannot be sure of getting the results in NSF’s timeframes.

But the PIs’main problem was that duration and size of most NSF grant awards creates churn in making them spend more time writing proposals to assure they win one or several simultaneous grants so they can keep their students and projects going. So much proposal-writing, with relatively long odds of being funded, takes time from research. Many preferred NIH’s grant terms of over 4 years and average annual amounts of $300,000. A few said they were under pressure to stop applying to NSF and concentrate on winning NIH’s better terms.

Second Question - Question 3.8

The second question asked what the PIs would do “if NSF provided you with what you need." Many said they would attack their scientific problems on a broader scope and reach for the big advances and contributions. Many said or implied that they could reach their goals more efficiently and perhaps sooner. A number said more secure funding would free up time now spent writing proposals, allowing them to make better direct contributions to research. One implication was that the quality of some NSF-funded research could be higher.

Many would use additional support to mentor more students and other junior personnel; with larger teams they would accomplish more research. A number said they would use additional funds to include more minority students. This was related to risk-taking, some explained, because with less tight funding they would feel freer to choose from a wider pool of recruits. Several said that added support would help them integrate research into their teaching. Many said they would use the funds for more collaboration. Several said they would use the added support to improve dissemination and web site development, supporting the theme that some PIs feel that NSF does not do enough to help them communicate their findings.

Third Question – Question 5.14

The third question asked the PIs to “write in any other comments you have about your experience with the NSF grant process that you think are important." One theme of importance is the discouragement expressed by those who believe that the odds of winning an NSF grant were getting longer and that resources from NSF grants are shrinking. Not one mentioned that the overall grant program had grown in the prior two years, nor that NSF’s overall average grant amounts had risen to $113,000. (The PIs’ grants evaluated in the survey averaged $112,000.)

Some PIs expressed the belief that NSF grants go to a network of established researchers, and that younger investigators and new institutions have a harder time winning awards. [In fact younger investigators continued to receive about one fifth of all grant awards since the 1990s; those whose chances of winning NSF grants have gotten harder are PIs with prior year awards.] Here as elsewhere, there were PIs who said NSF had funded their high risk proposals at the start of their careers, and were grateful for this spark and follow-on support.

The PIs also weighed in on how they preferred NSF staff to be proactive in advising them and shaping the award. Several criticized panel reviews; several said mail reviews were fairer. One theme was the wish for a chance to correct negative or erroneous reviewer comments before their proposal was put for decision. The biggest impression of the comments answering this question, however, is PI frustration with the amount of time spent writing new proposals or revising ones that had been declined, as an inefficiency in the system.

As for making the NSF grant program more efficient and productive, many assumed the total “pot" of funds would not grow significantly. Therefore several were concerned that if the average amount of grant awards were raised to, say, $140,000 per year, then NSF would make fewer awards and the odds of getting funded would be worse. Many proposed that NSF make more awards to increase the odds of being funded and foster more risky proposals. The preferred “right size" for grant term was 5 years; many wrote that 5 years would reduce churn, help them attract better graduate students, and encourage higher-risk research. However some had cautions about this change.

Many PIs offered unsolicited praise for NSF. Many thanked the agency’s staff for helping them work toward their education and research goals under a system which got mixed reviews.

The Appendices give the reader a first-hand look of how the PIs frame the important issues raised in this study. Appendix A contains four sample comments reprinted in full. Appendix B contains all the answers to the first question referring to NIH for which permission was given.

Ruts in "The Royal Road"

Report on Principal Investigators’ Responses to Written Questions in
NSF-Mathematica Policy Research Inc. Survey January–March 2002

Introduction

In FY2001 the National Science Foundation handled $2.1 billion in grants for science and engineering research to Principal Investigators (PIs) at academic institutions around the United States. This amount represented an increase over the FY2000 NSF grant total of $1.8 billion and the FY1999 grant total of $1.6 billion. The FY2000 and FY2001 increases were the latest effort to compensate for the slow growth in NSF grant awards during the 1990s; from FY 1992 to FY 1998 the NSF’s grant account remained between $1.2 and $1.5 billion. Meanwhile the costs of doing research such as pay and overhead rose through the 1990s. By decade’s end there were concerns in many quarters -- within the agency, by the investigators, their institutions, and Congress -- that the size of NSF grant awards was not keeping pace with costs.

Another set of concerns was the efficiency or inefficiency in the performance of research on NSF grants. NSF grant terms are shorter than the corresponding awards made by the National Institutes of Health, for example. NSF grant awards made in FY2001 averaged 2.9 years while NIH’s awards (R01 account) averaged over 4 years. Also, the average amounts of NSF grant awards remain lower than investigators wish. Through the 1990s NIH’s average annual grants have been on the order of $300,000 per year compared to NSF’s $100,000. So the NIH grant program appears more efficient than NSF’s in research performed for the administration involved.

Therefore, in June 2001, the Office of Management and Budget wrote to the Director of the National Science Foundation asking whether the “current size of NSF grants and their duration might be resulting in an inefficient process at U.S. academic institutions. Researchers might be spending too much time writing grant proposals instead of doing actual research." It asked if “positive impact on research output" could be documented from increased grant size and duration of some NSF grants. It asked the agency to work with U.S. academic institutions to develop efficiency measures and determine “what is the right size for the myriad types of research the agency funds."

In response to OMB’s query, the NSF asked Mathematica Policy Research Inc. to survey the entire group of approximately 6,000 PIs who received competitively reviewed NSF grant awards in FY2001. Mathematica’s survey was multiple choice. It also asked three open-ended questions to which the PIs could write answers. Fully 91% of those surveyed responded. The written answers alone comprise 6,160 individual comments printing out at 512 pages. This report is a qualitative summary by a science policy analyst of these written responses. The quantitative analysis of the entire survey is Mathematica’s report, “National Science Foundation Report on Efficiency of Grant Size and Duration: Principal Investigator FY 2001 Grant Award Survey and Institutional Survey."

Part I

First Written Question – Question 2.5

The survey’s first written question, Question 2.5, asked:

“Please describe any other impact(s) that resulted from the change in your 2001 NSF award or give more details on any in the list that need further explanation."

“The list “ referred to a multiple choice list of possible impacts of changes in their NSF award amounts and duration in the preceding Questions 2.1 - 2.4.

When deciding to make an award, NSF program directors and/or peer reviewers sometimes change what the PI originally proposed. Sometimes NSF changes the amount; it can change the duration, or change both, for example a smaller grant may fund two years of research instead of three. Of the grant awards analyzed by Mathematica for the survey 47% were funded up to 5% of the proposed amount. Just 2% had their awards increased over 5% from their request. Fully 51% had their awards lowered by 5% or more from what they proposed. Stretching total grant program dollars by lowering individual awards is common practice by NSF.

Fully 48% of those eligible to write answers to this question did so. Mathematica found that 6% of the written answers described positive impacts of awards that were larger or of longer duration than requested. Another 27% wrote that the change’s impact was to reduce their ability to meet goals and objectives. The largest fraction of answers, 35%, referred to negative effects such as on team building, students and collaboration.

Many PIs praised how NSF staff had helped to make their projects succeed, even when award amounts or terms were less than optimal.

Overall, PIs who had longer award terms and whose proposals were fully funded said these features helped research efficiency. Many to whom NSF gave less than they sought and had shorter award terms wrote that these features hindered their efficiency. Some said they trimmed the scope of their work and proposed less high-risk/high-payoff work to be able to meet shorter term deadlines for results. (See Parts II and III.) The PIs answering this first question, who had their grant awards lowered in funds or duration, stressed the negative impact on training and on the scope of research due to having less postdoctoral and student assistants.

A Context

Two important pieces of context are needed to understand why larger amounts and longer duration than NSF’s standard $100,000 and 3 years seem more efficient and why smaller grants and shorter duration impede efficiency, in the PIs’ view. The twin contexts are NSF’s mission and the character of academic research proposals.

A.1 NSF’s Mission in Research and Education

The agency’s mission is, in the language of the Act, “to initiate and support basic scientific research and programs to strengthen scientific research potential and science education programs at all levels." This dual mission of science and engineering research and education shapes how NSF research is performed. It means that most NSF grant awards go to PIs at academic institutions who arrange for some of the work to be done by graduate students working towards the MS or PhD degrees. Often key parts of a project are done by those with the doctorate, known as postdoctoral researchers or postdocs. (These are young researchers training for junior tenure track faculty appointments.) Undergraduates also participate under the guidance of graduate students or postdocs, whose formal employment can be as teaching assistants (TAs) or research assistants (RAs).

Thus the research project does two things at once: it produces scientific results and provides a forum for the PI to mentor personnel at different skill levels and career stages. For more than fifty years, research and education of the next generation of scientists and engineers has proceeded successfully in this integrated system.

One PI explained eloquently why the NSF system of integrated teaching and research is a “jewel beyond value and the envy of the world in terms of the positive returns it brings at so many levels." (See Appendix A) “University research is first and foremost an essential foundation of the process and functioning of education." No doubt we get “technology, scholarship, and knowledge-base benefits" from having research performed by junior personnel mentored by more senior personnel. But the greater “gravity" of the system is “the tremendous synergies we build into the education mission by having it so closely coupled to a preeminent world-class research enterprise. This is the take home message -- research is education!"

Thus when PIs consider the “impacts" of changes in NSF grant awards on their efficiency, they consider the effects on both endeavors: the performance of research, and publishing and disseminating it so the next scientific problems can be tackled, and their mission to mentor the best postdocs and students they can attract, so they become world-class themselves.

The above-quoted PI wrote: “[T]here really is a genius that underlies the integrated system of teaching and research that we operate in this country.….Research lies at the heart of how we do graduate education in the physical sciences. We bring the best students in the country (and the world for that matter) to this involvement. We develop in them the skills and envelop them in training experiences that, by the time they leave, place them at the very top of the national talent pool. …. This is a comprehensive system of training in the disciplines that works very well. It is the envy of the world and the NSF is the essential agency of national reach that preserves and nurtures it. We fool with this only at our peril."

But this PI was also concerned. “We have tweaked and fooled with [this system] to the point that it is now in grave danger of failing. The resources being invested in it are below what we need." This and other PIs’ concerns are summarized in this report.

A.2 PI Proposals Integrate Research with Training the Next Generation

Research proposals to NSF represent a package the PI has assembled, including allotments of their time, full- or part- time support for postdocs and / or students, use of instruments or field work, computing and sharing results through conferences, databases and web distribution. These plans rely on participation by junior personnel to do the supporting research, for its own sake and to learn, i.e. “research as education."

Rarely do PIs propose projects they would do alone. Overwhelmingly, they propose collaborations postdocs and students. A very large number of projects entail other PIs, sometimes at other institutions in the US and abroad. The proposal is a package, a precise mix of people, activities, equipment and collaboration. It often includes individuals the PI has lined up in advance, such as a starting graduate student who agrees to undertake a piece of the proposed research as their PhD thesis, and hopes to publish it when it is complete.

B Impacts of Increases in Award Amount and/or Duration

As mentioned, in FY2001 the agency gave increases of 5% or more above their grant requests to 2% of the PIs. What one PI did with the increase shows the coupling of research with education in the integrated NSF system.

The impact, they said, was “better support for research assistants (graduate students working on PhD), thereby attracting more high-talent young people to the field and getting them started on productive careers. (All the extra support from NSF in this case went into this category.) Of course, this will also be good for the research objectives themselves, but the main effect is to foster future growth through mentoring PhD students."

Those who got more than they asked for said it allowed broader scope and higher- risk/higher-payoff research. One PI wrote of their increase, “Mainly it allowed [me] to broaden the scope of the research and undertake a more innovative, ambitious (and time consuming) approach to the problem under consideration."

The PIs restated this theme elsewhere. (See below “Churn Relieved by 5 year Duration" and Part III.)

C Many PIs Adjust to Changes

In FY2001, 47% of PIs received competitively reviewed grant awards that were within 5% of the amounts they requested. Another 2% had increases of 5% or greater than their requests. This may explain why many PIs indicated they rebalanced their work plans adequately.

Their institutions sometimes make up for cuts in awards. One PI wrote that their large state university “stepped in to make up the difference," by paying for 5 students the PI had asked for and NSF had denied. But compensatory funding on this large scale seemed rare; most institutional help seems to be on the order of support for one student or for some travel.

Perhaps the easiest adjustment was when their home institution recomputed the overhead charge to be subtracted from the grant, netting the investigator what they sought in the first place.

In evaluating the impact of NSF’s trimmed awards, account should be taken of varying overhead requirements of institutions; Institutions with flexibility or lower overheads may adjust better when a PI gets less funds from NSF. On the other hand, institutions with higher overhead requirements, whose net to the researcher from an NSF grant is no longer worth the time and effort, may ask their PIs to apply to a more generous agency.

One PI wrote, for example: “The drastic decrease in budget and duration required that I take out several parts of the proposal which I then incorporated into another grant proposal to the NIH." Indeed, a common response to cuts from NSF is that PIs apply for compensatory funds from another agency or from private industry.

D Impacts of Decreases in Award Amount and/or Duration

The NSF’s dual mission in research and education explains what PIs do when NSF makes a grant award for significantly less funds and/or shorter duration than the PI had planned. Now they must undo the package they had assembled -- of co-PIs, post-docs, students, equipment, field work, computing, dissemination – to fit the reshaped award. Frequently the PIs said the impact was less research output, slower results, and concern they were not attracting the best junior personnel to mentor into the next generation.

The written answers to this first question show the PIs intensely concerned with having to cut personnel. A randomly chosen 10 pages of printout from the 106 pages of answers revealed that 51 of 138 PI comments, or 37%, mentioned the impacts on personnel, mainly students and postdocs. Of these 51 references, just two involved increased employment. The other 49 cited the costs and problems of not having those personnel the PI had planned.

D.1 Postdocs

To cut postdocs from a project is costly in scientific results, since postdocs have a career incentive to make discoveries on the cutting edge and become authors or first authors on resulting papers. The PIs, being tops in their fields themselves, sounded keenly aware of competition for the best postdoctoral researchers. As one wrote: “The curtailment of the postdoctoral funding makes it difficult to hire the best person. There is a real threshold here, i.e., you can’t just hire 85% of a person."

“It is the skill and independence of a postdoc," wrote another PI, “that truly permits us to launch into innovative (and high-risk) areas. I am fortunate to have an excellent postdoc.“ But due to NSF’s cut in the grant award, this postdoc would not do “the experiment initially planned (too time-consuming and high-risk, although it would have had huge potential returns). Instead she is focussing on more practical generic correlation measures."

Now from the viewpoint of that postdoc, if “their" PI can only offer work on “more practical generic…issues," when they had aspired to do “innovative (and high risk)" research, that postdoc may be less motivated to join the PI’s project. Not having this postdoc work at the cutting edge suggests an efficiency cost: the NSF grant will not train that person maximally and is less likely to bring the greatest scientific results.

As a result, postdoctoral participation in a new research initiative often acts as a catalyst for pushing the boundaries of the research. That is, the inclusion of a postdoc in a research program has more of an impact on that program than simply the research that he/she is able to accomplish on his/her own.

“By not providing sufficient funding for me to hire a postdoc, the NSF, I feel, has weakened my ability to progress rapidly on the proposed research, as too much of my time is now spent on direct stewar[d]ship of graduate researchers and too little time is spent on innovation and new insights."

This PI’s last point bears on efficiency. When a postdoc is cut from a project to meet a reduced budget, so the PI turns to a lower paid graduate student, the PI must supervise that student more. While this advances NSF’s education mission, if it means the PI spends less time directly on research, the project may yield fewer results.

D.2 Students – Graduate Students

“Basically, less money means fewer students. [F]ewer students means fewer avenues of research pursued.[I]t’s pretty simple!" said one PI bluntly. Another wrote that “[r]eduction in funding limited our ability to have post-doc mentor a graduate student within the proj[ect], which we find to be the best coupling for graduate education."

One PI had proposed “a paid research opportunity for one post-MS level graduate student. “Such students are more firmly grounded in research methodology. They also generally have a broader range of usable research skills. Eliminating the position increased orientation, training, and start-up time for the project. "

The PIs’ intimate familiarity with various mixes of postdocs, and students show that allocating them in grant proposals, and accounting for them administratively, are daily facts of life.

Money can be saved many ways. When NSF cuts a proposed award, some save by having a proposed graduate student paid as a Teaching Assistant instead of the preferred Research Assistant. This carries an efficiency tradeoff; a bit of salary is saved but then more of the student’s time is spent on teaching and less on research. From the standpoint of the student aspiring to a career in science, the TA position may be less attractive than the RA position, aside from the lower pay.

D.3 Students – Undergraduates, REU, RUI, Minorities

When NSF grants are awarded for less than the PI sought, another cost-saving move is to hire undergraduates instead of graduate students, paid technical staff or postdocs. Though some salary is saved, undergraduates cannot do the caliber of independent work a graduate student can. With rare exceptions, undergraduates cannot spot the insights that could be publishable discoveries the way postdocs can. A salaried skilled technician can be more efficient from the PI’s viewpoint than a series of undergraduates, who each need training for the technician’s work. When the PI directly supervises undergraduates workers, their educations are helped; but such teaming may not the most efficient way to get the research done.

When even undergraduate pay was cut as part of NSF- mandated reductions, the PIs often spoke of the educational loss from not having them involved. Several expressed gratitude for another NSF ‘program, Research Experiences for Undergraduates, or REU, which pays undergraduates to do research.

By design, some NSF grants got to PI faculty in undergraduate-only institutions. NSF awards help faculty there undertake original research and interact with colleagues elsewhere. By design, these grants include undergraduates, to help do the research and to use it in coursework. Historically, some of the nation’s outstanding scientists and engineers started their careers in undergraduate-only institutions.

Many of these institutions are incubators of future minority scientists and engineers. So NSF grant awards to PIs there affects the future pool of minorities in the field. One PI, for example, would have used the funds that were cut “to buy-out teaching time and hire underrepresented minorities as research assistants. It would also give me more time to work with undergraduates (I teach MBA students). Exposing minority undergraduates to economic research is essential to increasing the potential pool of future minority Ph.D.s."

The impacts of cutting back grants to PIs in these places are both loss of some research, the greater difficulty of the PI in working with colleagues elsewhere, and loss of the training of undergraduates.

“Even small grants go a long way toward inspiring, training and especially educating undergraduate students about science. Moreover, such grants give professors at smaller institutions the freedom and funding to stay connected to other researchers in their field." said a PI responding to the third question.

D.4 PIs Donate Their Time

Some PIs said they absorbed cuts in their original plans by donating some of their time and doing the research unpaid. But most of the comments to this effect, including the strongest, appeared in answers to the third question. (See Part III.)

D.5 Summer Salary Important

Many PIs who teach during the academic year use summer to devote uninterrupted time to research. When NSF cuts a month or two of summer salary from a proposed grant awards, some PIs wrote this caused inefficiencies. “I had to cut my summer salary….[I]t is in fact where I have some extra monies that, when funded, usually end up being used for ‘new ideas’ during the project or to buck up a postdoc salary etc."

Another said the cut “negatively impacted my family because I could not pay myself summer salary and achieve the goals of the grant." Another reported: “Summer support reduced from two months to one month. I am unable to work for free, so this will reduce the time available for research."

Also summer is the time some can visit with collaborators elsewhere for intensive work. One wrote: “I did not get compensation for the summer time which may reduce my possibilities to spend time elsewhere working together with my research collaborators."

D.6 Soft Money PIs

Among those PIs hurt by cuts were those who depend entirely on outside research funds, including NSF’s, for their pay. Career researchers not in tenure track positions are in this situation, and known as “soft money" researchers. They are most common in biomedical sciences, but found in other fields as well. When NSF cuts the amounts ‘soft money’ PIs sought, they must find other sources of income, or work for free, or not do the project.

Here was the impact of a reduced grant award for one soft money PI. “The biggest impact of the reduction in funds means that I could not support the graduate students on the project for the entire year, only 2 terms of the year, and that I have had to volunteer a great deal of my own time on this project to keep it going. This has meant personal sacrifice, as I live on soft money, and time spent as a ‘volunteer’ is time not spent on other projects or proposals for which I can be paid. What I see in the funding cuts is an erosion of the value attached to academic research employment."

E Duration, Continuity, Completeness

The context for this survey of all PIs with competitively reviewed grants in FY2001, as mentioned, was the fact that NSF had begun increasing the duration of some grant awards. Another context was anecdotal evidence that researchers prefer NIH’s standard term of more than 4 years (see below).

[Historically, the average term of NSF grant awards was 2.5 years. In the 1980s it moved upwards; it was 2.8 years for most of the 1990s. Since, NSF increased the duration of more grants, so the average was 2.9 years in FY2001. That year 85% of competitively reviewed grant awards were 3 years or less; 11% were for 4 years and 5% were for 5 years.]

In written answers, PIs often said the shorter terms made them less efficient. For example, one wrote: “Shorter term grants force you to generate additional funds from other sources, usually on other topics, that tends to reduce the coherence and synergism of your total research program." Shorter term awards forced the PIs back to the drawing board – that is, to the proposal-writing table – more often than they found efficient. (See F below and Part II B.2.)

Sometimes, PIs were grateful that NSF asked them to do a shorter project to bring their big plans into focus. One PI wrote that forcing their project into a briefer ‘proof of concept’ phase had helped. “We applied for a center grant and were given a planning grant; thus none of the research we proposed will be performed. On the other hand, the planning grant has been of great use to us in focusing our research activities…."

E.1 The 2- 3- and 5- Year Problem

A frequent claim was that NSF’s standard grant duration of 3 years caused them to propose safer inquiries instead of risk-taking. “A 3 year timescale is discouragement to working on those long term risky ideas that jut might end up producing an exceptional result," wrote one PI. “More restricted funding ultimately leads one to retrench within the more conventional ideas, those that have a higher probability of giving results for the next round of proposals," wrote another. The belief that NSF’s grants process encourages ‘safe science’ was a theme of later answers. (See Part II A.1 and B.1.)

But PIs wrote that 2- and 3- year grant terms give a prospective graduate student participant less assurance they would be supported through their theses, which take 4 years or more. Many PIs expressed concern that the duration of NSF grants made the work less attractive to the best students and that these promising recruits would go to more secure research environments – such as to a PI who had a 5 year award, or might leave academic science altogether. Anther inefficiency is churn even before a 3 year grant ends, PIs must apply for the next round; when it starts they must re-recruit and retrain, in effect rebalancing their teams mid-stream.

[The employment picture of the late 1990s helps explain PI concern with attracting the best junior personnel. The number of PhD graduates in science and engineering fields in the late 1980s was about 18,000 per year. By the late 1990s, US institutions granted about 28,000 of these degrees yearly.

[But the expansion was confined to biology-related fields and agriculture, while the numbers of MS and PhD degrees in engineering, physical and geosciences, and mathematics, staying about level. Even those with the BS in these fields had the prospect of lucrative careers in the private sector, in a commercial job market with heavy demand. Small wonder the faculty principal investigators in the survey were concerned they be able to offer adequate and secure career paths to attract the best. The PIs in biology-related fields faced pressure from lucrative private sector jobs, but other research careers funded by a fast-expanding NIH.]

The concern some PIs voiced that the quality of graduate apprentice students was not all they wished may be explained by this context. (See Parts II and III.)

One PI wrote: “The fact that the award is only for 2 years means (1) I’m reluctant to hire grad students who need 3 years support to complete their degrees and (2) I need shorter term goals so I can report results for the next proposal and (3) I’m going to have to write more proposals, which takes time away from the research. The fact that it’s funded at a lower level means I can’t take a chance on students whose talents and productivity are less certain."

Another PI wrote: “The change from 4 to 3 years makes it very hard to find and support a graduate student; there is not enough funding to last for the entire PhD period of the student."

How these factors interact was explained by another. “The reduced level of support limits (1) the pursuit of novel research when more time and resources are needed at the end of the project, when most productive and fruitful results are about to give birth; (2) the completion of a PhD student’s thesis: normally three to four years are needed to finish a PhD thesis. This two-year funding can only accommodate a Master’s candidate, thus reducing the level of research somewhat (making Master’s level performance for a PhD caliber research project); and (3) qualified personnel. …[T]alented students intending to [go] on to PhD work are not attracted to this two-year period in their 4-year career…. “

For a young scientist, the flip side of job insecurity is job security. One PI looked back on nostalgically on the impact of the Presidential Young Investigator award they received from the NSF years before. “The flexibility and security that award me in developing my research program as a new faculty member was invaluable. Years afterward I am still reaping the benefits of the research and teaching that I nucleated under that program.“

E.2 Grant Duration and Churn

Many PIs describe NSF’s 1- 2- and 3- year grants as creating churn in their professional lives that goes beyond the start-stop impacts on the research. Shorter terms hinder NSF’s co-equal mission of attracting the best would-be scientists and in mentoring them to the point of becoming world leaders themselves.

Another impact some cited was personal stress, such as the PI who concluded a long comment saying they wondered "daily" whether to “change jobs." (See Part II C.)

Churn was reduced for one PI whose grant term was increased by NSF. “At the time of the grant proposal I was about to experience a complete turnover in personnel. The result has been a rather slow restart…educating and working with my research group. Without the longer duration grant this would have had a disastrous impact, since there is a lag time in coming back up to ‘normal’ research productivity.

“In addition I have just started to form some research collaborations, and the longer grant will allow there to be some visible impact on my research results before the next grant proposal is due.

“The extended duration and generous funding has also allowed me to hire a top quality postdoc. I am very, very grateful for how my grant has been handled."

Small wonder that PIs answering this and later questions tended to say 5 years was the right grant term. “A 5 year timescale would be much more helpful," wrote one. One whom NSF had encouraged to ask for 5 years, and had the 5-year project approved, wrote: “[t]he longer period will encourage more speculative work and perhaps new collaborations."(See Part II B.4)

E.3 Cuts in Collaboration and Travel

Few PIs work alone. New knowledge becomes evident faster when researchers bring complimentary knowledge and skills together. A mathematician wrote: “It is very important for my research to be in constant contact with other mathematicians around the world (France, Germany, Russia, Japan). The less supp[ort] I get for travel/summer salary, the less contacts I have."

Increasingly American PIs collaborate with colleagues in other US institutions and abroad. New technology also makes field work and observations from big instruments more cross-national. So the need for collaborations is rising. But even as the Internet makes distant collaborations more necessary and feasible, face to face interaction remains important.

When one PI was awarded a budget that reduced the visiting time of a colleague from West Africa from 45 to 35 days, they wrote it “will reduce the time we can work on the GIS [global information systems] component of the project, which may have negative impact in terms of the analysis of results and timeliness in publication."

Assured travel funds for hoped-for collaborators can be critical to involving them in a US- based project, and thereby keeping American PIs at the hub of the world’s growing population of international scientific collaborations.

As for advancing students’ education and careers, the chance for them to present results at meetings advances their knowledge and NSF’s education role.

The flip side is that cutbacks in travel to conferences, or cutting travel by a co-PI, can set back efficiency. One wrote: “Loss of essentially all our travel money for conferences… means that our research results will not be disseminated to other researchers in the field as quickly as we would have liked [. (W]e will have to rely on refereed publications which have a 6+ month time lag between submission and publication). "

If a researcher does not go to a relevant conference, the informal and formal knowledge they would have gotten there is laborious, and in some respects impossible to obtain afterwards. (See Parts II and III.)

E.4 Impacts on Scope and Risk

Increases in dollar amounts sometimes helped PIs to widen the scope of their research. Decreases in student support decreased the scope, because less research could be performed. One PI wrote “loss of student support should merely delay the completion of the work, and somewhat reduce the overall scope."

More on how PIs said the NSF grant process impacts project scope and risk is in Part II B and Part III A.

F Time Writing Proposals v. Time for Research

Mathematica’s analysis of the multiple choice part of the survey showed how much time the PIs spend writing proposals. For initial submissions to NSF, 33% spent 80 hours or less preparing their proposasl; 29% spent 81 to 150 hours; and 29% said they spent more than 150 hours (3 1/2 40-hour weeks). The length of time went up for resubmission of proposals to NSF, with 36% saying they spent more than 150 hours on the revised versions.

A typical written comment was this complaint. “It is very discouraging that NSF makes it so difficult to obtain grants for longer than three years or to obtain significant budget increases for a single grant. The amount of time that I spend writing proposals is getting to be a larger fraction of the available research time. NSF is not keeping up with inflation as far as grant size is concerned and is not making the best possible use of researcher time as far as grant duration is concerned. The three-year grant cycle makes it harder to pursue the most difficult projects or attempt the most innovative approaches." When NSF cut another PI’s grant term to two years, they wrote “I’m going to have to write more proposals, which takes time away from the research." (See Part III D.4.)

The single most recurring concern about the inefficiency of NSF grants, is the amount of time spent writing proposals, which they claimed they would otherwise spend doing research.

G Comparisons to NIH

One reason for the survey was the chance that NSF PIs find NIH’s average grant of $300,000 per year and average term of over 4 years more efficient. Appendix B reprints all the PI comments for which permissions were given that referred in any way to NIH in answer to this first written open-ended question. This is to give readers a first hand understanding of how NSF PIs perceive NIH.

For example, a PI whose grant duration had been cut to 3 years from 4, wrote that as a result, “one of the postdoctoral fellows who had planned to come in year 3 from Australia, may not participate in the work. I had hoped to guarantee 2 years of support for that position in order to get this person to my lab.

“As a result, I have had to go to NIH and request funding for a different project, to try and make up the difference in funding level and length of award. …. I guess the summary is that even for a small lab, it is difficult to survive with a single NSF grant, and the 3 year cycle is not long enough to allow full development of the work in time to write the next renewal."

But the NSF PIs who were switching to NIH– or said they were under pressure to do so – were ambivalent about the change.

When NSF’s net to one PI was “tiny" due to their institution’s 110% overhead rate, they wrote: “I am therefore under pressure (both economic and from my director) to redirect my research grants to NIH. I do not want to do this as I have valued my NSF grant very much as it has enabled me to pursue the more imaginative aspects of my research program over the last 15 years of continuous funding from NSF. If the funding level at NSF does not increase in future, I shall be forced to terminate my relationship with NSF (as will other recipients at my institution)."

Another: “As NSF grants get smaller compared to, say, NIH or DOD grants, it is harder to justify putting as much time into these projects as one would if they were fully funded. In fact, in … it is hard to find time to spend on fundamental science, when it is so hard to generate funds to support it. So one feels pressure to spend less time on NSF-supported projects."

H Praise for NSF

Whatever concerns and frustrations the PI displayed in answer to this and later questions, their support for the agency’s program and unique value came through.

For example, one PI whose goals were reduced drastically, wrote nonetheless, “the reduced scope of the funded project allowed us to submit another proposal with expanded and non overlapping scope. The initial funding, though reduced, made possible pilot investigations to commence and to establish credibility. Furthermore, this experience helped us to sharpen our approaches and goals."

Part II

Second Written Question - Question 3.8

The survey’s second written question, Question 3.8, asked:

“Please describe any other impact(s) that would result if NSF provided you with what you need for what you want to accomplish, or give more details on any in the list that needs further explanation."

In the preceding multiple choice parts of this question (3.1 to 3.7) the PIs had been asked how much additional funding they would like over the next five years from all sources “to achieve what would like to with your ongoing body of research and educational activities." After providing a figure, the PIs were to rank as “positive," “none," or “negative" each of 18 possible impacts. Some of their written answers to Question 3.8 gave details of impacts from the NSF list.

Question 3.8 gave the PIs the chance to say generally what they would do “if NSF provided you with what you need." From the written answers, it seems that this part of the question triggered the unusually high response rate: 47% of those eligible to answer the question did so.

The largest fraction of PIs who wrote comments said, if they could have what they needed, they would attack their scientific problems much more broadly; they would reach directly for the big advances and contributions that are the goals of their present NSF research.

Many said or implied that with added support they could reach their goals more efficiently, perhaps sooner. A number said more secure funding would free up time now spent writing shorter term proposals and allow them to focus more on their own research. One implication was that quality of NSF funded research could be improved.

Many would use additional support to mentor more students and other junior personnel. As PIs explained in answer to the first question, when they have more students they get more research done (see Part I D). Answering this question, they see mentoring of more students as part of expanding the effort that would help them tackle broader research goals.

A striking number said they would use additional funds to include more minorities in their research activities. This was related to risk-taking; some explained that if funding were less tight they would be freer to choose from a wider pool.

The time relieved from proposal-writing they hoped to spend in equal measure mentoring junior personnel and working directly on research. Several said that added support would help them integrate research into their teaching.

Many said they would use the funds for greater collaboration. Instrumentation was also mentioned. Supporting the theme that PIs feel that NSF does not do enough to help them communicate their research (see Part I), many said they would use the added support to improve dissemination and web site development.

Several wrote, as many had answering the first question, how NSF’s present funding levels and grant duration limits the scope and risk-taking of their research. The work they propose for NSF support tends to be low-risk, they would rather propose higher-risk/higher-yield projects. (See also Part I E.4 and Part III A and B.)

The PI who wrote eloquently that NSF’s system of integrated research and education is a “jewel" and “the envy of the world"(see Part I A and Appendix A) was answering this second question. This PI also wrote the system is in “danger of failing" because it had been “tweaked" too much and “the resources being invested … are below what we need." Like many others, this PI wants to “increase grant sizes and funding balance across the portfolio of scientific interests served by the NSF," including “longer grant periods" to “rationalize" the system.

A Added Funds - Priorities

A.1 Scientific Visions

Mathematica’s report contains how much more funding the PIs said they needed, and the list of effects of such funding, based on the multiple choice answers to Question 3.1 to 3.7. The largest fraction of their written answers say that if they had the funds they need, they would attack scientific problems on a broader scope and reach directly for the big advances that are the goals of their careers.

Their exciting scientific visions can only be glimpsed through the brief descriptions. But since the public supports the cause of keeping the United States in the lead in many fields of science, it is useful to sample these impressive goals. The 5,989 answering these questions are, after all, among the finest scientists and engineers in the nation and the world.

“My field is Microbial Biology and we are doing ecological genomics. NSF-sized budgets are simply not big enough to do this type of work, so we have to scrounge for support from everywhere. We don’t have the equivalent of NIH available to us, thus we cannot run the large efficient labs that those in the biomedical community can. Raising funding is almost a full time job for me," said one.

Another said their work would have “important bearing on the tectonics of Asia as a whole" and will “help identify the ancient environments that were responsible for the formation of the gold and rare metal mineralization in the Trans-Baikal region."

A self-identified solar physicist hopes “to make the connection between the direction of the coronal magnetic field and the distribution of density in the corona. These goals could readily be achieved if NSF provided the funding we had originally requested. The funding would also lead to innovative techniques in studying the corona…."

“This is a very big project to survey galaxies in the distant universe. The NSF grant does not provide enough money to thoroughly analyze the data. It is also not large enough to allow us to develop the data archive we would like to prepare in order to share the data (which are unique) with the rest of the astronomical community…." wrote another.

Biomedical imaging could experience “a substantial impact" in the view of the PI who sought “efficient computational methods for improving images in three dimensional fluorescence microscopy …."

The relationship between genotype and phenotype is a long-standing mystery, but one PI is working with a postdoc on “the genetic correlation (tradeoff) between horns and eyes in beetles" in “the most rigorous attempt yet to quantify the impact of a developmental ‘constraint’ on trait evolution." The work would produce “a very big step in the link to fill in the ‘black box’ between genotype and phenotype!"

Another PI wants to “[a]dvance the knowledge of a new class of plant proteins (Ribosome-Inactivating Proteins-RIPs), and its integration into easy to use technologies." They hope to “[I]ncrease our understanding on the effect of these proteins in translation … available for humans, yeast, and Drosophilia. …Currently there are just three research groups (including mine) in the US that studies this interesting group of proteins…."

A PI interprets early climate and evolution to better understand the present distribution of species. A pollen database is used “as a tool to provide improved paleoclimatic and paleoecological interpretations of Amazonian and Andean datasets. These data need to be integrated into a larger initiative that merges climatology, human landuse studies, biology and education…."

Another wants “eventually" funded “a high energy physics experiment which will probe one of the most fundamental questions in the field. It could possibly have the effect of changing the standard model of particle physics. The impact of doing so would stimulate much theoretical and experimental activity throughout the field of elementary particle physics."

One PI wants to “[c]onvert nanoscience into nanotechnology for societal needs and define the emerging academic discipline of nanoengineering, including nanomanuracturing and nano processing."

An economist wrote: “We now know that the flow of information explains in part explains in part ‘irrational exuberance’ in markets. But …. I would like to explore other ideas such as: ….’Exactly what kind (lack of) information creates irrational exuberance and what information flow stabilizes markets?’"

“The genetics & genomics of plant resistance to insects … are overlooked but emerging areas.…a more comprehensive idea of the genomics of arthropod and disease pest resistance in general….." wrote another.

“The longer term impact to the global agriculture community potentially would be very high if the research led to…understanding…the …mechanism underlying apomixis [asexual reproduction in plants]."

“The results could lead to an environmentally safe technology to destruct harmful, toxic halogenated organic compounds," wrote another PI.

“Langmuir circulations occur in the upper ocean and lower atmosphere as long rolls of fluid which align with the wind; they greatly enhance the transfer of carbon dioxide from the atmosphere into the ocean. The intent of my work is to credibly model them. The work ultimately will impact our ability to predict, and to some degree control, global change via carbon dioxide."

Impacts on the Economy, Health

This sample shows how some PIs’ work could impact innovation, the economy, and health. Nanoengineering and nanomanfuacturing are very important to future US competitiveness, for example. Understanding the interrelationships between climate, landuse, and biology offers keys to humanity’s response to climate change. Carbon dioxide is a chief culprit in global warming; predicting how oceans absorb it is key to modeling climate change. Three-D florescence microscopy can have a big impact on biomedicine. NSF grants support much of the nation’s basic research in economics; they also support crucial basic research in computing and software not done by the private sector. So while most NSF research advances basic understanding with applications in the long term, some has near-term payoff.

Another benefit of giving the PIs what they need, some said, would be that the pace of US discovery would be stepped up in fields where other nations are closing in or surpassing us. Part III G takes up some PIs’ concerns that the United States is lagging internationally because of inefficiencies in the NSF research support system.

Some mentioned that longer term, more secure funding would improve quality. One PI wrote: "Anything NSF does that helps me spend more time doing science, thinking, and educating students to thin[k] and pursue their own ideas, will help produce higher quality science and scientists."

Additional funding would help some publish more, said one who was concerned that conflicts on their time had lowered production of papers from about 3-4 average in the five years they were supported by NIH, to 1 to 2 per year because now, "when I write, I have to stop the lab work."

A.2 Support More Students

More funds over a longer term would allow PIs to expand their mentoring of the next generation, many said. Some indicated it would improve the quality of junior personnel, a reminder that many PIs believe they are in a competitive market for the best talent.

For example, one PI wrote that more funds “would allow me to expand my current research group in a controlled manner, meaning I would be much more selective with respect to whom I chose to join my group. This would promote and enhance the quality of graduate education in this country." A self-identified senior physicist wrote that with added funds, “I could return to educating the very best post-docs in my field. I could take on the type of lon[g] range, innovative work that a senior physicist should lead." A significant increase in funding would enable “a more aggressive recruitment of quality graduate students," wrote another.

An engineer expressed the particular need to find PhD students, instead of MS students, to do more advanced research (see Part I D.2). “The opportunity to attract more PhD students … would in turn increase productivity, given that they stay longer than the MS students that predominate graduate programs in engineering."

Some PIs would use the funds to strengthen the educational aspect of their projects, rather than in effect, using students as mere employees. One PI would use added funds to “give even a stronger focus on the education process… . [V]ery often compromises are made to meet program/grant expectations that take away from the best possible educational path for a PhD student. For example, me just telling someone what to do to resolve a specific problem rather than letting them work through it."

Several PIs would use the funds to improve the undergraduate mentoring. One PI, self-identified as at “a small college," wrote: “Funding student research assistants is critical to both my research program and my teaching (as I incorporate my research findings and materials into the classroom)." Another wanted to “engage students in cutting edge research earlier in their collegiate experience….to enable sophomore students to work alongside graduate students and ….to integrate research experiences and findings into both undergraduate and graduate coursework."

A.3 Development of More Minority Scientists and Engineers

A striking number indicated they would use the added funds to enlarge the pool of minority scientists and engineers. “It is impossible to achieve worthwhile social goals (such as increasing underrepresented communities, enhance teaching, outreach to public) when the grants must be no-frills to get funded. This forces a researcher to do what has worked with who they know will work…." one wrote. Another wrote that if the had the funds they needed , they would bring a “ significant increase in the number of Native American students interested in the sciences…."

A PI self-identified as at a “young" university and a new laboratory, said “continued funding would mean a lot not only to my research but also for the morale of the university administration. Additionally this is a minority-institution trying to build a strong research university in an area which had no tradition. The NSF funds are helping a great deal in this regard."

A few commenters would expand the numbers of women, such as one whose “goal is simply to perform first rate research and to produce as many female physics PhD graduates as I possibly can, so that we can contribute to ‘changing the face of Physics.’"

A.4 Address Number of Students in the S& Education Pipeline

One argued it was an urgent national priority to use additional NSF grant funds to expand the ranks of scientists and engineers from the upcoming population of undergraduates. Over the next decade, one in three science and engineering faculty will retire, this PI said [most estimates place the fraction at one in four], while the ranks of undergraduates will swell.

“More than a third of the current faculty of the country will retire in the next 5 – 10 years. The enrollments of most public universities will go up by 15% over this same period of time. If we do not pay attention to and preserve the personnel development activities that our system of Federal research funding supports, we are going to be in very big trouble." (This is the “jewel" PI quoted in full Appendix A.)

A.5 More Cross Disciplinary and International Work

Some PIs specifically said that adequate funding would help them cross academic disciplines. For example, one wrote that more support “would give the ability to learn and pursue different disciplinary approaches to this research area (which lies at the boundary between several disciplines)."

But most PI comments concerning outreach said they wanted more collaborations with other institutions and internationally.

A striking comment, quoted here in full, shows how one PI sees globalization of training and collaboration as potentially speeding up scientific advance. "Given the sophistication of my area of research, I need to collaborate with several other groups to obtain the highest quality level of results. It would be extremely useful to have supplemental funds, or even a separate grants program, for collaborative research that spans different universities and different countries.

"An example is my last publication: five different countries are represented. My graduate student was effectively educated in the laboratories of three countries. A collaborative supplement or a collaborative grant would allow for the funding of a student who is quite literally 'shared' among a number of different laboratories.

"Although it is true that the costs can be shared, the benefit to US science would be enormous were there monies for this type of student. In effect, information flow would be toward the US rather than from the US.

Another wrote, “[a]dditional funding would provide for more collaborative contact and post field work analyses with Russian partners. T]here is value in bringing once isolated scientists in to the broad and open science arena…." In other words, U.S. science would benefit if it could draw more on formerly isolated talent abroad.

Another PI quoted in full in Appendix A, stressed competitiveness. Larger grant awards would allow them to “concentrate their efforts to make better progress over shorter frame of time. This change will allow US scientists to better compete with EU groups where a full professor is usually given fundings [sic] to support a 10-20 person group with minimal amount of paperwork." To this PI, the European Union’s support for science seemed more efficient (see Part I E.3 and Part III G.)

A.6 Dissemination, Web Sites, Distribution

Other forms of outreach, especially via the Internet, require findings and sometimes raw data to be prepared for distribution. This work is not cutting edge basic research, yet it is critical to achieve synergism among researchers. Several PIs wanted additional funds to complete their projects by making the results available by software and/or internet dissemination.

With added funding “the information stored in our collection could be made many times more accessible to researchers…." said one. Availability “will facilitate not only basic research, but innovative research involving meta-data and synthesis of existing research…. It will also help in our efforts to communicate ‘science’ to students and the general public."

"The principal impacts of these studies are to provide baseline data to other scientists on when climate changed, how much it changed, and what influence that had on natural and human communities…" wrote another.

Some mentioned dissemination to students, including elementary and secondary students known as K-12. For example one PI would use added funds for “creating and distributing software that is user friendly to collaborators, industry, and to K12 teaching institutions."

A.7 Instrumentation, Infrastructure

The relatively small size of NSF grant awards were not enough for some instrumentation needs. An experimentalist wrote: “As the state-of-the-art of instrumentation improves…the funds needed for instrumentation is far more than typically feasible with the current size of the grants." Also the NSF’s special announcements of awards for instrumentation monies “do not have the success rate needed to sustain genuine and well justified needs of the creative and active scientists."

Besides equipment, this PI’s field needed a “substantial infusion of funds" for “crystal growers, technical staff to maintain sophisticated equipment, and so on." Another wrote: “Access to new equipment would render our laboratory more competitive especially when compared to Asian or European laboratories where the technology infrastructure has bloomed recently."

Another PI wrote the added support could “potentially set up a nationwide network of 10-12 other universities using similar apparatus to share research developments."

B PIs Limit Scope and Risk of Proposed Research

Though this second question asked how they would be helped by having the resources needed from NSF, many PIs nonetheless wrote of the problems of present NSF grant duration and amounts – their theme in answering the first and third questions. Many wrote they follow, in effect, a slice-and-dice strategy that nibbles at the bigger scientific issues. (See also Part I E.4.)

B.1 “Failing is not an option"

PIs do not propose high risk aspects of their work to NSF, according to one PI. “I would not gamble the future funding of this program by focussing on high risk efforts. The Foundation must recognize that a baseline level of funding of ongoing programs will never lead to innovation, it will only provide enough for self preservation at renewal time….

“[T]his proposal was a case in point of how frustrating it is to move in new directions with NSF money. This was an accomplishment based renewal which I felt demonstrated clearly that we made excellent progress in the past granting period. Half of the proposal outlined how we will continue….However, we recognized opportunities in new directions that were the centerpiece of the application.

“Unfortunately, when the program officer can provide only can provide a 10% increase over the previous period (my general experience with NSF over the past 20 years) I cannot justify risking the project on the most uncertain directions."

Wrote another: “….More stable and longer term funding would provide a base from which to investigate innovative methods, and would permit the opportunity to fail. In the current environment, failing is not an option…." Another wrote: " ….A more secure source of funding would allow us to spend more time doing what we are paid to do. And we would take more risks in terms of expanding beyond our current area of expertise, and developing broader collaborations."

B.2 Time Writing Proposals v. Time On Research

The inefficiency of frequent proposal writing takes time away from research, many PIs insisted, as they had in answering the first question (see Part I F). This complaint directly answers OMB’s original question to NSF (see Introduction) and may be the most frequent comment PIs made in answering the three written questions in the survey.

Many proffered individual calculations of how long they took writing a proposal and how many proposals they must submit to yield enough income to support their work and students. They stressed that this took time away from research. Typical was the one who estimated “4-6 weeks of research time to put together a proposal."

Many PIs said they carve up their big aspirations into smaller parts that are sure of success in three years -- or sooner, because they lose time waiting for the funds and getting started. The critical PI quoted above (also in Appendix A) stressed such inefficiencies.

“I must write about 1 proposal every 3 weeks to procure enough fundings [sic] to support a group of 15 students and postdoc at a private research university because each proposal only funds 1- 1.5 students and my success rate, as a junior faculty member, is only about 12%. On average I spend about 30h a week dealing with the mechanics of proposal writings and submissions. This, however, amounts to only 30% of my time + overtime because I must work 100h/wk." After ticking off how the balance of the 100 hours is consumed in other chores, the PI went on, “rarely do I have the time to think about science and explore new ideas."

Another PI said that “If NSF would just give me the funds I need to run my group and do my research" they “could finally focus" on it “instead of spending 50% of my time for writing likely declined proposals."

Another version of this message was: “[T]he funding duration standard is [so] low (3 years) and the competition for funds so high (which causes underbidding because your livelihood depends on receipt of funds), it is difficult to put enough time into these complex problems to complete them efficiently. The result is a cycle of proposal writing, report writing and spreading one’s efforts, that ultimately create inefficiencies overall in the research process."

B.3 Loss of Continuity Creates Inefficiencies

Several PIs said they need to propose something new each time they approach NSF so that earlier work breaks off. One PI participated in a “coordinated field project/experiment in climate change“ that showed “exceptional potential for advancing the climate-change science for the benefit of mankind." They needed three years to finish post-experiment analysis, but the NSF cut funds for their proposed second year by half; and they had not even asked for the third year. This “is detrimental to my efficient relationship with this project…."

Another wrote NSF it does not provide the kind of long time continuity that enables me to go really deeply into a subject; Each new NSF grant is, to a large degree, a new grant."

“Starting up and shutting down research in response to changing funding is very detrimental to long-term research goals," a PI answering the third question, “These horrible gaps in funding where we’re trying to stretch out the last grant by penny pinching until we get the next one" create more inefficiency than grant size. This was why Canadian scientists who are “pretty much assured of continuous funding at …about $50 K per year in my field….are beating us to the questions – often coming up with answers before we get funded to do the very same thing." (See Appendix A.)

B.4 Preferred Award Size, Duration

Award Size

"NSF money is prestigious, but the grants are small relative to other sources," wrote a PI, echoing a common view. Those PIs who addressed grant size mainly said they should support a reasonable number of students and postdocs. A typical comment was this from a self-identified theoretical physicist. Researchers in their specialty “COMPLETELY rely on the NSF funding….If one is allowed to dream, the ideal NSF grant for a senior, well established theorist in their 40’s would be for 5-6 years and at the level of about $140K per year of REAL spending money. “ The reference to “real spending money" is to the overhead taken by each PI’s institution from the NSF grant total, discussed in the Mathematica report and in Part I C.

“This is what is needed to restart the engines of the US enterprise in basic natural sciences," this PI continued. “We are not biologists." As theoretical physicists“we do not need 1M $$$ grants. What we need is the ability to hire a postdoc or two, pay two or three graduate students, get the latest computer box, and travel to several key conferences/workshops a year…."

Duration - Five Years Preferred

Five years was preferred by those who mentioned an ideal grant duration. (See Part III H.) One explained: “Research always discovers new information, applications, and educational opportunities which cannot be addressed within the traditional [3 year] time constraint. Five year grants or two year (minimal effort) extensions would also allow for continuity in staff and more effective (among) grad student knowledge transfer. The working body of knowledge thus increases exponentially and not linearly."

NSF Should Follow NIH’s Terms

Here and later PIs referred to the tradeoff between assured long term support, which some said at some other agencies leads to stagnation, and the NSF system of intense, frequent recompetition for small short term grants. The NIH was the only federal agency whom the PIs wanted NSF to emulate. (See Part I G and Appendix B.)

Generally, PIs view NIH’s balance between longer term stability v. recompetition as more efficient and productive. “Even under a constant funding level you will see greatly increased productivity if the resources are granted with a longer time horizon (compare the longer NIH grant periods with ours)," wrote one.

Rising costs was a factor for many who wanted NSF to drastically increase award size and duration. Wrote one: NEW 22916 “Research has become very expensive over the past years. Therefore the limited size of NSF budgets compared to other granting agencies, e.g. NIH, makes it difficult to pursue a more extensive research program. also the hiring of qualified personnel becomes a problem. Graduate stipends along with tuition have also increased…."

C Other Impacts - Stress

Personal stress could be relieved by more secure funding, and less time writing proposals to hedge against the loss of funds. An interesting comment was the PI who wrote: “Five-year funding would allow me to move further from ‘safe science’. It would also decrease the stress in my life." Another wrote: “You have hit a tender spot with this questionnaire. I have watched closely as my most exciting colleagues, and I myself, have been stressed to the limit by the frantic process of funding and pursuing innovative research. We are losing young scientists and potential scientists who see this happening.

“The US is still in a unique position among modern nations, in that the people still believe in basic research, and the people in the agencies are advocates and often skilled scientists themselves.

“I am so lucky to have been funded by the NSF for many years, yet my 2 to 3 year grants are accompanied by continuous stress."

D Unique Value of NSF Grants

Many PIs singled out what was most valuable to them about the NSF grants program, values they want to preserve if any changes were made.

“NSF is the preferred funding agency for my research. It offers the simplest grant application and project administration procedures,“ wrote one.

“I certainly value the funding I DO have from NSF. “ wrote another. “It is wonderful ‘few-strings-attached’ funding that has supported all of the exciting things we have done in my lab."

“Having the NSF support high risk, innovative ideas allows the PIs to pursue knowledge in a way that positively impacts their creativity and they learn how to think outside the box. I very much value my NSF support and have leveraged my fundamental findings to secure additional monies that put our findings to practical use.“

The value to the nation of NSF grants for basic research in technology were stressed by PIs in this area. One who works on “aging wiring" wrote:

“We are working on a new method for detecting dangerous conditions in aging wiring (aircraft, nuclear power plants, large machinery, communication networks, etc.) which has been identified as an area of critical national concern. The funding provided by NSF so far has been the only place that I could try highly innovative (and slightly risky) technical advances." Other funding from government and industry has been for “simpler" development. But “NSF funding provides the ‘next generation of technologies" that are needed to take this project to optimal conclusion. It is amazingly difficult to get funding for this basic research any other way."

A good summary was this comment. “NSF, in my view, constitutes the premier funding source for high-quality university research. An increase in NSF funding will enable focusing brain–power for long term basic research, which is nowdays scattered in many directions because PIs want to have sufficient resources to carry out their research goals; e.g. PIs with ability to carry out fundamental research that should be funded by NSF [now] have to seek extra funds from other government agencies or industrial sources that of course some with other constraints that result in waste of brain power and time."

Part III

Third Written Question - Question 5.14

The third written question, Question 5.14, asked:

“Questionnaires by their nature are limited. Please write in any other comments you have about your experience with the NSF grant process that you think are important."

This question elicited the largest number of answers. Fully 2,521 PIs wrote responses, or half of the total. Printout of these remarks runs the longest of the three written questions, at 208 pages.

So 2,521 of the nation’s leading scientists and engineers sounded off about the good and the less good concerning the nation’s premier basic research agency and their circumstances. The agency would do well to mine these comments for feedback concerning individual programs, staff members, details of award timing and procedures and other aspects of its grants program.

What follows are samples from this rich lode that seem most relevant to the OMB-NSF-Mathematica study of the efficiency of the NSF grants program.

The first important theme is the discouragement of many PIs who wrote – rather, who assumed – that resources from NSF grants are shrinking. [This perception is striking because the average NSF grant award has increased in size in recent years.]

A second theme is the PIs’ view that younger investigators have a harder time winning NSF grants. Many wrote that major institutions or old-boy networks control distribution and edge out more innovative proposals new investigators and institutions. [This perception is also striking because it is not true overall: the success rates of PIs with prior NSF grants has been trending down while new applicants, who are mostly younger, have held their own.]

Though the PIs often praised NSF, their program directors, there were criticisms of panel reviews, how NSF handles multidisciplinary proposals and big- ticket proposals; both are believed to fare less well. As in earlier answers, PIs complained they spend too much time preparing proposals and revising them to meet reviewer’s demands. Many wrote that with 5-year instead of 3-year grants, they would run this gantlet less often and so spend more time doing research.

Some mentioned specific NSF practices they said hinder their efficacy, such as on summer salaries, overhead, rotator staff and others.

Yet the PIs gave high marks to the agency, most staff and the integrated system of teaching and research that the NSF grants program sustains. There were also concerns whether the best students are joining NSF projects, and about the quality of the research when fewer risks are taken. Here as earlier a few comments showed real stress, such as one who wrote that the process “makes me wonder if this is the best way to spend my time."

The summary ends with some of the PIs’ suggestions for change, including warnings about the move to longer grant duration and larger grant size favored by so many.

A Perception of Shrinking Resources and Lengthening Odds

Not a single comment in answer to these or earlier written questions mentioned the fact that the total NSF grant programs has risen in the past 3 years. [It remained in the $1.2 - $1.3 billion range from FY1992-FY1996; by FY1999 it had reached $1.6 billion. In FY2000 it rose to $1.8 billion and in FY 2001 to $2.1 billion. FY2001 was the year of the grant awards which the PIs describe in the survey. ]

[The size of the average annual grant bottomed out below $80,000 in FY1995. By FY2001 it had marched upward to $113,000. As mentioned, the mean duration rose to 2.9 years from 2.8 the year before, as the agency tried to award at least some grants for longer terms.

[Overall, a PI’s chances of getting funded by NSF’s grants program have gone up slightly from the early 1990s. In the 3-year period FY1992-FY1994, 42% of the PIs who proposed got funded; by FY1999-2001, 43.5% of PIs who proposed got funded. On a per-proposal basis, 29.5% of competitive proposals won grant awards from NSF in FY1992-FY1994; by FY1999-FY2001 29% of such proposals were funded. So the overall odds have stayed the same. Proposals to the comparable NIH program (RO1) have had a 31% success rate since 1998.

[New PIs – i.e. those who had not had prior NSF grants – held their own; their success rate ranged from 17% to 20% during the 1990s; in FY2001 19% of new PIs proposing to NSF were successful.

[The only group of PIs whose success rates have gone down are those who had prior year grants from NSF. In FY1992 41% of this group won new grant awards; by FY2001 only 32% did, according to NSF budget officials.]

Yet a widespread perception runs through these written comments that that the situation with NSF funding is getting worse. Reading some of them, one could imagine one is in the early 1970s when NSF’s resources shrank, setting off a feeding frenzy in which too many worthy proposals chased too few funds.

“In a downturn in resources for scientific research one could envision the scenario…where established research teams would find themselves spending more time writing proposals due to reduced probability of funding spending more time reviewing colleagues’ proposals (since they are caught in a similar bind) and thus less and less time spent in the lab and the field…making the discoveries…which is what we are supposed to be doing," one PI wrote.

Another wrote: “Having spoken with older, well respected scientists, I have been told that the competition for funds was considerably less fierce starting about 15 years ago, due to fewer scientists in the field. I am not sure this is true, per capita. Perhaps it is more likely that the standard of living for scientists has increased along with the rest of America, but the funding has not. As a result, to cover one’s research time within the relatively smaller budget standards, more proposals must be written to different groups…."

The explanation may be that the rise in NSF grant size has not kept pace with the PIs’ rising costs of doing research. A PI who said they had been supported by NSF for 15 years wrote: “[M]y yearly budget now is almost the same as it was 15 years ago! The money available has not been nearly enough to keep up with inflation. This has continually reduced the amount of work that I can do on the project over the years."

Another wrote:"The typical amount of award has remained practically unchanged in the last 10 years. Tuition and research assistant salaries have gone up. [I]n our institution graduate student salary went up by 50% over the past 8 years. At the same time, cost sharing from the university has significantly decreased. 2-PI proposals are practically impossible. The biggest impact is on the quality of graduate students. This makes our mission of educating quality graduate students through research – nearly impossible."

“[A] larger body of researchers requesting funds and a relatively flat budget" was making the “success ratio" too low, wrote another PI. “In the areas that I compete in it can be as low as 10% which makes me wonder if this is the best way to spend my time." Another wrote, “some people, after being rejected several times in a row, just give up. Much scientific talent has probably been sidelined."

B Perception Young and New Are Squeezed by Established Networks

“Many researchers, mainly young ones, are frustrated by the system and, after three or four rejections, give up on NSF," wrote one PI. Some said in some divisions the success rate was 1:15. Wrote another: “I almost shudder at the lack of encouragement that NSF has been able to offer the newer generation. Setting aside a certain portion of the NSF annual budget to ‘take care’ of minority applicants was permitted .…but what about a starting age/inexperience factor in setting aside a % of the annual budget?"

The corollary of the belief that young investigators face too-long odds is the belief that established networks have a lock on funds. One reported “in some parts of NSF…. the institution you are and the people you know often have more to do with funding decisions than do the quality of the science and the educational impact of a given proposal." Another wrote: “It would be worth a serious effort to reduce the level of cronyism and increase objectivity in the NSF process. I think the worst offenses are where a ‘community’ group gets NSF to earmark funds for a program which is their hobby horse – usually mediocre science."

[As mentioned, neither of these perceptions is borne out by the facts of the overall grant program. Investigators with prior NSF grants are the only group becoming less successful in competing for funds; these were 32% successful in FY2001 but had been 41% successful in FY1992. New PIs who have not had prior NSF grant awards, most of whom are within 7 years of the year of their PhD, have had the same success rate of one in five for many years.]

The agency might look at what PIs did before they received that first grant award. Some PIs told of submitting repeatedly, with excellent reviews, before the award came through. Aside from inefficiency in putting off research that was timely much earlier, such cycles of rejection and resubmission could be taken by the would-be grantee as “lack of encouragement."

C Small v. Large Institutions.

Another concern was, as one wrote, “that a few already well-heeled Universities and Institutions have the lion’s share of the NSF (and DOE and NIH) pie." Another PI wrote the “long term health" of mathematics required “that funding is not overly concentrated in the major institutions. …. Otherwise I’m afraid that the life blood will be circulating among a few institutions and will become ingrown. “ PIs from small institutions sometimes claimed they had difficulty breaking in.

But others insisted the agency had rewarded new ideas. One wrote: “The NSF staff recognized our innovation and provided long term support for myself and my lab members that significantly fostered my career." The work had had "repercussions in other fields" and “enabled me to mentor many students," several of which were minorities, they wrote. A number of undergraduate-only institution faculty members and those from minority institutions expressed gratitude for NSF grant support.

D Peer Review

Many PIs praised the NSF’s fairness. “My experience with NSF for the past 25 years allows me to say that the program directors have been doing excellent jobs and the grant process is very fair." Another PI was a good loser: “I have had very good experiences wit the NSF ... division, where I believed that the process was always fair – even when I did not get a grant funded."

There were concerns that some reviewers don’t take enough care. An otherwise satisfied PI wrote that “certain (not all) reviewers’ comments which revealed a lack of attention and/or elementary ability to relate to the ideas of another person."

“I find the review process frustrating. I know reviewers must remain [anonymous] but a single misunderstanding on their part, omission of a reference that should have been there in their opinion, and many hundreds of hours of work is sunk…" one wrote. Another wrote:

“[T]he peer evaluation system which has worked so well for a number of years since the inception of the NSF does not appear to be functioning well in recent years because of the prevailing low success-to-failure ratios. Any constructive comments offered in proposal reviews may be taken as criticisms and may result in failure of proposals.

“Some reviewers are overcritical and pick on minor points. A review board often has a hard time on the scientific basis in deciding where to draw the line between the pass and failure. Because of the highly limited budget at NSF, the line inevitably falls between proposals of similar scientific merits. This is discouraging for young as well as old scientists alike. Hence, rather than trying out a new idea, they tend to prepare for more conservative themes rather than innovative ones."

“I sense that NSF support has become much more competitive, and that many worthy proposals by top notch investigators must be submitted multiple times to be funded. A ‘queuing-up’ mentality seems to have crept into the funding process and I think this is deadly for innovative high risk research." said another PI. 20983 This one also wrote:

“[R]eviews have become really mean-spirited and nit-picky. I edit a major journal and process at least 2000 reviews each year. I rarely encounter reviews as negative as the ones I routinely receive from NSF on my own proposals. I believe in peer review, but when you get the same review time and again, its time for the program manager to intervene. I have discussed this with program managers who say the difference is money."

The churn of frequent submission, and resubmission to satisfy earlier rounds of reviewing, sparked further complaints that too much time was spent writing proposals. One can be representative. “The odds of getting funded are 20% in my area according to the numbers of submissions versus funded proposals. It is so competitive that one has to spend many weeks or months preparing a proposal. My last proposal had 12 pages of references so that I didn’t leave anyone out who worked in any area that I was covering.

“If a grant only funds one grad student project (seems to be the general rule) and one has 6 students[, t]hen it is possible to have to write 30 proposals to get them all funded. Fortunately my numbers are better than the 20% average. But still there is so much wasted time in this process. Can’t we have bigger grants for 4 and 5 years, so we can fund 2 or 3 students at once[?]"

D.1 Multidisciplinary research

“It can be difficult to obtain funding for interdisciplinary research," wrote one. Others said that reviewers for multidisciplinary projects were too focused on their own specialities.

D.2 Large proposals suffer

“[R]eviewers can be overly influenced by the budget of the grant they are reviewing." because “they know how small the total pool of funding is" said one PI. It should be noted that in answer to this and earlier questions, some PIs said they don’t propose durations of more than three years or more than standard amounts, since the odds of success are so long.

E. NSF Staff

The PIs wrote many testimonials to the helpfulness of NSF program officers in advising them during the proposal preparation, review and decision process. Many praised relationships with individual staff that had helped them over 15 or 20 years. But some other PIs complained they could not get through to staff or that staff were not helpful were rare. But many said their relationships with staff were critical to getting through the review and award gantlet; “if people don’t contact the NSF grant officer, the probability that they will have sufficient information to get an award is zero," wrote one. The Mathematica report quantifies the fraction of PIs who said they got lots of help from NSF staff, some help, and very little help.

The ideal NSF program officer, as described in the PIs’ written answers, is proactive and able to make things happen, by advising how the PI could change a proposal and facilitating arrangements once an award is made.

Having "rotators" from academia was praised as bringing fresh outlooks to the program offices. But an interesting critique of rotators was that “they lack the influence or contacts of permanent staff, thus putting their programs at a disadvantage…to programs administered by permanent staff."

“[I]n my opinion, the program officers have declined in quality, and have become more passive and more limited to proposal ‘logistics.’ They are less inclined to provide any overt leadership to the panels and to the science in general."

Another PI wrote: "[T]hese positions are staffed with quite passive individuals who rarely emerge from Washington except for national meetings." And another: “[I]t seems that program directors are overwhelmed. [T]he review process is somewhat ad hoc since there seem to be too many panels to participate in and the program directors are the ones who have to put it all together."

F. Core v. Newsy

Some wrote that, in an environment of too few funds and too many proposals, reviewers and staff prefer new research to proposals to continue work in core fields. Several criticized NSF for letting “fads" take precedence over more solid work – an echo of the need for more continuity that PIs mentioned answering the second question (see Part II B.3). Some PIs wrote they found it easier to get the initial grant to fund a research project than to get the follow on one. For example, one wrote: “NSF continues to be trendy in its funding. For example, while the ITR program is a good idea, the regular base programs have been neglected as a consequence.“

G. International Links

The NSF’s administration can mine these comments for feedback concerning many details of operations such as the Fastlane electronic proposal system, the CAREER program, and many specific program areas, such as the ocean drilling and the Antarctic programs.

In light of PIs’ earlier stated need for more collaboration, travel, and dissemination (see Part I E.3 and Part II A.6), their feedback about NSF’s international support is relevant. As usual, their comments are not completely consistent because individual experiences vary. Still, most who mentioned the international dimension wrote that NSF did not support it to the extent they needed to accomplish their best and retain U.S. world leadership in an era of globalized science.

One complained that drawn-out cycles of proposal resubmission and “change of reviewers every round" delayed their doing the work. It becomes frustrating for us …reinventing the wheel every time and also losing precious time to competitors from EU [European Union]. This has hurt us tremendously."

“[T]he review process needs improvement, I believe, because many of the reviewers are not fully aware of the leading edge research, especially what is happening in other major research countries, Europe and Japan," wrote another who was otherwise pleased with NSF peer review.

H. What NSF Should Do

The survey’s third written question asked for “comments you have about your experience with the NSF grant process that you think are important." It did not ask for solutions to the problems the PIs described. But it was clear from these and earlier written answers that many PIs have thought long and hard about alternatives.

One of the most striking results of Mathematica’s tally of the PIs’ multiple choice answers is their broad support for larger average award sizes, for more awards, and for grant awards of longer duration. The PIs supported these changes no matter what their experience was with their individual grant in FY2001. For example, 32% of those who had their FY2001 award increased said NSF should increase the size of awards; so did 44% of those who had their FY2001 awards decreased. Almost half, or 47%, of those who got increases of 5% or greater favored first of all increasing the number of awards NSF makes. And 40% of the PIs who got funded up to 5% of what they asked voted that NSF should increase the number of grant awards. Thus Mathematica’s multiple choice data, as well as these written responses suggests that NSF’s present dollar amounts, number of awards and durations of NSF grants do not seem “right" to its clientèle, the PIs.

H.1 Longer Grants, Simpler Renewals

Throughout the survey, the PIs generally preferred grant award lengths of 4 and 5 years, to fit with their graduate student’s timetables to get the PhD. Just as important, many wrote, was that five years could relieve their churn of constant proposal-writing (see Part II B.4). They also wanted average awards of $140,000 or higher, or NIH’s average of $300,000, so a single grant could support a research team. The Mathematica report spells out the financial implications.

An alternative would be for NSF to make present grants more easily extended. “Longer grant periods for established researchers with a track record of continued success is a good trend," wrote one. Another wrote: “If a project has a track record, say, averaging 10 papers per year, with solid contributions to developing human resources, then that ought to be worth more than treading water." The agency’s Accomplishment Based Renewal program was too limited, this investigator said.

Some raised cautions about 5 year terms. One noted there would be less money for new proposals if NSF directed more of the total to the added two years – though this result presumes no proportional increase in the program total. Another cautions was that scientists have difficulty planning beyond 3 years anyhow, so it may not be wise for reviewers and NSF to trust that the funds will be well spent in years 4 and 5. Some PIs argued that what would most relieve stress was a higher success rate for all proposers, no matter how achieved.

But increasing the success rates requires an increase in the total funds for grants at NSF. Though many assumed overall funding would not change much, the few who addressed this wanted NSF grants program to be funded much higher than at present.

H.2 Would Larger Awards Mean Fewer Share the Pie?

Only a few grappled with the implication that, if more NSF grants went to 5 year terms, and the average grant were $140,000 or $300,000, NSF most likely would support fewer projects.

One warned that this could create an even more demoralizing situation than at present, by shutting out more hopefuls. This PI wrote that morale had been “terrible" for the past decade “largely because it has been incredibly difficult to get grants and to keep good, important projects going." But “more money for longer" will “further depress morale…..

“It will create the same sort of ‘inheritance of wealth’ scheme…[T]hose who have will continue to get because they have the resources to insure their competitiveness. I am not sure that this will be good for our scientific progress overall, because I don’t believe that the top 50% of grant applicants are doing anywhere near 25% of the excellent research in my field."

This PI said that by “casting the net more broadly" instead “we increase our chances of capturing the scientists who will come up with something truly innovative and exciting, ensure funding for people who do really solid, important work that endures (where often the flashy, newsworthy science doesn't) and we'll be able to compete internationally on a broader scale." This PI saw the US falling behind Canadians and Europeans who are assured of stable funding over long periods, albeit at a low level of $50,000 per year. “They are beating us to the questions – often coming up with answers before we get funded to do the very same thing. As being first is often regarded as important, they win."

Another PI who preferred more grants to fewer larger ones, hoped this would change the “disconcerting" fact of “my brilliant and hard working former advisers stop getting any funding from NSF in the middle of their careers…."

H.3 Release Time for Writing, Summer Salaries

Here as in earlier answers, several PIs wrote that they wanted NSF to support time released from teaching to write books based on their NSF research. Some asked NSF make its rule allocating funds for summer salaries more flexible. (See Part I D.5.)

H.4 Peer Review

PIs also suggested changes in NSF’s merit review system. One wrote that NSF should “take the money spent on panels and use it instead to hire better program people, and give them greater scope to utilize the mail reviews in a way that furthers excellence in science and avoids being sidetracked by banal panel dynamics."

“I think NSF should seriously consider getting short 2 page letters of intent and then trying to do an initial assessment of the probable importance and success so the community is not spending more time and monies on trying to get grants than is really available," proposed another PI.

Another proposed a two-tier review system. The first panel review would be the same as now. Then NSF staff would examine the scores and writeups. “[I]f there is …inconsistency among primary reviewers, they can retain the proposal for a second tier review." The second tier “consists of a much smaller group of senior scientists who will review a small number of proposals with conflicting critiques." They may revise the score based on the “quality" of the reviewers. “The revised score will from then on serve as the basis for the subsequent consideration of funding of the proposal."

Another wanted NSF to use the NERC system. Investigators should submit short 5 page proposals instead of 15 page ones. They would offer PIs the chance to comment on reviews before they are seen by the review panel. This would fix the problem that, “[f]ar too often I have had funding rejected based on reviews that were ill informed about the work proposed."

Others asked for a regular appeal process. “We must be given the chance to offer a rebuttal to a frivolous review without resubmission and another six months of waiting."

H.5 Expand NSF’s Grants Budget

Many PIs seem to believe that the inefficiencies of NSF’s present grants program can be solved if the total pot of funds is larger. “[I]t is important that NSF be given the resources to fund larger numbers of qualified researchers," wrote one.

Two comments summarize the message of many PIs who wrote answers in this survey.

“…I think NSF is the greatest. But the size of the individual grants is just too small to really accomplish anything significant in my field… 200K per year will get you one post-doc and one student, and some materials and supplies and that is it. So you need at least three of these grants to run a decent sized group. This is a lot of grant-writing. I wish the NSF budget could be multiplied by 10. Then we might have something to work with."

“Clearly more money helps, but it is the TYPE of funding that NSF provides that is unique. So many other sources (including federal agencies) have a short-term focus on technological or economic deliverables. More NSF funds opens up a spectrum of possibilities: working on new ideas, allowing students to drive some of the research, building infrastructure and a base from which more funds can be obtained from other sources, more freedom to focus o[n] graduate and undergraduate educational goals rather than ‘research’ goals. The NSF is different than industry, and it needs to stay that way; and it needs to be bigger part of academic funding."

APPENDIX A

From Answers to Second Question – Question 3.8

“I believe the thing we must keep in mind here is that there really is a genius that underlies the integrated system of teaching and research that we operate in this country. It is a jewel beyond value and something that is the envy of the world in terms of the positive returns it brings at so many levels. The point to focus on here is that, in the physical sciences, University research is first and foremost an essential foundation of the process and functioning of education. It is the singular most important thing that supports the system of instructional delivery in our research Universities.

“ There is no doubt that the technology, scholarship, and knowledge-base benefits are real and of great value to the nation as well. They are not of the same gravity, though, as those that derive from the tremendous synergies we build into the education mission by having it so closely coupled to a preeminent, world-class research enterprise. This is the take home message—research is education! We bring into University at the graduate level, the absolute best undergraduates trained in the scientific and engineering disciplines. These students form an essential and irreplaceable component of the instructional team in every science and engineering department. In my University, for example, we teach more than 11,500 ‘instructional units’ in Chemistry (in non-jargon terms, that is an equivalent number of people taking courses in chemistry during the fall and spring terms). That it is possible to do this, to provide this level of service to the benefit of the children of the diverse citizenry of the state of Illinois, is utterly FANTASTIC! There is no way that this staggering involvement with instructional delivery could be achieved with this sort of scale, productivity, and efficiency of cost without the synergies that we derive from the research components of our mission.

Research lies at heart of how we do graduate education in the physical sciences. We bring the best students in the country (and world for that matter) to this involvement. We develop in them skills and envelop them in training experiences that, by the time they leave, place them at the very top of the national talent pool. These highly motivated young people, as an essential part of their graduate training, make enormous and irreplaceable contributions to our undergraduate instructional programs in the form of their service as instructors and teaching assistants. This is a comprehensive system of training in the disciplines that works very well. It is the envy of the world and the NSF is the essential agency of national reach that preserves and nurtures it. We fool with this only at our peril.

The sad news is that we have more than played with this system. We have tweaked and fooled with it to a point that it is now in grave danger of failing. The resources being invested in it are below what we need. Clearly the needs to increase grant sizes and funding balance across the portfolio of scientific interests served by the NSF are pressing ones. I want to emphasize as well that longer grant periods will rationalize in a needed way the duration of a grant and the training periods of the students who actually perform the work. As it is now, it is a very stressful process and too much time is lost in business activities (the pursuit of ever declining resources) and not enough on scholarly efforts. There is a real danger here. More than a third of the current faculty of the country will retire in the next 5-10 years. The enrollments of most public universities will go up by more than 15% over this same period of time. If we do not pay attention to and preserve the personnel development activities that our system of Federal research funding supports, we are going to be in very big trouble."

Also from answers to Second Question – Question 3.8

“Larger amount of funding/grant over a shorter time period will allow PIs to hire more qualified workers and to concentrate their efforts to make better progress over a shorter frame of time. This change will allow US scientists to better compete with EU groups where a full professor is usually given fundings to support a 10-20 person group with minimal amount of paperwork.

“As an example, I must write about 1 proposal every 3 weeks to procure enough fundings to support a group of 15 students and postdoc at a private research university because each proposal only funds 1-1.5 students and my success rate, as a junior faculty member, is only about 12%. On average, I spend about 30h a week dealing with the mechanics of proposal writings and submissions. This, however, amounts to only 30% of my time + overtime because I must work about 100 h/wk. In addition to all the bureaucratic craps (pls. pardon the language) that the university asks me to do (15 h/week), I must put in the extra 55 h/week on teaching (25h on teaching undergraduate), directing research in my group (15h, 1-1.5 h/each student[,] and fulfilling professional obligations (15h/wk on proposal review and manuscript review) if I want to do a good job on all of these tasks. Rarely do I have the time to think about science and to explore new ideas. The only time that I had to reflect on science is when I am in the taxi or on the airplane going somewhere.

“The system of "sharing the wealth" that NSF seems to be moving toward is not working. Not all university should have a Ph.D. program and not all professors should have a research group. It seems that there are quite a few people out there who continue to get fundings only because they are established and have political clouts even though they are still doing the same thing they did 40 years ago. Not only that they drain the resources from the system, they actually poison the new generation of scientists by taking in enthusiastic youngsters, teaching them outdated science, and transfering their complacency attitude towards these kids. I have been on Ph.D. committees where the candidate shouldn't have gotten the degree if he/she was in my own group.

“At the same time, there are young people who start out in academic [careers] as wonderful teacher and researchers with new ideas but cannot get fundings. I have a friend who is probably the best teacher and researcher at his university department who couldn't get any fundings after 5 yrs so he quit to go into industry. In contrast there is an old timer in his department who continues to get NSF fundings but has no student. This person managed to fool the system by publishing old data and laundering the money for his personal uses. There is a very interesting article by Greg Petsko in The Scientist (2002, 16(1, Jan 7), 72) that states exactly this problem but more eloquently than I can.

“Science should only be done by someone who will put his heart and soul into it, not by people who only hang on, waiting to retire. The more NSF continues to support those latter people, the worse will US Science become."

From Answers to Third Question - Question 5.14

“My answers should be interpreted with the understanding that I am a 35 yr old, 100% soft-money researcher with a family including two young children. Thus there is considerable pressure to perform across a spectrum of responsibilities while adjusting to them at the same time. This, of course, is just the situation I happen to be in now, and many have been in it before. This situation has caused me to consider the research process with some care, especially it's inefficiencies or ineffectual parts.

“Having spoken with older, well-respected scientists, I have been told that the competition for funds was considerably less fierce starting about 15 years ago, due to fewer scientists in the field. I am not sure this is true, per capita. Perhaps it is more likely that the standard of living for scientists has increased along with the rest of America, but the funding has not. As a result, to cover one's research time within the relatively smaller budget standards, more proposals must be written to different groups. The result is more time managing budgets and writing proposals. The latter factor should not be underestimated.

“For a soft money researcher needing to cover 12 months of his/her time and current budget standards pushing for 3 mos (or less) coverage for senior personnel per proposal (seemingly, this is only communicated obliquely) and an average of 3 year research duration this results in obtaining 4 funded proposals per 3 years. This would be a limited time sink if success was 100% (~ 4 weeks/year of proposal writing at 3 weeks/proposal). At 50% it is 8 weeks/yr writing proposals. At 33% success it is 12 weeks/yr and so on. It seems that a better balance can be found. Consider that many programs have funding rates of 10-33% (you tell me, I only hear these numbers, perhaps they are worst case).

“As a soft money researcher, I think having but one main project is too few, but having 4 main projects is too many, so the ideal is 2 or 3 (as opposed to side projects), which would require 4-6 months coverage for a PI per successful proposal. Using a salary of 70k/year for a senior person and a 2.25 multiplier on salary, the senior person, at 4 months salary is roughly 60k/year in the budget. This leaves about 30-40k or so for other personnel (post-docs, students, technicians), computers, travel and publications, etc, using the typical 90-100k/yr average for proposed work. With post-doc salaries around 40k one can see that the PI is resigned to 1) risking a higher-cost proposal that has a higher chance of failure (again, I think), or 2) reducing his number of man-months covered, if he/she wishes to support staff/students. Either choice results in more proposals and spreading oneself thinner.

“Personally, I feel I am able to maintain a good standard for my work, but projects take longer and in fact the quality and standard of the results could be still better if less time was spent on proposals and the dollar amounts were higher allowing for more focus on fewer projects.

“Another important issue is that many newer PIs are relatively savvy with word processing, figure preparation as well as their research activities. As a result, many of the activities that would be completed by assistants, are now completed by the PI resulting in less research time."

Also from Answers to the Third Question – Question 5.14

“The rumored budgetary rule of thumb--one RA, one PC, and one month of summer salary--is not enough to sustain a research program or promote scientific education. It is a strain to maintain an apprenticeship style of training in which senior and junior graduate students overlap on a project (and impossible to incorporate post doctoral associates). Since 1986, I have had 9 NSF grants. More funding each time would have made for a better educational environment.

“Last year's unfunded mandate that more money be devoted to children's research was misguided on several fronts. Children's research is funded amply at NICHD. The required funding was taken from an area--basic human perception and cognition--that already has precious little. One area that was spared, cognitive neuroscience, is one that again has funding opportunities at NIH.

“Accomplishment Based Renewal opportunities are too limited, both in frequency and level. If a project has a track record, say, averaging 10 papers per year, with solid contributions to developing human resources (e.g., dissertation research that appears in the top journals, student senior-authored publications, etc.), then that ought to be worth more than treading water.

“These complaints should be put in the context of what NSF does that remains special. It supports pure research and does not demand that there must be a payoff in medical or economic terms. For the ordinary scientist, pure research is always the royal road to understanding and NSF is often the only place to go if one wants funding to travel that royal road."

“Finally, with regard to the questionnaire, it would have been good to allow changes to answers; and the question on publications: I have coauthored 85 papers in the past 5 years. For team research, and with students taking leading roles, the question of how many first authorships is inappropriate."

APPENDIX B

Quotes in which Principal Investigators mention the National Institutes of Health(NIH) in answer to First Question – Question 2.5 *

--- “The decrease in funding forced my seeking additional support. Because this grant is for plant science, there are limited opportunities. NIH does not support this type of work. I received a 1-year grant (with a second year possible) from Monsanto Plant Science that enabled my hiring a technician to do work on the hormone promoter DNA binding project."

--- “The new NIH standards for graduate and post-doctoral salaries cannot possibly be supported on the severely trimmed budget. I am in a region of the country that is not geographically desirable and so we must offer competitive salaries if we are to have any hope of attracting top notch people. This becomes increasingly difficult with small NSF awards."

--- “The total average award for the funding of molecular and biochemical research from NSF to my knowledge averages about $100,00[0]. For the same kin[d] and level of research, the average award size at NIH is $325,000. I recognize the … award sizes have been kept at a lower level to support more investigators.

“In today’s research, we are trying to do state of the art research, ie ‘rocketship type laboratory’ with rotating ‘bicycle mechanics’. [T]his is the analogy that I use when I explain to folks about why I have to work 60 to 80 hrs a week, why I subsidize my research with my own personal funds and the difficulties in maintaining a state of the art research program. NSF needs to recognize that modern biological research takes more funding than 10 years ago and requires long term personnel to help maintain these sophisticated laboratories. This is also very important for the training of students.

“In addition, not only should the awards actually pay for the research that is being requested, the award periods in many cases should be 5 years and not 3 years. “

--- “Since I was funded for 3 years instead of 4, this means that one of the postdoctoral fellows, who had planned to come in year 3 from Australia, may not participate in the work. I had hoped to guarantee 2 years of support for that position in order to get this person to my lab.

“As a result, I have had to go to NIH and request funding for a different project, to try and make up the difference in funding level and length of award. That grant is pending. I guess the summary is that even for a small lab, it is difficult to survive with a single NSF grant, and the 3 year cycle is not long enough to allow full development of the work in time to write the next renewal."

--- “As new faculty, I appreciate very much the timely support from NSF. I hope that NSF will keep its tradition in nurturing researchers and teachers of the next generation. Unfortunately, the NIH does not have such a wisdom yet. An informal survey among new faculty members suggests that 80% of time are generally spent writing and rewriting NI[H] proposals. How could these new faculty have time to build a new lab, develop new courses, and do research? I salute you all at the NSF!

“I realize that the NSF budget is not big. A small increase in the amount of award would be helpful. This is needed because the salaries (including students’ stipends) nowadays are much higher than in the past. Yet the NSF limit of award remains the same level over many years.

“On the other hand, I also understand that Congress was not generous enough towards NSF. Under such conditions, if NSF wants to give support to as many young people as possible, I fully support it."

--- “As our indirect cost rate is 110% on salaries the effective award was tiny. I am therefore under pressure (both economic and from my director) to redirect my research grants to NIH. I do not want to do this as I have valued my NSF grant very much as it has enabled me to pursue the more imaginative aspects of my research program over the last 15 years of continuous funding from NSF. If the funding level at NSF does not increase in the future, I shall be forced to terminate my relationship with NSF (as will other recipients at my institution)."

--- “As NSF grants get smaller compared to, for example, NIH or DOD grants, it is harder to justify putting as much time into these projects as one would if they were fully funded. In fact, in this day and age it is hard to find time to spend on fundamental science when it is so hard to generate funds to support it. So one feels pressure to spend less time on NSF-supported projects."

--- “The drastic decrease in budget and duration required that I take out several parts of the proposal which I then incorporated into another grant proposal to the NIH."

--- “The submitted budget was very modest in keeping with my perception that NSF funds very small biology grants. The additional reduction in budget makes the award less than a third of a comparable biology grant funded by the NIH."

--- “My interpretation of the results of the handling of this application is that it was judged to be an important and innovative proposal in an areas supported by NSF, but one that can be translated to areas supported by NIH. The reduced support and reduced time frame were based on the premise that it would provide the required support to fully develop a proposal that, if the innovative aspects were demonstrated would be funded by NIH, therefor[e] it was supported more as a demonstration project rather than a fully funded NSF proposal.

“Therefor[e] we are pursuing the most innovative and collaborative aspects of the work with the anticipation of submitting the ‘renewal’ to the NIH and not to NSF…."

--- “We have had to scale down our research program. This has put us at a great disadvantage in our competition with other laboratories in the U.S. (especially those funded by NIH) and in for[ei]gn countries that are able to maintain research programs that are funded at considerabl[y] higher levels for longer periods of time. “

###