A Report on an Evaluation of the
National Science Foundation's
Instrumentation and
Laboratory Improvement Program
Table of Contents
 
 
Prepared under Contract
RED 94-52966
 
by
Paul Tuss
Joy Frechtling
Tom Ewing
Westat, Inc.
Mary Sladek, Division of Research, Evaluation and Communication
Duncan McBride, Division of Undergraduate Education
NSF Program Officers
 
 
 
 
 
 
 
NOTE: Any opinions, findings, conclusions, or recommendations expressed in this report are those of the participants.
February 1998
 
 

Table of Contents

Acknowledgments

Executive Summary

  1. ILI Evaluation
  2. Study Findings

  3. A.    Planting the Seeds: The Program's Scope and Coverage
    B.    Harvesting the Yield: Direct Program Impacts at Grantee Institutions C.    Propagating to New Areas: Far-Reaching Program Impacts
  4. Conclusions and Suggestions

  5.  
    About NSF
List of Appendixes

A Reflection on the ILI Evaluation

List of Tables
  1. Representation of women and minorities among principal investigators, by field and institution type: 1985-94
  2. Percentage of PIs reporting various equipment uses: 1990 and 1992 grants
  3. Numbers of undergraduate student users of ILI project equipment, by time interval, year of award, and type of use: 1990 and 1992 grants
  4. Average number per ILI project of undergraduate students using the project's equipment, by institutional setting and type of use: 1990 and 1992 grants
  5. Representation of women and minorities among ILI impacted undergraduates, by field and institution type: 1990 and 1992
  6. Financial and resource investments made by grantee institutions to support ILI projects: 1990 grants
  7. External sources of financial and resource investments to support ILI projects: 1990 grants
  8. Comparison of per-project benefits when PIs receive summer salaries or release time: 1990 grants
  9. Percentage of PIs employing various dissemination mechanisms, and average number of dissemination activities per award: 1990 and 1992 grants
  10. Forms of dissemination assessed in the tracer studies and the number of people impacted
List of Exhibits
  1. The Team of Disciplinary Experts
  2. Information-gathering activities
  3. Variables examined
  4. Project life cycles
List of Figures
  1. Number of ILI awards, by program year: 1985-94
  2. ILI coverage of institutions, by highest degree offered in science, mathematics, engineering, and technology: 1985-94
  3. Type and extent of PI-reported ILI project impacts on undergraduates at grantee institutions: 1990 and 1992 grants
  4. Type and extent of PI-reported impacts on faculty involved on the ILI project: 1990 and 1992 grants
  5. Type and extent of PI-reported ILI project impacts on the department or the institution as a whole: 1990 grants
  6. Overview of dissemination recipients from 15 tracer study projects
  7. Impacts associated with dissemination activities from Tracer Study 3: "Applications of Lasers in Chemistry"
  8. Outcomes of unsuccessful proposals/ideas submitted to ILI in 1990 and 1992 (N=3,064)
____________________________________________________________________________________________________________________

Acknowledgments

T he present evaluation study of the National Science Foundation's Instrumentation and Laboratory Improvement (ILI) program could not have taken place without our many colleagues who have labored tirelessly to help create a clearly presented, well-organized, and thoughtfully analyzed report. We would especially like to acknowledge Ken Burgdorf, who developed the evaluation design and managed the bulk of data collection activities. We are also indebted to Diane Ward, who served as survey operations director and assisted in managing the overall data collection effort, and Philip Cardillo, who served as research assistant throughout all phases of the project. Our technical editor was Carol Litman. Our desktop publishing specialist was Sylvie Warren. Ken Burgdorf, Paul Tuss, Thomas P. Ryan, and Gabriel A. Massaro served as site visitors. Laure Sharp also made significant contributions in reviewing early drafts of the report.

We are grateful to the team of disciplinary experts who worked closely with the Westat evaluation team throughout the design and implementation stages: Eleanor Baum, Thomas C. Farrar, James E. Parks, Anita Solow, George A. Timblin, and Eric Thomas.

We must also give enormous credit to the ILI project directors who hosted site visits, completed surveys, and otherwise helped educate us about the rewards and challenges of developing and implementing an ILI project.

Finally, we wish to acknowledge the essential support, advice, and critiques we received from NSF staff, particularly Mary Sladek, who served as Project Officer on the evaluation study; Duncan McBride, the Director of the ILI program, and Norman Fortenberry, Conrad Katzenmeyer, and James Dietz, who reviewed several drafts of this report.
 
 
 
 

Executive Summary

T his report presents the results of an evaluation of the National Science Foundation's Instrumentation and Laboratory Improvement (ILI) program. The evaluation study looked at the educational and research impacts of ILI projects and the report offers examples of how specific projects are using cutting-edge laboratory equipment to improve under-graduate education in science, mathematics, engineering, and technology (SMET). Evaluation results center around the program's scope and coverage; its educational and research impacts at grantee institutions and on the larger academic, social, and commercial communities beyond those institutions; its effect on individual ILI grant applicants, both successful and unsuccessful; the extent to which it is impacting K-12 students and teachers; and how successful it has been in serving women, members of underrepresented minorities, and people with disabilities.

Overall, the evaluation results show a complex mix of accomplishments and limitations. The program has provided improved educational opportunities for large numbers of undergraduate students, especially in areas at the frontier of undergraduate science education. In addition, ILI grants were found to stimulate extensive resource leveraging indicating that the laboratory improvement projects are highly valued. At the same time, the results raised questions about the participation rates of two-year institutions, the nature of impacts on the faculty reward system, and the extent of impact beyond the grantee institutions.

The ILI evaluation was carried out by Westat, Inc., a Rockville, Maryland, research and consulting firm, with assistance from a six-person team of science, mathematics, engineering, and technology educators who are knowledgeable about the program.
 
 
 
 

The ILI Program

The ILI program is a competitive, peer-reviewed grant program that provides institutions with funds to acquire laboratory equipment for use in improving undergraduate educational programs and opportunities. Each year, the program receives approximately 2,000 proposals and grants about 550 awards totaling about $20 million.¹ Individual awards range from $5,000 to $100,000 and are 100 percent (or more) matched by the recipient institution. Over the first decade of program operations, ILI awarded 4,704 grants totaling $158.6 million to 1,185 different institutions.

Nearly all doctorate-granting colleges and universities with undergraduate SMET programs (95+ percent) submitted proposals to ILI during the first decade of the program, and most (86 percent) received one or more awards. Most other four-year colleges and universities with undergraduate science programs also submitted proposals to the program (75 percent), and a majority (55 percent) received one or more awards.

Levels of participation in ILI have been lower for two-year institutions, which were not eligible for the program during its first three years, than for four-year institutions: among two-year colleges that award associate's degrees in science-related disciplines, 40 percent submitted one or more proposals to ILI by 1994, and 19 percent received one or more awards. The funding rate for two-year institutions (27 percent) has been essentially the same as that for four-year institutions (26 percent).

Table of Contents

 
__________________________________________________________________
¹These figures pertain to the Instrumentation Projects(IP) component of the ILI program, which was the focus of this evaluation.
 
 

Direct Program Impacts at

Grantee Institutions

According to data collected with the grantee survey and in site visits, a number of significant impacts have occurred at the grantee institutions.

Some case studies of ILI projects from the evaluation report illustrate these impacts.  

Table of Contents

 
Far-Reaching Program Impacts

Evaluation results also demonstrated that the influence of the ILI program can extend beyond those individuals and organizations that are direct beneficiaries of ILI awards.

The evaluation report includes case studies illustrating these far-reaching impacts. Table of Contents

 

Conclusions and Suggestions

The results of this evaluation show that overall, the ILI program is achieving its mission to increase the range and quality of modern laboratory equipment and to provide equipment-based learning opportunities for undergraduate students at grantee institutions. The evaluation identified five key areas in which the program has made the greatest impact:

However, the types and strength of impacts vary for different categories of undergraduate institutions, for different SMET fields, and for different target audiences. The evaluation identified five main areas where the program fell short of its goals: The pattern of outcomes suggests a number of tradeoffs that NSF might want to consider in shaping the program for the future such as encouraging two-year institutions to submit more applications or sacrificing broad coverage of institutions for the synergistic outcomes observed in those institutions that have received multiple awards. Whatever changes are ultimately made, this evaluation provides evidence that the program's first decade of activity has been rich and fertile, and that the seed money provided by NSF has yielded value well beyond its initial investments.
 
 
 
 
 

Table of Contents