This document has been archived.
I n the late 1980s, biology department faculty at Western Washington University feared that their hopes of launching a dramatically upgraded undergraduate marine science curriculum might have run aground. The institution had acquired the nearby Shannon Point Marine Center&-a potentially useful laboratory facility situated on Bellingham Bay just south of the Canadian border. But it lacked the financial resources to refurbish the structure, purchase new equipment, and develop the ambitious academic and research programs needed to attract topnotch instructors and the body of new students that would move the effort forward.
Project leaders requested initial funding support from a number of sources, including the Marine Facilities Development program of the National Science Foundation (NSF), but found that the NSF program does not provide startup funds for the acquisition of new laboratory equipment. Another NSF initiative, however-the Instrumentation and Laboratory Improvement (ILI) program-provided the seed money needed by the department to build a foundation of academic and research excellence that subsequently enabled Shannon Point faculty to break through and gain access to other, larger sources of development funds.
Relying heavily on the capabilities afforded by ILI-funded equipment, the center eventually reapplied to NSF's Marine Facilities Development program, and this time was successful in obtaining significant support. Since 1990, the Shannon Point project has attracted more than $1 million in facilities and program support from a variety of NSF programs and has brought in nearly $2 million from other external funding sources, including the National Oceanographic and Atmospheric Administration.
Today, Shannon Point Marine Center is a nationally recognized hub of marine science scholarship and laboratory investigation, attracting undergraduates from throughout the United States. Sophisticated equipment provided by the ILI grant has, for example, enabled the program to give every student hands-on experience in the use of radioisotopic and fluorometric techniques for observing such phenomena as the flow of energy through ecosystems. And every student, thanks to an ILI-purchased scintillation counter, is involved in laboratory exercises revealing the power and pitfalls of the techniques that this instrument supports.
Significant as well are impacts beyond Shannon Point's walls and its own student body. It is, for example, the core of a consortium involving two other universities and three community colleges in the state of Washington, a cooperative undertaking that enables students from other institutions to avail themselves of the center's programs and facilities. Also, a number of outreach programs, designed to meet the needs of younger students in the area, are seeing Shannon Point staff members give marine science presentations in local elementary and high schools. Frequently, these presentations are made by Shannon Point's minority student-scientists in the hope that they will serve as role models for younger minority students.
The jump-start that ILI gave to the Shannon Point Marine Center was invaluable, says its director, Dr. Stephen Sulkin. "It revolutionized the center," he says. "It validated what we were trying to do."
Established by NSF in 1985-as the College Science Instrumentation Program (CSIP)-ILI has awarded more than 4,700 seed-money grants, totaling approximately $160 million, to nearly 1,200 different institutions during its first decade. Through a highly competitive, annual, peer-reviewed grant competition, ILI provides funds for the purchase of laboratory equipment to be used in projects that improve undergraduate education in science, mathematics, engineering, and technology (SMET). Funded projects are expected to produce models for the improvement of undergraduate laboratory instruction nationally, as well as at the project site. Thus, ILI principal investigators (PIs) are expected to make conscientious efforts to share with others the innovative educational approaches and materials they develop in their projects.
Grants are in the $5,000-$100,000 range and must be matched with non-federal funds; what's more, the grant and matching funds must be used solely for purchase of instructional laboratory equipment. Since grantees are required to at least match ILI funding dollar for dollar, the program has stimulated investment of over $320 million in the acquisition of undergraduate laboratory equipment.
ILI is operated by the Division of Undergraduate Education (DUE)-a branch of NSF's Directorate for Education and Human Resources (EHR)-and is designed to complement other programs aimed toward enhancing the quality of instruction in colleges and universities, including two-year as well as four-year institutions.
The fundamental objective of the ILI program, as expressed by NSF, centers on the improvement of undergraduate education:
to support the development of experiments and laboratory curricula which improve the science, mathematics, engineering, and technology (SMET) education of undergraduate students, both SMET majors and non-majors. An important target audience is the students who are preparing to be K-12 teachers. Proposals are sought for the development of innovative methods for using laboratory exercises to improve student under-standing of basic scientific principles. These methods often involve the use of modern instrumentation, new technologies, or new applications that extend the instructional capacity of the equipment².
Implicitly, however, the program has several other goals relating to the projects and personnel at recipient institutions:
Furthermore, the ILI program has since its early days increasingly sought to broaden its focus beyond the support of improvements in the quality of instruction at individual institutions, with a growing interest in innovative curriculum-development projects that seek to elevate the level of undergraduate science, mathematics, engineering, and technology education nationwide.
According to the current DUE program announcement,³ "The ILI program seeks projects that will improve laboratory instruction nationally as well as at the project site and that will produce and adapt national models for the improvement of undergraduate laboratory instruction." Accordingly, program administrators discourage proposals that request equipment solely on the basis of an institution's peculiar financial need or increased enrollments.
Also, ILI has throughout its
existence encouraged proposals that involve provisions for equipment sharing
through consortia or those that would strengthen undergraduate education
by increasing the participation of women, underrepresented minorities,
and people with disabilities, especially if the project presents models
for increasing the numbers of students who will pursue careers in science,
mathematics, engineering, or technology.
______________________________________________________________________
³Program Announcement and Guidelines: Division of
Undergraduate Education, Directorate for Education and Human Resource.
National Science Foundation,NSF 97-29, p.25.
That ILI, which attracts more than 2,000 proposals annually and awards about $20 million per year in grants, is one of NSF's most visible programs is beyond question. The extent to which the program is succeeding in achieving its goals, however, is a most appropriate question.
In the late 1980s, an evaluation of ILI's first three years-based on mail surveys of grantees and applicants, and evaluator visits to selected project sites-indicated that the program was, indeed, making strides. The early grants appeared to be paying off in achieving their intended objectives and, in many cases, of generating spinoff projects and benefits of substantial value. Projects were providing students at grantee institutions previously unavailable opportunities for valuable hands-on learning activities, and ILI projects were viewed as being responsible for increased enrollments in SMET courses and degree programs and increased interest in related fields. Moreover, the funded projects were frequently said to be responsible for reinvigorating faculty morale and enthusiasm for teaching. Without ILI support, many awardees said, projects would have been significantly scaled back or not implemented at all, a finding verified in the survey of non-funded applicants.
Since this initial evaluation,
the ILI program has grown dramatically and matured considerably, now affecting
a much larger number and broader variety of institutions and individuals.
Recognizing the need to examine the expanded program and determine the
extent to which ILI is succeeding-or failing, if that is the case-in its
mission, NSF launched in 1994 a full-scale evaluation of the project, the
results of which are the focus of this report.
Table
of Contents
As with all NSF program evaluations, the ILI study was primarily guided by three fundamental questions: Is the program achieving its goals? Is it making an impact? And are there ways in which it can be improved? Thus, the agency's Directorate for Education and Human Resources initiated the study as a means for Foundation administrators to gain insights concerning the overall status of the program, and in response to a congressional mandate. With passage of the Government Performance and Results Act of 1993, accountability based on evaluation of a program's consistent results is no longer merely desirable-it is the law. With federal funding for NSF activities on the rise, EHR is required to account to Congress and the Administration for the effectiveness of its programmatic activities.
Currently, dozens of EHR programs, as well as a number of SMET projects that NSF shares with other government agencies, are under thorough review. A variety of assessments of programs in the NSF portfolio (approximately 30 in number) are to be completed within the next several years by third party, outside contractors working under the supervision of EHR's Division of Research, Evaluation and Communication (REC).
The ILI evaluation was carried out by Westat, Inc., a Rockville, Maryland, research and consulting firm that has conducted many other federal program evaluations and research studies focusing on postsecondary education. A six-person team of science, mathematics, engineering, and technology educators who are knowledgeable about the ILI program was appointed to advise Westat in the design and conduct of the project and in the assessment of findings4 (Exhibit 1).
|
|
|
|
| Biology: Eric Thomas, Vice President of Academic Affairs, Mississippi Valley State University, Itta Bena, Mississippi |
| Chemistry: Thomas C. Farrar, Professor of Physical Chemistry, University of Wisconsin, Madison, Wisconsin |
| Engineering: Eleanor Baum, Dean of Engineering, Cooper Union, New York, New York |
| Engineering Technology: George A. Timblin, Head, Department of Engineering and Advanced Technology, Central Piedmont Community College, Charlotte, North Carolina |
| Mathematics: Anita Solow, Dean of Academic Affairs, DePauw University, Greencastle, Indiana |
| Physics: James Parks, Director of Undergraduate Laboratories, University of Tennessee, Knoxville, Tennessee |
The evaluators gathered the bulk of quantitative and qualitative information for the ILI evaluation during the 1995-96 academic year. Generally, the data collection was designed to shed light on whether the program has been successful in achieving its objectives during the past decade overall and how well it is meeting the needs of specific categories of undergraduate institutions-that is, two-year colleges and four-year institutions that grant either associate, baccalaureate, or graduate degrees.
The evaluation centered around the program's scope and coverage; its educational and research impacts at grantee institutions and on the larger academic, social, and commercial communities beyond those institutions; the effect of the program on individual ILI grant applicants, both successful and unsuccessful; and the extent to which the program is impacting K-12 students and teachers and how successful it has been in serving women, members of underrepresented minorities, and people with disabilities.
Throughout, the evaluators also sought to identify ways in which the program might be improved.
To address comprehensively these and other issues, the evaluation involved several information-gathering activities designed to provide both a broad and deep look at the extent to which the program is achieving its objectives (Exhibit 2).
Database Review of NSF ILI Proposals and Awards (1985-94). A thorough examination was made of existing NSF data concerning all ILI proposals and awards during the program's first decade (1985-94) to describe the program's scope and coverage.
Mail Survey of ILI Grantees (1990 and 1992). A mail survey was conducted of all ILI grantees from the 1990 and 1992 cycles of the program to gather information on issues related to implementation and impact5. These two programs cycles were selected as being generally typical of the numbers and kinds of ILI projects that have been supported in recent years while of an age that would allow adequate appraisal of their impacts both within and beyond the grantee institutions.
_______________________________________________________________
5For all but one of the 1,136 ILI grantees
in 1990 and 1992, we were able to find a person who is currently responsible
for the ILI project, to whom the survey questionnaire could be directed.This
was usually, but not always, the original PI.The response rate was 95 percent
for 1992 and essentially the same(94 percent) for 1990 grantees.
Site Visits to Exemplary Projects (1990 and 1992). Nineteen site visits were carried out to institutions housing ILI projects that were identified as having been exemplary in one or more ways, based on information in the survey questionnaires. Selection criteria include evidence of wide-ranging curriculum impacts, substantial spinoff projects, expanded educational and research programs, extensive dissemination of project-generated materials, and/or extensive community outreach. Site visit trips were conducted jointly by a Westat evaluator and by a member of the team of disciplinary experts whose field of expertise matched the project under investigation.
Site Visits to ILI-Rich Institutions (1985-94). Six additional visits were conducted to institutions that had received an especially large number of ILI awards (10 or more) to assess the cumulative effects of multiple ILI projects on the quality, reputation, and growth of programs at such institutions. These visits were also conducted jointly by a Westat evaluator and by a member of the team of disciplinary experts.
Tracer Studies (1985-94). A series of 15 tracer studies were conducted to document the impacts of selected projects that were identified from the full portfolio of ILI awards by NSF program staff or by questionnaire responses as achieving significant impacts beyond the initiating/host institution. The intent of the tracer studies is not to generalize but to challenge assumptions that the impacts of ILI are limited to the grantee institutions. In the studies, selected recipients of disseminated procedures and products were contacted through telephone or electronic mail to learn about the uses they have made of these materials and about the impacts that have occurred.
Mail Survey of Unsuccessful Applicants (1990 and 1992). Finally, a survey was conducted of unsuccessful ILI applicants from the 1990 and 1992 program cycles (250 from each year) to obtain their views about the program and its impacts on them.6
The findings from these complementary
information-gathering techniques are presented in Chapter
2. In light of the fact that ILI is seen as providing seed money, the
results are organized around three major themes:
A) Planting the Seeds: The
Program's Scope and Coverage
B) Harvesting the Yield: Direct Program Impacts at Grantee Institutions
C) Propagating to New Areas: Far-Reaching Impacts
Selected vignettes, which appear as sidebars throughout the report, illustrate findings from site visits to exemplary projects.
The final chapter (Chapter 3) contains the evaluators' conclusions and recommendations.
_________________________________________________________________________
6From the approximately 3,000 unsuccessful
ILI applicants in 1990 and 1992, a systematic random sample of 500 were
selected (250 from each program year) for inclusion in the survey.We were
able to locate and contact 92 percent of the sampled unsuccessful applicants
from the 1992 program and 87 percent from the 1990 program. Among the unsuccessful
apllicants contaced,the survey cooperation rate was 95 percent in both
years.
| Table of Contents | Start of Chapter 1 |
| Start of Chapter 2 | Start of Chapter 3 |