This document has been archived.
A Report on an Evaluation
of the
National Science Foundation's
Instrumentation
and
Laboratory
Improvement Program
Table
of Contents
Prepared under Contract
RED 94-52966
by
Paul Tuss
Joy Frechtling
Tom Ewing
Westat, Inc.
Mary Sladek, Division
of Research, Evaluation and Communication
Duncan McBride, Division
of Undergraduate Education
NSF Program Officers
NOTE: Any opinions, findings,
conclusions, or recommendations expressed in this report are those of the
participants.
February 1998
Table of Contents
Acknowledgments
Executive
Summary
-
ILI
Evaluation
-
Study
Findings
A.
Planting the Seeds: The Program's Scope and Coverage
B.
Harvesting the Yield: Direct Program Impacts at Grantee Institutions
C.
Propagating to New Areas: Far-Reaching Program Impacts
-
Conclusions
and Suggestions
About
NSF
List of Appendixes
A Reflection
on the ILI Evaluation
List of Tables
-
Representation
of women and minorities among principal investigators, by field and institution
type: 1985-94
-
Percentage
of PIs reporting various equipment uses: 1990 and 1992 grants
-
Numbers
of undergraduate student users of ILI project equipment, by time interval,
year of award, and type of use: 1990 and 1992 grants
-
Average
number per ILI project of undergraduate students using the project's equipment,
by institutional setting and type of use: 1990 and 1992 grants
-
Representation
of women and minorities among ILI impacted undergraduates, by field and
institution type: 1990 and 1992
-
Financial
and resource investments made by grantee institutions to support ILI projects:
1990 grants
-
External
sources of financial and resource investments to support ILI projects:
1990 grants
-
Comparison
of per-project benefits when PIs receive summer salaries or release time:
1990 grants
-
Percentage
of PIs employing various dissemination mechanisms, and average number of
dissemination activities per award: 1990 and 1992 grants
-
Forms
of dissemination assessed in the tracer studies and the number of people
impacted
List of Exhibits
-
The
Team of Disciplinary Experts
-
Information-gathering
activities
-
Variables
examined
-
Project
life cycles
List of Figures
-
Number
of ILI awards, by program year: 1985-94
-
ILI
coverage of institutions, by highest degree offered in science, mathematics,
engineering, and technology: 1985-94
-
Type
and extent of PI-reported ILI project impacts on undergraduates at grantee
institutions: 1990 and 1992 grants
-
Type
and extent of PI-reported impacts on faculty involved on the ILI project:
1990 and 1992 grants
-
Type
and extent of PI-reported ILI project impacts on the department or the
institution as a whole: 1990 grants
-
Overview
of dissemination recipients from 15 tracer study projects
-
Impacts
associated with dissemination activities from Tracer Study 3: "Applications
of Lasers in Chemistry"
-
Outcomes
of unsuccessful proposals/ideas submitted to ILI in 1990 and 1992 (N=3,064)
____________________________________________________________________________________________________________________
Acknowledgments
T he
present evaluation study of the National Science Foundation's Instrumentation
and Laboratory Improvement (ILI) program could not have taken place without
our many colleagues who have labored tirelessly to help create a clearly
presented, well-organized, and thoughtfully analyzed report. We would especially
like to acknowledge Ken Burgdorf, who developed the evaluation design and
managed the bulk of data collection activities. We are also indebted to
Diane Ward, who served as survey operations director and assisted in managing
the overall data collection effort, and Philip Cardillo, who served as
research assistant throughout all phases of the project. Our technical
editor was Carol Litman. Our desktop publishing specialist was Sylvie Warren.
Ken Burgdorf, Paul Tuss, Thomas P. Ryan, and Gabriel A. Massaro served
as site visitors. Laure Sharp also made significant contributions in reviewing
early drafts of the report.
We are grateful to the team
of disciplinary experts who worked closely with the Westat evaluation team
throughout the design and implementation stages: Eleanor Baum, Thomas C.
Farrar, James E. Parks, Anita Solow, George A. Timblin, and Eric Thomas.
We must also give enormous
credit to the ILI project directors who hosted site visits, completed surveys,
and otherwise helped educate us about the rewards and challenges of developing
and implementing an ILI project.
Finally, we wish to acknowledge
the essential support, advice, and critiques we received from NSF staff,
particularly Mary Sladek, who served as Project Officer on the evaluation
study; Duncan McBride, the Director of the ILI program, and Norman Fortenberry,
Conrad Katzenmeyer, and James Dietz, who reviewed several drafts of this
report.
Executive
Summary
T his
report presents the results of an evaluation of the National Science Foundation's
Instrumentation and Laboratory Improvement (ILI) program. The evaluation
study looked at the educational and research impacts of ILI projects and
the report offers examples of how specific projects are using cutting-edge
laboratory equipment to improve under-graduate education in science, mathematics,
engineering, and technology (SMET). Evaluation results center around the
program's scope and coverage; its educational and research impacts at grantee
institutions and on the larger academic, social, and commercial communities
beyond those institutions; its effect on individual ILI grant applicants,
both successful and unsuccessful; the extent to which it is impacting K-12
students and teachers; and how successful it has been in serving women,
members of underrepresented minorities, and people with disabilities.
Overall, the evaluation results
show a complex mix of accomplishments and limitations. The program has
provided improved educational opportunities for large numbers of undergraduate
students, especially in areas at the frontier of undergraduate science
education. In addition, ILI grants were found to stimulate extensive resource
leveraging indicating that the laboratory improvement projects are highly
valued. At the same time, the results raised questions about the participation
rates of two-year institutions, the nature of impacts on the faculty reward
system, and the extent of impact beyond the grantee institutions.
The ILI evaluation was carried
out by Westat, Inc., a Rockville, Maryland, research and consulting firm,
with assistance from a six-person team of science, mathematics, engineering,
and technology educators who are knowledgeable about the program.
The ILI
Program
The ILI program is a competitive,
peer-reviewed grant program that provides institutions with funds to acquire
laboratory equipment for use in improving undergraduate educational programs
and opportunities. Each year, the program receives approximately 2,000
proposals and grants about 550 awards totaling about $20 million.¹
Individual awards range from $5,000 to $100,000 and are 100 percent (or
more) matched by the recipient institution. Over the first decade of program
operations, ILI awarded 4,704 grants totaling $158.6 million to 1,185 different
institutions.
Nearly all doctorate-granting
colleges and universities with undergraduate SMET programs (95+ percent)
submitted proposals to ILI during the first decade of the program, and
most (86 percent) received one or more awards. Most other four-year colleges
and universities with undergraduate science programs also submitted proposals
to the program (75 percent), and a majority (55 percent) received one or
more awards.
Levels of participation in
ILI have been lower for two-year institutions, which were not eligible
for the program during its first three years, than for four-year institutions:
among two-year colleges that award associate's degrees in science-related
disciplines, 40 percent submitted one or more proposals to ILI by 1994,
and 19 percent received one or more awards. The funding rate for two-year
institutions (27 percent) has been essentially the same as that for four-year
institutions (26 percent).
Table of Contents
__________________________________________________________________
¹These figures pertain to the Instrumentation Projects(IP)
component of the ILI program, which was the focus of this evaluation.
Direct Program
Impacts at
Grantee Institutions
According to data collected
with the grantee survey and in site visits, a number of significant impacts
have occurred at the grantee institutions.
-
Revitalizing Undergraduate
Education. ILI-funded equipment is being used
to support the development of experiments and laboratory curricula that
improve the SMET education of undergraduate students, resulting in improved
courses and research opportunities. Principal investigators for (PIs) 98
percent of the 1990 and 1992 ILI grants reported using the equipment acquired
with the grants to reconfigure, expand, and update existing laboratory
courses. In addition, most PIs used ILI equipment to provide hands-on research
opportunities for undergraduate students (84 percent), and about half used
the equipment to either create new courses or degree programs.
-
Reaching Undergraduate Students.
Student usage of ILI equipment is high and extends over time. An average
of about 470 undergraduates per project received direct, hands-on use of
ILI equipment in their coursework during the first four years of operation.
Student use of equipment for research was far more limited, with an average
of 30 students per project using the equipment for that purpose in the
first four years. All but six of the 543 grantees from the 1992 program
cycle reported that their principal ILI-supported equipment was still being
used for undergraduate instruction (three years after their grants were
awarded), as did nearly all 1990 grantees (five years after grants were
awarded).
-
Serving the Needs of Women
and Minority Students. ILI program efforts are
compatible with NSF objectives to encourage more widespread participation
in the sciences by groups that have been historically under-represented.
Of the undergraduate hands-on users of ILI equipment, 46 percent were women
and 16 percent were underrepresented minorities. These figures are consistent
with the percentage of women in the overall population of undergraduate
students enrolled in SMET fields and slightly above the percentage of minorities
in the overall enrolled population.
-
Invigorating Faculty. A
majority of the PIs reported that their projects benefit faculty at the
grantee institution by improving instructional effectiveness and quality,
improving their capacity to teach additional scientific or technical courses
and topics, and improving morale and enthusiasm for teaching. On average,
five faculty members at the grantee institution used the ILI equipment
for teaching during a project's first four years, and two faculty members
used it for research. Impacts on the faculty reward system were negligible;
little or no impact was reported by half the PIs in the area of improved
prospects for tenure, promotion, and salary increases.
-
Stimulating Increased Investments.
In 1990, ILI grants averaged $34,000. This
seed money typically stimulated several kinds of investments, both in terms
of funds and reallocation of resources. On average, in the first five years
of operation these grants generated $140,000 of internal and external financial
investments, over four times the original grant amount.
-
Intensifying Institutional
Impacts Through Multiple Awards. Site visits to
institutions that have received an unusually large total number of awards
(10 or more) over the past decade found more comprehensive curricular reforms
being implemented and additional synergistic, campuswide effects that consistently
surpassed the impacts of individual projects.
Some case studies of ILI projects
from the evaluation report illustrate these impacts.
-
Humboldt State University.
A pair of ILI grants in 1985 and 1992 enabled Dr. William Allen to provide
his school with its first significant amount of instructional laboratory
equipment and facilities for cell and molecular biology study by undergraduates.
The school's biology depart-ment now offers undergraduate courses in cell
biology, genetics, and immunology-plus a graduate course in biotechnology.
As the result of the infusion of ILI-funded laboratory equipment, there
has been a doubling of students interested in cellular and molecular biology
at Humboldt State University over the past few years.
-
Bryn Mawr College.
An ILI grant allowed Dr. Rhonda Hughes to purchase the equipment necessary
for her to launch the new courses, Dynamical Systems and Chaos, and also
to start a summer mathematics program relying heavily on computers. According
to Dr. Hughes, the mathematics computer laboratory established with the
help of ILI is regarded as having stimulated a campuswide commitment to
the use of computers for undergraduate instruction in several departments.
The ILI project is said to have been instrumental in obtaining several
million dollars in donations from such sources as the Pew Charitable Trusts
(for a computer-supported chemistry and mathematics lecture hall), the
Keck Foundation (for a geology facility similar to the math laboratory),
and the Hughes Foundation (for the renovation of the school's biology laboratory).
-
Rensselaer Polytechnic Institute.
A project partially funded by ILI has enabled Dr. Frank DiCesare to replace
a 15-year-old course on the basics of microprocessor systems, which centered
around a series of rote experiments, with a new course that helps students
interactively design, build, test, and optimize a specific microprocessor
control application. He uses an approach involving the integrated use of
several teaching methods and materials, including lectures, a laboratory
manual, a computer-based hypermedia teaching and learning environment,
and hands-on experiments supervised by laboratory assistants. A significant
spinoff of the project has been the creation of a multimedia software company
by Dr. DiCesare and several former students who were involved in the development
of the new course.
-
Hope College.
The cumulative impacts of 16 ILI awards at this baccalaureate-granting
institution in Holland, Michigan, were found to transcend departmental
boundaries. For example, the PI for one of the early ILI projects at Hope
College, a project integrating biochemistry and molecular biology laboratories,
reported that ILI not only led to a new Biochemistry major on the campus,
but also initiated a climate of collaboration in proposal writing and the
receipt of several additional multidisciplinary ILI awards. Conversations
with other PIs confirmed that ILI-funded instrumentation has stimulated
changes in curriculum and instruction that have affected every campus program,
not just those that received ILI grants.
Table of Contents
Far-Reaching
Program Impacts
Evaluation results also demonstrated
that the influence of the ILI program can extend beyond those individuals
and organizations that are direct beneficiaries of ILI awards.
-
Disseminating ILI-Generated
Products. ILI PIs are expected to make conscientious
efforts to share the innovative educational approaches and materials they
develop in their projects with their colleagues. Data from the grantee
survey show that they fulfill this expectation to a very impressive extent.
In addition to responding to an average of about 7 external requests for
information about their ILI projects, PIs report an average of about 5
PI-initiated dissemination products or activities. This includes presentations
of papers at regional or national conferences (more than half of PIs report
one or more activities in this category), publications in journals and
other professional media (about one-third of PIs), speeches or colloquia
at other institutions (also about one-third), and workshops or short courses
to acquaint others with project-developed approaches and materials (one-quarter
of PIs).
-
Stimulating Innovation at
Other Institutions and Organizations. Although
the cumulative national and worldwide impacts on undergraduate SMET education
of the PI-initiated innovation dissemination activities are difficult to
quantify precisely, the results of the tracer studies conducted by the
evaluation team showed that ILI instructional innovations have been widely
adopted by workshop attendees and other identifiable second-stage users
of project-generated products throughout the nation. In most of the tracer
study projects, dissemination activities had reached at least 100 undergraduate
professors per project. Among the respondents for a given tracer study
project, the percentage of recipients of ILI-related dissemination who
reported incorporating the information and materials that they received
into their undergraduate courses ranged from 40 to 100 percent.
-
Impacting Unsuccessful Applicants.
The stimulative effects of ILI extend well beyond the direct and indirect
impacts of the projects that are actually funded in any given program cycle.
Although approximately 75 percent of ILI proposals will not be funded,
the survey of unsuccessful applicants revealed that 70 percent of the curricular
improvement proposals that were rejected by ILI in 1990 or 1992 ultimately
became implemented to some degree, either with subsequent ILI funding support
(19 percent) or with support from other sources-most often the institutions
themselves (51 percent).
The evaluation report includes
case studies illustrating these far-reaching impacts.
-
University of Texas, Austin.
An ILI project to modernize the way geography is taught has spawned an
online curriculum in geographical techniques employing active-learning
strategies and hypermedia techniques benefiting hundreds of young scholars,
instructors, and professional researchers worldwide. In the online curriculum,
students receive the bulk of their instructional material via the Internet
and present the results of their learning-formal reports, oral presentations,
examination responses, and so forth-on the World Wide Web. The program
is now used by at least 50 other geography departments throughout the world;
users also include government agencies such as NASA and commercial enterprises.
-
James Madison University.
The PI for a 1986 ILI project entitled "Applications of Lasers in Chemistry"
has initiated a broad range of dissemination activities including the development
and distribution of 20 laboratory experiments suitable for undergraduate
chemistry courses, distribution of low-cost laser apparatus kit plans,
publication of a newsletter, and facilitation of an annual short course
on laser experimentation. The impacts of this project were far reaching-every
respondent to the tracer study followup survey who had participated in
the laser short course, and most that were subscribers to the newsletter,
have reported using the materials and information in an undergraduate teaching
and research setting quite successfully.
Table of Contents
Conclusions and
Suggestions
The results of this evaluation
show that overall, the ILI program is achieving its mission to increase
the range and quality of modern laboratory equipment and to provide equipment-based
learning opportunities for undergraduate students at grantee institutions.
The evaluation identified five key areas in which the program has made
the greatest impact:
-
The ILI program has reached a
wide range of institutions and fields.
-
ILI awards have been equitably
distributed across individuals and SMET fields.
-
The laboratory equipment and
curricular innovations supported by ILI grants serve large numbers of undergraduate
students in the grantee institutions through improved coursework opportunities.
-
ILI grants were found to stimulate
extensive resource leveraging at the grantee institutions.
-
PIs receiving ILI awards spread
the word about their projects through a wide range of dissemination activities.
However, the types and strength
of impacts vary for different categories of undergraduate institutions,
for different SMET fields, and for different target audiences. The evaluation
identified five main areas where the program fell short of its goals:
-
Two-year institutions did not
participate as fully in the program as four-year institutions. This is
largely because of their relatively lower application rates.
-
ILI proposals contain relatively
few requests to fund equipment-sharing proposals or consortia, despite
the fact that in the program years under investigation, the ILI program
announcement made a special point of encouraging such proposals.
-
Reported impacts on the faculty
reward system were negligible.
-
The synergistic benefits of receiving
multiple awards have been limited to a few institutions.
-
Although the tracer study examination
of exemplary dissemination efforts showed instances of instructional innovations
being adopted by workshop attendees and other identifiable second-stage
users of project-generated products throughout the nation, most ILI PIs
are unable to confirm whether and to what extent their disseminated information
and materials have been adopted at institutions beyond the host institution.
The pattern of outcomes suggests
a number of tradeoffs that NSF might want to consider in shaping the program
for the future such as encouraging two-year institutions to submit more
applications or sacrificing broad coverage of institutions for the synergistic
outcomes observed in those institutions that have received multiple awards.
Whatever changes are ultimately made, this evaluation provides evidence
that the program's first decade of activity has been rich and fertile,
and that the seed money provided by NSF has yielded value well beyond its
initial investments.
Table of Contents