File: | evalhigh.pdf |
---|
Pages: | 1 to 35 of 35 |
---|
Document Body | Page Navigation Panel |
AUGUST 1996
N
S
F
E
VA
L
UAT
I
O
N
H
I
G
H
L
I
G
H
T
S
Directorate
for
Education
and
Human
Resources
Division
of
Research,
Evaluation
and
Communication
An
REC
Report
on
Evaluation
N
S
F
E
VA
L
UAT
I
O
N
H
I
G
H
L
I
G
H
T
S
Dear
Dr.
Williams:
I
am
pleased
to
transmit
NSF
Evaluation
Highlights:
A
Report
on
the
National
Science
Foundations
Efforts
to
Assess
the
Effectiveness
of
Its
Education
Program.
This
report
is
an
edited
compilation
of
evaluation
reports
produced
since
1992,
when
you
created
an
Evaluation
unit
(now
housed
in
the
Division
of
Research,
Evaluation
and
Communication,
or
REC)
within
the
Directorate.
It
was
prepared
with
the
assistance
of
Westat,
Inc.
This
report
reflects
how
EHR
program
evaluations
inform
what
we
know
about
the
impacts
of
NSFs
education
investments;
how
the
results
of
these
independent
evaluations
provide
information
relevant
to
the
shaping
and
management
of
the
EHR
portfolio;
and
how
the
ongoing
schedule
of
evaluations
demonstrates
staff
and
program
accountability
to
the
NSF
leadership,
the
National
Science
Board,
the
Congress,
and
the
American
public.
Readers
will
draw
their
own
conclusions
about
the
utility
of
this
evaluation
work.
What
cannot
be
denied
is
NSFs
dedication
to
demonstrating
how
collections
of
projects
in
localities
throughout
the
Nation
subsumed
under
EHR
program
names,
supported
through
merit-based
competitions,
and
systematically
monitored
postaward
allow
us
to
gauge
progress
in
improving
science,
mathematics,
engineering,
and
technology
education
at
all
levels
of
the
system,
kindergarten
to
participation
in
the
workforce.
We
are
not
merely
claiming
to
make
a
difference.
Program
evaluations
try
to
measure
the
magnitude
and
pace
of
that
difference
in
teaching
and
learning.
It
is
therefore
with
pride
that
I
submit
this
report.
It
is
a
testament
to
your
decision
to
require
program
evaluation
as
an
integral
part
of
EHRs
business.
I
hope
you
find
NSF
Evaluation
Highlights
a
useful
horse
in
the
EHR
accountability
stable.
Sincerely yours,
Daryl
E.
Chubin
Division
Director
for
Research,
Evaluation
and
Communication
programs.
The
purpose
of
the
evaluation
initiative
which
is
orchestrated
by
the
Directorates
Division
of
Research,
Evaluation
and
Communication
(REC)
is
essentially
twofold:
to
provide
EHR
officials
with
information
that
will
help
them
manage
more
effectively
the
approximately
30
programs
in
the
Directorates
portfolio;
and
to
report
to
Congress
and
the
public
on
the
effectiveness
of
the
Foundations
science
and
mathematics
education
programs.
To
date,
the
evaluation
effort
has
yielded
more
than
a
dozen
reports,
including
several
extensive
evaluations
that
have
been
completed
and
briefer
studies
focusing
on
the
ongoing
impact
of
programs
whose
evaluations
are
in
progress.
EHR
is
observing
a
schedule
that
calls
for
full
evaluations
of
all
of
its
science
and
mathematics
education
programs
to
be
accomplished
within
the
next
few
years.
The
first
section
of
this
four-part
document
(The
Value
of
Evaluation)
presents
an
overview
of
RECs
evaluation
efforts;
the
second
(Highlights)
summarizes
the
reports
completed
thus
far;
the
third
(Evaluation-Related
Activities)
describes
RECs
participation
in
evaluation-oriented
projects
that
extend
beyond
conducting
program
evaluations
for
EHR;
and
the
concluding
section
(Future
Evaluation
Efforts)
discusses
the
Directorates
future
objectives
and
schedule
for
fulfilling
its
evaluation
objectives.
iii
Young
Scholars
Program
(YSP)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
21
Informal
Science
Education
Program
(ISE)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
24
n
Higher
Education
Instrumentation
and
Laboratory
Improvement
Program
(ILI)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
27
Course
and
Curriculum
Development
Program
(CCD).
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
29
Undergraduate
Faculty
Enhancement
Program
(UFEP).
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
32
Visiting
Professorships
for
Women
Program
(VPW)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
35
Experimental
Program
to
Stimulate
Competitive
Research
(EPSCoR)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
37
Evaluation-Related
Activities
41
Future
Evaluation
Efforts
51
Order
Form
57
v
Foundation
(NSF)
in
1991
launched
a
major
effort
called
the
Statewide
Systemic
Initiatives
Program
(SSI).
This
program,
initiated
by
the
Foundations
Directorate
for
Education
and
Human
Resources
(EHR),
was
created
on
the
premise
that
positive
reform
in
K-12
science
and
mathematics
education
will
effectively
be
achieved
if
pursued
by
means
of
a
systemic
effort,
coordinated
nationwide
on
the
state
level
rather
than
piecemeal.
The
aim
of
systemic
reform
is
for
all
states
ultimately
to
participate
in
a
comprehensive
endeavor
to
elevate
teaching
standards,
to
enrich
the
instructional
materials
that
are
offered
to
students,
and
to
refine
the
methods
employed
in
measuring
students
grasp
of
science-
and
mathematicsrelated
subjects.
Central
to
the
systemic
reform
concept
is
that
all
students
should
benefit,
regardless
of
gender,
ethnic,
linguistic,
or
disability
status.
To
bolster
this
ambitious
undertaking,
EHR,
through
its
SSI
program,
has
over
the
past
5
years
provided
up
to
$2
million
annually
to
individual
states
offering
substantive
plans
for
implementing
systemic
reform.
Is
the
EHR
initiative
paying
off?
Is
science
and
mathematics
education
in
the
Nations
primary
and
secondary
schools
getting
better?
If
so,
how
much
K-12
improvement
is
attributable
to
SSI?
And
how
might
EHR
enhance
the
program
to
hasten
achievement
of
its
conceptual
goals?
To
explore
such
issues
and
to
monitor
the
program
as
it
evolves,
EHR
initiated
a
large-scale
evaluation
of
SSI
in
1992.
3
The
ultimate
aim
of
the
evaluation
initiative
is
to
make
sure
EHR
is
fulfilling
its
mission
to
improve
the
Nations
science,
mathematics,
engineering,
and
technology
education.
Currently,
dozens
of
EHR-administered
programs,
as
well
as
a
number
of
science
and
mathematics
education
projects
that
the
Foundation
shares
with
other
government
agencies,
are
being
assessed;
full
evaluations
for
the
entire
EHR
portfolio
are
to
be
completed
within
the
next
several
years.
n n n
The
evaluation
initiative
was
launched
by
EHR
in
response
to
a
growing
awareness
building
through
the
1980s
and
eventually
emerging
as
a
mandate
for
action
that
the
attention
being
paid
to
monitoring
EHRs
science
and
mathematics
education
programs
was
deficient.
The
Directorates
approach
to
monitoring
its
programs
needed
to
be
revised.
Traditionally,
programs
had
been
judged
largely
according
to
their
impact
on
individual
grantees
reports,
for
example,
of
an
awardees
satisfaction
with
a
particular
program
or
records
of
career
advances
or
scientific
achievements
that
could
be
attributed
to
his
or
her
NSF
affiliation.
By
1992
it
was
clear
that
a
broader
based,
outcomes-oriented
process
of
evaluation
had
to
be
implemented.
As
EHRs
program
portfolio
expanded,
administrators
had
to
have
richer,
fuller
data
to
monitor
the
progress
of
their
programs,
to
strive
for
their
continuous
improvement,
and
to
measure
their
gains
over
time.
No
longer
was
it
sufficient
to
know
how
an
individual
grant
awardee
had
used
his
or
her
funding
and
what
specific
benefits
had
resulted.
Rather,
it
was
decided,
evaluations
must
reveal,
among
other
things,
the
overall
progress
of
a
program
in
relation
to
its
stated
mission.
5 4
Although
the
evaluation,
conducted
by
outside
contractors,
is
due
to
be
completed
in
1997,
it
already
has
yielded
three
extensive
reports
and
a
wealth
of
useful
insights.
Judging
from
the
evaluators
reports,
SSI
is
indeed
succeeding
overall.
Most
participating
states
are
developing
new
and
more
effective
science
and
mathematics
curricula;
they
are
working
toward
the
clarification
of
student
achievement
criteria;
more
effort
is
being
made
to
elevate
the
quality
of
science
and
mathematics
teaching;
and
more
state
and
local
funding
is
being
attracted
to
complement
NSFs
financial
support
of
systemic
reform.
Not
so
gratifying,
however,
are
revelations
also
stemming
from
the
evaluations
that
the
SSI
program,
with
all
of
its
virtues,
is
lagging
in
some
respects.
Some
states,
for
example,
have
been
slow
to
develop
strategies
in
response
to
the
SSI
challenge,
while
others
have
failed
to
galvanize
public
support
for
improvement
in
science
and
mathematics
education.
Today,
SSI
continues
as
a
major
component
of
the
EHR
program
portfolio
its
objectives,
expectations,
financial
viability,
and
other
matters
constantly
undergoing
the
scrutiny
of
the
evaluation
team.
This
evaluation,
however,
is
but
one
example
of
an
unprecedented
NSF
initiative
currently
underway
to
subject
all
of
the
EHR
directorates
programs
to
sound,
rigorous,
and
thorough
examination
by
outside
analysts.
The
evaluations
are
primarily
guided
by
three
fundamental
questions:
Is
the
program
under
study
achieving
its
goals?
Is
it
making
an
impact?
And
are
there
ways
in
which
the
program
can
be
improved?
The
ultimate
aim
of
the
evaluation
initiative
is
to
make
sure
EHR
is
fulfilling
its
mission
to
improve
the
Nations
science,
mathematics,
engineering,
and
technology
education.
Broadly
speaking,
EHRs
responsibility
entails
the
development
and
administering
of
programs
designed
to
ensure
that
high-quality
science
and
mathematics
education
is
available
to
every
child
in
the
United
States;
that
the
educational
system
yields
individuals
who
can
meet
the
needs
of
the
Nations
science,
mathematics,
The
evaluations
are
primarily
guided
by
three
fundamental
questions:
Is
the
program
under
study
achieving
its
goals?
Is
it
making
an
impact?
And
are
there
ways
in
which
the
program
can
be
improved?
EHR
today
is
required
to
account
to
Congress
and
the
Administration
for
the
effectiveness
of
its
programmatic
activities.
n n n
Since
1992,
developing
and
implementing
thorough
and
insightful
EHR
program
evaluations
has
been
the
responsibility
of
the
Directorates
Division
of
Research,
Evaluation
and
Communication
(REC),
which
was
formed
in
1991.
To
fulfill
its
evaluation
mandate,
REC
has
for
the
past
4
years
pursued
three
broad
objectives:
To
design
and
oversee
periodic
evaluations
of
the
approximately
30
programs
currently
in
EHRs
portfolio;
To
elevate
the
practice
of
science
and
mathematics
education
evaluation
by
developing
new
and
better
methods;
and
To
build,
through
workshops
and
other
means,
the
capacity
of
NSF,
other
government
agencies
whose
missions
include
science
education,
and
individual
grant
awardees
to
conduct
rigorous
evaluations.
In
addition
to
implementing
evaluations
of
EHR
programs,
REC
has
during
the
past
several
years
engaged
in
evaluation
activities
associated
with
science,
mathematics,
engineering,
and
technology
education
programs
administered
by
other
federal
agencies.
To
carry
out
its
evaluation
work,
the
REC
staff
focuses
on
developing
three
kinds
of
studies:
Evaluations:
systematic
examinations
conducted
by
external
evaluators
to
determine
the
merit
or
worth
of
programs
and
ways
in
which
they
can
be
improved;
7 6
Also
by
1994,
EHRs
annual
federal
financial
support
had
grown
dramatically
to
$569
million
from
$71
million
in
1981
(see
Exhibit
1).
As
the
funding
mounted,
Congress
became
increasingly
intent
on
keeping
a
closer
eye
on
where,
exactly,
tax
money
was
going
and
how
effectively
EHR
was
using
it.
With
passage
of
the
Government
Performance
and
Results
Act
of
1993,
accountability
based
on
evaluation
of
a
programs
consistent
results
was
no
longer
Exhibit
1:
Funding
for
NSF
and
for
the
Directorate
for
Education
and
Human
Resources
(EHR),
1956-94
Year,
1956-1994
*
1956
Total
NSF
funding
=
16
million
1956
EHR
funding
=
3.5
million
Source: Indicators of Science and Mathematics Education 1995
56 58 60 62 64 66 68 70 72 74 76 78 80 82 84 86 88 90 92 94
Dollars
in
millions
Total NSF funding
Total
EHR
funding
*
0
500
1000
1500
2000
2500
3000
3500
Since
1992,
evaluations
of
varying
scope
have
been
completed
for
four
programs;
more
than
a
dozen
others
are
either
in
progress
or
in
the
planning
stage.
Additionally,
REC
has
played
a
leadership
role
in
several
other
evaluation-related
projects,
including
interagency
program
evaluations
and
how-to
documents
focusing
on
the
evaluation
process
itself.
9 8
Impact
studies:
briefer,
quicker
examinations
of
programmatic
effectiveness
that
yield
reports
more
limited
in
their
focus,
data
collection,
and
analysis
than
a
full
evaluation;
and
Monitoring:
ongoing
collection
and
analysis
of
data
on
the
status
of
EHR
projects.
A
staff
of
REC
evaluation
officers
oversees
the
design
and
execution
of
evaluations
by
independent
contractors,
who
are
retained
to
ensure
the
objectivity
with
which
a
program
is
examined.
Most
of
these
studies
take
up
to
2
years
to
complete,
at
a
cost
ranging
from
$100,000
to
$1
million.
In
fiscal
year
1994,
program
evaluation
represented
an
EHR
investment
of
about
$12
million
roughly
2
percent
of
the
Directorates
total
budget.
The
Directorate
has
placed
all
30
of
its
programs
on
a
5-year
evaluation
cycle,
so
that
at
least
one-third
of
the
portfolio
is
being
evaluated
at
all
times
(see
Exhibit
2).
Exhibit 2: Evaluation Status of EHR's 30 Programs
8 Not yet evaluated
2 Monitoring underway
11 Evaluation/impact study underway
4 Evaluation/impact study completed
5 Evaluation/impact study in planning
The
Directorate
has
placed
all
30
of
its
programs
on
a
5-year
evaluation
cycle.
evaluation effort.
The
reports
published
to
date,
examining
the
efficacy
of
nine
EHR
programs
differ
from
one
another
in
purpose,
scope,
level
of
detail,
and
length.
Several
evaluations,
for
example,
have
yielded
extensive
reports,
intended
to
serve
as
cumulative
assessments
of
their
subjects
progress
over
an
extended
period
of
time;
others
are
comparatively
brief
impact
studies,
intended
to
inform
NSF
administrators
of
their
subjects
progress;
and
one
is
a
preliminary
report
of
a
full-scale
evaluation
that
is
scheduled
for
completion
in
1996.
Following
are
highlights
of
the
reports
published
thus
far
descriptions
of
the
programs
that
have
undergone
evaluation,
accounts
of
the
evaluation
process
associated
with
each,
and
a
brief
sampling
of
findings
arising
from
the
evaluation.
Statewide
Systemic
Initiatives
Program
(SSI)
Description.
SSI
was
launched
by
EHR
in
1991
as
a
major
endeavor
toward
improving
science
and
mathematics
education
for
all
children
in
the
United
States
through
comprehensive,
state-level
systemic
reform.
The
goal
of
the
program
is
to
move
from
independently
devised
reform
efforts
to
state-initiated
measures
involving
the
coordinated
improvement
of
many
aspects
of
the
education
system,
including
teacher
preparation,
instructional
materials,
and
assessment
of
student
learning.
To
seed
and
scale
interventions
in
the
teaching
and
learning
of
science
and
mathematics,
EHR
participates
in
funding
arrangements
called
cooperative
agreements
awards
made
to
states
of
up
13
SSI
states
take
seriously
the
idea
that
a
vision
of
good
practice
must
guide
their
efforts.
The
methodology
in
years
1
and
2
of
the
evaluation
combined
an
extensive
review
of
documents
(including
annual
reports
from
the
funded
states
to
NSF)
with
supplemental
telephone
interviews
with
state
officials.
Also,
indepth
case
studies
including
week-long
visits
were
conducted
in
nine
states
selected
to
represent
a
variety
of
educational
and
demographic
conditions
and
varying
approaches
to
systemic
reform.
In
addition,
site
visits
were
conducted
in
three
non-case-
study
states.
To
date,
the
following
SSI
evaluation
reports
have
been
published:
Evaluation
of
the
National
Science
Foundations
Statewide
Systemic
Initiatives
(SSI)
Program:
First
Year
Report,
Volume
I,
Technical
Report
(SRI
International
1994).
Evaluation
of
the
National
Science
Foundations
Statewide
Systemic
Initiatives
(SSI)
Program:
Second
Year
Report:
Cross-Cutting
Themes
(SRI
International
1995).
Evaluation
of
the
National
Science
Foundations
Statewide
Systemic
Initiatives
(SSI)
Program:
Second-Year
Case
Studies:
Connecticut,
Delaware,
and
Montana
(SRI
International
1995).
Findings.
Given
the
wide
latitude
that
states
have
in
implementing
systemic
reform,
generalizations
about
program
emphases
and
activities
vary
considerably,
but
some
common
elements
can
be
observed.
For
example:
SSI
states
take
seriously
the
idea
that
a
vision
of
good
practice
must
guide
their
efforts.
Many
are
developing
new
statewide
curriculum
frameworks,
and
there
is
growing
agreement
that
a
central
goal
of
mathematics
and
science
education
is
to
promote
critical
thinking,
conceptual
understanding,
and
problem-solving
ability.
15 14
to
$2
million
per
year
for
1
to
5
years
for
the
purpose
of
forging
partnerships
among
educational
institutions
that
will
lead
to
ambitious,
coherent,
and
comprehensive
approaches
to
statewide
reform.
To
be
eligible,
states
must
demonstrate
both
commitment
to
reform
and
the
support
of
its
government
officials
and
educators,
since
such
support
is
an
essential
element
in
the
success
of
science
and
mathematics
education
reform.
In
1991,
an
initial
competition
resulted
in
cooperative
agreements
with
10
states;
a
second
competition
in
1992
resulted
in
11
additional
awards;
and
a
third
competition
in
1993
resulted
in
awards
to
five
more
states
(see
Exhibit
3).
Evaluation.
In
1992,
NSF
contracted
with
a
consortium
of
research
organizations
led
by
SRI
International
of
Menlo
Park,
California,
to
conduct
a
5-year
evaluation
to
examine
individual
projects
and
determine
how,
in
the
aggregate,
they
are
performing
to
promote
and
sustain
systemic
reform.
During
the
first
year,
the
evaluation
team
concentrated
on
descriptive
activities,
collecting
data
on
each
funded
states
approach
to
systemic
reform
and
its
plan
for
1991
Awardees
1992
Awardees
1993
Awardees
Connecticut
California
Arkansas
Delaware
Georgia
Colorado
Florida**
Kentucky
New
Jersey
Louisiana
Maine
New
York
Montana
Massachusetts
South
Carolina
North
Carolina
Michigan
Nebraska
New
Mexico
Ohio
Puerto
Rico
Rhode
Island*
Texas
South
Dakota
Vermont
Virginia**
*
cooperative
agreement
discontinued
1994
**
cooperative
agreement
discontinued
1996
Exhibit 3: Recipients of Statewide Systemic Initiatives (SSI) Awards, 1991-93
All
SSI
states
have
developed
strategies
for
improving
educational
equity
by
providing
access
for
all
students
to
high-quality
science
and
mathematics
education.
The
strategies
and
the
groups
targeted
for
service
vary;
most
commonly,
the
focus
is
on
women
and
minorities.
Sources
other
than
SSI
are
providing
(in
the
aggregate)
more
than
a
dollar-for-dollar
match
to
NSFs
investment.
17 16
All
SSI
states
are
seeking
to
develop
and
articulate
clear
goals
for
what
students
should
know
and
be
able
to
do.
These
goals
are
fairly
uniform
across
states
for
mathematics,
but
less
so
for
science.
Reform
strategies
vary
greatly
by
state,
but
a
shared
emphasis
has
been
on
inservice
training
of
teachers
(see
Exhibit
4).
More
than
one-third
of
SSI
funds
Findings
n
SSIs
spent
the
greatest
portion
of
their
NSF
dollars
on
the
professional
development
of
practicing
mathematics
and
science
teachers,
providing
such
services
to
nearly
50,000
teachers
during
the
past
year
(about
8%
of
the
public
school
teachers
in
SSI
states).
n
NSF's
funds
leveraged
$83
million
from
other
sources
for
the
SSI
program.
n
SSIs
use
a
wide
combination
and
variety
of
reform
strategies,
ranging
from
aligning
state
policies
to
supporting
model
schools
to
developing
new
instructional
materials.
n
Many
SSIs
are
working
to
strengthen
the
education
infrastructure
to
support
the
long-term
improvement
of
mathematics
and
science
instruction
beyond
the
end
of
the
SSI
program.
n
Three
states
target
the
introduction
of
new
instructional
materials
as
a
primary
SSI
strategy;
many
others
disseminate
information
about
existing
materials
that
they
consider
effective.
n Five states support model schools in their reform strategy.
n Ten states target local districts and/or communities as part of their SSI change strategy.
Problems
and
Issues
n
A
number
of
state
policies,
such
as
those
for
assessment
of
student
learning,
often
remain
out
of
alignment
with
states'
reform
goals.
n The SSIs give little attention to local education policy systems.
n
A
strength
of
the
SSIs
is
their
focus
on
building
the
capacity
of
practicing
teachers
to
improve
the
teaching
of
mathematics
and
science,
yet
the
level
of
effort
is
often
insufficient
to
meet
the
challenges
of
reform.
n States are paying relatively little attention to preparing the next generation of teachers.
n
SSI
states
most
often
rely
on
persuasion
and
incentives
to
motivate
educators
to
participate
in
their
reform
agenda,
although
there
also
have
been
real
changes
in
authority
relations
in
a
few
states.
n
The
task
of
mobilizing
public
opinion
is
a
difficult
one
and
often
requires
skills
not
readily
available
within
the
mathematics,
science,
and
education
communities
promoting
the
reform
agenda.
Exhibit
4:
Statewide
Systemic
Initiatives
(SSI)
Change
Strategies:
Findings
and
Problems
Teacher
Professional
Development
(Inservice)
36%
Training
of
New
Teachers
(Preservice)
5%
Public
Awareness
5%
Other
9%
Administration
&
Coordination
21%
Program
Evaluation
6%
Development
of
New
Statewide
Curriculum
Frameworks
3%
Curriculum
&
Instructional
Materials
9%
Student
Assessment
6%
Exhibit
5:
Statewide
Systemic
Initiatives
(SSI)
Expenditure
of
NSF
Funds
in
FY
1993
by
Purpose
of
Activity
Presidential
Awards
for
Excellence
in
Science
and
Mathematics
Teaching
Program
(PAESMT)
Description.
PAESMT
was
established
in
1983
under
the
sponsorship
of
NSFs
Division
of
Elementary,
Secondary
and
Informal
Education
with
the
goal
of
providing
national
recognition
to
outstanding
elementary
and
secondary
school
teachers
in
science
and
mathematics.
The
program
also
seeks
to
encourage
active
leadership
roles
in
science
and
mathematics
education
by
former
awardees.
Each
year,
after
a
lengthy
nomination
and
selection
process,
four
mathematics
and
science
teachers
two
elementary
and
two
secondary
are
chosen
for
the
award
in
every
state
and
U.S.
territory.
Each
award
is
accompanied
by
an
NSF
grant
of
$7,500,
to
be
used
at
the
teachers
discretion
to
improve
science
or
mathematics
education
at
the
awardees
home
institution.
The
awardees
are
brought
to
Washington,
D.C.,
for
a
week
of
activities
and
professional
interaction.
During
the
first
decade
of
the
programs
existence,
more
than
1,600
teachers
from
around
the
Nation
received
national
recognition
through
this
program
for
their
subject-matter
expertise
and
teaching
skills.
Evaluation.
From
the
time
of
its
inception
through
1994,
no
evaluation
of
this
program
had
been
undertaken.
However,
in
1994,
the
Foundation
contracted
with
Westat,
Inc.,
to
engage
in
a
small-scale
study
focusing
primarily
on
the
personal
and
professional
impacts
of
the
award.
Telephone
conversations
were
conducted
with
115
of
the
430
teachers
who
had
received
awards
in
1990
and
1991.
An
interview
protocol
was
developed
to
guide
the
conversations,
but
the
exact
questions
varied
among
respondents,
and
the
awardees
were
given
considerable
latitude
in
shaping
the
content
of
the
interview.
19 18
On
the
other
hand,
progress
in
achieving
some
goals
may
be
lagging,
and
states
may
have
to
revise
some
of
their
strategies.
For
example:
NSF
has
targeted
equity
as
a
key
objective,
but
while
the
participating
states
have
adopted
various
strategies
to
accomplish
this
goal,
some
have
not
yet
developed
coherent
plans.
States
are
finding
the
need
to
mobilize
public
support
for
systemic
reform
greater
than
was
anticipated
and,
therefore,
must
increase
efforts
to
attract
support.
The
alignment
of
all
state
policies
to
achieve
systemic
reform
is
proving
difficult.
In
particular,
technical,
financial,
and
political
barriers
to
changing
methods
of
assessing
student
learning
present
a
major
impediment
to
change.
Issues
that
have
largely
been
neglected
to
date
include
state
strategies
for
transferring
lessons
from
model
sites
to
schools;
attention
to
local
education
systems,
including
school
districts
and
boards;
and
attention
to
the
need
for
changes
in
the
preparation
of
the
next
generation
of
teachers.
Overall,
while
measuring
systemic
change
and
assessing
the
impact
of
a
broad
program
such
as
SSI
(with
its
large
expenditures
and
multiplicity
of
jurisdictions)
is
a
complex
undertaking,
this
evaluation
has
played
an
important
role
in
providing
NSF
and
its
partners
with
a
useful
indication
of
education
reforms
introduced
in
classrooms,
schools,
districts,
and
across
states.
NSF
is
implementing
a
number
of
changes
in
response
to
the
year
1
and
2
evaluation
reports,
including
the
provision
of
a
full
range
of
technical
assistance
services.
To
this
end,
the
technical
assistance
provider
is
helping
states
develop
strategies
to
inform
the
public
about
what
has
been
accomplished
by
these
reform
efforts
and
to
share
lessons
with
other
NSF
systemic
initiatives
such
as
those
in
urban
and
rural
areas.
In
addition,
the
evaluation
has
redoubled
The
SSI
evaluation
has
played
an
important
role
in
providing
NSF
and
its
partners
with
a
useful
indication
of
education
reforms.
PAESMT
program
awardees
reported
a
renewed
sense
of
validation
for
their
efforts
and
reinforcement
of
their
motivation
to
continue
to
teach.
Negative
comments
were
received
by
the
evaluators
regarding
only
two
aspects
of
this
program.
The
first
had
to
do
with
the
application
process,
described
by
several
interviewees
as
so
burdensome
that
it
might
discourage
some
worthy
candidates.
A
number
of
interviewees
also
felt
that
the
existence
of
the
program
and
the
nomination
process
are
only
known
to
persons
active
in
professional
associations.
They
recommended
increased
dissemination
of
information
concerning
the
program
to
bring
about
wider
participation
by
all
segments
of
the
teaching
profession.
In
response
to
these
comments,
NSF
is
considering
ways
in
which
the
application
process
can
be
simplified
and
awareness
of
the
program
increased.
Overall,
the
evaluators
summarized
their
assessment
of
PAESMT
by
calling
it
a
program
with
significant
impact
on
the
teachers
recognized;
these
teachers,
in
turn,
have
impacted
their
colleagues
in
some
very
significant
ways.
Young
Scholars
Program
(YSP)
Description.
Launched
in
1988
by
NSFs
Division
of
Elementary,
Secondary
and
Informal
Education,
YSP
typically
funds
summer
projects
of
from
3
to
8
weeks
duration
in
universities
and
other
organizations
that
conduct
scientific
research.
Junior
and
senior
high
school
students
are
eligible
to
participate,
the
goal
of
the
program
being
to
excite
students
about
science,
mathematics,
and
technology
and
to
encourage
them
to
investigate
and
pursue
careers
in
these
fields.
Related
21 20
In
addition
to
the
impact
of
the
award
on
recipients,
other
topics
that
arose
during
the
phone
interviews
included
their
use
of
the
NSF
grant
money
and
additional
financial
support
that
many
of
the
awardees
received
from
other
donors
in
connection
with
the
NSF
award.
The
evaluation
also
sought
awardees
views
of
the
nomination
and
selection
process
and
their
suggestions
for
strengthening
the
program
and
broadening
its
coverage.
This
study
resulted
in
a
publication
covering
all
aspects
of
the
evaluation:
Short
-Term
Impact
Study
of
the
Presidential
Awards
for
Excellence
in
Science
and
Mathematics
Teaching
(Westat
1994).
Findings.
PAESMT
was
found
to
be
an
overwhelming
success
in
terms
of
its
impact
on
participants
and
the
recognition
it
provides
to
the
importance
of
good
mathematics
and
science
instruction.
Strong
positive
effects
were
found
at
the
professional
and
personal
levels
for
those
who
have
been
recognized,
with
awardees
reporting
a
renewed
sense
of
validation
for
their
efforts
and
reinforcement
of
their
motivation
to
continue
to
teach.
Within
an
awardees
local
community,
there
was
much
pride
and
greater
interest
in
effective
mathematics
and
science
instruction
on
the
part
of
students,
parents,
and
other
teachers.
A
majority
of
awardees
reported
that
the
award
increased
their
opportunities
to
make
improvements
in
their
schools
mathematics
and
science
programs;
many
of
them
became
involved,
following
their
recognition
by
NSF,
in
curriculum
development
and
the
supervision
of
teacher-training
workshops.
Opportunities
for
involvement,
even
leadership,
at
the
state
level
through
participation
on
panels
and
committees
dealing
with
education
also
had
increased
for
some
grant
recipients
(see
Exhibit
6).
In
a
number
of
cases,
the
award
and
the
publicity
associated
with
it
led
to
gifts
or
awards
from
other
organizations.
23
83
74
64
57
Federal
level
State
level
Teacher
associations
Public
media
Private
sector
Exhibit
6:
Percent
of
Presidential
Awards
for
Excellence
in
Science
and
Mathematics
Teaching
(PAESMT)
Program
Awardees
from
1990
to
1991
Reporting
Positive
Effects
on
Opportunities
for
Leadership
at
Various
Levels,
1994
PAESMT
was
found
to
be
an
overwhelming
success
in
terms
of
its
impact
on
participants
and
the
recognition
it
provides
to
the
importance
of
good
mathematics
and
science
instruction.
Preliminary
findings
suggest
that
the
YSP
has
been
especially
encouraging
or
reinforcing
for
African
American
students
and
least
encouraging
or
reinforcing
for
female
participants.
Among
the
high
school
seniors
in
the
1991
program
almost
all
of
whom
were
still
in
college
at
the
time
of
the
1994
evaluation
65
percent
were
majoring
in
science,
mathematics,
or
engineering,
and
24
percent
were
majoring
in
health-related
fields.
However,
data
also
revealed
similar
college
choices
among
students
who
had
applied
but
did
not
participate
in
the
1991
program.
Findings
suggest
that
the
program
has
been
especially
encouraging
or
reinforcing
for
African
American
students
(see
Exhibit
7).
It
was
23 22
goals
are
to
acquaint
them
with
various
career
options
in
the
sciences,
engineering,
and
technology;
to
increase
their
awareness
of
the
academic
preparation
needed
for
such
careers;
to
acquaint
them
with
academic
and
research
environments;
and
to
contribute
to
their
confidence
in
their
ability
to
make
career
decisions.
The
program
strongly
emphasizes
student
participation
in
the
process
of
scientific
discovery
through
interaction
with
practicing
scientists
in
the
laboratory
and
other
research
environments.
In
recent
years,
the
program
has
supported
more
than
140
projects
each
year,
involving
more
than
6,000
students
annually
at
sites
throughout
the
United
States.
Supporting
NSF
grants
are
awarded
to
colleges,
universities,
or
other
organizations
and
institutions
whose
members
are
primarily
university
faculty
members
or
researchers.
Evaluation.
The
first
formal
evaluation
of
this
program,
a
shortterm
impact
study,
was
conducted
by
Westat,
Inc.,
in
1994.
The
study
was
intended
to
be
small-scale
and
exploratory,
designed
primarily
to
obtain
information
about
participating
students
perceptions
of
their
experiences
with
the
program,
about
their
educational
and
career
plans,
and
about
the
perceived
impact
that
their
YSP
experiences
had
on
those
plans.
In
addition
to
student
self-reports,
evaluators
sought
the
opinions
of
parents.
Also,
information
about
interest
in
science
and
career
choices
was
obtained
from
students
who
applied
to,
but
did
not
attend,
a
YSP
project.
The
evaluators
conducted
telephone
interviews
with
215
participants
and
70
nonparticipants,
all
of
whom
had
participated
or
applied
in
1991.
Interviews
were
also
conducted
with
52
parents.
One
publication
was
issued
in
connection
with
this
evaluation:
Short-Term
Impact
Study
of
the
National
Science
Foundations
Young
Scholars
Program
(Westat
1994).
Findings.
The
study
indicates
that
the
YSP
experience
was
a
very
positive
one
for
the
overwhelming
majority
of
participants.
They
gained
awareness
of
a
science-related
community
of
professionals,
learned
a
good
deal
about
various
fields,
and
discovered
much
about
themselves
and
their
professional
interests.
Of
course,
given
the
programs
selection
process,
which
seeks
to
identify
students
of
high
ability
who
have
an
interest
in
science,
mathematics,
or
engineering,
41
33
52
21
24
28
18
33
20
11
11
11
Science,
Mathematics,
Engineering,
and
Technology
Health professions
All
other
professions
Undecided
Exhibit
8:
Percent
of
1991
Young
Scholars
Program
(YSP)
Participants
Reporting
Various
Career
Plans,
by
Gender,
1994
Women
All participants
Men
Key
81
44
13
21
7
23
0
12
Science,
Mathematics,
Engineering,
and
Technology
Health professions
All
other
professions
Undecided
Exhibit
7:
Percent
of
1991
Young
Scholars
Program
(YSP)
Participants
Reporting
Various
Career
Plans,
by
Race,
1994
White
African American
Key
The
study
indicates
that
the
YSP
experience
was
a
very
positive
one
for
the
overwhelming
majority
of
participants.
The
evaluation
methodology
developed
by
the
contractor
combines
surveys,
case
studies,
and
21
site
visits
(15
to
ISE-funded
projects
sites
and
6
to
science
education
project
sites
not
funded
by
the
program).
Interviews
with
past
and
present
project
directors
and
other
knowledgeable
sources
also
are
planned.
In
addition,
a
meta-analysis
of
existing
pertinent
evaluations
is
planned.
As
a
first
step,
the
evaluator
inventoried
and
analyzed
the
types
of
activities
and
institutions
supported
by
ISE.
In
addition
to
technical
reports
dealing
with
the
evaluation
plan,
a
report
summarizing
some
preliminary
findings
has
been
issued:
Informal
Science
Education
Program:
Evaluation
Design
Brief
and
Report
of
Preliminary
Findings
(Cosmos
1995).
Findings.
From
1984
to
1995,
funding
for
ISE
increased
from
less
than
$5
million
to
$35
million.
Of
the
total
grant
funds
awarded
($183
million),
more
than
three-fourths
went
to
support
museum
exhibits
and
the
production
of
television
programs.
The
remainder
of
the
funds
was
awarded
to
a
variety
of
activities,
including
after-
school
and
community
programs,
radio,
and
film
production.
Nearly
80
percent
of
the
funds
went
to
science
and
natural
history
museums,
zoos,
and
childrens
museums
and
to
media
organizations,
such
as
production
companies
and
television
stations
(see
Exhibit
9,
page
26).
Preliminary
survey
data
show
that
young
people
involved
in
informal
science
activities
were
predominantly
between
the
ages
of
12
and
17
and
that
NSF-funded
projects
were
especially
influential
in
providing
access
to
science
education
for
previously
underserved
populations
and
in
introducing
fields
of
science
to
the
public.
The
case
studies
offer
evidence
that
program
goals
are
being
met:
For
example,
in
one
California
county,
two
eighth
grade
girls
have
25 24
least
encouraging
or
reinforcing
for
women
in
their
decisions
to
pursue
careers
in
the
sciences
and
engineering
(see
Exhibit
8,
previous
page).
Eighty-one
percent
of
African
American
participants
reported
career
intentions
in
science,
mathematics,
engineering,
and
technical
(SMET)
fields.
Just
33
percent
of
female
participants
indicated
such
a
choice.
The
study
raised
a
number
of
issues
concerning
the
program
that
should
be
considered
in
the
future.
Chief
among
these
is
the
advisability
of
structuring
the
program
to
ensure
its
attractiveness
and
availability
to
young
women,
members
of
racial
minorities,
and
students
of
high
potential
as
well
as
those
whose
high
ability
has
been
established.
In
response,
NSF
evaluation
staff
has
recommended
a
followup
study
to
provide
more
information
about
how
the
program
can
provide
greater,
more
positive
impacts
on
these
students,
especially
the
young
women.
Informal
Science
Education
Program
(ISE)
Description.
Since
the
early
1980s,
the
ISE
has
functioned
with
the
goal
of
advancing
science
learning
for
the
general
public.
The
program,
administered
by
EHRs
Division
of
Elementary,
Secondary
and
Informal
Education,
is
devoted
to
supporting
public
and
commercial
television
and
radio
programming,
the
creation
of
films
and
videos,
museum
exhibits,
science
and
technology
center
exhibits,
professional
development
activities,
and
community-
and
youth-oriented
organizations.
The
program
was
designed
to
provide
rich
and
stimulating
environments
outside
of
the
school
setting
where
individuals
of
all
ages,
backgrounds,
and
interests
can
increase
their
appreciation
and
understanding
of
science,
mathematics,
and
technology.
ISEs
goals
have
been
sharpened
in
recent
years
with
an
emphasis
on
establishing
linkages
between
formal
and
informal
education
and
increasing
the
number
of
young
people
involved
in
science-related
activities,
especially
those
from
minority
or
otherwise
underserved
groups.
Nearly
80
percent
of
ISE
funds
went
to
science
and
natural
history
museums,
zoos,
and
childrens
museums
and
to
media
organizations,
such
as
production
companies
and
television
stations.
ISE-funded
projects
were
especially
influential
in
providing
access
to
science
education
for
previously
underserved
populations
and
in
introducing
fields
of
science
to
the
public.
The
evaluators
final
report
will
present
the
first
summative
analysis
of
the
NSF
investment
in
ISE.
Instrumentation
and
Laboratory
Improvement
Program
(ILI)
Description.
ILI
was
launched
in
1985
by
NSFs
Division
of
Undergraduate
Education
(DUE),
whose
mission
is
to
strengthen
and
ensure
the
vitality
of
undergraduate
education
in
science,
mathematics,
engineering,
and
technology.
The
ILI
program
(one
of
three
DUE
efforts
on
which
reports
have
been
completed)
supports
projects
aimed
toward
generating
new
and
improved
approaches
to
laboratory-
and
field-based
instruction.
Above
all,
the
program
is
designed
to
stimulate
student
interest
in
science-
and
technologyrelated
courses
and
degree
programs
and
to
support
the
development
of
national
models
for
the
improvement
of
undergraduate
laboratory
instruction.
ILI
has
become
one
of
the
Foundations
largest
and
most
visible
programs
in
the
area
of
science
education,
attracting
more
than
2,000
proposals
annually
and
awarding
about
$20
million
a
year.
From
its
inception
in
1985,
it
has
made
more
than
4,700
grants
to
1,200
different
educational
institutions.
And
since
grantee
institutions
are
required
to
match
ILI
funds
by
at
least
100
percent,
the
program
has
generated
an
additional
$316
million
in
support
of
laboratory
improvement.
Evaluation.
In
October
1994,
the
Foundation
began
conducting
a
full-scale
evaluation
of
ILI,
scheduled
for
completion
by
Westat,
Inc.,
in
1996.
It
will
assess
how
successful
the
program
has
been
in
reforming
undergraduate
laboratory
instruction
and
in
meeting
the
needs
of
various
kinds
of
institutions
and
individual
disciplines.
The
evaluation
also
seeks
to
address
the
effect
of
the
program
on
curricula,
faculty
27 26
taken
the
initiative
to
develop
hands-on
geology
materials
after
participating
in
an
ISE
activity
run
by
the
University
of
California
at
Davis,
a
result
that
relates
to
the
ISE
goal
of
promoting
science
interest
in
students
from
groups
that
are
traditionally
underrepresented
in
science
in
this
case,
girls.
Another
stated
ISE
goal
that
of
establishing
new
relationships
between
formal
and
informal
science
education
was
addressed
in
Michigan,
where
a
school
district
conducted
a
summer
program
using
The
Magic
School
Bus,
an
ISE-funded
project
that
produced
a
popular
book
and
television
series.
Furthermore,
videos
dealing
with
environmental
issues
that
were
produced
with
ISE
support
Media
Producers,
TV
Stations,
and
Radio
Networks/Stations
39%
Natural
History
and
Children's
Museums,
Zoos,
and
Planetariums
39%
University/Basic
Research
Organizations
6%
Professional
Organizations
(not-for-profit)
6%
Community-Based
and
Youth
Group
Organizations
4%
Other
Institutions*
6%
*
Other
institutions
include
aquariums;
botanical
gardens
and
arboretums;
for-profit
organizations;
general
museums;
government
agencies;
and
nature
centers.
Exhibit
9:
Types
of
Institutions
Receiving
Support
from
Informal
Science
Education
Program,
1984-95
(Total
=
$183
million)
Matching
funds
were
frequently
generated
by
private
and
industrial
sources
that
regard
an
institutions
winning
an
NSF
ILI
grant
as
a
sign
of
quality
assurance.
Course
and
Curriculum
Development
Program
(CCD)
Description.
CCD,
another
program
funded
by
DUE,
awards
grants
to
universities
for
developing
undergraduate
science
and
mathematics
units,
courses,
and
sequences
of
courses.
Its
aim
is
to
support
major
instructional
changes
at
the
undergraduate
level
that
have
potential
national
impact.
Two
major
goals
are
pursued:
To
increase
the
understanding,
interest,
and
comfort
of
students
engaged
in
science-
and
mathematicsrelated
studies,
especially
those
who
are
members
of
minorities
and
other
groups
traditionally
underrepresented
in
those
fields;
and
To
contribute
to
a
shift
in
academic
culture
that
will
foster
the
placement
of
greater
value
on
undergraduate
teaching
and
learning.
This
program
especially
encourages
development
of
innovative
introductorylevel
science
and
mathematics
courses
for
both
majors
and
nonmajors.
Grants
provide
for
the
implementation,
assessment,
and
dissemination
of
projects
designed
to
improve
curricula
and
the
learning
environment
and
to
develop
new
materials,
software,
and
technologies,
as
well
as
courses.
29 28
teaching
practices,
and
on
the
training
of
future
elementary
and
secondary
teachers.
The
evaluation
covers
the
period
1985-94.
Data
sources
include
the
analysis
of
existing
ILI
records;
a
mail
survey
of
ILI
grantees
from
1990
and
1992;
a
survey
of
individuals
whose
ILI
applications
were
unsuccessful;
tracer
studies
documenting
the
impacts
of
innovative
projects
developed
by
grantees
on
other
institutions;
and
site
visits
to
selected
colleges
and
universities.
To
date,
the
ongoing
ILI
evaluation
has
yielded
a
preliminary
document:
A
Short-Term
Impact
Study
of
the
National
Science
Foundations
Instrumentation
and
Laboratory
Improvement
Program
(ILI)
(Westat
1996).
Findings.
The
preliminary
report
answers
one
important
evaluation
question:
Does
the
program
reach
the
intended
audience?
The
program
has
been
generally
successful
in
this
respect.
Most
important,
undergraduates
from
both
4-year
and
2-year
institutions
who
major
in
NSF-supported
disciplines
are
being
reached
by
the
program;
and
89
percent
of
bachelors
degree
recipients
in
these
fields
studied
at
institutions
that
received
one
or
more
ILI
awards
(see
Exhibit
10).
Information
gathered
during
early
site
visits
points
to
other
noteworthy
program
outcomes.
For
example,
matching
funds
were
frequently
generated
by
private
and
industrial
sources
that
regard
an
institutions
winning
an
NSF
grant
as
a
sign
of
quality
assurance;
many
students
have
pursued
advanced
science/technology
studies
or
careers
as
a
result
of
their
experiences
in
ILI-enabled
laboratories;
and
the
presence
of
ILI-funded
laboratory
equipment
has
attracted
new
faculty
at
some
institutions.
4-Year
Institutions
1,500
1,236
936
2-Year
Institutions
1,160
468
220
Exhibit
10:
Number
of
Postsecondary
Institutions
That
Applied
for
and
Received
Instrumentation
and
Laboratory
Improvement
(ILI)
Program
Awards,
1985-94
Number
submitting
1
or
more
proposals
Number
receiving
1
or
more
awards
Number of academic institutions in the U.S.
Key
Substantial
changes
have
occurred
in
the
thinking
and
behavior
of
the
great
majority
of
faculty
members
in
CCD
participating
departments.
Many
students
have
pursued
advanced
science
/
technology
studies
or
careers
as
a
result
of
their
experiences
in
ILIenabled
laboratories.
Project
directors
also
saw
marked
gains
in
students
understanding,
competence,
and
comfort
in
dealing
with
science
and
mathematics
subjects
(see
Exhibit
11).
The
site
visits
yielded
evidence
that
the
greatest
gains
were
often
experienced
by
students
who
had
difficulty
in
the
past
with
traditional
instruction
practices.
Although,
according
to
the
interim
report,
the
CCD
program
was
not
yet
having
a
profound
impact
on
departmental
policies
(such
as
increased
commitment
to
undergraduate
education),
the
program
was
having
significant
success
in
vitalizing
undergraduate
teaching
and
encouraging
new
approaches
to
learning.
Regarding
barriers
to
student
gains,
some
students
had
difficulty
adopting
new
learning
behaviors
and
skills,
which
some
projects
attempted
to
remedy
through
appropriate
course
materials
or
tutorial
centers.
Barriers
to
project
implementation
included
psychological,
cultural,
and
structural
factors:
for
example,
the
evaluators
found
that
many
faculty
members
enjoy
their
roles
as
sources
of
wisdom
in
traditional
teacher
settings
and,
thus,
are
reluctant
to
play
what
they
may
view
as
a
mere
facilitator-
of-learning
role.
Other
barriers
to
program
success
include
the
traditional
placement
of
higher
value
on
substantive
research
activities
than
on
research
into
teaching
practices
and
the
reluctance
of
some
university
departments
to
allocate
funds
for
nontraditional
programs.
The
evaluators
suggested
a
number
of
strategies
for
overcoming
these
barriers
to
implementation.
They
plan,
in
their
final
CCD
evaluation
report,
to
recommend
ways
in
which
the
program
will
continue
to
be
successful
in
the
future.
31 30
The
first
CCD
grants
were
awarded
in
1988.
For
the
first
3
years,
awards
were
made
for
projects
focusing
on
calculus,
precalculus,
and
engineering.
Evaluation.
An
evaluation
of
this
program,
covering
its
first
5
years
(1988-93),
is
currently
being
conducted
by
Network,
Inc.,
of
Andover,
Massachusetts.
During
this
5-year
period,
approximately
400
awards
were
made.
The
evaluation
methodology
consists
of
three
major
components:
Surveys
of
all
principal
investigators
who
received
awards
and
a
sample
of
individuals
who
submitted
proposals
but
were
not
funded;
Site
visits
to
27
grant-recipient
institutions;
and
Twelve
case
studies.
A
final
report
is
due
in
the
summer
of
1996.
An
interim
report
was
issued
in
October
1995,
based
on
partial
data
available
at
that
time:
Evaluation
of
the
Division
of
Undergraduate
Educations
Course
&
Curriculum
Development
Program,
Interim
Report
(The
Network,
Inc.
1995).
Findings.
The
evaluators
have
specified
four
questions
that
they
are
seeking
to
answer
with
data
from
the
surveys
and
site
visits.
Two
of
the
questions
deal
with
project
impact
on
faculty,
students,
and
departments;
and
two
questions
deal
with
barriers
that
interfere
with
success
and
the
ways
in
which
successful
projects
have
overcome
these
barriers.
The
interim
report
focused
chiefly
on
the
first
two
questions.
The
study
revealed
that
substantial
changes
have
occurred
in
the
thinking
and
behavior
of
the
great
majority
of
faculty
members
in
participating
departments.
Project
directors
generally
felt
that
compared
to
their
preaward
instructional
methods,
faculty
members
were
relying
less
on
lecturing,
were
using
new
instruction
materials
and
new
methods
of
student
assessment,
and
were
spending
more
time
teaching.
At
the
same
time,
students
were
more
often
Percent
of
project
directors
holding
this
opinion
Student
gained
competence
in
applying
concepts,
principles,
or
theories
81
Student
gained
competence
in
using
methods
and
equipment
76
Student
gained
understanding
or
familiarity
with
the
scientific
approach
to
problems
72
Student
developed
greater
interest
in
or
comfort
with
the
science
taught
71
Exhibit
11:
Four
Areas
in
Which
1988-93
Project
Directors
Felt
That
Students
Benefit
from
Projects,
1995
Barriers
to
CCD
program
success
include
the
traditional
placement
of
higher
value
on
substantive
research
activities
than
on
research
into
teaching
practices.
Assessment
of
the
National
Science
Foundations
1988-1990
Undergraduate
Faculty
Enhancement
Program:
Final
Report
(Westat
1993).
Assessment
of
the
National
Science
Foundations
1988-90
Undergraduate
Faculty
Enhancement
Program:
Executive
Summary
(Westat
1993).
Assessment
of
the
National
Science
Foundations
Undergraduate
Faculty
Enhancement
Program:
Interpretive
Overview.
A
Statement
from
the
Assessment
Advisory
Committee
(Westat
1993).
Findings.
From
the
vantage
point
of
both
participants
and
project
directors,
the
first
objective
meeting
participants
needs
was
effectively
met.
Both
groups
expressed
enthusiasm
for
the
projects
in
which
they
were
involved,
finding
them
either
highly
valuable
or,
at
least,
very
worthwhile.
As
benefits
derived
from
the
projects,
respondents
singled
out
increased
knowledge
of
their
field,
personal
growth
or
renewal,
increased
motivation
to
aspire
for
excellence
as
teachers,
and
increased
contacts
with
colleagues
from
other
institutions
(see
Exhibit
12).
33 32
Undergraduate
Faculty
Enhancement
Program
(UFEP)
Description.
This
program
was
inaugurated
by
EHRs
DUE
in
1988
in
response
to
a
1986
National
Science
Board
task
force
report
addressing
undergraduate
education
in
mathematics,
engineering,
and
the
sciences.
The
report
recommended
that
NSF
establish
a
comprehensive
set
of
programs
to
catalyze
and
stimulate
national
efforts
to
assure
a
vital
faculty
in
these
areas
of
study.
Accordingly,
UFEP
was
designed
to
assist
faculty
members
who
are
primarily
engaged
in
the
instruction
of
undergraduates
to
gain
experience
with
recent
advances
in
their
fields,
with
new
experimental
techniques,
and
with
ways
of
incorporating
these
into
undergraduate
instruction.
Projects
are
regional
or
national
in
scope
and
typically
consist
of
hands-on
short
courses
or
workshops,
along
with
followup
activities
that
encourage
sustained
interaction
among
participants.
A
major
component
of
the
program
is
regional
coalitions
between
2-
and
4-year
institutions.
Proposals
are
accepted
from
any
organization
with
the
scientific
expertise
and
facilities
to
conduct
these
projects.
Subject
matter
of
faculty
enhancement
activity
may
pertain
to
any
field
of
science,
engineering,
or
mathematics
normally
supported
by
NSF.
During
its
first
3
years
of
operation
(1988-90),
the
program
awarded
approximately
$6.7
million
to
92
projects,
each
of
which
supported
one
or
more
workshops
or
short
courses
and
served
nearly
3,000
participants.
Evaluation.
To
assess
the
progress
of
UFEP,
NSF
contracted
with
Westat,
Inc.,
of
Rockville,
Maryland,
to
evaluate
the
programs
performance
during
its
first
3
years.
The
evaluation
was
designed
to
consider
effectiveness
mainly
in
three
areas:
(1)
Did
the
projects
that
were
funded
meet
the
participants
needs?
(2)
Did
the
program
as
implemented
meet
the
needs
of
the
profession?
(3)
Were
the
programs
and
criteria
as
defined
by
NSF
appropriate
to
meet
program
goals?
In
approaching
the
evaluation,
Westat
obtained
data
through
questionnaires
completed
by
directors
of
91
of
the
92
projects
that
Exhibit
12:
Percent
of
Faculty
Enhancement
Program
(UFEP)
Participants
from
1988
to
1990
Who
Selected
Above-Average
Ratings
When
Asked
About
Specific
Benefits
Derived
from
Their
Experience
with
the
Program,
1993
Increased knowledge of the field
Increased
motivation
or
stimulation
for
teaching
excellence
Personal
growth
Increased
contacts
with
colleagues
from
other
institutions
New
perspectives
on
teaching
and
learning
Information
about
other
resources
for
use
in
teaching
Increased scholarly activity
Knowledge
and
skills
acquired
about
new
instructional
procedures,
materials,
or
equipment
76
73
73
64
60
59
43
58
Both
participants
and
project
directors
expressed
enthusiasm
for
the
UFEP
projects
in
which
they
were
involved,
finding
them
either
highly
valuable
or,
at
least,
very
worthwhile.
Awards
are
usually
granted
for
periods
of
6
to
15
months,
with
recipients
expected
to
spend
approximately
70
percent
of
the
period
on
research
activities
and
30
percent
on
teaching
and
other
interactive
activities.
Between
1982
and
1992,
approximately
100
women
applied
each
year,
and
29
percent
of
them
received
awards.
Evaluation.
The
Foundation
contracted
with
SRI
International
at
the
close
of
the
programs
first
decade
to
evaluate
the
outcomes
of
the
visiting
professorship
program
and
also
to
identify
the
barriers
that
women
perceive
as
retarding
their
progress
in
science
and
engineering
careers.
Thus,
the
evaluation
was
a
two-phase
undertaking
the
first
half
dealing
with
career
advancement
for
women
in
science,
the
second
focusing
on
visiting
professorship
experiences
of
the
awardees
and
evidence
of
program
effects.
In
conducting
the
evaluation
of
the
programs
effectiveness,
SRI
International
gathered
quantitative
and
qualitative
data
through
mail
questionnaires.
Respondents
included
more
than
200
VPW
awardees
and,
for
purposes
of
comparison,
more
than
300
applicants
who
did
not
receive
an
award
(declinees),
plus
more
than
300
women
who
were
recipients
of
NSF
grants
other
than
those
offered
by
the
visiting
35 34
The
majority
of
participants
reported
modifying
their
teaching
methods,
introducing
new
content,
and
acquiring
new
equipment.
Findings
were
less
clear-cut
regarding
two
other
questions
that
the
evaluation
sought
to
answer:
Is
the
program
furthering
the
needs
of
the
science
and
mathematics
teaching
professions
in
general?
And
is
the
program
structured
suitably
to
meet
its
goals?
When
asked
to
assess
the
impact
of
the
program
on
their
profession,
educational
organization
members
and
people
holding
faculty
positions
varied
in
their
responses,
depending
on
their
disciplines.
Chemistry
instructors,
for
example,
were
very
affirmative,
while
physics
instructors
were
considerably
less
so.
And
concerning
UFEP
goals,
the
Assessment
Advisory
Committee
expressed
some
concerns:
participants
included
a
high
proportion
of
individuals
who,
even
without
program
encouragement,
were
already
very
active
professionally,
judging
from
their
abundant
publications
as
well
as
their
frequent
attendance
at
meetings,
seminars,
and
workshops.
Furthermore,
the
committee
concluded
that
UFEP
administrators
might
be
able
to
do
a
better
job
of
encouraging
the
participation
of
women
and
minorities
in
the
program.
Overall,
however,
the
evaluation
indicated
that
the
program,
despite
its
weaknesses,
has
been
of
some
value
in
faculty
development
and
renewal.
The
Assessment
Advisory
Committee
recommended
that
a
wider
net
be
cast
in
recruiting
participants,
especially
women,
minorities,
and
others
who
are
not
among
the
most
professionally
active
members
of
their
profession.
In
response
to
the
evaluation,
NSF
made
several
changes
to
the
UFEP
funding
guidelines.
First,
to
ensure
the
participation
of
those
faculty
who
are
most
in
need
of
professional
development,
NSF
now
strongly
encourages
UFEP
proposers
to
solicit
the
participation
of
faculty
at
2-
year
institutions
and
faculty
just
starting
their
academic
careers.
Second,
the
guidelines
were
modified
to
place
more
emphasis
on
the
participation
of
women
and
members
of
underrepresented
racial
and
ethnic
groups
in
UFEP
workshops.
Finally,
the
eligibility
requirements
of
the
program
were
expanded
to
include
scientific
societies
and
associations
as
UFEP
host
organizations.
The
majority
of
UFEP
participants
reported
modifying
their
teaching
methods,
introducing
new
content,
and
acquiring
new
equipment.
Approximately
90
percent
of
awardees
rated
VPW
as
having
a
positive
impact
on
their
professional
development,
research
activity,
research
career
generally,
and
scientific
reputation.
The
evaluators
concluded
that,
for
participants,
the
main
benefits
were
increased
research
productivity,
enhanced
professional
development,
and
an
expanded
network
of
professional
support
(see
Exhibit
13).
The
evaluation
revealed
that
a
disproportionately
small
fraction
of
VPW
grants
had
been
awarded
to
women
from
underrepresented
racial
and
ethnic
groups.
In
response,
the
NSF
evaluation
staff
recommended
that
the
program
be
refashioned
to
better
promote
diversity
within
the
scientific
workforce.
In
addition,
while
the
evaluation
demonstrated
that
the
program
had
succeeded
in
expanding
research
careers
of
the
awardees,
NSF
felt
the
impact
of
the
program
on
the
education
of
young
women
through
provision
of
quality
teaching,
advising,
and
mentoring
could
be
improved.
This
realization
contributed
to
NSFs
decision
to
move
the
program
into
the
Division
of
Human
Resource
Development
and
consolidate
it
with
NSFs
activities
for
girls
and
women
at
the
kindergarten
through
undergraduate
levels.
Experimental
Program
to
Stimulate
Competitive
Research
(EPSCoR)
Description.
EPSCoR
came
into
being
in
1979
in
response
to
national
concerns
about
the
geographic
distribution
of
research
and
development
funding.
Originally
directed
at
enhancing
the
capabilities
of
universities
that
previously
had
not
competed
well
for
NSF
awards,
program
goals
have
been
broadened
to
include
enhancement
of
educational
opportunities
for
underrepresented
faculty
and
students.
Thus,
improvement
of
science
education,
coupled
with
human
resource
development
through
the
participation
of
women,
minorities,
and
persons
with
disabilities,
has
become
central
to
the
programs
objectives.
EPSCoR
is
a
partnership
program,
with
NSF
funding
intended
to
stimulate
local
action
and
investment
that
will
result
in
lasting
37 36
professorship
program.
The
study
was
completed
in
1993;
two
publications
have
resulted
from
it:
The
Visiting
Professorships
for
Women
Program:
Lowering
the
Hurdles
for
Women
in
Science
and
Engineering:
Final
Report
(SRI
International
1993).
The
Visiting
Professorships
for
Women
Program:
Lowering
the
Hurdles
for
Women
in
Science
and
Engineering:
NSF
Summary
and
Comments
(SRI
International
1994).
Findings.
The
study
was
primarily
devoted
to
assessing
the
first
program
goal:
providing
advancement
opportunities
for
participants.
The
evaluation,
based
on
participants
selfreports,
documents
a
number
of
benefits.
Approximately
90
percent
of
awardees
rated
VPW
as
having
a
positive
impact
in
four
areas:
their
professional
development,
research
activity,
research
career
generally,
and
scientific
reputation.
Also,
more
than
80
percent
of
questionnaire
respondents
reported
a
positive
impact
on
their
relationships
with
colleagues
and
on
their
selfesteem.
Awardees
considered
the
chief
benefit
of
the
program
to
be
the
improvement
in
the
quality
and
theoretical
grounding
of
their
research
and
the
opportunity
to
devote
large
blocks
of
time
to
it,
an
opportunity
they
were
not
in
the
habit
of
enjoying
at
their
home
institutions.
Compared
with
VPW
applicants
who
had
not
received
awards
and
women
who
had
received
other
NSF
grants,
VPW
awardees
had
authored
more
published
papers
following
their
time
as
visiting
professors
and
had
also
more
often
progressed
to
institutions
with
reputations
for
research
excellence.
However,
because
awardees
were
older
and
more
often
held
tenured
positions,
there
were
fewer
gains
with
respect
to
tenure
and
advancement
among
awardees
than
among
66
45
39
20
14
Research
facilitation
and
accomplishments
Research
competence;
increased
skills
and
knowledge
Career
advancement;
enhanced
status
Expanded
network;
professional
contacts
Opportunity
to
provide
support
for
female
students
Exhibit
13:
Percent
of
Visiting
Professorships
for
Women
(VPW)
Program
Awardees
from
1982
through
1992
Reporting
Various
Types
of
Program
Benefits
They
Experienced,
1993
The
main
benefits
of
VPW
were
increased
research
productivity,
enhanced
professional
development,
and
an
expanded
network
of
professional
support.
At
each
site
addressed
in
the
study,
students
are
exposed
to
a
wide
range
of
educational
opportunities,
including
working
in
research
laboratories
and
copublishing
with
principal
investigators.
Additionally,
students
have
been
offered
new
and
revised
courses,
new
degree
programs,
and
access
to
state-of-the-art
equipment
as
a
result
of
EPSCoR.
The
study
also
found
that
EPSCoR-supported
students
have
earned
numerous
honors
and
awards,
grants
and
scholarships,
and
noteworthy
placements
in
graduate
schools
or
the
professional
work
place.
High
proportions
of
supported
graduate
and
postdoctoral
students
have
authored
or
coauthored
professional
publications
(see
Exhibit
14).
Moreover,
EPSCoR
funds
have
supported
women
and
minorities
in
two
ways:
individual
support
for
students
participating
as
part
of
EPSCoR
research
teams;
and
programmatic
support
for
university
initiatives,
such
as
conferences,
travel
grants,
and
mentoring
activities.
To
varying
degrees,
these
efforts
have
succeeded.
For
example,
although
the
proportion
of
women
among
EPSCoR-supported
personnel
in
such
fields
as
the
physical
sciences
is
higher
than
the
proportion
of
women
involved
in
these
fields
nationally,
the
number
of
women
in
EPSCoR-supported
engineering
and
mathematics
programs
is
still
quite
low.
Likewise
regarding
the
participation
of
minorities,
although
Asians
are
well
represented,
blacks
and
Hispanics
are
underrepresented,
with
the
latter
two
groups
constituting
about
7
percent
of
EPSCoR-supported
graduate
students
about
the
same
as
their
representation
in
science
nationwide.
39 38
improvements
to
a
states
research
infrastructure
and
increased
national
research
and
development
competitiveness.
The
program
currently
operates
in
18
states
and
in
Puerto
Rico.
Of
the
61
institutions
involved
in
the
program,
the
great
majority
are
colleges
or
universities.
During
the
programs
history,
NSF
has
invested
more
than
$120
million,
while
states
have
provided
more
than
$275
million
in
supporting
EPSCoRs
objectives.
Evaluation.
An
impact
study
conducted
by
Cosmos
of
Bethesda,
Maryland,
in
1995
was
confined
to
the
educational
impacts
of
EPSCoR
and
was
primarily
descriptive.
(A
comprehensive
evaluation
of
the
program
is
currently
under
way
and
due
for
completion
in
1997.)
Two
categories
of
program
effects
were
examined
system
inputs
and
outputs.
System
inputs
included
the
number
of
enrolled
students,
outreach
activities
to
recruit
targeted
student
populations,
and
educational
opportunities
and
experiences
offered.
System
outputs
were
defined
as
diversity
(proportion
and
accomplishments
of
women
and
minorities)
and
educational
achievements
(degrees
awarded
and
other
accomplishments
of
EPSCoR-supported
students).
Data
for
the
impact
study
was
primarily
obtained
from
existing
sources,
especially
two
extensive
databases
that
NSF
uses
in
program
planning
and
evaluation.
In
addition,
information
was
obtained
from
annual
reports
submitted
by
states
and
site
visits
to
two
award
sites
(Montana
and
South
Carolina)
that
were
made
specifically
for
the
impact
study.
The
evaluation
yielded
one
publication:
A
Quick
Impact
Study
of
the
Educational
Impacts
of
the
EPSCoR
Program
(Cosmos
1995).
Findings.
This
study
uncovered
a
wide
range
of
positive
educational
impacts
that
may
justifiably
be
attributed
to
the
EPSCoR
initiative.
The
study
found,
for
example,
that
awardees
are
supporting
either
partially
or
in
their
entirety
a
considerable
number
of
diverse
outreach
activities
ranging
from
science
fair
to
K-12
teacher
training
all
of
which
are
aimed
at
increasing
the
number
of
students
studying
college-level
sciences.
1993 1994
Percent of Students
Exhibit
14:
Number
and
Percent
of
Students
Participating
in
Experimental
Program
to
Stimulate
Competitive
Research
(EPSCoR)
Who
Have
Produced
One
or
More
Publications,
1993
and
1994
Graduate
Undergraduate
Postdoctorate
Key
0
10
20
30
40
50
60
70
80
90
100
1993 1994
Number
of
Students
0
50
100
150
200
250
Students
have
been
offered
new
and
revised
courses,
new
degree
programs,
and
access
to
state-of-the-art
equipment
as
a
result
of
EPSCoR.
40
In
general,
this
impact
study,
documenting
a
wide
range
of
educational
impacts
associated
with
EPSCoR,
suggests
that
the
program
is
strengthening
the
educational
resources
and
competitive
status
of
the
institutions
it
has
funded.
However,
the
evaluators
stress
the
importance
of
keeping
in
mind
that
other
programs
and
activities
supported
by
NSF
and
by
state
and
private
organizations
may
have
contributed
to
some
of
the
beneficial
changes
observed
in
the
study.
Although
the
impact
study
found
that
substantial
progress
had
been
made
by
EPSCoR
states
in
improving
the
participation
of
women
and
ethnic
minorities,
the
NSF
evaluation
staff
felt
that
a
greater
opportunity
was
being
missed.
It
recommended
that
future
EPSCoR
funding
solicitations
explicitly
address
the
goal
of
increasing
the
diversity
of
the
scientific
and
technological
workforce
in
these
states.
The
EPSCoR
impact
study
suggests
that
the
program
is
strengthening
the
educational
resources
and
competitive
status
of
the
institutions
it
has
funded.
overseen
or
coordinated
many
other
activities
that
support
not
only
its
own
science
and
mathematics
education
assessment
efforts
but
also
extend
the
value
of
these
efforts
beyond
its
own
domain.
For
example,
REC
has
disseminated
a
number
of
documents
presenting
approaches
that
recipients
of
grants
from
EHR
and
other
agencies
might
take
in
developing
evaluations,
the
goal
being
to
elevate
the
skills
of
evaluation
in
general.
Also,
REC
has
coordinated
several
interagency
reports
and
has
contributed
substantially
to
the
development
of
various
teacher
enhancement
studies
and
curriculum
surveys
that
shed
light
on
the
extent
to
which
progress
is
being
made
in
science
and
mathematics
education
reform.
Tools
for
Conducting
Evaluations.
The
evaluations
required
by
EHRs
many
functions
and
responsibilities
cannot
be
carried
out
using
a
single
model.
Some
projects
have
clear
short-term
goals
such
as
increasing
the
number
of
minority
graduates
who
obtain
bachelors
degrees
in
science
or
engineering
in
one
or
two
colleges
or
universities
and
can
be
performed
using
traditional
evaluation
methods.
Program
goals,
on
the
other
hand
such
as
increasing
the
interest
of
women
in
science
careers
are
long-
term
and
difficult
to
measure
or
attribute
to
specific
program
activities.
Evaluating
changes
in
educational
systems,
which
requires
systemic
evaluation,
is
an
even
more
complex
task.
To
date,
REC
has
begun
to
deal
with
these
issues,
as
is
evident
in
two
NSF
publications:
User-Friendly
Handbook
for
Project
Evaluation
was
developed
to
provide
principal
investigators
and
project
evaluators
with
a
basic
understanding
of
the
43
REC
has
disseminated
a
number
of
documents
presenting
approaches
that
recipients
of
grants
from
EHR
and
other
agencies
might
take
in
developing
evaluations.
Interagency
Support.
NSF
is
only
one
of
several
federal
agencies
that
make
no
substantial
investments
in
science,
mathematics,
engineering,
and
technology
education.
The
need
for
enhanced
coordination
of
federal
activities
in
this
area,
and
increased
evaluation
efforts
by
all
agencies,
has
been
addressed
by
several
coordinating
bodies.
In
1993,
for
example,
REC
supported
an
expert
review
panel
and
published
a
report
on
the
panels
findings
titled
The
Federal
Investment
in
Science,
Mathematics,
Engineering
and
Technology
Education:
Where
Now?
What
Next?
which
emphasized
the
need
for
evaluation
of
these
programs
throughout
the
federal
government.
REC
subsequently
coordinated
the
evaluation
of
projects
supervised
by
eight
agencies
that
provided
opportunities
for
teachers
from
around
the
Nation
to
study
at
federal
facilities.
Participants
spent
4
weeks
in
hands-on
education
programs
in
such
areas
as
environmental
studies,
materials
science,
and
space
research.
The
goal
was
to
improve
the
science
knowledge
of
the
participating
teachers,
and
thus
improve
middle
and
secondary
school
science
learning.
REC
issued
a
report
on
this
project,
titled
Evaluation
Report:
FCCSET/DOE:
1993
Summer
Institutes.
Currently,
a
multi-agency
study
of
teacher
enhancement
and
development
programs
is
being
conducted
by
a
group
of
researchers
45 44
evaluation
process
and
the
tools
to
carry
out
the
evaluations
required
of
EHR
grantees.
These
evaluations
should
be
used
by
project
directors
and
principal
investigators
to
improve
their
projects
as
they
develop
and
progress,
as
well
as
to
document
the
results
that
were
achieved.
Because
most
projects
have
measurable
short-term
goals,
the
emphasis
in
the
handbook
is
on
quantitative
evaluation
methods,
including
sampling,
data
collection,
and
report
writing.
Frequent
regional
workshops
using
the
handbook
are
offered
to
project
directors
and
staff,
most
of
whom
have
no
prior
experience
in
conducting
evaluations.
A
videotape
illustrating
the
handbooks
highlights
was
also
prepared.
Both
the
handbook
and
the
workshops
have
been
very
well
received
by
the
intended
audience.
Footprints:
Strategies
for
Non-Traditional
Program
Evaluation
is
a
collection
of
commissioned
papers
authored
by
experts
in
the
educational
research
community
who
had
been
asked
for
fresh
ideas
and
new
methodologies
that
might
inform
the
design
of
EHR
evaluations.
The
papers
were
first
presented
and
discussed
at
a
conference
held
under
NSF
auspices.
Footprints
were
defined
as
evidence
of
a
programs
impact
on
the
field
of
education,
on
scholarship,
on
other
institutions,
and
on
educational
practice.
The
contributors
brought
different
backgrounds,
philosophies,
and
approaches
to
the
task,
and
they
offered
a
variety
of
innovative
evaluation
schemes.
The
book
offered
strong
suggestions
to
adopt
mixed
evaluation
strategies
that
incorporate
both
quantitative
and
qualitative
elements.
Innovating
and
Evaluating
Science
Education:
NSF
Evaluation
Forums
1992-94
contains
selected
papers
from
a
series
of
evaluation
forums
sponsored
by
REC
REC
has
coordinated
several
interagency
reports
and
has
contributed
substantially
to
the
development
of
various
teacher
enhancement
studies
and
curriculum
surveys.
Teacher
Enhancement
for
Elementary
and
Secondary
Science
and
Mathematics:
Status,
Issues
and
Problems.
This
book,
edited
by
Abt
Associates
and
published
by
NSF
in
1994,
consists
of
chapters
written
by
11
experts
in
the
field
who
examined
a
number
of
issues,
including
student
learning
and
curriculum
reform,
that
affect
teacher
enhancement
efforts.
The
book
seeks
to
help
educators
who
are
responsible
for
planning
teacher
enhancement
programs
by
sharing
what
has
been
learned
about
successful
programs.
The
major
themes
are
the
need
for
teachers
to
understand
the
ways
in
which
students
actually
learn
and
the
impact
of
curriculum
changes
and
educational
reform
on
teachers
instructional
practices.
An
Assessment
of
the
Capacity
of
Federal
Laboratories
to
Provide
Inservice
Teacher
Enhancement
Programs
was
prepared
by
Abt
Associates
and
Quantum
Research
Corporation
in
1995.
Laboratories
of
nine
federal
agencies
were
surveyed
regarding
the
operation
of
programs
specifically
designed
to
enhance
the
skills
of
K-12
teachers
and
broaden
their
future
capacity
to
offer
such
programs.
By
their
own
estimates,
the
federal
laboratories
already
sponsoring
teacher
47 46
from
Westat,
SRI,
and
the
National
Center
for
Improving
Science
Education.
The
purpose
of
this
study
is
to
further
the
understanding
of
the
impact
of
these
projects,
starting
with
those
that
reflect
what
are
believed
to
be
best
practices.
A
report
on
the
first
phase
of
the
project
Teacher
Enhancement/Development
Study:
A
Look
at
Best
Practices:
Phase
I
has
been
completed.
Other
Teacher
Enhancement
Studies.
A
large
portion
of
the
federal
investment
in
science
and
mathematics
education
is
directed
at
students
in
grades
K-12,
and
especially
at
inservice
education
for
their
teachers.
Within
EHR
alone,
close
to
one-third
of
the
fiscal
1995
budget
($200
million
out
of
$600
million)
was
spent
on
programs
in
elementary
and
secondary
education.
About
half
of
this
amount
was
allocated
to
inservice
projects
to
improve
the
qualifications
and
effectiveness
of
mathematics
and
science
teachers.
The
projects
emphasize
activities
that
support
NSFs
agenda
for
reform
in
science
and
mathematics
education.
Other
government
agencies
and
organizations
are
also
investing
significantly
in
teacher
enhancement.
Although
a
full-impact
study
of
the
teacher
enhancement
is
still
in
planning,
REC
has
published
a
number
of
studies
that
have
provided
useful
background
data
and
perspectives
for
program
managers
and
policymakers.
These
completed
studies
include
descriptive
information
about
the
activities
that
the
teacher
enhancement
program
has
supported
in
recent
years,
a
broad
investigation
of
issues
affecting
teacher
improvement,
and
an
examination
of
the
characteristics
and
effects
of
teacher
enhancement
activities
carried
out
by
public
and
private
agencies
in
the
United
States
over
the
past
four
decades.
Teacher
Enhancement
Programs:
A
Perspective
on
the
Last
Four
Decades,
a
report
prepared
by
Westat,
Inc.,
is
based
on
extensive
bibliographic
research.
The
study
included
a
historical
perspective
(1950-90)
on
teacher
enhancement
programs
and
examined
the
major
programs
currently
funded
in
terms
of
their
goals
and
impacts.
This
review
concluded
that
teacher
enhancement
programs
provide
significant
benefits
in
terms
of
several
goals:
enhancing
teachers
acquisition
of
new
knowledge,
renewal,
and
professional
leadership.
The
review
pointed
out
a
paucity
of
studies
that
assess
the
impact
of
such
programs
on
student
learning
and
performance
and
suggested
the
need
for
this
focus
in
the
future.
It
is
hoped
that
the
REC
evaluation
efforts
noted
above
and
others
now
underway
or
in
the
planning
stage
will
put
EHR
program
managers,
grantees,
and
contractors
in
a
better
position
to
achieve
the
Foundations
goal
of
improving
the
Nations
science
and
mathematics
education
at
all
levels.
49 48
enhancement
programs
could
expand
the
number
of
teachers
served
from
the
53,000
involved
in
1994
to
almost
90,000
teachers
served
annually.
The
authors
considered
the
laboratories
estimates
to
be
conservative
and
consequently
used
a
mathematical
model
to
estimate
capacity
under
alternative
scenarios.
Many
nonparticipating
laboratories
reported
that
they
could
sponsor
teacher
enhancement
activities
that
would
increase
the
percentage
of
participating
laboratories
from
53
percent
to
81
percent.
The
report
also
provides
descriptive
information
about
teacher
enhancement
programs
currently
provided
and
factors
influencing
the
ability
of
the
laboratories
to
reach
maximum
numbers
of
teachers
in
the
future.
Surveys
of
Science
and
Mathematics
Curricula.
Additionally,
several
REC-sponsored
surveys
have
yielded
a
considerable
body
of
pertinent
data
on
the
extent
to
which
school
governing
bodies
and
individual
educators
are
responding
to
the
call
for
improved
science
and
mathematics
education.
These
studies
have
been
far-ranging,
covering
such
issues
as
the
ways
in
which
new
scientific
knowledge
is
incorporated
into
classroom
teaching,
the
attempts
being
made
by
teachers
to
enhance
their
skills,
and
the
gains
that
students
are
showing
as
a
result
of
reform
efforts.
A
Profile
of
Science
and
Mathematics
Education
in
the
United
States:
1993,
based
on
surveys
conducted
by
Horizon
Research,
suggests
that
much
remains
to
be
done
to
enhance
the
training
of
K-12
teachers
in
science
and
mathematics.
Thus,
only
26
percent
of
elementary
teachers
feel
sufficiently
qualified
to
teach
science,
and
only
about
one-third
of
elementary
and
middle
school
teachers
had
spent
more
than
15
hours
on
mathematics
and
science
inservice
education
in
the
last
3
years.
Several
REC-sponsored
surveys
have
yielded
a
considerable
body
of
pertinent
data
on
the
extent
to
which
school
governing
bodies
and
individual
educators
are
responding
to
the
call
for
improved
science
and
mathematics
education.
Training
in
evaluation
methods,
based
on
the
UserFriendly
Handbook
for
Project
Evaluation,
was
initiated
for
the
Foundations
internal
evaluation
staff
and
project
directors;
Interagency
leadership
and
cooperation
in
evaluation
was
successfully
implemented;
and
A
research
program
supporting
evaluation
activities
was
launched.
The
major
task
now
facing
the
evaluation
program
staff
is
to
complete
the
evaluations
of
all
EHR
programs.
Studies
about
to
get
underway
will
examine
the
following
programs:
Urban
Systemic
Initiatives
Rural
Systemic
Initiatives
Calculus
Reform*
Graduate
Research
Traineeships
Undergraduate
Faculty
Enhancement
Advanced
Technological
Education
Institution-wide
Reform
of
Undergraduate
Education*
53
* An initiative of the Course and Curriculum Development Program
Evaluation
training
also
is
taking
a
new
direction
in
a
joint
project
between
the
Foundation
and
the
National
Aeronautics
and
Space
Administration
(NASA),
which
is
exploring
the
use
of
educational
performance
indicators
in
evaluating
its
programs.
This
effort,
emphasizing
the
management
of
information
in
a
NASA
database,
is
expected
to
generate
an
evaluation
training
manual
and
workshop
materials
that
will
be
useful
to
federal
agencies
in
addition
to
NASA.
Moreover,
REC
will
continue
to
explore
opportunities
with
other
agencies
as
well
for
example,
a
new
project
that
is
being
carried
out
in
collaboration
with
the
Department
of
Energy
and
the
Department
of
Education.
This
evaluation
will
look
at
the
impact
of
federal
programs
that
increase
the
participation
of
underrepresented
groups
in
science
and
mathematics
and
how
these
programs
can
be
improved.
Finally,
EHR
will
continue
to
expand
its
grants
program
in
evaluation,
the
goal
being
to
improve
evaluation
models
and
techniques
and
to
ensure
that
evaluations
are
undertaken
of
major
mathematics
and
science
education
initiatives.
55 54
As
the
cycle
of
specific
program
evaluations
approaches
completion,
RECs
evaluation
efforts
will
have
greater
flexibility
and
resources
to
address
the
broader
educational
issues
associated
with
several
EHR
programs,
such
as
teacher
preparation
and
programs
for
young
people.
A
good
example
of
such
issues
is
teacher
enhancement.
Although
the
effort
to
improve
the
scientific
and
pedagogical
skills
of
the
Nations
science
and
mathematics
teachers
is
the
specific
focus
of
one
EHR
program,
related
activities
play
a
major
role
in
a
variety
of
other
programs,
such
as
the
Statewide,
Urban,
and
Rural
Systemic
Initiatives;
Alliances
for
Minority
Participation;
and
Collaboratives
for
Excellence
in
Teacher
Preparation.
Thus,
the
subject
of
teacher
enhancement
is
too
broad
to
be
covered
in
a
single
evaluation
design.
Recognizing
this
fact,
REC
plans
to
accumulate
data
on
projects
that
are
oriented
specifically
toward
teacher
enhancement;
to
augment
those
findings
with
information
gathered
from
other
projects
yielding
pertinent
teacher
enhancement
evaluation
data;
and
to
support
new
projects
that
will
help
fill
the
gaps
in
this
synthesis
of
data.
Although
RECs
major
focus
will
continue
to
be
on
the
evaluation
of
EHRs
programs,
the
Divisions
encouragement
of
better
and
more
frequent
evaluations
of
EHR
projects
is
among
its
long-term
goals.
A
series
of
evaluation
awareness
workshops
for
NSF
staff
members
and
principal
investigators
is
underway.
A
second
series
is
being
planned.
It
is
hoped
that
these
workshops
will
clarify
to
NSF
staff
the
Foundations
evaluation
expectations.
Also
in
progress
is
the
development
of
a
second
handbook
for
project
evaluators,
which
will
explore
the
use
of
quantitative
and
qualitative
models.
This
handbook
is
being
created
in
response
to
requests
from
current
workshop
attendees
that
there
be
more
discussion
of
how
to
use
qualitative
techniques
in
a
rigorous
manner.
Several
other
activities
also
will
contribute
to
long-term
improvement
of
NSF
project
evaluations.
Through
a
grant
from
the
American
Educational
Research
Association,
a
program
will
be
initiated
in
the
fall
of
1996
to
support
selected
graduate
students
with
mathematics
and
science
backgrounds
in
preparing
for
careers
in
evaluation.
Additionally,
REC
will
encourage
the
use
of
stronger
RECs
evaluation
efforts
in
the
future
will
have
greater
flexibility
and
resources
to
address
the
broader
educational
issues
associated
with
several
EHR
programs.
Moreover,
REC
will
continue
to
explore
opportunities
with
other
agencies.
See bottom of the back of this form for necessary ordering information.
Elementary
and
Secondary
Education
Evaluation
of
the
National
Science
Foundations
Statewide
Systemic
Initiatives
(SSI)
Program:
First
Year
Report,
Volume
I,Technical
Report
(SRI
International
1994).
Evaluation
of
the
National
Science
Foundations
Statewide
Systemic
Initiatives
(SSI)
Program:
Second
Year
Report:
Cross-Cutting
Themes
(SRI
International
1995).
Evaluation
of
the
National
Science
Foundations
Statewide
Systemic
Initiatives
(SSI)
Program:
Second-Year
Case
Studies:
Connecticut,
Delaware,
and
Montana
(SRI
International
1995).
Short-Term
Impact
Study
of
the
Presidential
Awards
for
Excellence
in
Science
and
Mathematics
Teaching
(Westat
1994).
Short-Term
Impact
Study
of
the
National
Science
Foundations
Young
Scholars
Program
(Westat
1994).
Informal
Science
Education
Program:
Evaluation
Design
Brief
and
Report
of
Preliminary
Findings
(Cosmos
1995).
Higher
Education
A
Short-Term
Impact
Study
of
the
National
Science
Foundations
Instrumentation
and
Laboratory
Improvement
Program
(ILI)
(Westat
1996).
Evaluation
of
the
Division
of
Undergraduate
Educations
Course
&
Curriculum
Development
Program,
Interim
Report
(The
Network,
Inc.
1995).
Assessment
of
the
National
Science
Foundations
1988-1990
Undergraduate
Faculty
Enhancement
Program:
Final
Report
(Westat
1993).
Assessment
of
the
National
Science
Foundations
1988-90
Undergraduate
Faculty
Enhancement
Program:
Executive
Summary
(Westat
1993).
Assessment
of
the
National
Science
Foundations
Undergraduate
Faculty
Enhancement
Program:
Interpretive
Overview.
A
Statement
from
the
Assessment
Advisory
Committee
(Westat
1993).
The
Visiting
Professorships
for
Women
Program:
Lowering
the
Hurdles
for
Women
in
Science
and
Engineering:
Final
Report
(SRI
International
1993).
The
Visiting
Professorships
for
Women
Program:
Lowering
the
Hurdles
for
Women
in
Science
and
Engineering:
NSF
Summary
and
Comments
(SRI
International
1994).
A Quick Impact Study of the Educational Impacts of the EPSCoR Program (Cosmos 1995).
Tools
for
Conducting
Evaluations
User-Friendly
Handbook
for
Project
Evaluation
Footprints:
Strategies
for
Non-Traditional
Program
Evaluation
Innovating
and
Evaluating
Science
Education:
NSF
Evaluation
Forums
1992-94
Continues
on
back
s
Other
Teacher
Enhancement
Studies
Teacher
Enhancement
Programs:
A
Perspective
on
the
Last
Four
Decades
A
Study
of
NSF
Teacher
Enhancement
Programs
(TEP)
Participants
and
Principal
Investigators:
1984-89
Teacher
Enhancement
for
Elementary
and
Secondary
Science
and
Mathematics:
Status,
Issues
and
Problems
An
Assessment
of
the
Capacity
of
Federal
Laboratories
to
Provide
Inservice
Teacher
Enhancement
Programs
Surveys
of
Science
and
Mathematics
Curricula
A
Profile
of
Science
and
Mathematics
Education
in
the
United
States:
1993
State
Curriculum
Frameworks
in
Mathematics
and
Science:
How
are
They
Changing
Across
the
States?
ORDERING
INFORMATION
If
ordering
by
mail
or
fax,
please
legibly
print
your
name
and
mailing
address
in
the
space
provided.
MAIL:
NSF
Evaluation
Reports
Center,
Room
855
4201
Wilson
Boulevard
Arlington,
Virginia
22230
FAX: (703) 306-0434
NAME:
__________________________________________________________________________
ADDRESS:
__________________________________________________________________________
__________________________________________________________________________
__________________________________________________________________________
__________________________________________________________________________
If
ordering
by
e-mail,
include
your
name,
mailing
address,
and
titles
of
the
reports
you
are
ordering
in
your
message.
E-MAIL: REC@NSF.GOV
T
he
Foundation
provides
awards
for
research
in
the
sciences
and
engineering.
The
awardee
is
wholly
responsible
for
the
conduct
of
such
research
and
preparation
of
the
results
for
publication.
The
Foundation,
therefore,
does
not
assume
responsibility
for
the
research
findings
or
their
interpretation.
The
Foundation
welcomes
proposals
from
all
qualified
scientists
and
engineers
and
strongly
encourages
women,
minorities,
and
persons
with
disabilities
to
compete
fully
in
any
of
the
research-related
programs
described
here.
In
accordance
with
federal
statutes,
regulations,
and
NSF
policies,
no
person
on
grounds
of
race,
color,
age,
sex,
national
origin,
or
disability
shall
be
excluded
from
participation
in,
be
denied
the
benefits
of,
or
be
subject
to
discrimination
under
any
program
or
activity
receiving
financial
assistance
from
the
National
Science
Foundation.
Facilitation
Awards
for
Scientists
and
Engineers
with
Disabilities
(FASED)
provide
funding
for
special
assistance
or
equipment
to
enable
persons
with
disabilities
(investigators
and
other
staff,
including
student
research
assistants)
to
work
on
NSF
projects.
See
the
program
announcement
or
contact
the
program
coordinator
at:
(703)
306-1636
The
National
Science
Foundation
has
TDD
(Telephonic
Device
for
the
Deaf)
capability,
which
enables
individuals
with
hearing
impairment
to
communicate
with
the
Foundation
about
NSF
programs,
employment,
or
general
information.
To
access
NSF
TDD
dial:
(703)
306-
0090
for
FIRS
dial:
1-800-
877-8339
Official
Business
Penalty
for
private
use
$300
NSF 96-140