Evaluation and Assessment Capability (EAC)
Program Monitoring and Evaluation
EAC was established to provide NSF with enhanced capability to operate from a basis of evidence in program and policy decisions; to more consistently assess the influence of its investments; and to establish a culture of evidence-based planning and policy making. EAC has been responsible for establishing mechanisms for NSF-wide leadership and coordination in program and portfolio evaluation, providing expert support and resources for data collection, integration, and management, and improving NSF-wide evaluation capacity.
In FY 2016, EAC staff managed the completion of two independent studies of the INSPIRE program that were conducted by SRI and Abt Associates. These studies confirmed the decision on the evolution of INSPIRE to RAISE, which retains the principles that support transformational, interdisciplinary research.
In FY 2017, EAC staff is evaluating five activities using its own resources and has negotiated nine evaluation/monitoring contracts in collaboration with lead-directorates of the programs. Each of these projects is summarized below.
- Broader Impacts (BI). This study focuses on the nature of BI evidence in proposals, review panel summaries, and annual reports. Findings will inform guidance on how principal investigators, reviewers and COV members consider broader impacts of the research.
- Intergovernmental Personnel Act (IPA). The purpose of this project is to assess the effect of the policy change requiring a mandatory 10 percent cost share on NSF’s IPA program. Comprehensive results from the pilot, expected before the end of FY 2018, will inform NSF about the effects of that policy.
- Innovations at the Nexus of Food, Water and Energy Systems (INFEWS). The purpose of this study is to determine to what extent and which ways the scientific community has addressed all of the systems in their responses to the solicitation and what changes this examination of the portfolio suggests for the next solicitation. Final results are anticipated before the end of FY 2017.
- National Academies (NA). The purpose of this project is to assess the impact of NSF-funded workshops and conferences convened by NA. Using data provided by NA and information gleaned from public websites, EAC is compiling information on number of report downloads, mention of NA studies in publications and legislation and influence of such studies on program solicitations. This work is ongoing and will be continually updated as NSF funds new awards to NA.
- Patents. This study examines patent data from the US Patent and Trademark Office for links to NSF awards. The results, such as time between the award of the grant and the subsequent patent, provide insights into the potential economic impacts of NSF investments.
- Broadening Participation (BP). This study, conducted by Windrose Vision, examines how BP investments are informing research, through empirical investigations of broadening participation issues or through broadening participation implementation research on what is making a difference. Final results are anticipated before the end of FY 2018.
- Data Asset Inventory. This study, conducted by ICF, will develop an inventory and assessment of the data assets currently available to support inquiry of NSF investments in human capital, particularly graduate education (excluding GRFP) and workforce development. The overarching purpose is to determine if data elements can be added to ongoing collections or standardized across collections to reduce the burden of future monitoring and evaluation efforts. Final results are anticipated before the end of FY 2017.
- Graduate Research Fellowship Program (GRFP). This activity, conducted by the National Opinion Research Center (NORC), encompasses the development of a data collection system that can be used to describe the fellows’ graduate school experiences and track career outcomes. This system, with a potential expansion to describe the education and career trajectories of all graduate students funded by NSF, is expected to be in place before the end of FY 2020.
- NSF Innovation Corps (I-CorpsÔ) Teams Program. This longitudinal evaluation of the I-CorpsÔ Teams Program, conducted by MSG, assesses the impact of the program on the team members as well as their academic institutions. The results of this evaluation will shed light on how I-CorpsÔ extends the focus of the researchers beyond the research environment. Final results are anticipated before the end of FY 2018.
- Technical assistance for Inclusion across the Nation of Communities of Learners of Underrepresented Discoverers in Engineering and Science (NSF INCLUDES). The purpose of this project, conducted by EDC, is to offer evaluation-related technical assistance to support the initial development of design, implementation, and assessment support activities for the NSF INCLUDES Launch Pilots. Technical assistance services may consist of, but are not limited to, coaching and training, experienced consultation/facilitation, tools and resources, technology and peer learning. This technical assistance will be provided until the end of FY 2019.
- Evaluation of Inclusion across the Nation of Communities of Learners of Underrepresented Discoverers in Engineering and Science (NSF INCLUDES). This comprehensive, developmental program-level evaluation, conducted by 2M Research, provides formative feedback to support continuous learning and improvement during the inaugural phase of the NSF INCLUDES initiative. It will assess the processes and progress for all Launch Pilot, Alliance and Backbone efforts for the next few years. Final results are anticipated before the end of FY 2019.
- Research Experience for Undergraduates (REU). The primary purpose of this effort, conducted by Mathematica, is to design, build, pilot, test and present findings for a web-based longitudinal data collection system for tracking REU Site participants. Building an evaluation framework around this data collection effort will also facilitate the measurement of participant outcomes. Final results and a proposed evaluation framework are anticipated before the end of FY 2019.
- Secure and Trustworthy Cyberspace (SaTC). This study builds on STPI findings from a review of historical data from investments in cybersecurity from 2008 to 2011. The primary emphasis of the evaluation will be on data from the inception of SaTC in 2012 to the present. An understanding of how and in what ways SaTC makes collective progress toward its goals and objectives will inform the use of these findings to refine existing and future SaTC program level activities. Final results are anticipated before the end of FY 2019.
- Science, Engineering and Education for Sustainability (SEES). This evaluation of the SEES portfolio, conducted by MSG, seeks to measure the success in terms of (1) the development of new knowledge and concepts that advance the overarching goal of a sustainable human future; (2) new and productive connections made among researchers in a range of disciplines; and (3) the development of a workforce capable of meeting sustainability challenges. Final results are anticipated before the end of FY 2018.
In FY 2017, EAC has initiated three new evaluation contracts, each of which is summarized below.
- Centers for Chemical Innovation (CCI). The purpose of this comprehensive assessment is to evaluate the effectiveness of the CCI program in achieving its stated goals. Key concepts of interest are influence of CCIs on the culture of and the nature of collaborative practices in the chemical sciences. The results of this study will be used to communicate the impact of the program and to strengthen the design and operation of the program. Final results are anticipated before the end of FY 2019.
- Established Program to Stimulate Competitive Research (EPSCoR). The purpose of this evaluation is two-fold: (1) to develop a flexible framework to explore, define and measure research competitiveness in relation to the unique jurisdictional contexts of each EPSCoR awardee; and (2) to collect and use evidence of jurisdictional progress toward research competitiveness over time for strategic program improvement. An understanding of how and in what ways progress is made towards increased research competitiveness will inform the use of these findings to refine existing and tailor future EPSCoR program level activities. Final results are anticipated before the end of FY 2020.
- GeoEd. This evaluation will inform strategic direction by describing the extent to which the GeoED portfolio is contributing to and progressing toward the achievement of program goals. The purpose of this evaluation is three-fold, to: (1) develop a flexible framework to define, measure, and explore value and impact; (2) provide evidence of the range, synergies, and variability across factors contributing to impact over time; and (3) strengthen the practice of evaluative inquiry for program improvement among GeoED decision makers and stakeholders. Final results are anticipated before the end of FY 2018.