SUMMARY OF ASSESSMENT AND EVALUATION PROCESSES
NSF makes use of qualitative information as well as quantitative data in determining progress toward achieving the Foundation’s goals. During FY 1999, NSF established new reporting systems and procedures, reporting guidelines, and templates to enable the collection, assessment, and analysis of the information and data necessary for reporting performance. Data sources for each goal as well as data limitations were identified. Steps to alleviate limitations were also identified. NSF has put in place methods to ascertain data quality problems and is developing methods for correction of the problems. For many of the goals, the data sources are central systems while other goals have required creation of new data/information systems to track programs. NSF has identified areas where complete data was not available for FY 1999 reporting, and is taking steps to modify its data systems, where appropriate. NSF will continue striving to improve its data quality and accuracy by improving processes, systems and data input and collection.
| NSF will continue striving to improve its data quality and accuracy by improving processes, systems and data input and collection. |
NSF’s programs and plans are reviewed throughout the year on a continuing basis by internal staff, senior management, and external advisory committees, to determine whether changes are necessary. Regular reviews also provide useful information for establishing new goals and redirecting efforts. Changes to programs and plans may be necessitated by inability to meet a goal, lack of appropriateness of a goal, or inability to measure a goal. These changes are handled through the performance plans and internal management. Major changes, when necessary, are handled through revisions to the strategic plan.
NSF receives and maintains performance information in the form of reports from the external community through Committees of Visitors (COVs) and Advisory Committees (ACs). COVs and ACs are committees made up of independent external experts from academia, industry, government and the public sector. COVs are subcommittees of directorate advisory committees, whose meetings are subject to Federal Advisory Committee Act rules.
Each year, approximately one-third of NSF’s programs are reviewed, so that all programs have been reviewed at the end of a three-year period. COVs and ACs produce reports, which are used in assessing NSF’s performance, and provide advice to NSF on needed improvements. These COV reports and advisory committee reports combined with data from internal databases, and with appropriate integration by NSF management, form the basis for NSF’s performance reports, performance plans, and strategic plans.
Program Assessment by Committees of Visitors (COVs)
COVs review NSF programs for quality of process and results on a 3-year cycle. COVs have access to all of NSF’s information and systems. COVs report to NSF on the integrity and efficiency of the processes for proposal review and the quality of resultsthe outputs and outcomes of NSF’s programs as they appear over time. In FY 1999, COVs were asked to judge whether NSF was successful in achieving Goals 1-4, and in implementing the new merit review criteria (Goal 7). COVs used a standardized reporting template with a set of questions addressing process, program management, and quality of outcomes to conduct their assessment of NSF programs. COVs also report on the noteworthy achievements of each year, ways in which projects have collectively affected progress, and expectations for future performance. The recommendations of COVs are reviewed by management and taken into consideration by NSF when evaluating existing programs and future directions for the Foundation. COV reports are public documents, available upon request.
Directorate Assessment by Advisory Committees
Directorate Advisory Committees, composed of external experts who have broad experience in academia, industry, and government, annually review the COV reports, available external assessments, and directorate annual reports, judging program effectiveness, and describing strengths and weaknesses. Directorate annual reports are a balanced representation of the directorate’s activities for the fiscal year, including results of COV reports and results from programs not reviewed by COVs in the fiscal year. Advisory committees have full access to a variety of data sources. Advisory committees use the combined COV reports and directorate annual reports as the basis for a strategic planning discussion. NSF management reviews the advisory committee reports and integrates the assessments into the NSF Annual Performance Report.
Integrating NSF’s Performance Report
The COV reports and advisory committee reports form the basis for reporting outcome goal results. Examples of outcomes are also selected from directorate annual reports, and other sources such as project reports, newspaper articles, publications, or other reports. NSF management integrates the results of the investment process goals and management goals with the outcome goals to produce this report. The resulting information is also used in preparing the annual performance plans and strategic plans.
