text-only page produced automatically by LIFT Text
Transcoder Skip all navigation and go to page contentSkip top navigation and go to directorate navigationSkip top navigation and go to page navigation
National Science Foundation HomeNational Science Foundation - Environmental Research and Education
Environmental Research and Education
design element
ERE Home
About ERE
Funding Opportunities
Awards
News
Events
Discoveries
Publications
Advisory Committee
See Additional ERE Resources
View ERE Staff
Proposals and Awards
Proposal and Award Policies and Procedures Guide
  Introduction
Proposal Preparation and Submission
bullet Grant Proposal Guide
  bullet Grants.gov Application Guide
Award and Administration
bullet Award and Administration Guide
Award Conditions
Other Types of Proposals
Merit Review
NSF Outreach
Policy Office
Additional ERE Resources
Follow ERE on Twitter
ERE Funding Opportunities


January 2004
Subchapter 300-Committee of Visitors Reviews

310     INTRODUCTION
320     COV REVIEW CYCLES, SCHEDULES, AND TIMING
330     COV RELATIONSHIP TO ADVISORY COMMITTEE AND FACA
340     ESTABLISHING AND SUPPORTING A COV
350     COV MEMBERSHIP
360     SCOPE OF COV REVIEW
370     INFORMATION ACCESSIBLE TO COVS
380     COV REPORTS AND RESPONSES TO COV REPORTS
390     RESPONSIBILITIES

300     COMMITTEE OF VISITORS REVIEWS

This subchapter provides guidelines to NSF staff for scheduling and conducting Committee of Visitors (COVs) reviews, and distributing COV reports.

310     INTRODUCTION

        311.   COV reviews provide NSF with external expert judgments in two areas: (1) assessments of the quality and integrity of program operations and program-level technical and managerial matters pertaining to proposal decisions; and (2) the degree to which the outputs and outcomes generated by awardees have contributed to the attainment of NSF's mission, strategic goals, and annual performance goals.

        312.   COV reviews are to be conducted at regular intervals of three years for programs and offices that recommend or award grants, cooperative agreements, and/or contracts and whose main focus is the conduct or support of NSF research and education in science and engineering.

        313.   The portfolio of activities under review by a COV should include a balanced representation of all types of actions administered by the program for the period under review including: examples of the core activities of the program, examples of NSF-wide and cross-directorate activities and special initiatives, examples of Small Grants for Exploratory Research (SGER's), center and facility activities, and other activities handled by the program.

        314.   Oversight of the COV review mechanism and schedule is the responsibility of the Office of Integrative Activities (OIA). On behalf of the NSF Director, OIA will facilitate the efforts of NSF directorates and offices to comply with the COV guidelines, and will provide periodic reports to the Director on the status of the COV mechanism and schedule.

320     COV REVIEW CYCLES, SCHEDULES, AND TIMING

        321.   Programs and offices that recommend or award grants, cooperative agreements, and/or contracts whose main focus is the conduct or support of research and education in science and engineering are to be reviewed by a COV at regular intervals of no more than three years.

        322.   NSF-wide activities and special initiatives are to be evaluated periodically in a broader context to determine their impact and/or progress. Those evaluations are described in the management plans for those activities and are to be scheduled separately from the COV reviews.

        323.  The scheduling of COV meetings throughout the fiscal year should be timed in accordance with the reporting requirements for Advisory Committee reviews and reporting requirements for the Government Performance and Results Act (GPRA). When determining the schedule, staff may take into account dates of prior reviews, new and reorganized programs, significant personnel changes, major budgetary changes, and other circumstances that have affected or are expected to affect the programs.

        324.   The timing of COV reviews should be arranged to the extent possible to ensure that there is an even distribution of program reviews conducted within a directorate or O/D office each year.

        325.   Programs that are related administratively or programmatically and can be grouped together within a cluster, section, or division, should be scheduled for COV review at the same time. Assistant Directors and O/D Office Directors will select the level of aggregation at which COVs will be conducted according to the specific needs of the division, directorate, or office, and to provide the appropriate coverage of program management.

        326.   A three-year schedule of COV reviews will be prepared by October 1 of every year by each Assistant Director and O/D Office Director who recommends awards for all such programs under his or her purview. The directorate/office COV schedule will be prepared annually in coordination with the appropriate Division Directors.

        327.   Assistant Directors and Office Directors will provide a copy of the three-year schedule of COVs to OIA at the beginning of each fiscal year. Subsequent changes to the COV schedule must be reported to OIA.

        328.   Each Assistant Director and O/D Office Director who recommends awards for programs under her or his purview will prepare, by October 1 of every year, for each COV conducted during the previous two fiscal years:

  • A list of recommendations made by the COV; The response to each of these recommendations by the Assistant Director or O/D Office Director; Actions already taken by the Assistant Director or O/D Office Director with respect to the response to each recommendation;
  • As appropriate, actions remaining to be taken.
This information will be made available to the relevant, incoming COV, program staff, and Division Director in a timely fashion or prior to the next COV meeting.

330     COV RELATIONSHIP TO ADVISORY COMMITTEE AND FACA

        331.  The Federal Advisory Committee Act (FACA) of 1972 requires that committees providing advice directly to the management of Federal agencies be chartered and operated in conformance with the procedural requirements of the FACA (see NSF Manual 1, Chapter IV, Subchapter 100-Management of Advisory Committees and Review Panels).

        332.  Subcommittees, such as NSF's COVs, that report to the chartered "parent" Advisory Committee, rather than the agency, do not meet the definition of an advisory committee in the FACA, and thus are not subject to the procedural requirements of the Act.

        333.  NSF COV meetings need not be announced in the Federal Register, nor are they required to be open to the public.

        334.  The COV chairperson, or designee, reports the COV findings and recommendations to the affiliated chartered parent Advisory Committee.

        335.  The affiliated, chartered parent Advisory Committee must review and discuss each report from an unchartered COV. The Advisory Committee discussion of the COV report may take the form of a face-to-face meeting or video teleconference under regular FACA procedures.

        336.  The OIA should be consulted for procedural information if an office determines a chartered COV is to be established which reports directly to NSF management independent of an Advisory Committee.

340     ESTABLISHING AND SUPPORTING A COV

        341.   An unchartered COV review, whether supported by NSF staff or by a contractor working under the guidance of NSF staff, must be established and accomplished in accordance with all of the provisions of this subchapter.

        342.   Information and guidelines concerning use of program funds to support the costs associated with the COV process are stipulated in NSF Manual 17, "Financial Management Policy Manual (FIN)", Section 171, "Program Support Costs."

        343.  The make-up of the COV, including the COV membership and the COV chair, is the responsibility of the Directorate or Office leadership.

        344.  Directorates and Offices will make logistical arrangements, provide background materials and agency guidelines for the COV, and will facilitate the efficiency and effectiveness of the COV review.

        345.  Directorates and Offices will appoint a person who will have responsibility for approving the COV agenda, for orienting the COV members and reviewing the COV responsibilities with the COV members (including confidentiality, conflict-of-interest, and Core Questions), and for ensuring that NSF Form 1230P, Conflict-of-Interests and Confidentiality Statement, is filed by each COV member and reviewed before the COV meeting, and appropriate records are kept.

        346.  Directorates and Offices may choose to select a contractor to support planning and implementation of a COV. The contractor's duties may include, but are not limited to, recruiting and processing COV members selected by the Directorate or Office, preparing background materials for the COV, making logistical arrangements, supporting the meeting on site, and assisting in the editing and formatting of the final report.

350     COV MEMBERSHIP

        351.  The selection of credible, independent experts who are able to provide balanced and impartial assessments of NSF programs and activities is critical to the credibility of the COV mechanism. Assistant Directors and O/D Office Directors are responsible for the selection of COV members. They should consult with the appropriate Division Director in guiding the selection of COV members, and also consult with the Chair of the affiliated Advisory Committee when selecting COV members.

        352.  Assistant Directors and Office Directors are responsible for ensuring that the selection of COV members contributes to a balanced, independent review that reflects a diversity of perspectives and is free from disqualifying conflicts of interest.

        353.   To the fullest extent possible, the composition of the committee should be balanced with respect to programmatic coverage, types of institutions under review, and geographic representation.

        354.  At least twenty-five percent of the COV membership should include qualified individuals who are not currently serving on any NSF Advisory Committee and have not been applicants to the program under review for at least five years. Any person with an action pending in a program under review may not participate as a COV member for that program. COV membership should also include at least one member of the affiliated Advisory Committee.

        355.  COV membership should include representatives of those disciplines, fields and activities that are impacted or affected by the outputs and outcomes resulting from NSF awards recommended by the program(s) under review. Examples include but are not limited to: separate but related disciplines, private industry, government agencies and laboratories, educators, foreign scientists and engineers, and other potential users.

360     SCOPE OF COV REVIEW

        361.   Each COV review should provide NSF with results and other information that can be integrated at the Foundation level, as specified in NSF's GPRA Strategic Plan and annual Performance Plan. The cognizant Assistant Directors and Office Directors may also use the COV mechanism to gather evaluation information on other aspects of program management and organizational performance and on issues such as investment strategies or priority setting of importance to the divisions and directorates.

        362.  The COV review of program management will consider proposal actions that were completed during the previous three fiscal years.

        363.  The COV review of awardee results will consider examples of the direct accomplishments of projects supported by the programs under review that are either currently active at the time of the COV review or were closed out during the previous three fiscal years.

        364.  A set of Core Questions and a Reporting Template for COV use, reflecting noteworthy changes in the annual NSF Performance Plan, will be issued and revised periodically by the Office of the Director. The revised set of questions will be issued in the form of an O/D Staff Memorandum and will be maintained by OIA. http://www.inside.nsf.gov/od/gpra/. OIA will provide assistance to all Directorates and Offices using these questions with COVs.

        365.  For each COV review, the cognizant Assistant Director or Office Director will, in collaboration with the Chair of the Advisory Committee or equivalent affiliated organization, prepare a written charge to the COV consisting of the Core Questions as well as other items of a general and program-specific nature of interest to the NSF.

        366.  The COV Core Questions and Reporting Template will be applied to the program portfolio and will address the proposal review process used by the program, program management, and the results of NSF investments. Specific questions to be addressed and reported on are:

  1. the integrity and efficiency of processes used to solicit, review, recommend and document proposal actions, including such factors as:
    1. selection of an adequate number of highly qualified reviewers who are free from bias and/or conflicts of interest;
    2. appropriate use of NSF merit review criteria;
    3. documentation related to program officer decisions regarding awards and declines, and the scope, duration and size of projects;
    4. balance of awards in terms of subject matter; emerging opportunities; high risk and innovation; size versus number of awards; new investigators; diversity of underrepresented groups; geographic distribution of principal investigators; and
    5. overall technical management of the program.
  2. the relationships between award decisions, program goals, and Foundation-wide programs and goals;
  3. results, in the forms of outputs and outcomes of NSF investments for the relevant fiscal years, as they relate to the Foundation's current strategic goals and annual performance goals.
  4. the significant impacts and advances that have developed since the previous COV review and are demonstrably linked to NSF investments, regardless of when these investments were made. Examples might include new products or processes, or new fields of research whose creation can be traced to the outputs and outcomes of NSF-supported projects over an extended period of time.
  5. response of the program(s) under review to recommendations of the previous COV review (see 378).

        367.   NSF staff may assist a COV technically in the preparation or review of a COV report only to the extent of ensuring that the confidentiality of NSF material is not compromised and to correct errors of a factual nature.

370     INFORMATION ACCESSIBLE TO COVS

        371.   COV members will be informed that the names and comments of reviewers are confidential, and that information related to declinations is confidential.

        372.  COV members will not review any proposal jacket with which they have been determined to have a conflict-of-interest. COV members will not review any proposal from their own institution or any proposal in which they collaborated or participated.

        373.   COVs will have access to:

  • documentation of completed proposal actions for the period under review;
  • a list of all projects closed out during the period under review;
  • annual program reports covering the period under review which include summary data on the portfolio relevant to activities in the program (per Section 366 and 375);
  • examples of results in the form of outputs and outcomes from current and recently completed projects;
  • examples of connections between outputs and outcomes from NSF projects and significant impacts or advances resulting from NSF-supported projects; and
  • the most recent COV report.

        374.   COV members will have access to all program documentation completed during the period under review, including proposals. The COV members will review a balanced sample of proposals representative of the portfolio of the program, including awards, declines, borderline proposals, proposals returned as inappropriate, and proposals that were withdrawn. The COV members may choose to review proposals through a random selection process of their own.

        375.   To facilitate the COV review, each program will prepare an annual summary report on program performance that covers the period under review and reflects the portfolio of activities for the program, including Foundation-wide activities. Centralized NSF data modules should be used where possible. Examples of the information and data that should be provided in the annual report include:

  1. Total number of proposals received in the program;
  2. Numbers of proposals awarded, declined, withdrawn, and returned as inappropriate;
  3. Data and funding rates for categories of PIs;
  4. Methods of merit review used;
  5. Data pertaining to categories of reviewers (including new PIs and members of underrepresented groups);
  6. Information on issuance of new solicitations and guidelines;
  7. Participation and role of the program in cross-cutting and NSF-wide activities;
  8. Number of new projects, continuing projects, and projects closed out;
  9. Examples of awardee results (with award number) reported during the period under review including:
    1. outputs and outcomes resulting from projects in progress or closed out during the period under review;
    2. significant impacts and advances linked to the outputs and outcomes of NSF investments made at other times; and
    3. results relevant to NSF's mission, and to outcome goals and areas of emphasis that apply to the period under review.

        376.   NSF program officers will provide appropriate examples of awardee results for each year under review in the annual program report (see Sections 363 and 375). Results may be selected from the Annual and Final Project Reports of NSF funded projects that were on-going or closed out during the period under review, and examples of results from other sources. The information provided might vary depending on the type of project supported and how it relates to the goals of the activity being reported and/or NSF-wide goals. Examples selected should reflect the most significant accomplishments in a program's portfolio of support within the reporting period, and may identify topics or areas requiring special consideration or attention by the program. It is not necessary to select samples from each project closed out during the previous three fiscal years: COV members will be provided with a list of all projects closed out during the period under review, and will have access to final project reports upon request.

        377.   Results from NSF supported research and education projects may lead to discoveries and achievements in areas beyond the immediate project and may eventually have broader impacts such as service to society. The connections between the initial project and broader impacts may be anecdotal. In preparing annual program reports throughout the year, program staff should look for examples of links and connections between results of awards and NSF goals, and strive to present examples of such connections in annual program and performance reports and for COV reviews whenever possible.

        378.   The COV members will be provided with the most recent COV report and the related unit's response to that report.

380     COV REPORTS AND RESPONSES TO COV REPORTS

        381.   The COV chairperson will furnish a written report to the parent Advisory Committee or its delegated representative. The parent Advisory Committee, or delegated representative, will transmit the COV report(s) to NSF management and request NSF management to respond to recommendations made in the COV report(s).

        382.   The COV report:

  1. will address the COV charge, describe the method of review, including the number of files it examined and method for selecting them, and the nature of other information provided by the programs under review, and will include any COV findings, recommendations, or suggestions;
  2. will include all components of a standard COV Reporting Template as provided in the Core Questions, which are approved by the NSF Director and maintained by OIA http://www.inside.nsf.gov/od/gpra/;
  3. may also include additional sections specific to the programs under review.

        383.  The COV report will be transmitted by the COV chairperson to the appropriate Advisory Committee (or delegated representative) within two weeks of the COV meeting, and at least four weeks prior to the Advisory Committee meeting at which the report will be reviewed.

        384.  When received, the Advisory Committee chairperson, or designated representative, will transmit the COV report to the cognizant Assistant Director or Office Director and request a written response.

        385.  The Assistant Director or Office Director will

  1. prepare a written response to the COV report. The response should describe action(s) to be taken on each suggestion or recommendation where action is appropriate.
  2. prepare a document describing the diversity, independence, and balance represented by the members of the COV, and the resolution of real or apparent conflicts of interest.
  3. transmit the written response and the document describing diversity and conflict resolution to the Advisory Committee and NSF senior management.
  4. return any COV report to the Advisory Committee, or delegated representative, which does not adequately address the Core Questions.

        386.  The following COV documents will be made available by the Assistant Director or Office Director in electronic format within 15 days of the Advisory Committee review of the COV report. Notification of access to documents should be provided to the NSF Director, Chief Operating, Financial and Information Officers, the NSF Inspector General, Office of Integrative Activities Director, and the NSF Committee Management Officer:

  1. the COV report, including a list of COV members and charge to the COV;
  2. the Assistant Director's or Office Director's report on COV membership and resolution of conflicts as specified in Section 385.b;
  3. the Assistant Director's or Office Director's response to the COV report(s);
  4. the parent Advisory Committee's written comments relevant to the COV report(s), if any; and
  5. a list of Advisory Committee members.

        387.   The Assistant Director or Office Director will also make available to the public, upon request, printed copies of the documents specified in Section 386.

390     RESPONSIBILITIES

        391.  Each Assistant Director or appropriate Office Director is responsible for:

  • ensuring that procedures described in this subchapter and in FACA regulations are met (Section 330);
  • establishing and supporting each COV in accordance with all the provisions of this subchapter (Section 340);
  • reviewing and approving plans for synchronizing COV reviews of programs at the program, cluster, section, or division level (Section 320);
  • preparing and submitting a three-year schedule of COV reviews to OIA at the beginning of each fiscal year (Section 320);
  • informing OIA of any changes in COV meeting schedules during the current fiscal year (Section 320);
  • consulting with the Division Director and Chair of the cognizant Advisory Committee, if appropriate, to guide the selection of COV members and to define the goals and issues that the COV will address (Sections 340, 350 and 360);
  • ensuring that a document describing the diversity and independence of the COV's members and the resolution of real or apparent conflicts of interest accompanies the COV Report (Sections 350 and 380);
  • responding in writing to recommendations made in each COV report (Section 380); and
  • providing access to the COV report, with a description of COV membership, the Advisory Committee's review and approval, if appropriate, and providing his or her response to the COV report to the parties listed in Section 380, and in written form upon request (Section 380).

        392.   OIA will assist the Director in meeting the guidelines described in this subchapter by:

  • monitoring and facilitating the COV process during each fiscal year and providing periodic reports to the Director (Section 314);
  • maintaining a record of the three-year schedule of COV reviews submitted annually by each directorate (Section 327);
  • maintaining standard COV Reporting Templates, approved by the Director, for COV Core Questions (Section 364);
  • assisting NSF staff in their efforts to follow the procedures described in this subchapter (Sections 314, 336 and 364); and
  • assisting the Director in preparing an annual summary report to the National Science Board about COV operations (Section 314).

        393.   The NSF Committee Management Officer located in the Division of Human Resource Management is responsible for:

  • facilitating compliance with FACA regulations by NSF staff charged with overseeing COVs;
  • providing advice and guidance to NSF officials about NSF Advisory Committees (as described in NSF Manual 1, Chapter IV, Subchapter 100, and in the Committee Management Guide); and
  • distributing eight copies of each Advisory Committee reviewed COV report to the Library of Congress as required by the FACA.

Other Versions: WORD ~ PDF   


Dr. Fae L. Korsmo (Email: fkorsmo@nsf.gov)
National Science Foundation ~ Office of Integrative Activities (OIA)
4201 Wilson Boulevard, Room 1270 ~ Arlington, Virginia 22230 USA
Tel: 703.292.8040 ~ Fax: 703.292.9040

Web Publisher: Betty Wong (Email: bwong@nsf.gov) ~ Last update: 1/29/04

 

Email this pagePrint this page
Back to Top of page