Title: Centers for Ocean Sciences Education Excellence 
       (COSEE) FAQ's
Date: 02/04/08


Frequently Asked Questions:

Background: The COSEE National Advisory Committee met in November,
2007 and concluded that the tasks outlined in the recent NSF program
solicitation (NSF 08-509) for the COSEE National Network Evaluator
would require more than the budgeted $75,000/year. It recommended
that NSF increase the amount of funding available for the evaluation
activities. The intent of this FAQ is to clarify the scope of the
required evaluation by specifying that COSEE Network activities that
affect the NSF-funded oceanographic research community are of
highest importance and by describing the constraints on how to
respond to the Academic Competitive Council expectations.

   * Are there any specific goals that should take priority in the
     evaluation?

   * What should the evaluator focus on with relation to the
     Academic Competitiveness Council (ACC) Report (U.S. Department
     of Education, Report of the Academic Competitiveness Council,
     Washington, D.C., 2007 at
     http://www.ed.gov/about/inits/ed/competitiveness/acc-mathscience/report.pdf)?

   * The ACC identifies a hierarchy of evaluation strategies with
     randomized control trials being the "strongest"
     method. Does the COSEE evaluation need to be based on
     randomized control trials?



  1. Are there any specific goals that should take priority in the
     evaluation?

     Answer: NSF is interested in the impact and effectiveness
     of COSEE on all audiences, as stated in the program
     solicitation. Background documents for the planning and
     implementing the National Network Evaluation include the
     COSEE Blueprint
     (http://www.vims.edu/adv/cosee/COSEEBlueprintJan2007.pdf).
      The five Blueprint goals should be evaluated primarily
     from the viewpoint of  COSEE's impact on the
     NSF-supported oceanographic research, education, and
     public outreach community.

     Evaluation of the internal functions of the COSEE Network,
     i.e. the ability of the Centers to function as a network,
     cross-network collaborations, etc. is not considered a
     priority for this study.

  2. What should the evaluator focus on with relation to the
     Academic Competitiveness Council (ACC) Report (U.S. Department
     of Education, Report of the Academic Competitiveness Council,
     Washington, D.C., 2007 at
     http://www.ed.gov/about/inits/ed/competitiveness/acc-mathscience/report.pdf)?

     Answer: For the purposes of reporting to the Academic
     Competitiveness Council, COSEE has been identified as an
     "Outreach and Informal Education Program" by
     the NSF. The Informal Education and Outreach working group
     of the ACC identified two national and program-level
     goals: National Goal 1: Public Awareness-In the
     context of informal education and outreach, increase the
     awareness, interest, engagement, and understanding of STEM
     concepts, processes, and careers of the general public and
     other target populations. National Goal 2: Professional
     Audiences-Improve practice and build professional
     and institutional capacity by funding efforts that
     generate, develop, and apply innovative ideas and models
     for the informal science education field. These two goals
     should be used to develop evaluations of COSEE. Metrics
     for these two goals are identified on page 44 of the
     report.

  3. The ACC identifies a hierarchy of evaluation strategies with
     randomized control trials being the "strongest"
     method. Does the COSEE evaluation need to be based on
     randomized control trials?

     Answer: The report states that "The hierarchy does
     not include study designs appropriate for education
     research in areas other than measuring the impact of an
     educational activity on student outcomes." (p. 15).
     The report also notes, "The hierarchy is a statement
     of general principles and does not address all
     contingencies that may affect a study's ability to
     produce valid estimates of a project's
     impact." (p.16). The ACC report concludes that the
     nature of informal education and outreach makes it
     difficult to conduct rigorous evaluation (p. 26).
      Evaluators should consider the hierarchy as guidance for
     evaluation development, but should propose evaluation
     methods that are most appropriate for the COSEE goals and
     the stated ACC goals for Informal Education and Outreach.