NSF 08-585: Research and Evaluation on Education in Science and Engineering (REESE)

The proposal types allowable under this solicitation have been updated to enable more flexible mechanisms for conducting research relevant to this program (see Section II. D.). The proposal type and its research strand must be specified in the title on the cover page of all proposals and in the project summary, preferably in the first sentence. The proposal types now include the following:

• Knowledge Diffusion proposals are small projects for the synthesis of existing knowledge on a topic of critical importance to STEM learning and education or for the diffusion of research based knowledge. The maximum award size for knowledge diffusion projects is $250,000 for duration of up to two years.
• Empirical Research proposals are studies that focus on important issues in STEM learning. They may be either exploratory or full studies and have maximum award sizes of $1,000,000 over three years.
• Large Empirical Research proposals are intended for more complex projects focused on important issues in STEM learning. The projects could involve teams of experts from multiple disciplines working on conceptually related projects, or they might be longitudinal or randomized control studies of large samples of participants.  The maximum award for Large Empirical Research projects is $2,000,000 with duration of up to five years.

The deadline for the submission of proposals has been moved earlier in the year.

Research and Evaluation on Education in Science and Engineering  (REESE)

The Division of Research on Learning in Formal and Informal Settings (DRL) in the Directorate for Education and Human Resources (EHR) of the National Science Foundation (NSF) supports basic and applied research and evaluation that enhance science, technology, engineering, and mathematics (STEM) learning and teaching.  The Research and Evaluation on Education in Science and Engineering (REESE) program aims at advancing research at the frontiers of STEM learning, education, and evaluation, and at providing the foundational knowledge necessary to improve STEM teaching and learning at all educational levels and in all settings.  This solicitation calls for three types of proposals--Knowledge Diffusion, Empirical Research, and Large Empirical Research.

The goals of the REESE program are: (1) to catalyze discovery and innovation at the frontiers of STEM learning, education, and evaluation; (2) to stimulate the field to produce high quality and robust research results through the progress of theory, method, and human resources; and (3) to help coordinate and transform advances in education, learning research, and evaluation.  REESE pursues its mission by developing an interdisciplinary research portfolio focusing on core scientific questions about STEM learning in current and emerging learning contexts, both formal and informal, from childhood through adulthood, and from before school through to graduate school and beyond into the workforce.  REESE places particular importance upon the involvement of young investigators in the projects, at doctoral, postdoctoral, and early career stages, as well as the involvement of STEM disciplinary experts.  In addition, research questions related to educational research methodology and evaluation are central to the REESE activity.

About the National Science Foundation and the Directorate for Education and Human Resources

The National Science Foundation (NSF) is charged with promoting the vitality of the nation's science, technology, engineering and mathematics (STEM) research and education enterprises. As part of this mission, the Directorate for Education and Human Resources (EHR) has primary responsibility for providing national and research-based leadership in STEM education. EHR promotes five themes in fulfilling this responsibility through:

1. Broadening participation to improve workforce development;
2. Promoting cyber-enabled learning strategies to enhance STEM education;
3. Enriching the education of STEM teachers;
4. Furthering public understanding of science and advancing STEM literacy; and
5. Promoting learning through research and evaluation.

To address these themes, the Directorate sponsors programs in the Divisions of Research on Learning in Formal and Informal Settings (DRL), Undergraduate Education (DUE), Graduate Education (DGE), and Human Resource Development (HRD).

About the Division of Research on Learning in Formal and Informal Settings

DRL invests in projects to enhance STEM learning for people of all ages. Its mission includes promoting innovative and transformative research, development, and evaluation of learning and teaching across all STEM disciplines by advancing cutting-edge knowledge and practices in both formal and informal learning settings. DRL also promotes the broadening and deepening of capacity and impact in the educational sciences by encouraging the participation of scientists, engineers, and educators from the range of disciplines represented at NSF. Therefore DRL's role in the larger context of Federal support for education research and evaluation is to be a catalyst for change--advancing theory, method, measurement, development, evaluation, and application in STEM education. The Division seeks to support both early, promising innovations, as well as larger-scale adoptions of proven educational innovations. In doing so, it challenges the field to create the ideas, resources, and human capacity to bring about the needed transformation of STEM education for the 21st century.

Because NSF is a catalyst for supporting basic research at the frontiers of discovery in the STEM fields, DRL takes as a central principle that new and emerging areas of STEM must figure prominently into efforts to improve STEM education at all levels and in all settings. Its programs should reflect this through the integration of cutting-edge STEM content and the engagement of STEM researchers in all DRL initiatives.

The Division's programs offer a set of complementary approaches for advancing research, development, and field-based improvement strategies.

• The Research and Evaluation on Education in Science and Engineering (REESE) program aims at advancing research at the frontiers of STEM learning, education, and evaluation, and at providing the foundational knowledge necessary to improve STEM teaching and learning at all educational levels and in all settings.
• The Discovery Research K-12 (DR-K12) program seeks to enable significant advances in K-12 student and teacher learning of the STEM disciplines, through research and development of innovative resources, models, and technologies for use by students, teachers, administrators and policy makers.
• The Informal Science Education (ISE) program builds on educational research and practice and seeks to increase interest in, engagement with, and understanding of STEM by individuals of all ages and backgrounds through self-directed STEM learning experiences.
• The Innovative Technology Experiences for Students and Teachers (ITEST) program is intended to enhance participation in the U.S. STEM and Information and Communication Technology (ICT)-intensive workforce, through the design, implementation, scale-up and testing of strategies for students and/or teachers, and through research studies that depend understanding of issues related to STEM workforce participation.

Each of these programs is intended to improve the capacity of their respective fields to further STEM learning. They are central to NSF's strategic goals of Learning and Discovery, helping to cultivate a world-class, broadly inclusive STEM workforce, expanding the scientific literacy of all citizens, and promoting research that advances the frontiers of knowledge.

All research and development activities within DRL aim at generating knowledge and transforming practice in STEM education. DRL's programs are designed to complement each other within a cycle of innovation and learning (see Figure 1) that forms the conceptual framework for its programs (adapted from RAND, 2003, American Statistical Association, 2007, NSF EHR, 2005). All DRL programs are concerned with all five components of the cycle, to different degrees.

Figure 1. Cycle of Innovation

Each part of the cycle, represented by the activities of DRL's programs, forms the vital and compelling foundation for transition to the next part of the cycle; the research, development, and implementation activities need to be rigorous, as appropriate. From challenging the STEM educational and research communities with transformative ideas, to conducting the pioneering and pragmatic research necessary to advance those goals, to developing world-class instructional materials and resources for teachers and students to advance their knowledge of STEM teaching and learning, to engaging all citizens and residents of the United States in learning and as future technologists, scientists and engineers, DRL is providing the ideas, resources, and human capacity to advance STEM learning and education in the 21^st century.

The major distinction between DR-K12 and REESE is that DR-K12 focuses specifically on issues of K-12 learning and projects will involve either a substantial development component, or will study the implementation of particular resources, models and technologies for the purpose of informing future design and implementation--the design, develop, and test and implement, study efficacy, and improve components of the cycle. REESE focuses primarily on building theory and knowledge through research and evaluation, across learning contexts and ages--the synthesize and theorize; scale-up and study effectiveness; and hypothesize and clarify components in the cycle ( NRC, 2002, and AERA, 2007). The outcomes of DR-K12 projects will be resources, models, or technologies that are grounded in or informed by research or practice, as well as research findings about the implementation and impact of K-12 STEM education resources, models and technologies. The primary outcomes of REESE projects will be research findings, methods, and theoretical perspectives.

References

American Educational Research Association (2007). Estimating causal effects using experimental and observational designs. Washington, DC: American Educational Research Association.

American Statistical Association (2007). Using statistics effectively in mathematics education research. Retrieved July 9, 2007 from http://www.amstat.org/research_grants/pdfs/SMERReport.pdf.

RAND Mathematics Study Panel (2003). Mathematical proficiency for all students: Toward a strategic research and development program in mathematics education. (MR-1643.0-OERI) Santa Monica, CA: RAND.

National Research Council. (2002). Scientific research in education. Washington, DC: National Academy Press.

National Science Foundation (2005). The mathematics education portfolio brief, (NSF 05-03). Retrieved July 9, 2007 from https://www.nsf.gov/pubs/2005/nsf0503/nsf0503.pdf.

The goals of the REESE program are: (1) to catalyze discovery and innovation at the frontiers of STEM learning, education, and evaluation; (2) to stimulate the field to produce high quality and robust research results through the progress of theory, method, and human resources; and (3) to help coordinate and transform advances in education, learning research, and evaluation efforts. REESE pursues its mission by developing an interdisciplinary research portfolio focusing on core scientific questions of STEM learning in current and emerging learning contexts, both formal and informal, from childhood through adulthood, and from before school through to graduate school and beyond into the workforce. REESE places particular importance upon the involvement of young investigators in the projects, at doctoral, postdoctoral, and early career stages, as well as the involvement of STEM disciplinary experts. In addition, research questions related to educational research methodology and evaluation are central to the REESE activity.

This solicitation calls for three types of proposals: knowledge diffusion, empirical, and large empirical. All REESE proposals, regardless of their type, must be responsive to one of two broad topical strands, Emerging Research or Contextual Research, as described below. The proposal type and its research strand must be specified in the project summary, preferably in the first sentence.

1. Research on Emerging Topics in STEM Education

Emerging research that broadens knowledge in the field often challenges existing assumptions about learning and teaching within or across STEM disciplines.The REESE program is committed to supporting transformative education research in STEM education through novel answers to foundational questions about what STEM concepts can be learned by whom, at what age, and how and where that can happen.

REESE seeks proposals that have the potential to transform existing fields of STEM learning and education through pioneering research that defies disciplinary boundaries in pursuit of emerging knowledge in STEM learning. Through  Emerging Research projects, REESE challenges scholarly communities to put forward groundbreaking ideas, concepts, theories, modes of research and development, and the measurement and methodological technologies needed to understand and measure the impact of the proposed innovations. Emerging research is by its nature uncertain, so high-risk/high-gain proposals and potentially transformative ideas are welcome.

Emerging Research proposals will seek to contribute to far-reaching and longer-term developments in knowledge and theory. They may be especially oriented toward the design, develop, and test components of the cycle in Figure 1. Please note: Emerging Research proposals are limited to one or more of the following areas of inquiry:

1. Neural basis of STEM learning

Fundamental aspects of STEM learning are beginning to be understood in terms of neural processes and biological context. Discoveries in these and other areas are influencing our understanding of behavior, cognition, and the nature of human learning. REESE will support studies focused on human learning in the STEM fields drawing on a wide range of theoretical approaches and empirical techniques. It is incumbent upon those submitting proposals to make explicit the implications their work has for current theories of learning and instructional methods, however long-term and indirect they may be. For example, neuroscientific studies of attention or inhibition could constrain theories about the learning of specific STEM content or help explain why some misconceptions are robust and difficult to overcome. They could similarly inform the creation of principles of design for the development of instructional materials, informal learning opportunities, or the education of teachers in the STEM fields.

In order to gain traction on fundamental questions of mind and brain as related to STEM learning, REESE supports innovative combinations of theory, methods, and levels of analysis from a wide range of disciplines. An important aspect of these activities is to build capacity in neuroscience related to complex human learning and education, and to identify trajectories by which multidisciplinary research anchored in the biological basis of human learning can inform STEM educational practice. The involvement of researchers familiar with STEM educational practice will be of benefit both in helping to set the cognitive and neuroscientific research agendas in learning as well as in helping to disseminate relevant literatures across disciplines.

2. Cognitive processes underlying STEM learning and teaching

The REESE program encourages proposals that push the boundaries of existing knowledge about the cognitive processes underlying the learning and teaching of complex STEM content at all age levels and in all learning contexts. The program seeks to foster interdisciplinary collaboration among cognitive scientists, educational researchers, and STEM disciplinary educators, bringing their respective literatures into more systematic and productive contact. To that end, investigators must make a clear case for how the proposed research has the potential to lead to significant advancements in our understanding of STEM learning and teaching, even if such advancement is by no means assured. In particular, studies must identify the STEM content of focus and argue for its importance. Similarly, assumptions, whether implicit or explicit, about STEM learning must engage relevant theoretical developments and empirical findings, whether in the cognitive science, education research, and/or STEM education literatures.

This is a call for researchers to attempt to make substantial progress on fundamental intellectual and scientific questions about the nature of learning, teaching, and knowing, at all education levels, that bear upon developing expertise in STEM fields. For example, investigators might pursue questions about the role of students’ goals and beliefs about STEM learning as they relate to STEM performance, or they could take advantage of recent developments at the intersection of mathematics and cognitive science that seek to create probabilistic models of reasoning, memory, language, categorization, and learning in complex STEM domains. They might address such problems as whether and which aspects of knowledge of the natural world have early-arising conceptual biases that influence the course of learning throughout the life span, affecting which STEM concepts appear to be commonsense and which seem counterintuitive. By contrast, investigators might address claims about which aspects of understandings of the natural world are relevant to a particular social or linguistic context and how they arise, or how prior opportunities to learn relate to what is developmentally appropriate.

3. Measurement, modeling, and methods for research and evaluation

The REESE program is committed to advancing the state of the art in STEM learning, education research, and evaluation by supporting proposals to improve or develop new qualitative and quantitative methods, measures, tools and analytic techniques. Investigators studying problems in this area must make a clear case for the technical, analytic, methodological, or measurement problem to be addressed, and plans for how the proposed methods will be developed. An argument should be included about why the particular methodological advance will be applicable within the context of education, learning, or evaluation research in one or more specific STEM content areas.

For instance, researchers and evaluators studying complex STEM education phenomena need methods for reducing data while maintaining their validity, reproducibility, and inherent richness. Some methodologists are experimenting with hybrid forms of qualitative and quantitative techniques based in game and risk theory within more traditional experimental designs, for application to STEM education problems. Moreover, continued work is needed in methods of combining and aggregating different forms of evidence within a single design or across multiple studies through such methods as meta-analytic or synthetic techniques, mixed qualitative-quantitative techniques, and modeling data derived from qualitatively diverse perspectives in causal logic. New evaluation models to examine STEM program and project impact may be developed and validated.

REESE encourages the development of measures of STEM learning that can increase the validity and reliability of conclusions drawn from effects estimated through a range of scaling and other construct measurement techniques. Instruments that are designed to measure complex and meaningful constructs in STEM learning and education can be developed and tested in various field settings, within the contexts of research, implementation and program evaluation, and by a variety of users of different backgrounds. In the tradition of evaluation in particular, opportunities exist for a more synergistic union between measurement work deeply rooted in one or more STEM content areas (often absent from education evaluations) with methods designed to assess the impact of programmatic or policy implications at various organizational levels.

REESE also encourages the submission of proposals to validate or construct major theoretical or analytical approaches in evaluation. Proposals should justify the potential contribution of these methods to evaluate the effects of STEM education on policy, programs, and practices. Priority will be given to proposals that provide new topics and methods for investigating causal relationships between implementation, outcomes, and long term impact. Projects that address evaluation research in informal settings are also encouraged (see insci.org/docs/Eval_Framework.pdf).

In addition, the STEM education research and evaluation communities remain in need of appropriate and robust ways to measure and model constructs at higher programmatic and organizational levels and within nested logic structures. Often these phenomena cannot be accurately depicted by simply aggregating individual level phenomena. Measurement and scaling techniques often make analytic assumptions about the parametric form of data and about nested phenomena themselves that fail to capture the complexity of many learning environments and that fail to meet classical statistical assumptions. Research is encouraged that seeks ways that measurement and modeling techniques can become more intellectually responsive to education and learning theory and more robust to modeling assumptions, so that they can be applied to STEM learning and education questions.

In the area of modeling and related developments for data mining, sharing, and manipulation, some fields of science and engineering are tapping creative solutions to representation. These solutions emerge from large-scale, distributed data and other authentic resources now becoming available due to advances in computing power, pattern recognition, graphical imagining and representation, and other web-based venues and technologies. Techniques such as these might be extended and adapted for use in modeling learning trajectories, making inferences about particular large-scale interventions, or in diffusion of innovations at various levels educational or informal learning systems. REESE is interested in proposals to adapt and advance these techniques for application to STEM learning settings.

4. Cyberlearning and teaching

Ongoing investments by the National Science Foundation to advance our nation's cyberinfrastructure have provided the foundation from which to re-conceptualize traditional models of teaching and learning in school-based and informal learning environments. The re-conceptualization of how, when, and where learning can take place has strong implications for how to effectively educate 21st century learners who are already digital natives.

Cyberlearning is defined by NSF as learning that is mediated by networked computing and communications technologies. Cyberlearning is learning that occurs through one or more types of cyber-enabled networks and communications technologies, and may comprise an entire learning experience. Cyberlearning is potentially transformative in that it may provide learning experiences that may highly motivate STEM learning or enable the learning of new STEM content, or allow for teaching that reaches new levels of effectiveness. As a result, cyberlearning has the potential to enhance and enrich the learning process throughout the school years and into adulthood, as a lifelong chronicle--potentially improving the effectiveness with which knowledge is gained over an entire life span.

REESE invites proposals for research to test claims that cyberlearning promotes significantly different ways of learning STEM content, or allows for the learning of different STEM content. Research is needed that will enable the potentially transformational promise of technology to be realized as a means to improve educational opportunity. Accordingly, REESE welcomes proposals that study learning across the entire cyberlearning landscape. REESE encourages projects that: investigate the cognitive implications of cyberlearning; advance the teaching and learning of STEM content through an array of cyber-enabled technologies at all age levels; explore the intersection of human-computer interactions; study types of STEM content that can be taught, learned, and assessed through cyberlearning technologies and the conditions under which these occur; investigate the effects of STEM learning through the use of (1) visual technologies, such as visualizations, simulations, and games, (2) types of virtual environments, and (3) virtual humans; and, study adaptive learning technologies, cognitive tutors, and the highly social networks of virtual organizations.

REESE seeks to understand the cognitive, social, organizational, and technological effects (both positive and negative) that cyberlearning has on teaching and learning of STEM content in school-based classrooms, in a variety of out-of-school environments, and in virtual environments, and encourages proposals that can make a clear case for how the proposed research represents the potential for making a significant advance in cyberlearning.

2. Contextual Research Topics in STEM Education

The Contextual Research strand encourages proposals that address central problems and topics in STEM education, teaching and learning, and evaluation, for all age groups and in all settings--problems that must be addressed in order for substantial progress to be made in educating the STEM workforce of tomorrow and ensuring the STEM literacy of all. Research in this area is often multidisciplinary, drawing on the expertise of STEM content experts, STEM education researchers, cognitive and social scientists, computer scientists, and potentially those from other areas of praxis and scholarship. It may also draw on international research trends and theoretical perspectives.

In contrast to Emerging Research strand, which is limited to the specified topics above, the Contextual Research strand of REESE offers three broad areas for transformative solutions to persistent problems: research on teaching and learning in formal and informal settings, research on policy and systems, and evaluative research as domains in which many of the problems central to advancing STEM learning and teaching are located. Investigators are welcome to draw on other key elements of current contexts for STEM education in arguing for the importance of other particular research topics. The REESE program expects that Contextual Research proposals will more typically address problems that are current and widely visible within STEM teaching and learning, with nearer-term, more-direct implications for use in the context of policy and practice than is the case for Emerging Research projects.

The research findings, prototypes, or other output of these contextual projects should be of use to communities of researchers, policy analysts, and developers who seek research to develop curricula, improve teacher education programs, or provide guidance to policymakers or other stakeholders. Although a project may involve a specific national initiative, curriculum, program, or policy as a test case, the intellectual merit and broader impact of the project should be carried by the potential progress it suggests from the research questions it engages. (For investigators interested in developing resources, models, or technologies, such as curricula, refer to the Discovery Research K-12 (DRK-12) program solicitation).

Examples of the type of work invited under this strand follow, although they do not constitute an exhaustive or mutually exclusive set of priorities.

1. STEM teaching and learning in formal and informal settings.

REESE invites proposals that advance understanding of the broad role that teachers and faculty, teaching and instruction, curriculum and learning environments, and assessment play for learning and education in STEM content areas. Topics such as recruitment, preparation, continuing development, and retention of STEM educators (e.g., K-12 teachers, graduate teaching assistants, higher education faculty, informal science educators) are central concerns of educational organizations of many types. REESE encourages research on the knowledge that STEM professionals need in order to enable their clientele to learn and engage with particular STEM topics and how that knowledge affects learning outcomes. It also encourages research on teacher or faculty understanding of learner knowledge in particular STEM domains, and on how learners' pre-existing conceptual understandings or misunderstandings affect their learning of more sophisticated STEM content.

REESE is strongly committed to supporting projects that lead to further understanding STEM learning, of a variety of aspects of STEM content, in all of its contexts. Projects are encouraged to examine the implications that learning of particular content, in particular social contexts (such as classrooms, across cultural and linguistic groups, undergraduate courses, graduate programs, museums, web sites, or games) has for individual learning and achievement. REESE encourages proposals on STEM learning in informal settings (e.g., museums, science centers, zoos and aquariums, and from media) due to the continued growth of these activities, their increased importance to people's out-of-school and lifelong learning experiences, and the blurring of the boundaries in society as to where, when, and how people learn. REESE considers proposals on STEM learning in settings such as out-of-school programs, programs for at-risk students, alternative organizational designs for education and learning, home schooling, parent-child interactions, emergent social learning structures such as are available over the Internet, and linkages between formal and informal settings. REESE encourages proposals that examine the non-cognitive aspects of learning such as what motivates and sustains learner interest in STEM, and what fosters engagement and persistence.

REESE invites proposals for research that can help provide a foundation for methods for assessing learners' knowledge in STEM content domains, ranging from the earliest learners through adults. The learning of specific STEM content must be an integral aspect of these proposals and the particular content domain must be made explicit. Such research may address how to characterize student understanding, broadly defined, for multiple uses by teachers, instructors, administrators, parents, students, and policymakers.

Recent reports call for research on the effectiveness of various instructional interventions and teaching practices, such as the National Mathematics Advisory Panel's "Foundations for Success and the National Research Council's Taking Science to School." REESE invites proposals that systematically study the teaching of STEM content, especially studies that investigate the effectiveness of instructional interventions and practices. Such studies should move beyond descriptions of teaching practice to build the knowledge base of promising instructional practices and characteristics that promote student learning under different instructional interventions. Rigorous research is needed to ascertain the effectiveness of different instructional strategies (e.g., peer tutoring in the elementary grades, cooperative learning at the undergraduate level). We encourage rigorous research that takes up questions of cause and effect, including studies that employ multi-level methods of causal inference. We especially encourage proposals from cross-disciplinary teams of researchers that include disciplinary experts.

Finally, REESE encourages proposals that unite research in teaching, learning, and assessment through the study of particular learning trajectories or progressions of STEM content across age bands or grade levels. These learning progressions research projects may seek to test conceptual models for what may be needed for the effective teaching, learning, and assessment of the novel, deep, or foundational content proposed. Such research studies aim to understand the dynamic interplay affecting the relationships among learner, teacher or instructor, and content in testing key conjectures and theoretical ideas about teaching, learning, and/or assessment. We encourage learning progressions studies of advanced STEM content at critical transition areas (e.g., middle grades to high school, high school to college, undergraduate to graduate study).

2. Education policy and systems studies

The REESE program is interested in research on the role of institutions and organizational dynamics as they pertain to STEM learning and education. For example, policies and standards shaping large-scale testing programs at the state level affect the opportunities students have to learn STEM content, the selection of curriculum and instructional materials (and so what is taught at which grade level and how it is taught), and rewards and incentive structures for organizational change. The data-analytic and interpretation capabilities of schools, administrators, and teachers may have implications for the implementation and use of such assessment programs. Similarly, general education requirements at the postsecondary level and graduation requirements at the secondary level may have important benefits, opportunity costs, and individual and organizational responses.

Systems studies can include such entities as K-12 school systems, informal educational organizations, and institutions of higher education (including graduate education) and requisite governance authorities. It also includes broader conceptions and organizational designs for learning such as those distributed in both virtual and real environments, those connected with larger national and international systems of innovation, or those linked across the various levels of the education system. In all cases, an argument for why the particular research questions are relevant for STEM learning must be provided. Systems studies may examine the mechanisms for the spread of research findings or other innovations. REESE welcomes research that deepens understanding of how systems-related resources can be engaged to advance the STEM learning of students who are members of underserved populations, perhaps identifying and modeling points within an organizational system that are particularly vulnerable or resistant to change.

REESE encourages research that seeks to understand the ways individuals, organizations, and whole systems respond to education laws, regulations, and other interventions across various levels (i.e., international, national, state, district, school, or university and college) as they relate to STEM learning. Issues of adaptive and emergent organizational behavior are of interest in producing theoretical, descriptive, and potentially predictive models of change in STEM education and learning. REESE is also interested in projects that conduct secondary analyses of large-scale data sets.

Organizational and systems research problems are likely to require collaborative teams of researchers and practitioners, including social psychologists, sociologists, systems and institutional theorists, organizational sociologists, policy experts and economists, STEM education researchers, and STEM disciplinary experts.

3. Evaluation studies

REESE invites proposals for projects that simultaneously undertake evaluation studies and advance theory, knowledge, methods, or implementation of evaluation. Projects to address relevant evaluation questions for STEM-education initiatives at the national level are being sought for this competition. Some of these programs are NSF-supported initiatives, and others are funded from other sources, but they share goals to advance STEM learning in the K-12, undergraduate, or informal arenas. For example, there are specific initiatives or programs to increase the supply, retention, and quality of STEM teachers; efforts to reform the preparation of STEM teachers; efforts to introduce new instructional materials in K-12 classrooms; and efforts to change assessment or standards at the state and national levels. The REESE program invites proposals to conduct evaluation studies of such national initiatives. Such studies would need to clearly identify the initiative being evaluated and provide a letter of support from the leaders/funders of the program demonstrating mutual agreement to the project. Proposals in this category must make an argument for why an evaluation study of this initiative (or a focused component of it) is important for informing national policy and practice. In addition, proposals should describe the key evaluation questions being pursued, explain the methodologies to be used, and provide evidence of how the evaluation will have access to relevant data and programmatic information. Evaluation studies need to advance DRL’s understanding of the evaluation of STEM education initiatives, and need to argue why the findings of such studies will be useful for advancing new initiatives to improve STEM learning.

Proposers should contact a Program Director before submitting an Evaluation studies proposal.

3. Information Applicable To All Proposals

Research design and methodology: REESE expects investigators to propose rigorous and replicable research methods that are well-justified, are suited to the particular research questions being studied, and that have the likelihood of yielding significant knowledge in pursuit of core problems in STEM education and learning. Each supported project must meet the following basic requirements:

1. The proposed topics, questions, methodologies, and research settings must be consistent with the overall goals of the REESE program. Investigators should pose research problems of compelling national importance deeply rooted in one or more STEM fields. Research questions must be clear and specific and must be answerable through the means proposed.
2. The investigators must demonstrate how the proposed research program builds upon existing evidence obtained from relevant prior research. All proposals must draw on the existing educational and learning literatures and on the education-related literature in one or more other domains such as the physical and biological sciences, engineering, cognitive science, neuroscience, statistics, mathematics, and information science.
3. The investigators must explicitly describe the research design including the methods, sample selected for study, instruments, and all means of data collection. A range of research designs appropriately matched to the nature of the research problem and questions are encouraged in REESE. Information must also be provided on the reliability, validity, and appropriateness of proposed measures and instruments. If the reliability and validity of the instruments are initially unknown, the applicant must include specific plans for establishing these measurement properties.
4. The investigators must provide a specific data analysis plan, including procedures to code and (if necessary) reduce qualitative data, details on how potential threats to internal and external validity will be addressed, power analyses (when appropriate) demonstrating the adequacy of proposed cell sizes, and plans for estimating effect sizes as appropriate. Proposals are strengthened by the reporting of pilot results.

Project personnel and management: The research and management roles of each of the senior personnel on the project must be described in brief within the project description. Collaborative teams representing multiple disciplines are typical in REESE projects. In addition, at least one of the senior personnel must be designated as the methodology and measurement leader of the project. In single-investigator projects, this person will necessarily be the principal investigator. In multi-investigator projects, this person must be listed among senior personnel and may or may not be the principal or a co-investigator. All projects should address the role to be played by STEM disciplinary experts, as appropriate.

Where projects request time for students and other trainees, specific plans must be discussed for how any postdoctoral associates, graduate students, undergraduates, or others will benefit in their education and training in connection to the proposed research. Involvement of students is encouraged as a means of building capacity in STEM education research.

REESE does not necessarily expect the same team of investigators to conduct research across all components of the cycle of innovation. However, investigators are expected to conduct research so that relevant models, frameworks, data and measures are well-documented, replicable, and usable by other research teams wishing to work on similar problems from other vantage points or by using other research designs. It is the intention of the REESE program to encourage investigative teams to work simultaneously, as part of a larger knowledge community, on a given problem of national importance. See the Large Empirical proposal discussion under Eligible Project Types for related information.

Dissemination: All REESE projects are expected to accumulate and communicate knowledge to the relevant research, policy, and practitioner communities. As part of DRL's strong and unwavering commitment to the broader impacts of funded research, reports from successful REESE projects must be published in peer-reviewed professional or scholarly journals, and findings (positive or negative) must be disseminated through appropriate means to audiences relevant to the goals of the project. Projects are encouraged to seek out appropriate audiences across disciplinary boundaries. Projects will also be expected to share research designs, findings, and overall project information with policymakers and the REESE Diffusion and Evaluation Network, and possibly report annually to an on-line data system.

Project Evaluation: All projects must have an evaluation plan that includes measures that the project team intends to use in assessing its success and meeting its milestones and objectives. It is critical that all projects have a substantive external expert review mechanism that provides regular feedback on the project's research methods and progress, analysis procedures, interpretation of data into findings, and dissemination activities. Proposals must make a clear argument for what steps will be taken to ensure that the proposed evaluation is distant from the project and is objective, and must describe how evaluation input will be used to shape the project.

4. Eligible proposal types

This solicitation calls for three types of proposals: Knowledge Diffusion, Empirical Research, and Large Empirical Research. The content of all proposals, regardless of their type, must be responsive to one or more topics in the Emerging Research or Contextual Research strands described above. The proposal type and its research strand must be specified in the project summary, preferably in the first sentence.

1. Knowledge Diffusion proposals

Knowledge diffusion projects are small grants for the synthesis of existing knowledge on a topic of critical importance to STEM learning, education, and/or evaluation, or for the diffusion of research-based knowledge. Synthesis proposals should identify areas where the knowledge base is sufficiently robust to support strong scientific claims, identify areas of importance to education research, evaluation or practice, and propose rigorous methods for synthesizing findings and drawing conclusions from a range of relevant literatures. Proposals should identify the criteria to be used for including or excluding studies in the synthesis. Investigators are permitted to propose workshops and other meetings in pursuit of the diffusion of research-based knowledge or to provide training on topics of advanced research or evaluation methods, analysis, modeling, or measurement. Emphasis will be placed on the proposed dissemination plan. Maximum award size for Knowledge Diffusion proposals is $250,000 for duration of up to two years.

2. Empirical Research proposals

Empirical Research proposals should identify areas that have the potential for advancing discovery and innovation in STEM learning. These projects are designed to support the collection of new empirical data or to conduct secondary analyses from existing state, national or international databases. Such projects are expected to be based deeply in the STEM disciplines. Maximum award size for most Empirical Research proposals is $1,000,000 for duration of up to three years.

3. Large Empirical Research proposals

REESE will support a limited number of projects up to $2,000,000 for up to five years. Proposals must carefully justify why a budget of this size would be required to carry out the research. The proposals will generally involve teams of multi-disciplinary experts working on conceptually related projects. For example, one team could seek to develop a new behavioral measure of learning in a content area of particular STEM importance, while a second team studied the neural underpinnings of learning in the area. A proposal may have one team generating a mature prototype, while another team might test the hypotheses about learning in a randomized control trial. Another example would be one team conducting largely theory-generating work from an ethnographic approach, while other teams conduct complementary quantitative studies. Such proposals must also include a Coordination Plan that provides (1) a description of how the separate activities are conceptually interlinked, (2) the agreements for data sharing among the partners, (3) a description of how samples or data collection will be complementary or will use parallel data definitions, (4) a discussion of how data will be jointly modeled or analyzed or how findings will be aggregated across teams, (5) plans for joint publication and dissemination, and (6) a plan for ongoing dialogue, communication, and scholarly exchange. The Coordination Plan should be described in no more than five pages and submitted as Supplementary Documentation.

Other types of proposals that might be appropriate for a large award would be a longitudinal study of a large sample of participants, or a randomized control trial of an intervention whose efficacy has been established in more limited conditions. These projects do not require a Coordination Plan.

5. Conferences and Workshops

REESE may support a few well-focused conferences or workshops related to the goals of the program. Budgets are expected to be related to the duration of the event and the number of participants, but normally the total cost will not exceed $100,000. Please see the GPG Section II. D. for additional information about conference and workshop proposals. Proposals may be submitted at any time, generally at least one year in advance of when the conference would be held. All conference proposals should provide for an evaluation of the impact of the conference done 18 months after the conference.

Estimated program budget, number of awards and average award size/duration are subject to the availability of funds. NSF expects to make standard or continuing grant awards. The estimated number of awards will be 30 to 45 per year for each competition in FY 2009 and FY 2010, pending availability of funds. It is anticipated that about 10-15 Knowledge Diffusion awards, 15-20 Empirical awards, and 5-10 Large Empirical awards will be made. The anticipated funding amount is $30,000,000 per year for each competition in FY 2009 and FY 2010, pending availability of funds. The maximum award for Knowledge Diffusion projects is $250,000 with duration of up to two years. The maximum award Empirical project is $1,000,000 with duration of up to three years. The maximum award for Large Empirical research projects is $2,000,000 with duration of up to five years.

What is the intellectual merit of the proposed activity?
How important is the proposed activity to advancing knowledge and understanding within its own field or across different fields? How well qualified is the proposer (individual or team) to conduct the project? (If appropriate, the reviewer will comment on the quality of the prior work.) To what extent does the proposed activity suggest and explore creative, original, or potentially transformative concepts? How well conceived and organized is the proposed activity? Is there sufficient access to resources?

What are the broader impacts of the proposed activity?
How well does the activity advance discovery and understanding while promoting teaching, training, and learning? How well does the proposed activity broaden the participation of underrepresented groups (e.g., gender, ethnicity, disability, geographic, etc.)? To what extent will it enhance the infrastructure for research and education, such as facilities, instrumentation, networks, and partnerships? Will the results be disseminated broadly to enhance scientific and technological understanding? What may be the benefits of the proposed activity to society?

Examples illustrating activities likely to demonstrate broader impacts are available electronically on the NSF website at: https://www.nsf.gov/pubs/gpg/broaderimpacts.pdf.

Integration of Research and Education
One of the principal strategies in support of NSF's goals is to foster integration of research and education through the programs, projects, and activities it supports at academic and research institutions. These institutions provide abundant opportunities where individuals may concurrently assume responsibilities as researchers, educators, and students and where all can engage in joint efforts that infuse education with the excitement of discovery and enrich research through the diversity of learning perspectives.