Frequently Asked Questions (FAQs) for NSF 13-126, Common Guidelines for Education Research and Development

Do these Guidelines apply to all education projects at NSF?

These Guidelines are most relevant for NSF programs and projects that undertake education research and development (R and D) activities. Some solicitations will explicitly reference the Guidelines. The Guidelines are not developed to be relevant for programs or projects that conduct outreach activities of various kinds and scholarship programs are not included.

Why are these Guidelines necessary?

There is an interest across the federal government in increasing coherence and consistency across programs with similar goals. The education R and D programs in both NSF and the Department of Education will benefit from the use of a common framework for the projects that they fund. The Guidelines are intended to help the PI community share common language to describe key points related to relevant research literatures, outcomes and external feedback. Finally, each agency will be able to build on the investments of the other in more coherent ways.

Will these Guidelines preclude projects that are at the cutting edge of innovation?

The Guidelines should not hurt projects that are at the cutting edge of innovation. The key point of the Guidelines is to ensure that projects are explicit about their research questions, methods and analytic approaches in their proposals. These criteria should be relevant for all types of education R and D efforts, including those that may be at the cutting edge of innovation. Further, these are guidelines intended to help PIs in their proposal preparation. They should not be viewed as a straitjacket that hinders creative thinking. Rather, PIs should review them, and make sensible use of them as they describe the education R and D activities they propose.

How do the guidelines affect NSF's Merit Review criteria?

The Guidelines are not intended to supplant the Merit Review criteria. One element of the intellectual merit criterion for proposals is whether the project can advance knowledge and understanding. In addition, the intellectual merit criterion calls for a well-reasoned, well organized plan based on a sound rationale along with a mechanism to assure success. The Guidelines are consistent with this criterion.

Does NSF intend to study the impact of the Guidelines? Are revisions to the guidelines anticipated?

Conversations are underway about the best approach to studying the impact of these Guidelines. One example would be to conduct portfolio analyses of education R and D projects to see how they fall into the six study types outlined in the Guidelines and to examine how the distribution changes over time. Another example would be to analyze how program solicitations change as a result of implementing the Guidelines.

After the Guidelines have been in place for a while, it is reasonable to anticipate that changes will be made as a result of what is learned through the implementation process.

Do the Guidelines preclude or privilege any research methodologies?

The Guidelines don't preclude or favor any research methods, but they do require that the methods be well described, justified, and appropriate to the research questions that are posed. They are consistent with such broad ranging discussion of research methods as Scientific Research in Education, (NRC, 2002). Both qualitative and quantitative approaches can be used in all of the six research genres that are described in the Guidelines. For example, a small-scale randomized trial might be used in a design and development study and a qualitative study might be embedded in an efficacy or effectiveness study.

How are the guidelines intended to relate to research in emergent technologies?

Research in emergent technologies may be concerned with phenomena that are at scale rather than moving to scale (e.g., MOOCs). Even so, in studying these phenomena, issues of the quality of the research must be attended to and the descriptions of the research questions, methods and analytic approaches must be clearly described.

To what extent will these Guidelines represent a culture shift to NSF PIs?

For some NSF PIs, these Guidelines will not represent a shift. The Guidelines essentially codify what is considered to be reasonable practice for education research and development. For others, perhaps particularly for those who have been engaged primarily in development work, the Guidelines will represent more of a shift as PI teams will have to more explicitly address research questions, methods and approaches for analysis of data.

How will reviewers be informed about the Guidelines?

Reviewers will be informed of the Guidelines through multiple approaches: the Guidelines will be posted on the NSF website, referred to in program solicitations, discussed in reviewer webinars and orientations, and presented at PI and other professional meetings.

What are the implications of the Guidelines for external evaluation of projects?

The Guidelines include recommendations for all types of studies that call for external feedback on the work being proposed. However, they are more expansive than requiring a third-party evaluation. External feedback can include a number of approaches including third party evaluation, program officer evaluation, and/or regular feedback from advisory groups. It will be up to the proposer to argue for whatever kind of external feedback they identify as appropriate, aligned with program requirements.

What was the input from the field for these guidelines?

The Guidelines were sent out for external review to five experts through a process organized by the Department of Education's Institute for Education Sciences. In addition, a Presidential Invited session on the Guidelines was held at the 2013 American Education Research Association (AERA) annual meeting in San Francisco. At that session, there were round table discussions about the implications and implementation of the Guidelines.

What are the plans for future input from the field?

We anticipate that sessions that include discussion of the Guidelines will occur at 2014 professional meetings of organizations such as the American Education Research Association (AERA), National Association for Research on Science Teaching (NARST), National Council on Teachers of Mathematics (NCTM) and the Society for Research on Educational Effectiveness (SREE), among others. The Guidelines will also be discussed at relevant PI meetings.

Do the Guidelines have implications for the inclusion of education research expertise on education development proposals?

Proposals that focus on education development activities should be certain to include personnel with appropriate expertise in research design and methods”as key personnel or advisors.

How might these Guidelines help community building among research, policy and practice community?

Proposers should anticipate the policy and/or practice significance of the projects they propose in any of the six research genres. Even if the implications for policy or practice are distal, projects should describe their potential relevance to policy or practice. In addition, the Guidelines can help practitioners and policymakers develop a better understanding of what various types of education research should be expected to produce. The Guidelines are intended to support better-informed decisions based on the levels of evidence provided.

Will the use of the Guidelines lead to cookie cutter approaches in terms of projects that are funded?

We do not anticipate that the use of the Guidelines will lead to cookie cutter approaches in terms of the research and development projects that are funded. The Guidelines outline six different genres of research that call for different kinds of evidence. In addition, the Guidelines recommend supporting projects that span the genres. These Guidelines will not restrict new or original approaches.

Do the Guidelines apply to PROGRAMS as well as proposals in both agencies?

The Guidelines were developed to apply to education research and development proposals in order to help the agencies make more strategic decisions on individual projects. They were not designed to apply to programs. It might be possible to build on the ideas in the Guidelines to develop a comparable document for program evaluation, but that would entail undertaking a separate and different set of activities. We caution others from using the Guidelines for purposes beyond what is intended.