Science and Engineering Indicators 2012

Development and Content of NAEP Technology and Engineering Literacy Assessment

Beginning in 2014, the National Assessment of Educational Progress (NAEP) will administer the first nationwide student assessment in technology and engineering literacy. The framework defines key terms such as technology and engineering literacy, determines the content to be assessed, specifies the types of assessment questions to be asked, and guides the development of the assessment instrument (WestEd 2010).

Although the federal No Child Left Behind Act of 2001 requires that every student be "technologically literate by the time the student finishes the eighth grade," the law itself is vague in defining what technological literacy is, leaving states to determine what it means and how it should be assessed. Some states require engineering/technology education for students in at least some grades, but few have adopted formal assessments in this area (Metiri Group 2009). Technology- and engineering-related courses are typically offered in middle and high schools as electives or are embedded in other subject areas, such as science or social studies (WestEd 2010). Overall, coursetaking in these subjects is not widespread: in 2009, about 3% of high school graduates had taken an engineering course and 6% an engineering/science technology course (Nord et al. 2011). Currently, there are no national standards for K–12 engineering or technology education. Implementing such standards is difficult given limited experience with engineering/technology education at the K–12 level and insufficient numbers of teachers qualified to deliver instruction in this area (National Academy of Engineering 2010).

Definitions of Technology and Engineering Literacy. For the purpose of developing national assessments in this area, the NAEP framework defines technology, engineering, and technology and engineering literacy as follows (WestEd 2010, pp. 1–4):

• Technology is any modification of the natural or designed world done to fulfill human needs or desires.
• Engineering is a systematic and often iterative approach to designing objects, processes, and systems to meet human needs and wants.
• Technology and engineering literacy is the capacity to use, understand, and evaluate technology as well as to understand technological principles and strategies needed to develop solutions and achieve goals.

Areas To Be Assessed. The 2014 NAEP assessment of technology and engineering literacy will test students in the following three areas:

• Technology and Society covers the interaction of technology and people; effects of technology on society and the natural world; and questions of ethics, equity, and responsibility that arise from those effects.
• Design and Systems includes the nature of technology; the engineering design process by which technologies are developed; and basic principles of dealing with everyday technologies, including maintenance and troubleshooting.
• Information and Communication Technology involves computers and software learning tools; networking systems and protocols; and the selection and use of hand-held digital devices and other technologies for accessing, creating, and communicating information and for facilitating creative expression.

For examples of questions, see http://www.nagb.org/publications/frameworks/prepub_naep_tel_framework_2014.pdf (in chapters 3 and 4). Note that the grade level for these sample questions has not yet been determined.

Mathematics and Science Achievement in Charter Schools

Charter schools are public schools that provide elementary or secondary education to students under a specific charter granted by the state legislature or other appropriate authority (Hoffman 2008). These schools are independent of direct control by local school districts and operate free of many regulations applicable to traditional public schools. Data from the National Alliance for Public Charter Schools (http://www.publiccharters.org/dashboard/home) show that between 2000 and 2010, the number of charter schools more than tripled and the number of students attending these schools increased almost fivefold. In 2009–10, there were about 5,000 charter schools in 40 states and the District of Columbia with a total of 1.6 million students (3.4% of all U.S. public school students).

Comparison of student performance in charter versus traditional public schools is difficult because students in charter schools are self-selected (Garcia 2008; Grady and Bielick 2010). Some parents may enroll their children in charter schools because their children are struggling academically. Other parents may desire greater parent involvement or control. Still others may choose charter schools because they are dissatisfied with some aspect of local public schools. These selection factors may result in student populations in charter schools that are different from those in traditional public schools.

The data from the National Assessment of Educational Progress show that although average mathematics performance of fourth and eighth graders in charter schools improved from 2000 to 2009, charter school students overall had consistently lower scores than their counterparts in traditional public schools, and the gaps persisted over time (figure 1-A). In 2009, the average mathematics score of 12th graders in charter schools was also lower than that of their counterparts in traditional public schools. No measurable difference in average science scores, however, was found between students in charter and noncharter public schools (special NSF tabulations).

To mitigate the effects of selection factors, researchers have employed various research designs to control for different student characteristics in charter and noncharter schools (Abdulkadiroglu et al. 2009; Berends et al. 2010; Braun, Jenkins, and Grigg 2006; CREDO 2009; Hoxby, Murarka, and Kang 2009; Lubienski and Lubienski 2006; Zimmer et al. 2009). These studies produced mixed results on the effectiveness of charter schools, with impacts ranging from small (either positive or negative) to statistically insignificant (Betts and Tang 2008). There is wider variation in performance among charter schools than among public noncharter schools (Braun, Jenkins, and Grigg 2006). This may be due in part to wide variation in charter schools' operation and organizational structure (Buddin and Zimmer 2005; Zimmer et al. 2003).

Sample Items from PISA

Sample Items for Mathematics

1) A result of global warming is that the ice of some glaciers is melting. Twelve years after the ice disappears, tiny plants, called lichen, start to grow on the rocks. Each lichen grows approximately in the shape of a circle. The relationship between the diameter of this circle and the age of the lichen can be approximated with the formula:

where d represents the diameter of the lichen in millimeters, and t represents the number of years after the ice has disappeared. Using the formula, calculate the diameter of the lichen, 16 years after the ice disappeared.

Correct answer: 14 mm.

Difficulty level: Correct answer corresponding to 484 score points on the PISA mathematics scale ranging from 1 to 1,000.

2) In Mei Lin's school, her science teacher gives tests that are marked out of 100. Mei Lin has an average of 60 marks on her first four Science tests. On the fifth test she got 80 marks.

What is the average of Mei Lin's marks in Science after all five tests?

Correct answer: 64.

Difficulty level: Correct answer corresponding to 556 score points on the PISA mathematics scale ranging from 1 to 1,000

Sample Items for Science

1) Mary Montagu was a beautiful woman. She survived an attack of smallpox in 1715 but she was left covered with scars. While living in Turkey in 1717, she observed a method called inoculation that was commonly used there. This treatment involved scratching a weak type of smallpox virus into the skin of healthy young people who then became sick, but in most cases only with a mild form of the disease. Mary Montagu was so convinced of the safety of these inoculations that she allowed her son and daughter to be inoculated. In 1796, Edward Jenner used inoculations of a related disease, cowpox, to produce antibodies against smallpox. Compared with the inoculation of smallpox, this treatment had less side effects and the treated person could not infect others. The treatment became known as vaccination.

What kinds of diseases can people be vaccinated against?

A. Inherited diseases like haemophilia.
B. Diseases that are caused by viruses, like polio.
C. Diseases from the malfunctioning of the body, like diabetes.
D. Any sort of disease that has no cure.

Correct answer: B. Diseases that are caused by viruses, like polio.

Difficulty level: Correct answer corresponding to 436 score points on the PISA science scale ranging from 1 to 1,000.

2) Regular but moderate physical exercise is good for our health.

Is this an advantage of regular physical exercise:
Physical exercise helps prevent heart and circulation illnesses. Yes / No
Physical exercise leads to a healthy diet. Yes / No
Physical exercise helps to avoid becoming overweight. Yes / No

Correct answer: Yes, No, Yes in that order.

Difficulty level: Correct answer corresponding to 545 score points on the PISA science scale ranging from 1 to 1,000.

For additional sample questions, see http://www.pisa.oecd.org/dataoecd/47/23/41943106.pdf.

Common Core State Standards

To ensure that students graduate from high school adequately prepared for college and employment, a group of 48 states, led by the National Governors Association's Center for Best Practices and the Council of Chief State School Officers, has developed the Common Core State Standards Initiative (CCSSI) (NGA 2009). The standards outline a body of knowledge and skills students must master at each grade level to graduate from high school ready for college and career in the 21st century. The standards clarify what students are expected to learn in each grade, permit cross-state comparisons, and seek to improve student achievement by increasing the rigor of courses required to meet the standards (Fine 2010).

To date, CCSSI has sponsored development of standards for English language arts (ELA) and mathematics for grades K–12. (Detailed information on the ELA and mathematics standards is available on the CCSSI website at http://www.corestandards.org/the-standards.) The National Research Council is currently working on a framework for new national science standards for grades K–12 that states will have the opportunity to include in their common core standards when the standards become available in 2012 (Achieve, Inc. 2011).

Of the 48 states participating in CCSSI (Texas and Alaska do not participate), 44 states and the District of Columbia had adopted the standards by the end of 2010 (Gewertz 2010). States adopted the standards for a variety of reasons, including their rigor, the opportunity for cross-state comparisons, and increased chances of securing Race to the Top funds (EdSource 2010; Kober and Rentner 2011; The Opportunity Equation 2011). According to a recent survey, a majority of the states adopting the standards plan to develop new assessments, curriculum materials, instructional practices, teacher induction and professional development programs, and teacher evaluation systems based on the standards (Kober and Rentner 2011).

Taking Algebra I Before High School

Algebra I is considered a "gateway" course leading to more advanced coursetaking in mathematics and science and to higher levels of achievement (Loveless 2008). An increasing number of educators and researchers are calling for more students to take algebra I before high school (Ma and Wilkins 2007; Matthews and Farmer 2008; National Mathematics Advisory Panel 2008).

High school transcripts indicate credits earned for high school courses taken before ninth grade. According to HSTS data, 26% of high school graduates took algebra I before high school in 2009, up from 20% in 2005 (table 1-A). Percentages of both male and female graduates taking algebra before high school increased, though females (27%) slightly outpaced males (25%) in 2009. Upward trends occurred in all racial/ethnic groups as well, with black, Hispanic, and white graduates posting increases of 4, 7, and 6 percentage points, respectively. Asian/Pacific Islander students outpaced their peers by increasing their rates of completing algebra 1 before high school from 30% in 2005 to 48% in 2009.

HSTS identifies three curriculum levels based on the types of courses students take: standard, midlevel, and rigorous. A rigorous curriculum includes 4 years of mathematics including up to at least precalculus and 3 years of science, which must include biology, chemistry, and physics. HSTS data show that nearly two-thirds of graduates who completed a rigorous high school curriculum took algebra I before high school (Nord et al. 2011).

Measuring Teaching Quality

No research has conclusively identified the most effective teachers or the factors that contribute to their success, but efforts to improve measures of teaching quality have proliferated in recent years. For example, 21 states and over 100 teacher preparation programs have joined the Teacher Performance Assessment Consortium (TPAC) to develop a teacher evaluation instrument. The evaluation will be based on assessments embedded in teachers' preparatory coursework and on documentation of teaching and learning during multi-day lessons.

Another effort has focused on establishing a composite indicator for effective teaching by measuring student gains on test scores, quality of teaching practice, teachers' pedagogical content knowledge, student perceptions of the classroom environment, and teachers' perceptions of working conditions and instructional support at their schools (Measures of Effective Teaching 2010). Through the Measures of Effective Teaching project, researchers have analyzed data in large school districts nationwide to identify effective teachers and teaching practices. Data collection began in the 2009–10 academic year and continued in 2010–11.

A similar effort focused on mathematics teaching quality is underway at the National Center for Teacher Effectiveness, which seeks to identify practices and characteristics that distinguish effective mathematics teachers and to develop practical instruments and training tools for school districts. The center's core project, Developing Measures of Effective Mathematics Teaching, will combine measures of teacher characteristics, practice, and content knowledge and measures of student engagement and learning to build a composite measure of teaching effectiveness in mathematics. Data collection in approximately 50 schools and 200 classrooms began in 2010 and will continue through 2013.

These projects are among the largest efforts to incorporate gains in student test scores into the measurement of quality, but they are not the first. Several researchers have sought to develop so called "value-added" models that link teacher effectiveness to student gains in achievement test scores (Hanushek and Rivkin 2010; Hanushek et al. 2005). These models do not directly measure variation in teaching practices; rather, they compare test score gains of students with similar background characteristics and initial scores within the same school and attribute students' differences in progress to their teachers (Baker et al. 2010). Although some studies have validated the value-added approach (Jacob and Lefgren 2008; Kane et al. 2010; Kane and Staiger 2008), researchers have raised concerns about nonrandom assignment of students to teachers within a school; the use of standardized tests that do not adequately measure students' knowledge, skills, and progress; and family support or other factors outside of school that contribute to students' achievement (Baker et al. 2010; Hanushek and Rivkin 2010; Rothstein 2008).

Despite these concerns, there seems to be consensus that these models can contribute to current efforts to evaluate teaching when used along with other observable measures. However, researchers have not yet arrived at a comprehensive model for measuring teaching quality.

More information on the Teacher Performance Assessment Consortium is available at http://aacte.org/index.php?/Programs/Teacher-Performance-Assessment-Consortium-TPAC/teacher-performance-assessment-consortium.html. More information about the Measures of Effective Teaching project is available at http://www.metproject.org/. More information about the Developing Measures of Effective Mathematics Teaching project is available at http://www.gse.harvard.edu/ncte/projects/project1/default.php.

Teacher Attrition

Concerns about K–12 teacher shortages, teaching quality, and the need to retain high-quality instructors in the nation's elementary and secondary schools have led to considerable research on rates of attrition among teachers (Borman and Dowling 2008; Boyd et al. 2009; Ingersoll and Perda 2009; Jalongo and Heider 2006). A recent national study revealed that from 1988 to 2008, 5–9% of public school mathematics and science teachers left the teaching profession each year (figure 1-B) (Keigher and Cross 2010). The annual attrition rates of mathematics and science teachers are not higher than the average for all teachers (8–9% versus 8% in 2008, for example). Mathematics and science teachers who left teaching were also no more likely than other teachers who left to take noneducation jobs (appendix table 1-26).

Another study found large school-to-school differences in mathematics and science turnover (defined as teachers leaving their schools by either moving to another school or leaving teaching altogether) (Ingersoll and May 2010). High-poverty, high-minority, and urban public schools had among the highest mathematics and science teacher turnover rates. Reasons prompting mathematics teachers to leave their schools included lack of individual classroom autonomy, student discipline problems, and the extent to which teachers received useful content-focused professional development. For science teachers, the strongest factors included the maximum potential salary, student discipline problems, and the extent to which teachers received useful content-focused professional development (Ingersoll and May 2010).

More research is needed to establish conclusively links between how teachers enter the profession and attrition, but some has suggested that teachers who enter through alternative programs may be more likely to leave their schools or the profession than traditional-pathway teachers (Boyd et al. 2006; Kane, Rockoff, and Staiger 2006; Smith 2007).

State Student Tracking Systems

For the most part, existing state data systems are cross-sectional and do not track students over time. Statewide longitudinal data systems (SLDS) are designed to follow individual students from early childhood through high school and into postsecondary education and employment. The impetus for these new data systems comes from the need for more comprehensive and reliable data for accountability and evidence-based decisionmaking in education (DQC 2011a).

In 2005, the Institute of Education Sciences of the U.S. Department of Education introduced the SLDS Grant Program to encourage the development of these systems (IES 2011a). At the same time, a group of prominent education stakeholders launched the Data Quality Campaign to provide a national forum for discussions about SLDS implementation and to avoid duplication of effort and encourage collaboration across states (DQC 2011b). Although several states had been developing SLDS before 2005, most began designing their systems with the first round of federal funding in 2005, and many have made significant progress over the past 6 years (DQC 2011c). As of early 2011, for example, all states and the District of Columbia had collected student-level data on graduation and dropout rates (DQC 2011a).

Since 2005, 41 states and the District of Columbia have received at least one SLDS grant through one of four federal funding opportunities, including the American Recovery and Reinvestment Act (ARRA) (IES 2011b). To obtain ARRA funds, all governors and most legislatures agreed to implement SLDS that link preschool, K–12, postsecondary education, and workforce data and that conform to the requirements outlined in the America Competes Act by 2013 (U.S. Department of Education 2009). In addition, some states are linking their education data with data on corrections and social welfare assistance (Carson et al. 2010).

SLDS not only improve the quality of secondary and postsecondary education data, but also expose problems, such as the misalignment of state programs and inconsistencies in articulation of the data, that can then be addressed to improve education. SLDS are limited, however, by their inability to track students across state borders and into private colleges. A pilot project in Florida, Georgia, and Texas aims to develop a possible remedy for this problem by linking state data with college enrollment data from the National Student Clearinghouse (Bill & Melinda Gates Foundation 2010).