KDI Home    Contact Us   

IEA Web site

Investigators Test Automatic Essay Scoring in Tutoring Systems

Researchers at New Mexico State University have been testing the patented Intelligent Essay Assessor (IEA) software for its applicability to automated training systems. Their results have demonstrated the promise of IEA as a tool for helping students master course content and improve their writing skills.

The IEA is a technique for scoring responses to essay questions that is based on a theory called Latent Semantic Analysis (LSA). It assesses the quality of an essay by characterizing the meaning of the words contained in the essay and comparing that characterization with the conceptual relevance and content of essays of known quality. LSA, the theory behind the IEA, and the IEA technology have been presented at a number of conferences and have inspired much interest.

IEA scores on essays are very much like those of human graders. They have been found to agree with those of human experts as much as those experts agree with each other. Other essay grading programs have been developed that assess grammar, spelling, and punctuation rather than content. The IEA software can determine the student's mastery of the material and give students constructive feedback. Unlike other essay grading programs, the IEA software can help students learn.

Peter W. Foltz, one of the professors who developed the IEA, and Adrienne Y. Lee, both of NMSU, were principal investigators on a recent KDI grant. They looked at the potential for using the IEA software as a tool not only for assessing performance but also for helping students learn from automated training systems.

Foltz and Lee developed training systems for use in a series of psychology courses. They evaluated the merits of the IEA software in this context by comparing the relative performance of students who responded to multiple-choice, essay, and reflection questions. The students who responded to the essay questions received feedback from the IEA software and were allowed to rewrite and resubmit their essay as many times as they wanted. Those students mastered the material best and experienced the greatest improvement in their writing skills.

These results are very promising. According to Foltz, "They show that incorporating writing into automated training systems is both feasible and can result in improved learning over traditional assessment methods. Automated writing assessment has the potential to be incorporated into almost any area of training in which a student must reason about a particular content area. This can include school topics, such as biology, physics, and history, as well as such areas as corporate or military training."

Additional NSF-funded research work is under way to refine the application of LSA to training systems and to test additional topics in science in which writing can be incorporated into automated training systems.

Back to Top of Page