Investigators Test Automatic Essay Scoring in Tutoring
Researchers at New Mexico State University have been testing
the patented Intelligent Essay Assessor (IEA) software for its applicability to
automated training systems. Their results have demonstrated the promise of IEA
as a tool for helping students master course content and improve their writing
IEA is a technique for scoring responses to essay questions that is based on a
theory called Latent Semantic Analysis (LSA). It assesses the quality of an
essay by characterizing the meaning of the words contained in the essay and
comparing that characterization with the conceptual relevance and content of
essays of known quality. LSA, the theory behind the IEA, and the IEA technology
have been presented at a number of conferences and have inspired much interest.
IEA scores on essays are very much like those of human
graders. They have been found to agree with those of human experts as much as
those experts agree with each other. Other essay grading programs have been
developed that assess grammar, spelling, and punctuation rather than content.
The IEA software can determine the student's mastery of the material and give
students constructive feedback. Unlike other essay grading programs, the IEA
software can help students learn.
Peter W. Foltz, one of the professors who developed the IEA,
and Adrienne Y. Lee, both of NMSU, were principal investigators on a recent KDI
grant. They looked at the potential for using the IEA software as a tool not
only for assessing performance but also for helping students learn from
automated training systems.
Foltz and Lee developed training systems for use in a series
of psychology courses. They evaluated the merits of the IEA software in this
context by comparing the relative performance of students who responded to
multiple-choice, essay, and reflection questions. The students who responded to
the essay questions received feedback from the IEA software and were allowed to
rewrite and resubmit their essay as many times as they wanted. Those students
mastered the material best and experienced the greatest improvement in their
These results are very promising. According to Foltz, "They
show that incorporating writing into automated training systems is both
feasible and can result in improved learning over traditional assessment
methods. Automated writing assessment has the potential to be incorporated into
almost any area of training in which a student must reason about a particular
content area. This can include school topics, such as biology, physics, and
history, as well as such areas as corporate or military training."
Additional NSF-funded research work is under way to refine
the application of LSA to training systems and to test additional topics in
science in which writing can be incorporated into automated training systems.
Back to Top of Page