text-only page produced automatically by Usablenet Assistive Skip all navigation and go to page content Skip top navigation and go to directorate navigation Skip top navigation and go to page navigation
National Science Foundation
Awards
design element
Search Awards
Recent Awards
Presidential and Honorary Awards
About Awards
Grant Policy Manual
Grant General Conditions
Cooperative Agreement Conditions
Special Conditions
Federal Demonstration Partnership
Policy Office Website



Award Abstract #0746556

CAREER: Learning to Generate American Sign Language Animation through Motion-Capture and Participation of Native ASL Signers

NSF Org: IIS
Div Of Information & Intelligent Systems
divider line
Initial Amendment Date: April 11, 2008
divider line
Latest Amendment Date: May 8, 2013
divider line
Award Number: 0746556
divider line
Award Instrument: Continuing grant
divider line
Program Manager: Ephraim P. Glinert
IIS Div Of Information & Intelligent Systems
CSE Direct For Computer & Info Scie & Enginr
divider line
Start Date: June 1, 2008
divider line
End Date: May 31, 2014 (Estimated)
divider line
Awarded Amount to Date: $638,496.00
divider line
Investigator(s): Matt Huenerfauth matt.huenerfauth@rit.edu (Principal Investigator)
divider line
Sponsor: CUNY Queens College
65 30 Kissena Blvd
Flushing, NY 11367-1575 (718)997-5400
divider line
NSF Program(s): Cyber-Human Systems (CHS)
divider line
Program Reference Code(s): 1045, 1187, 7367, 9215, 9251, HPCC
divider line
Program Element Code(s): 7367

ABSTRACT

American Sign Language (ASL) is the primary means of communication for about 500,000 people in the United States. ASL is a distinct language from English; in fact, a majority of deaf U.S. high school graduates have only a fourth-grade (age 10) English reading level. Consequently, many deaf people find it difficult to read English text on computers, in TV captioning, or in other settings. Software to translate English text into an animation of a human character performing ASL would make more information and services accessible to deaf Americans. Unfortunately, however, essential aspects of ASL are not yet modeled by modern computational linguistic software. Specifically, ASL signers associate entities under discussion with 3D locations around their bodies, and the movement of many types of ASL signs changes based on these locations: pronouns, determiners, many noun phrases, many types of verbs, and others. When do signers associate entities under discussion with locations in space? Where do they position them? How must ASL sign movements be modified based on their arrangement? Creation of robust software to understand or generate ASL requires answers to questions such as these. The PI's goal in this research is to discover techniques for generation of ASL animations that automatically predict when to associate conversation topics with 3D locations, where to place them, and how these locations affect ASL sign movements. To these ends, he will create the first annotated corpus of ASL movement data from native signers (in a motion-capture suit and gloves), annotate this corpus with features relating to the establishment of entity-representing locations in space, use machine learning approaches to analyze when/where these locations are established and how 3D motion paths of signs are parameterized on those locations, incorporate the models into ASL generation software, and recruit native ASL signers to evaluate the 3D animations that result. This work will advance our linguistic knowledge relating to little-understood yet frequent ASL phenomena, and so will lay the foundation for software to produce a huge variety of ASL signs and constructions that are beyond the ability of current techniques to generate. This will in turn lead to ASL generation systems that produce higher quality animations that are more grammatical and understandable to deaf users, which will greatly benefit accessibility applications for deaf users and ASL machine translation.

Broader Impact: The ASL motion-capture corpus and an ASL generator that automatically handles spatial phenomena will enable more computational linguistic researchers to study ASL. This research also has applications for sign languages used in other countries (most with similar phenomena), and for the generation of animations of human gestures (for which empirical techniques developed in this work will apply). The PI is committed to finding ways to encourage deaf high school students to pursue science careers, and to creating Ph.D. research opportunities for deaf students. He will give presentations in ASL at local deaf high schools about computing research, make available summer research experiences for deaf high school students (using ASL skills to annotate the corpus, conduct evaluation studies, and inform the Deaf community about computing), recruit native ASL signers as Ph.D. and undergraduate researchers, and create courses on people-focused computer science research and careers (to attract diverse students to the field) and on assistive technology research (to interest and train Ph.D. students). These educational activities will be enabled by the PI's conversational ASL skills, by the research's relevance to deaf students, and by the Queens College's unique proximity to five local high schools for deaf students.


PUBLICATIONS PRODUCED AS A RESULT OF THIS RESEARCH

Note:  When clicking on a Digital Object Identifier (DOI) number, you will be taken to an external site maintained by the publisher. Some full text articles may not yet be available without a charge during the embargo (administrative interval).

Some links on this page may take you to non-federal websites. Their policies may differ from this site.


(Showing: 1 - 10 of 11)
  Show All

Matt Huenerfauth. "A Linguistically Motivated Model for Speed and Pausing in Animations of American Sign Language," ACM Transactions on Accessible Computing, v.2, 2009, p. 1. 

Matt Huenerfauth. "Improving Spatial Reference in American Sign Language Animation through Data Collection from Native ASL Signers," Proceedings of the International Conference on Universal Access in Human-Computer Interaction (UAHCI). San Diego, CA. July 2009. In C. Stephanidis (Ed.), Universal Access in HCI, Part III, HCII 2009, Lecture Notes in Computer Science., v.5616, 2009, p. 530. 

Matt Huenerfauth, Pengfei Lu. "Accurate and Accessible Motion-Capture Glove Calibration for Sign Language Data Collection," ACM Transactions on Accessible Computing, v.3, 2010, p. 1. 

Pengfei Lu, Matt Huenerfauth. "Data-Driven Synthesis of Spatially Inflected Verbs for American Sign Language Animation," ACM Transactions on Accessible Computing, v.4, 2011. 

Matt Huenerfauth, Pengfei Lu. "Effect of Spatial Reference and Verb Infection on the Usability of American Sign Language Animations," Universal Access in the Information Society, v.11, 2012, p. 169-184. 

Hernisa Kacorri, Pengfei Lu, Matt Huenerfauth. "Effect of Displaying Human Videos During an Evaluation Study of American Sign Language Animation," ACM Transactions on Accessible Computing, v.5, 2013. 

Matt Huenerfauth. "A Linguistically Motivated Model for Speed and Pausing in Animations of American Sign Language," ACM Transactions on Accessible Computing, v.2, 2009. 

Matt Huenerfauth, Pengfei Lu. "Accurate and Accessible Motion-Capture Glove Calibration for Sign Language Data Collection," ACM Transactions on Accessible Computing, v.3, 2010. 

Matt Huenerfauth, Pengfei Lu. "Effect of Spatial Reference and Verb Inflection on the Usability of American Sign Language Animations," Universal Access in the Information Society, v.11, 2012, p. 169. 

Pengfei Lu, Matt Huenerfauth. "Data-Driven Synthesis of Spatially Inflected Verbs for American Sign Language Animation," ACM Transactions on Accessible Computing, v.4, 2011. 


(Showing: 1 - 10 of 11)
  Show All




CONFERENCE PROCEEDINGS PRODUCED AS A RESULT OF THIS RESEARCH

Lu, PF; Huenerfauth, M. "Accessible Motion-Capture Glove Calibration Protocol for Recording Sign Language Data from Deaf Subjects," in 11th International ACM SIGACCESS Conference on Computers and Accessibility., 2009, p. 83-90. 

Huenerfauth, M. "Improving Spatial Reference in American Sign Language Animation through Data Collection from Native ASL Signers," in 5th International Conference on Universal Access in Human-Computer Interaction held at the HCI International 2009., v.5616, 2009, p. 530-539. 

BOOKS/ONE TIME PROCEEDING

Pengfei Lu, Matt Huenerfauth. "Accessible Motion-Capture Glove Calibration Protocol for Recording Sign
Language Data from Deaf Subjects", 06/01/2009-05/31/2010, "Proceedings of the 11th International ACM SIGACCESS Conference on Computers
and Accessibility (ASSETS 2009), Pittsburgh, Pennsylvania, USA."
,  2009, "New York: ACM Press.".

Pengfei Lu, Matt Huenerfauth. "Collecting an American Sign Language
Corpus through the Participation of
Native Signers", 06/01/2010-05/31/2011, "Proceedings of the International
Conference on Universal Access in
Human-Computer Interaction (UAHCI)"
,  2011, "Springer".

Matt Huenerfauth, Pengfei Lu. "Modeling and Synthesizing Spatially
Inflected Verbs for American Sign
Language Animations", 06/01/2010-05/31/2011, "Proceedings of the 12th International
ACM SIGACCESS Conference on
Computers and Accessibility (ASSETS
2010)"
,  2010, "DOI=10.1145/1878803.1878823".

Pengfei Lu, Matt Huenerfauth. "Collecting a Motion-Capture Corpus of
American Sign Language for Data- Driven
Generation Research", 06/01/2010-05/31/2011, "Proceedings of the First Workshop on
Speech and Language Processing for
Assistive Technologies (SLPAT), HLT-
NAACL conference"
,  2010, "Association for Computational Linguistics, Stroudsburg, PA, USA, pages 89-97.".

Matt Huenerfauth, Pengfei Lu. "Eliciting Spatial Reference for a Motion-Capture Corpus of American Sign
Language Discourse", 06/01/2010-05/31/2011, "Proceedings of the Fourth Workshop
on the Representation and Processing
of Signed Languages: Corpora and Sign
Language Technologies, LREC
conference"
,  2010, "European Language Resources Association, pp. 121-124.".

Pengfei Lu, Matt Huenerfauth. "Collecting an American Sign Language
Corpus through the Participation of
Native Signers", 06/01/2011-05/31/2012, "Proceedings of the International
Conference on Universal Access in
Human-Computer Interaction (UAHCI)"
,  2011, "Springer".

Matt Huenerfauth, Pengfei Lu. "Modeling and Synthesizing Spatially
Inflected Verbs for American Sign
Language Animations", 06/01/2011-05/31/2012, "Proceedings of the 12th International
ACM SIGACCESS Conference on
Computers and Accessibility (ASSETS
2010)"
,  2010, "DOI=10.1145/1878803.1878823".

Pengfei Lu, Matt Huenerfauth. "Collecting a Motion-Capture Corpus of
American Sign Language for Data- Driven
Generation Research", 06/01/2011-05/31/2012, "Proceedings of the First Workshop on
Speech and Language Processing for
Assistive Technologies (SLPAT), HLT-
NAACL conference"
,  2010, "Association for Computational Linguistics, Stroudsburg, PA, USA, pages 89-97.".

Matt Huenerfauth, Pengfei Lu. "Eliciting Spatial Reference for a Motion-Capture Corpus of American Sign
Language Discourse", 06/01/2011-05/31/2012, "Proceedings of the Fourth Workshop
on the Representation and Processing
of Signed Languages: Corpora and Sign
Language Technologies, LREC
conference"
,  2010, "European Language Resources Association, pp. 121-124.".

Pengfei Lu, Matt Huenerfauth. "Synthesizing American Sign Language
Spatially Inflected Verbs from Motion-
Capture Data", 06/01/2011-05/31/2012, "Proceedings of the Second International
Workshop on Sign Language Translation
and Avatar Technology (SLTAT), at the
13th International ACM SIGACCESS
Conference on Computers and
Accessibility."
,  2011, "http://vhg.cmp.uea.ac.uk/demo/SLTAT2011Dundee/".

 

Please report errors in award information by writing to: awardsearch@nsf.gov.

 

 

Print this page
Back to Top of page
  FUNDING   AWARDS   DISCOVERIES   NEWS   PUBLICATIONS   STATISTICS   ABOUT NSF   FASTLANE  
Research.gov  |  USA.gov  |  National Science Board  |  Recovery Act  |  Budget and Performance  |  Annual Financial Report
Web Policies and Important Links  |  Privacy  |  FOIA  |  NO FEAR Act  |  Inspector General  |  Webmaster Contact  |  Site Map
National Science Foundation Logo
The National Science Foundation, 4201 Wilson Boulevard, Arlington, Virginia 22230, USA
Tel: (703) 292-5111, FIRS: (800) 877-8339 | TDD: (800) 281-8749
  Text Only Version