KDI Home    Contact Us   

House Ear Institute Web site Qualisys™ 3-D Motion Capture System UCLA Phonetics Lab

3D Optical and Magnetometer Signals for Speech

Neuroscientists have applied a new tool in exploring the visual cues people display while speaking. These cues are vital to our ability to understand speech in noisy environments and enhance communication for the hearing impaired. The Qualisys™ 3-D Motion Capture System (http://www.qualisys.se/) is now being used by neuroscientists to capture 3-D placement of reflector targets on the talking subject. Use of this system may lead to applications such as enhancing the comprehensibilty of computer generated speakers, facilitating assimilation of a second language, and increasing the ease of communication for the hearing impaired.

Qualisys™ 3-D Motion Capture System

When people with normal hearing and vision talk to one another, they produce both acoustic and optical speech signals and process both types, leading to effective communication of nuance and understanding. Visual cues become especially important to the listener under noisy conditions or if the speaker is mumbling.

There has also been considering recent research in the area of Automatic Speech Recognition (ASR) for use with computers. However, very little information is currently available about optical speech signals. This information is vital for controlling synthesizers and for understanding human speech processing.

A joint project between four laboratories (Department of Communication Neuroscience and Electrophysiology Department at House Ear Institute, Phonetics Laboratory at UCLA, and Speech Processing and Auditory Perception Laboratory at UCLA) funded by the National Science Foundation and headed by Dr. Lynne Bernstein, has developed a novel system for recording both 3-D visual and auditory data for use in these fields. Recordings include acoustic, optical (with retroreflector labeling on the faces), and physiologic signals (Electromagnetic Midsaggital Articulography -- EMA), using both deaf and hearing adult subjects.

Deaf students associated with the National Center on Deafness, California State University Northridge participate as lipreaders.

In addition to the QualisysTM System, a new facility has been developed to record synchronized streams of audio, video, 3D motion (Qualisys™), and electromagnetic midsaggital articulography (EMA) data.

These tools promise to bring significant support to multiple areas of research in engineering, computer science, medicine, and education, leading to enhanced communication and learning across disciplines. For more information on this project, please go to http://www.hei.org/research/projects/comneur/speechdata.htm.

Back to Top of Page