Person and equipment with text - force feedback tactile glove, gaze tracker, smart controller for tactile glove, microphone array, speech synthesis, automatic speech recognition, multimodal user input, text input, activation outputs, action nodes, connection weights, word detector nodes, action

Fig.6. Multimodal language acquisition. The user communicates with the machine in the sensory dimensions of sight, sound, and gesture. The machine responds to the input then receives feedback as to the appropriateness of its response. The system changes its language model/user model based on a semantic-level error signal (multimodal feedback from the user).

Next slide

Back to list of slides.