Fig.6. Multimodal language acquisition. The user communicates with the machine in the sensory dimensions of sight, sound, and gesture. The machine responds to the input then receives feedback as to the appropriateness of its response. The system changes its language model/user model based on a semantic-level error signal (multimodal feedback from the user).
Back to list of slides.