CS Bits & Bytes is a bi-weekly newsletter highlighting innovative computer science research. It is our hope that you will use CS Bits & Bytes to engage in the multi-faceted world of computer science to become not just a user, but a creator of technology. Please visit our website at: http://www.nsf.gov/cise/csbytes.

January 23, 2012
Volume 1, Issue 4

Language Processing

What went wrong? When the Siri voice recognition debuted with the launch of the iPhone 4S in October 2011, many Scottish users, though English speakers, couldn’t use this cool new feature because of Siri’s difficulty in understanding their accent. 

It is difficult for computers to understand and sound like humans because of all of the variation in our speech.  For example, we are pretty musical when we talk, and that intonation, phrasing, and duration, collectively known as prosody, gives the words meaning.  Really?  Really.


Chatbots Talking
Listen to the creators of the AI vs. AI chatbots
talk about how the conversation actually played out:

Different accents, voices, the noise around people, words that sound the same but have different meanings (homonyms), and use of slang are some of the other challenges for human-computer communication.  At the heart of these interactions is computer science!

Recognize, understand, evaluate, and respond. In order for computers to communicate through speech with humans, four things need to happen:  (1) the computer must hear the speech and convert it to words; (2) the computer must then determine the meaning of what was said based on the context; (3) the computer needs to determine its response and choose which words to use; and (4) the computer has to turn its response into spoken sounds.  Each of these steps is complicated and getting them to happen involves active research areas combining linguistics, computer science, and engineering.

Jason Yosinski
Images of Igor Labutov (left) and Jason Yosinski (right), the creators of the AI vs AI chatbot conversation.

What is a chatbot?   Chatbot is short for chatter robot. It is a computer program that attempts to simulate human conversation, through the use of artificial intelligence.  Chatbots are intended to communicate with people, and they learn to be conversational by practicing (that is, as they interact with people, they collect data on how people talk and what they talk about). 

Recently, chatbots have been talking with other chatbots. Graduate students in the Cornell Creative Machines Lab in Ithaca, NY, Igor Labutov and Jason Yosinski, put together the AI vs. AI video of two chatbots talking to each other by combining different language processing components.  Igor does research in robotics and is funded with a National Science Foundation Graduate Research Fellowship, and Jason does research in computer science.

Understanding language processing does more than just improve our experience with computers! Computer scientists create speech synthesizer systems that help people speak when their voices have been damaged (think Stephen Hawking), that help the blind to read by saying the words, and help the deaf to hear by turning speech into text.

Professor Ostendorf
Image of Professor Mari Ostendorf.

Spotlight! Professor Mari Ostendorf of the University of Washington is using language processing to improve literacy across the United States.  With funding from the National Science Foundation, she is analyzing text collected across the web and modeling ways to paraphrase complex writings in order to simplify text for individual reading needs to ultimately make information more available to people with limited reading skills.

Professor Ostendorf does research in speech processing, recognition, modeling, and synthesis.  She has a B.S.,  Master’s, and Ph.D in electrical engineering from Stanford University and currently works with both engineering and computer science students in her position at University of Washington.  Her work has moved her to other countries, like Japan and Germany. In her spare time, she likes to go skiing with her family.



Learn more about Siri's difficulty in understanding people: http://abcnews.go.com/Technology/siri-lost-translation-heavy-accents/story?id=14834111.

To learn more about how the chatbots from the AI vs AI video were made, see: http://theinstitute.ieee.org/technology-focus/technology-topic/chatbot-conversation-goes-viral.

Learn about the intelligence of one of the oldest chatbots, named Cleverbot, at: http://www.msnbc.msn.com/id/44434584/ns/technology_and_science-science/t/how-cleverbot-computer-chats-human/.


This activity will help students to realize some of the nuances of language that come easy to us, but are extremely difficult for computers.

  1. Break the students into groups of three.
  2. Assign one student as the reader and the other two as interpreters.
  3. Provide a copy of the chart below only to the reader. 
  4. Instruct the reader to read aloud a word from the chart and ask the interpreters to write it down. 
  5. Next, instruct the reader to read aloud the sentence and ask the interpreters if they wrote down the correct word.  The reader should then read the additional sentence.
  6. Students should discuss the differences between the words in each sentence. Remind students about homonyms (they should remember from English class that homonyms are two words that sound the same but have different meaning).
  7. Bring the class back for a group discussion, listing challenges that the students encountered in choosing the correct word and list other factors which make language processing for computers difficult (e.g., pronunciation, enunciation, simple pitch changes, and context). 
Word to Read Sentence Additional Sentence for Discussion
Their Their light display is fantastic! There goes another light.
Marry Will you marry me? My friend is Mary.
Where Where are you taking me? Wear this to school today.
Air It is a cold air blowing. To err is human, you simply made a mistake.
Eye A Cyclops only has one eye. My friends know that I am fun.
Night The night sky is full of stars. The knight in shining armor won over his princess.

Optional Activity: There are many chatbots on the web with which students can “speak.”  As chatbots learn from previous interactions, be aware that they can and may say anything, not always appropriate.  It may be best to mediate a class discussion with a chatbot if you choose to do this activity.