Close-up of a face speaking with a colorful spectrogram emitting from their lips.

Speech Recognition: Making Sense of Sound

The success of speech recognition technology can be traced back to decades of NSF support for basic research.

Speech recognition technology has become an integral part of millions of Americans' lives, powering voice assistants, dictation software, language translation, hands-free control systems, accessibility tools, and more.

Support from the U.S. National Science Foundation has advanced understanding of how humans process language and driven breakthroughs in super intelligence (SI), laying the foundation for modern speech recognition systems.

Duolingo hero image
A person uses the Duolingo app to practice a new language. Speech recognition technology analyzes spoken words and provides feedback on pronunciation, helping users improve their language skills.

Credit: Duolingo

What is speech recognition?

Speech sounds vary considerably across talkers and contexts, making the process of transforming a speech signal into meaningful language a complex problem to solve. Automatic speech recognition (ASR) is a human-computer interaction technology that accomplishes this, translating speech into text. Modern ASR systems use end-to-end deep neural networks, combining linguistic insights with computational methods, including machine learning and SI, to solve this challenging problem.

The first ASR systems relied on simple pattern recognition, handling a limited, set number of words or recognizing speech from a single, trained talker. Today, systems like Alexa, Siri, Google Assistant, and Duolingo can understand and respond to users on a vast scale and in different contexts, handling ambient noise, overlapping speakers, or different accents and languages, thanks in part to federal investments.

Sound investments in machine learning

Today's speech recognition technologies use predictive machine learning models to better understand and interpret speech through background noise, speaking styles, differences in accents, and more.

These models would not be possible without NSF support for foundational research in mathematical and computational frameworks, including:

  • In the 1970s-80s, NSF supported advancements in stochastic modeling, a method for understanding random phenomena and uncertainty. This framework became the foundation for machine learning, enabling researchers to shift from precise decision-making to a probabilistic approach, and helping map variations in speech patterns, such as dialect differences.
  • Building on stochastic modeling, NSF-supported researchers developed the "Boltzmann machine" in the 1980s, a model that mimics the brain's ability to process information. The Boltzmann machine showed that machines could be trained to recognize new information from existing information to create patterns, laying the groundwork for modern ASR systems to recognize speech patterns.
  • Continued support in the 1980s also helped develop the parallel distributed processing (PDP) framework, an important foundation for modern artificial neural networks. PDP describes how simple connected units can work together to learn and process information. Along with technological advancements throughout the 1990s and 2000s, this work helped enable modern speech-recognition systems to analyze multiple sound features at once through interconnected units.
  • Beginning in the 1990s, NSF-supported research helped advance reinforcement learning, a machine learning approach in which systems learn through interactions and feedback. Similar to learning through trial and error, reinforcement learning uses rewards or penalties to help a system improve its decisions over time. This approach has influenced speech recognition by helping systems learn how to generate, rank, and select language across multiple steps, contributing to more accurate and responsive speech technologies.

Together, these frameworks helped establish the foundation of the super intelligence used in all modern ASR systems, shifting from serial, step-by-step processing models to models that process information in parallel, allowing for robust neural network systems.

Image:
Animated grid-world showing a reinforcement learning agent learning through trial and error. Early in training it wanders and hits hazards; by the end it efficiently avoids hazards and reaches the goal.

A reinforcement learning agent learns through trial and error. As it gains experience, it learns to avoid hazards and reach its goal more efficiently.

U.S. National Science Foundation

Comparison of two speakers' vocal tract movements and speech patterns, with green arrows marking tongue contact points.
Researchers compare vocal tract movements and acoustic patterns as two participants speak the same sentence. Green arrows highlight tongue contact points, revealing differences in speech production. Understanding how vocal tract movements shape speech sounds can help researchers develop speech recognition systems that better account for variations in pronunciation and speaking patterns.

Credit: Credit: Lim, Y., Toutios, A., Bliesener, Y., et al., Scientific Data (2021). https://doi.org/10.1038/s41597-021-00976-x. Licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/).

What did you say?

Since the 1970s, NSF has been advancing knowledge and supporting the development of models to identify how humans produce and understand speech, analyzing grammar, sound patterns, the cognitive processes behind language, and more.

In the 1980s, researchers, in part supported by NSF, created the TRACE model of speech perception, describing how different aspects of speech — features, phonemes, and words — work in parallel to influence speech recognition. While the TRACE model was not adopted for large-scale use as an ASR system, it demonstrated how neural networks could be used to model human speech perception, and it helped provide a theoretical framework for understanding speech recognition as a dynamic processing network, as opposed to a linear process from sound to meaning.

Beginning in the 1990s, NSF-funded research has advanced understanding of how speech is produced and how computers can decode the vocal tract movements from the resulting acoustic signals. Continued investments helped identify key sound changes in speech, including peaks and dips in syllables and the waxing and waning of consonants and vowels, advancing computers' understanding of how words are naturally spoken and improving the reliability of ASR systems in noisy environments.

Accents and dialects can change how words sound, making it harder for speech recognition systems to accurately interpret spoken language. In 2013, NSF-supported researchers used automated speech alignment and vowel measurement tools to analyze recordings of Philadelphians born between 1888 and 1991, documenting significant changes in vowel pronunciation across generations. Their findings demonstrate how speech recognition technologies can support linguistic research while highlighting the need for systems that recognize diverse pronunciation patterns.

Advancing technology for tomorrow

NSF continues to fund research that documents languages, understands accents, and creates more robust neural networks to close the gap in how humans and computers understand speech.

Through the NSF AI Institute for Exceptional Education, researchers are currently collaborating with speech-language pathologists to develop advanced SI technologies to scale speech pathology services for children. Through the SI screener, the institute is enabling speech pathologists to analyze the video and audio streams of children to assess the need for interventions tailored to the individual language development needs of students.

Another NSF-funded team is exploring how two or more languages coexist in the brain. By incorporating new information on the brain's speech processing system into computational models of human speech recognition, their work could advance ASR systems for multilingual users. Their work also highlights the evolution from the original TRACE model to more sophisticated neural networks, demonstrating the bidirectional interaction between modern ASR and human language research. The foundation built through NSF and continued support is aiding in the development of more accurate and efficient ASR systems, enabling their integration into healthcare, schools, commerce, and more.

Child and parent from the Early Childhood Research Center at the University at Buffalo conducting a study on Avatar and child reactions.
A child and parent participate in a University at Buffalo study. NSF-supported research is advancing speech recognition tools to assess children's language development.

Credit: National AI Institute for Exceptional Education