Hearing is not understanding
When you ask a voice assistant a question, two separate jobs take place. The first is speech recognition, which converts the sound of your voice into written words. The second is working out what you meant and what to do about it. This article is about the first job: getting from audio to text. It is a harder problem than it looks, because people do not speak in neat, separated words.
From sound to numbers
A microphone turns air vibrations into an electrical signal, which is then measured thousands of times per second and stored as a long list of numbers. That raw waveform is not very useful directly. Systems typically process it in short slices, often a few hundredths of a second long, and convert each slice into a compact description of which frequencies are present and how strong they are. You can picture it as a rolling snapshot of the sound’s pitch and tone content.
Different speech sounds have different frequency patterns. Vowels, for instance, show characteristic bands of energy that differ from one vowel to another, while sounds like “s” produce a hissy, high-frequency pattern.
From numbers to speech sounds
The next step is to figure out which basic sounds, called phonemes, are being spoken. English has roughly forty of them, depending on how you count and which accent you use. Words are built from sequences of phonemes: the word “cat” is something like three sounds in a row.
Older systems used statistical models for this, combined with hand-built dictionaries of pronunciations. Modern systems mostly use neural networks trained on large collections of recorded speech paired with transcripts. Some are built in stages, with an acoustic model that maps audio to sounds and a separate model that handles word choice. Others are trained end to end, mapping audio directly to text. In either case, the system learns patterns from examples rather than following rules someone wrote, which makes it a form of machine learning.
Choosing words that make sense
Sounds alone are ambiguous. “Recognize speech” and “wreck a nice beach” sound very close. To decide between candidates, the system uses a language model, which estimates how likely a sequence of words is in ordinary language. “Recognize speech” is a more probable phrase than “wreck a nice beach” in most settings, so it usually wins.
Context can help further. If the assistant knows you are asking about music, it may favor a band name over a similar-sounding common word. Some systems also adapt to your own voice or vocabulary, such as names in your contacts.
The decoding step weighs many possibilities at once and picks the most probable overall sentence, rather than deciding one word at a time.
Why errors happen
Even good systems make mistakes. Common sources include:
- Background noise. Traffic, music, running water or other people talking can mask the sounds the system needs.
- Distance and microphone quality. A faint, echoey recording from across a room carries less information than speech close to a microphone.
- Accents and dialects. Systems tend to work best on the kinds of speech they have heard most in training. If some accents or ways of speaking are underrepresented in the data, error rates for those speakers can be higher. This is an example of the training-data problem and a case of possible algorithmic bias.
- Homophones. Words like “their” and “there” sound identical, so only context can separate them.
- Unusual words. Rare names, technical terms and new slang may not be in the vocabulary, so the system substitutes something similar.
- Fast or mumbled speech. Words blend together, and sounds at the edges disappear.
- Overlapping speakers. When two people talk at once, separating them is difficult.
- Speech differences. Stuttering, speech disorders, age and illness can all change how speech sounds, and many systems handle these less well.
- Different languages in one sentence. Switching languages mid-sentence is hard for many systems.
Accuracy figures reported for speech recognition depend heavily on the test conditions, so a single percentage should be read with caution. A system may do very well on clear, read-aloud speech and worse on casual conversation in a noisy room.
Waking up and listening
Many assistants wait for a trigger phrase. A small, simple model listens continuously for that phrase, and only after it detects it does the device begin the heavier processing. Where the full processing happens varies: some of it may be on the device itself and some may be sent to remote servers. This matters for data privacy, since recordings may be stored or reviewed, depending on the service and its settings. Checking the privacy options on your device is a sensible habit.
False triggers also occur, when the device mistakes ordinary sounds or conversation for its wake phrase and starts listening when you did not intend it to.
What happens after the text appears
Once speech has become text, other components try to work out intent: is this a question, a command, a request to play something? That stage uses language understanding, and the reply may be generated or retrieved. The assistant can answer incorrectly even when it transcribed you perfectly, and the reverse can also happen, so it helps to know which stage failed.
Getting better results
Some simple habits help, although none guarantees success: speak at a natural pace rather than shouting, reduce background noise when you can, be close to the microphone, and use full phrases that give the system context. If a name is repeatedly misheard, check whether the device allows you to add or correct it.
Speech recognition has improved a great deal over the years, and it is useful for dictation, accessibility and hands-free tasks. Yet it is still a statistical guess about what sounds were made, and anything important, such as a message to a person or a financial command, deserves a quick check of what the system thought you said.