Can a phone understand speech without sending the audio anywhere?

Why recognizing speech is a lighter job than running a language model

A speech recognizer maps audio onto words and may use language-specific acoustic and linguistic information. Model size and accuracy vary across systems, languages and configurations; a generative language model has a different role and resource profile.

Speech arrives in real time, so a recognizer generally needs to keep pace with a person talking. Text generation can use more compute to produce an answer sooner; actual rates vary by system and task.

What actually runs on the phone while you talk

Speech recognition systems often process audio in short overlapping frames and reduce it to features before inference. The representation and timing vary by system; the model does not necessarily process raw audio samples directly.

A wake word is handled separately, by a very small always-on detector that often sits on a low-power co-processor so the main chip can stay asleep. Recognition proper is usually streaming: the decoder emits a best guess while you are still speaking and revises it as more audio arrives. That is why dictated words sometimes change a beat after they appear.

Local recognition avoids a remote request for that recognition step, which can reduce network delay and allow it to work without coverage. The actual response time and supported languages depend on the device, feature and configuration.

Where local recognition still struggles

The hard cases are not caused by running locally. They are the hard cases for speech recognition anywhere, and a phone simply has less room to throw hardware at them. Four of them account for most of the errors people actually notice.

Recognition accuracy depends on the system, language, speaker, recording conditions and evaluation set. Clean, close speech can be easier than noisy or spontaneous speech, but a result only applies to the conditions in which it was measured.

• Noise and distance. A voice from across a reverberant room is a far weaker signal than a voice near the microphone, and multiple microphones with beamforming help without solving it.
• Rare words. Names, medical and technical vocabulary, and anything thin in the training data. Recognizers can be biased toward a supplied word list, which a phone can build from contacts and app names it already holds.
• Languages and accents. Accuracy tracks how well a voice is represented in training. A phone usually carries a few language packs rather than every language at once, so adding one is a download.
• Formatting. Punctuation, capitalization and stripping out false starts are typically a separate step from recognition, and they are where a transcript most obviously looks machine-made.

Recognition is not understanding

Getting the words right is the first half. Deciding that a sentence about calling the landlord on Thursday is a reminder, with a person in it and a date in it, is the second half, and that part needs a language model rather than an acoustic one. The two are separate systems with very different appetites, and only one of them is a solved problem on a phone.

Whether audio is recognized on the device is also a different question from whether it is recorded or kept. A system can transcribe locally and still upload the transcript, or hold audio for review. Where the recognition runs tells you where the sound was processed, not what happened next.