Design a Voice Assistant

The ask

A person speaks a request to a phone, a speaker or a car system.

Three stages do the work. Speech recognition turns the spoken words into written text.

A language model reads that text and works out the answer, or the next step.

Speech synthesis turns the written answer into spoken sound. It is the text-to-speech step, aimed at a live conversation.

The assistant also carries out simple actions. Setting a timer and playing music are typical requests.

Long answers stay out of scope. Spoken answers must be brief to be useful.

Estimate the scale

The service expects 50,000,000 voice requests a day.

50,000,000 requests ÷ 86,400 seconds = 579 requests per second.

Each request is a live interaction. Capacity must cover people talking at the same time, not just the request rate.

Take one full turn, from first word to last spoken word, as about 10 seconds.

579 requests per second × 10 seconds = 5,790 turns in progress at any moment.

Peak use clusters at waking, commuting and cooking times. The fleet is sized above the average rate.

Data and features

Each stage trains on its own data. Recognition needs recorded speech paired with exact written transcripts.

The language model trains on text, and on past conversations with names and private details removed.

Synthesis needs recorded speech from speakers who gave clear permission.

Context is a feature of the whole system. Time, place and the current conversation all change what a short request means.

A request like 'play it again' means nothing without the conversation before it.

The strongest signal is failure in real use. A request repeated louder usually means the first hearing was wrong, and corrections are labelled for the stage that failed.

High-level design

The device records the request and streams the audio to an app server.

Recognition turns the audio into text. It also returns a confidence score, a measure of how sure it is.

The text goes to the language model as a prompt, together with the recent conversation.

Where the request needs an action, the agent pattern takes over. The model picks a tool, the tool acts, and the result returns to the model.

The answer text goes to speech synthesis. The sound streams back to the device.

Every stage runs as its own service. Each stage can scale and improve on its own.

Training the models

Three models means three training pipelines. Each model is trained, tested and released on its own schedule.

Recognition is tested on many accents, ages and noisy rooms before release.

The language model is fine-tuned for short spoken answers. A long written style is a failure when the answer will be heard.

Synthesis voices are trained only from consented recordings. Each voice is checked for misread words.

A stage can be rolled back alone. A bad recognition release never forces the other stages back.

Serving predictions

Speech recognition gets about 200 milliseconds.

The language model gets about 500 milliseconds to its first token.

Speech synthesis gets about 300 milliseconds to first sound.

200 + 500 + 300 = about 1,000 milliseconds before the answer starts.

Work overlaps where it can. Synthesis starts on the first sentence while the model writes the rest.

If one stage overruns, the others cannot win the time back. Budgets are enforced per stage, not just in total.

Deep dive: acting on a request

Questions end with an answer. Actions change the world, like sending a message or booking a ride.

The agent pattern keeps actions bounded. The model may only use listed tools, each with a narrow job.

Risky or costly actions ask the person first. The assistant states what it is about to do and waits for a yes.

Every action is logged with the request that caused it. A wrong action can be traced, and undone where the tool allows.

A tool failure is spoken plainly. The assistant says what failed and offers the next best step, never a silent retry loop.

When hearing fails

Noise is the normal case. Kitchens, streets and cars all fight the microphone.

Low confidence triggers a short question, not a guess. The assistant asks one short clarifying question and listens again.

On-screen text is the fallback where a screen exists. Showing the heard text lets the person spot the wrong word fast.

Names are the weakest point. The assistant spells back unusual names before acting on them.

Repeated failure ends the turn politely. The person is offered the option to type instead.

Privacy and voice data

Voice recordings are personal. They can reveal who a person is, and how that person sounds and feels.

Listening starts after a trigger phrase or a button press. The device does not silently listen at all times.

Recordings are kept only as long as the product needs them. The person can delete them.

Training use of recordings is a separate choice. It stays off unless the person turns it on.

Access to recordings inside the company is logged and limited.

Evaluation and trade-offs

Each stage is measured alone. Recognition is scored by how many words it gets wrong.

The whole chain is measured by task success. A set timer or a sent message counts; a fluent wrong answer does not.

Latency is measured to first sound. A perfect answer after a long silence still feels broken.

Online experiments change one stage at a time. Changing two stages together hides which one helped.

Bigger models hear and answer better, and eat the budget. Asking more clarifying questions cuts wrong actions and tries patience. Keeping less voice data protects privacy and starves training.

The answer budget