Building a voice AI agent: the speech-to-text, LLM and text-to-speech pipeline

The pipeline sounds simple.

Speech goes in, text comes out, a model decides what to say, and a voice reads the answer back. Most builders sketch it as three boxes and a couple of arrows, then get surprised when the demo works and the live version doesn't. The three components are not the hard part. What breaks agencies is everything that happens at the boundaries between them: latency stacking up across three separate API calls, context getting lost between turns, and a business process underneath that was never designed for a phone conversation in the first place.

The latency budget nobody sets

Each hop in the pipeline adds delay. Transcription has to wait for a pause in speech before it commits to text. The model has to receive that text, reason over it, and generate a reply. The voice engine has to synthesize audio from that reply before anything plays back. Chain three services from three different vendors and the gaps compound in ways that don't show up until the workflow is actually connected end to end.

A slow round trip from spoken question to spoken answer makes callers wonder if the line has dropped. It usually happens when nobody budgets latency as a design constraint. The delay is the sum of three components each built to be correct, not fast, with nobody owning the total.

The fix is treating end-to-end response time as a number to protect from day one, the same way you'd protect a budget or a deadline, rather than something to notice after a client complains. That means testing the full round trip early, not testing each service in isolation and assuming they'll compose cleanly.

Language and dialect are not solved by picking a good model

The pipeline's English performance tells you almost nothing about how it will handle another language, and this catches people who've only ever tested in their own tongue. Speech recognition, the underlying model, and the voice synthesis were all trained overwhelmingly on English data, so accuracy degrades unevenly as you move away from it.

An agent that works well in English can stumble in another language at every layer: recognition misses accents, replies lose nuance, and the synthesized voice sounds off. Any one of those gaps is enough to unsettle a client, let alone all three at once. If a client's market runs in anything other than English, that has to be tested as its own project phase, not assumed to inherit from the English build.

State is the part that quietly breaks first

A voice agent isn't just answering one question. It's tracking where a caller is in a multi-step process. This is where systems most often go wrong in ways that are hard to spot in a quick test. When the agent and the workflow disagree about where the conversation stands, the caller experiences it as the agent not listening. The fix is keeping one record of progress that both sides read, not patching the prompt.

Build the plumbing before the personality

It's tempting to spend early effort on how the agent sounds and what it says, because that's the visible, demoable part. The plumbing underneath, on the other hand, decides whether any of that matters: telephony that reliably connects calls without getting flagged as spam, a CRM that updates in step with the conversation, and a scheduling system that reflects reality a second after the call ends. A single missed sync on any of those and the caller has a bad experience regardless of how natural the voice sounds.

The one thing to hold onto is this: a voice pipeline is judged on its weakest connection, not its strongest component. Build and test the seams before you polish the voice.