The real-time voice AI held a conversation, answered from the knowledge base and transferred to humans, but it felt slow. On any question that needed the knowledge base the caller would ask, then sit in silence for five to seven seconds before the AI began answering. On a phone call that is an eternity, and callers assumed the line had dropped.
The hunt
Latency in a voice pipeline can hide in many places: audio buffering, voice-activity end-pointing, the network hops to the model, the model's own think time. Rather than guess, we instrumented each hop with timestamps and watched where the seconds actually went.
The audio path was fine. Turn detection was fine. The time was vanishing inside our own knowledge-search tool, which was the last place we had thought to look.
The root cause: a hidden second model call
Here is the trap. For convenience, our knowledge tool called the full answer chain. But the full chain's final stage is itself a language-model call that composes a written answer. So every time the real-time voice model asked its tool a question, retrieval ran quickly, then a whole extra model round-trip composed written prose, the tool returned that prose, and only then did the voice model read it and compose the spoken answer.
That is two sequential model calls per turn, when the caller only ever hears the output of the second one. The first call's entire job was to write prose that the second model immediately rewrote into speech. Pure waste, and several seconds of it.
The fix was to split a retrieval-only path out of the chain that stops after the retrieval stages and returns raw context, with no composition call. The voice model does the only composition, which is where the caller's audio comes from anyway. With the curated-FAQ fast-path in front, most turns do not even touch the vector store, and the dead air collapsed from several seconds to a beat.
The second bug: getting interrupted by the room
With latency fixed, a subtler problem surfaced. The AI would stop mid-sentence because it thought the caller had started talking, when really it was a television in the background, a cough or line hiss.
Two session settings fixed it. Far-field noise reduction filters ambient phone-line noise before it reaches voice-activity detection. And a semantic turn-detection model decides a turn has ended by understanding speech, rather than reacting to raw loudness, so a bump of background sound no longer counts as an interruption.