Add a large language model to a live experience and you feel it immediately: a lag between what the user does and what the system says. The reflex is to chase a faster model or a smaller prompt, which helps at the margin but misses the real lever. In a well-designed real-time system the language model's latency barely matters -- because the model is not on the critical path at all.
Separate what must be instant from what can wait
The first move is to stop treating the pipeline as one uniform stream of work. A live conversation system actually has layers with very different latency requirements. The live transcript must be instant. A running sentiment read can lag slightly. Rich, generated guidance can take a second or two, because it is advice, not a blocking step.
Once you see these as separate lanes, the design almost writes itself. You do not make the language model instant; you make it optional, occasional, and off the path the user is waiting on.
The fast lane and the triggered lane
The live transcript is produced by streaming speech recognition and rendered the moment words are recognised. Nothing in this lane waits on a language model. This is what keeps the experience feeling alive: the user always sees immediate response regardless of what heavier processing is happening elsewhere. The transcript is the heartbeat, and the heartbeat never pauses to think.
The intelligence -- coaching, objection handling, corrective guidance -- runs in a second lane that fires on triggers, not on every utterance. Most moments in a conversation do not need a generated suggestion. A cheap check decides when one is warranted, and only then does the heavy model run.
Because it fires occasionally and asynchronously, its latency is felt rarely and never blocks the conversation. The suggestion simply arrives a beat later and appears alongside the live transcript.
Four techniques that make the triggered lane feel instant
Within the triggered lane, a handful of techniques compress perceived latency to near nothing. Trigger-gating uses a lightweight, often local classifier to decide whether a moment needs the model at all, so the expensive call is rare rather than constant. Model tiering uses a small, fast model for frequent, simple judgements and reserves a larger model for the occasional rich generation -- never the largest model in the hot path.
Streaming and short outputs matter too. Stream tokens so the user sees the answer begin within a few hundred milliseconds, and keep outputs short, since generation time scales with length; perceived latency becomes time-to-first-token, not total time. Finally, caching and precomputation help: cache the stable parts of the prompt so only the new context is processed, and pre-generate responses to common, predictable situations so they return as a near-instant lookup rather than a fresh generation.
Latency becomes an orchestration problem
Put these together and the result is striking. The live transcript never stalls, and generated guidance begins appearing within a fraction of a second of the moment that warranted it. The language model may take a second to finish, but the user never waits on it, because it was never standing between them and the response.
Latency stops being a model problem and becomes an orchestration problem, which is exactly the kind of problem you can engineer away. This two-speed pattern is the backbone of every responsive live-AI system we design. The question is never how to make the model fast enough for the hot path; it is how to keep the model out of the hot path entirely -- and once you ask it that way, real-time AI stops feeling like a compromise.