If you have compared speech models, you have noticed that some emit words the instant you speak them while others -- often the most accurate, like Whisper -- make you finish talking before anything appears. This is not a tuning setting or a bug; it is a direct consequence of the model's architecture. Understanding the three main families explains which models can power a live experience and which cannot.
The core divide: online vs offline
Speech recognition splits into two modes. Streaming, or online, models process audio as it arrives and emit partial results continuously, which is ideal for anything real-time. Batch, or offline, models take a whole segment of audio and transcribe it in one pass: more accurate, but unable to show a word until the segment ends.
The three dominant architectures each land on one side of this divide by design. Streaming architectures emit words during speech; batch architectures produce output only after the pause. Knowing which family a model belongs to tells you, before any benchmark, whether it can drive a live experience.
CTC and RNN-T: the streaming families
Connectionist Temporal Classification, or CTC, predicts a token for each small frame of audio and then collapses repeats and blanks into text. Because each frame is handled as it arrives, CTC is naturally streaming and very fast. Its weakness is that it treats frames largely independently, with a limited sense of longer context, so on its own it can be less accurate than richer approaches; it remains a workhorse, often paired with a language model to sharpen results.
The RNN-Transducer, or RNN-T, and its modern relatives such as the Conformer-Transducer were designed specifically for low-latency streaming. A transducer couples an acoustic encoder with a prediction network that models what has been said so far, letting it emit words as audio flows in while still accounting for context.
This is the architecture behind most production real-time systems, including fast open models such as streaming Zipformer and NVIDIA Parakeet. If you need words on screen while the speaker is mid-sentence, this is the family you want.
Attention encoder-decoder: accuracy, but batch
Attention-based encoder-decoder models, the architecture behind Whisper and NVIDIA Canary, read an encoded representation of a whole audio segment and then decode text from it, attending across the entire input. This global view is exactly why they are so accurate -- and exactly why they are not natively streaming. The decoder wants to see the segment before committing.
You can approximate streaming by feeding overlapping chunks, but that is a workaround with latency and accuracy costs, not true online recognition. Whisper is not slow because it is heavy; it is batch because attention models look at the whole segment before they answer.
Choosing for a real-time product
The architecture dictates the product experience, so match it to the job. For a live transcript and in-the-moment coaching, use a streaming transducer such as Zipformer or Parakeet, where words appear as the speaker talks. For post-call analytics, summaries and training-data generation, use a batch attention model like Whisper, where a second or two of latency is irrelevant and accuracy is paramount.
If you need both, run a streaming model for the live feed and a batch model offline for the high-accuracy record. Many mature systems do exactly this. The common failure is reaching for the most accurate model on a benchmark -- usually an attention model -- and then fighting its architecture to force it into a live loop.
We treat the streaming decision as a first-class architectural choice, not an implementation detail, because picking the wrong family is the kind of mistake you feel in every second of latency, on every call, forever.