There is a quietly radical option in speech recognition that many teams overlook: run the model where the audio already is. Instead of streaming every conversation to a data centre, a small, efficient recogniser runs on the user's own device or a local box at their site. The audio never leaves, the server cost for transcription approaches zero, and for a global, multi-location product this turns out to solve cost, privacy and latency at the same time.
Small models are good enough -- especially with help
The old objection was accuracy: small models that run locally were not good enough. Two things have changed. Modern compact models are dramatically better than their predecessors and run in real time on an ordinary processor, with no dedicated accelerator required.
And in a system where a language model sits downstream to interpret and correct, the local recogniser does not have to be perfect -- the understanding layer absorbs its imperfections. A small local model plus downstream intelligence delivers quality that a small local model alone never could.
The economics are hard to beat
Transcription is the continuous, always-on part of a voice system, and continuous work is where per-minute cloud costs accumulate. Move that work onto hardware the user already owns and the marginal cost of another minute of audio is essentially zero. You are no longer renting compute to process silence and small talk.
The only costs that remain are the occasional, high-value ones -- the language-model calls that fire on the moments that matter -- which is exactly where you want your spend to go. The cheapest minute of audio to transcribe is the one you never had to upload.
Privacy becomes the default, not a feature
When recognition happens on-site, sensitive conversation audio simply never crosses the network. That is a fundamentally stronger privacy posture than any policy or encryption promise about data in a third-party cloud, and for regulated industries it can be the difference between a viable product and a non-starter.
Privacy stops being a feature you bolt on and becomes a property of the architecture. The strongest guarantee is not a better promise about data in transit; it is the data that was never sent.
Hub-and-spoke, and when the edge is the right call
Edge deployment does not mean abandoning the benefits of a central system. The effective pattern is hub-and-spoke: each location runs its own real-time recognition -- fast, local, private -- while a central hub coordinates models, aggregates the insights that are safe to share, and pushes improvements back out to every site. Inference lives at the edge; learning and analytics live at the centre. Each site gets low latency and data locality, the organisation gets a shared and improving system, and because every location contributes to the centre, the platform gets better for everyone as it grows.
On-device and edge recognition is not the answer to every product, but it is the strongest fit for a specific and common profile: real-time, cost-sensitive, privacy-conscious, and deployed across many locations. We reach for edge-first designs whenever a product's shape allows it, because they resolve the usual tensions -- cost versus privacy versus latency -- instead of trading them off. The cheapest, most private, most responsive transcription is the one that happens where the conversation already is.