When a speech model underperforms, the reflex is to fine-tune it. But fine-tuning is the heaviest lever available: it needs labelled data you may not have, GPU time, specialised expertise, and a careful evaluation regime to avoid regressions. Before committing to a training project, it is worth knowing that a whole toolbox of inference-time techniques can lift accuracy substantially with no retraining at all -- several of them more effective, and far cheaper, than a fine-tune.
Start with scale and contextual biasing
Before anything clever, ask whether the model is simply too small. A larger, more diverse base model handles accents, noise and vocabulary that a small one cannot, purely because it has seen more of the world. Swapping up a model size often erases the problem people were about to spend weeks fine-tuning away. It is not glamorous, but it is frequently the highest-return move.
The next lever is contextual biasing. Every recogniser lets you nudge it toward expected words -- names, products, industry terms -- at decode time. Feed in the vocabulary you expect on a given call and the model resolves ambiguous audio toward those terms instead of guessing.
Made dynamic, pulling the relevant vocabulary per call from a CRM or knowledge base, this becomes a retrieval-flavoured technique that fixes precisely the words general models get wrong, with zero training. It improves the model; it does not replace it.
LLM post-correction: the most practical lever
The most practical modern technique is to run the recogniser and then hand the raw transcript to a language model to clean up using context. Current research frames it as a judge-editor over the recogniser's top hypotheses: keep the spans where the guesses agree, rewrite only the uncertain ones, and never invent names or numbers.
This repairs accent and domain errors without touching the acoustic model at all, and it works for every language and accent because the correction happens in the text domain. If a language model already sits downstream, the transcript does not need to be perfect; you let the system, not one model, carry accuracy.
One caveat is worth respecting. Naive zero-shot correction does not always help -- giving the model a few examples of the corrections you want makes it markedly more reliable.
Retrieval-augmented recognition, ensembling, and rescoring
A newer frontier borrows directly from how large language models learn. Retrieval-augmented approaches pull relevant examples from a datastore at inference and give them to an LLM-based recogniser, which adapts in-context with no fine-tuning. Related work shows that simply showing a model a few samples of a speaker or a language variety lets it adapt on the fly, in a strikingly human-like way. For a global product this is powerful: you adapt per speaker at runtime rather than baking one accent into the weights.
Finally, you can combine signals. Run more than one recogniser, or take several hypotheses from one, and let a language model pick or merge the best answer. Rescoring a short list of candidates with a stronger model reliably lifts accuracy, again with no training of the acoustic model.
When fine-tuning is still worth it
None of this makes fine-tuning obsolete. When you have a stable, high-volume domain and the data to support it, adapting the model to your vocabulary -- not to an accent -- is a durable investment, especially if you want a small, cheap model to run at the edge. But it should be a deliberate choice made after the cheaper levers, not the first reflex.
The right order is to scale up, bias, post-correct, retrieve -- and only then, if the numbers justify it, train. Our rule of thumb is to exhaust the inference-time toolbox before you spin up a training run. Most accuracy problems dissolve long before a fine-tune is warranted, and the ones that remain are far easier to target once you know exactly what the cheap levers could not fix.