For an AI support bot one question dominates all others: does it answer from the business's real knowledge, or does it improvise? A confident, fluent, wrong answer is worse than no answer. Our answer engine is therefore grounded first and generative last, and the language model is only invited once we have facts to hand it.
The four stages
A customer's question passes through four stages. First, curated FAQs. Hand-written answers to the most common questions are checked first, because a hit here is the cheapest, fastest and most predictable path, with no embeddings and no language model. For a great deal of support traffic, such as opening hours or how to reset a password, this is all you need.
Second, semantic vector search. On a miss we search the knowledge base by meaning. Each knowledge chunk is embedded once at ingest time and its vector stored; at query time we embed the question and pull back the nearest chunks along with their similarity scores. Third, documents and PDFs. The same retrieval surface covers uploaded manuals, policies and guides, which are chunked, embedded and made searchable alongside everything else, with a keyword index kept as a hybrid fallback for exact-term queries such as product codes where embeddings under-perform.
Fourth, and only now, the language model enters. It is given a tight brief: answer the question using the supplied context, and if the context does not cover it, say so. The retrieved chunks are that context. The model's job is to phrase, not to know.
Confidence, and knowing when to stop
Every answer carries a confidence signal and its sources. Two rules ride on that signal. If the best answer's confidence falls under a per-tenant threshold, we do not send a shaky reply; we escalate to a human with the transcript attached. And the sources always travel with the answer, so a chat reply can cite the exact document it drew on and an agent reviewing the bot can see precisely what it used.
This is what separates a demo from something a business can put in front of real customers. The bot is never forced to guess, and there is always a door open to a person when grounding runs thin.
The retrieval-only variant that saved us later
One subtle but important refinement concerns the voice channel. Running the full chain, which itself ends in a language-model call, inside another model's tool call meant two language-model round-trips per turn, and the latency showed badly.
So we exposed a retrieval-only entry point that returns facts rather than a composed answer. The real-time voice model then composes the spoken reply itself from those facts. One model, not two. That decision is the subject of a later war story, but it started here, as a clean seam in the retrieval design.