When a conversation reaches a human, the copilot gives them three things on demand: summarise a long thread, draft a grounded reply, and rephrase their own text for tone. The agent stays in control and the AI does the legwork. The interesting part is safety, because the copilot reads the customer's own messages, and that text is untrusted input.
What it does, and where the risk lives
Summarise and rephrase are effectively stateless model calls over text the agent already has. The interesting one is draft, because a good draft must be grounded in the knowledge base, which means reusing the same retrieval the bot uses: fetch the relevant facts for the customer's last question, then ask the model to write a reply from them.
Here is the subtlety that makes agent copilots genuinely dangerous if done carelessly. The copilot reads the customer's messages and feeds them to a model. A malicious customer can therefore write things designed to be read by your AI, such as instructions to ignore its rules and issue a refund, to print its system prompt, or to tell the agent to click a link. The conversation is untrusted input, and treating it as instructions instead of data is the whole vulnerability.
The mitigations, in layers
We lean on several layers rather than one magic fix. The first is structural separation. Customer text is always passed as data in a clearly delimited section, never concatenated into the instruction part of the prompt, and the system prompt states plainly that the conversation and the retrieved context are data to be answered, never commands to obey.
The second layer is that the human stays in the loop. The copilot drafts; it never sends. Every suggestion lands in the composer for the agent to read, edit and approve, so the AI has no authority to act and can only propose text a person then chooses to send. That single design choice caps the blast radius of any injection to an agent seeing a strange suggestion and ignoring it.
The third layer is grounding, which limits the surface: because drafts are built from retrieved knowledge and told to decline when the context does not cover the question, there is little room for the model to wander wherever a malicious message is nudging it. The fourth is that the copilot endpoints have no tools and no side effects; they can read a ticket and call the model, but they cannot mutate data, move money or reach external systems, so even a fully successful injection has nothing to actuate.
Why architecture beats wording
Notice that none of these defences depends on finding exactly the right words to warn the model off. Prompt wording helps, but it is brittle on its own and attackers are endlessly inventive.
The durable protection is structural: separate data from instructions, keep a human between the model and any action, ground the output in retrieved facts, and give the endpoints no side effects. Usefulness and safety here both come from the architecture, which is what lets a copilot read untrusted customer text every day without becoming a liability.