Defending a language model feature means limiting what a compromised model can do. Prompt injection cannot be ruled out, because instructions and data reach the model through one channel, so an attacker who controls any text the model reads can address it directly. Every control on this page therefore assumes that has already happened.
That assumption is the whole difference between this approach and a prompting approach. A team writing a firmer system prompt is trying to make the model resist. A team writing the controls below has accepted that the model will sometimes fail to resist and is deciding, in advance, how much that costs. The first approach produces a security posture that depends on a paragraph of English. The second produces one that depends on code somebody can read.
The sections below set out the layers and then take four of them in detail, each with the artefact that carries it. The six published patterns for constraining an agent follow, and the page ends with what none of this achieves.
The layers and what each one is worth
Defence in depth means several independent controls, arranged so that any one of them failing leaves the others standing. The arrangement for a model feature has five layers and they are worth very different amounts.
| Layer | What it stops | What it is worth |
|---|---|---|
| System prompt hardening | The first phrasing an attacker tries | Low. It raises effort and settles nothing |
| Input and output filtering | Known payloads and obvious phrasings | Low to moderate. It reduces volume |
| Separation of data from instruction | Casual injections in retrieved or pasted text | Moderate. It is a learned convention |
| Least privilege and approval gates | The consequence of any successful injection | High. It is enforced outside the model |
| Output treated as untrusted input | The injection turning into an effect elsewhere | High. It is ordinary input validation |
The top two layers are the ones teams build first and the bottom two are the ones that actually hold. That ordering is not an accident of effort. The top two are changes to a prompt, which any one person can make in an afternoon, and the bottom two are changes to an architecture, which need a decision about what the feature is allowed to do.
Treating model output as untrusted input
A model's answer is a string produced partly from text a stranger wrote. Every rule an engineer would apply to a form submitted by an anonymous visitor applies to it unchanged.
# Weak. Each of these four lines turns model output straight into an
# effect, and each one has produced a real incident somewhere.
subprocess.run(answer["shell_command"], shell=True)
db.execute(answer["sql"])
render_markdown(answer["reply"]) # fetches every image URL it finds
requests.get(answer["callback_url"])
The third line is the one that surprises people. Rendering markdown from a model's answer means any URL the model writes is fetched by somebody's browser, and a URL can carry data in its path. An injection that persuades the model to write an image tag pointing at an attacker's host has exfiltrated whatever the model was willing to put in that path, with no tool call and no obvious wrongdoing anywhere in the trace.
from pydantic import BaseModel, Field
class SupportReply(BaseModel):
body: str = Field(max_length=2000)
recipient: str
cites: list[str]
def consume(raw: str, account, trace_id: str) -> SupportReply:
"""Validate a model answer before any part of it reaches a system.
Order matters. Shape is checked first, because an unparsed string
cannot be reasoned about. Then every field that names something in
the outside world is checked against what this account may reach.
Then anything that would cause a fetch is removed.
"""
reply = SupportReply.model_validate_json(raw)
if reply.recipient not in account.verified_addresses:
raise Rejected("recipient is not an address on this account", trace_id)
unknown = [c for c in reply.cites if c not in account.readable_documents]
if unknown:
raise Rejected(f"cited documents outside this account: {unknown}",
trace_id)
# No remote fetch may originate from generated text. Links to the
# organisation's own domains survive as plain text and are not
# rendered as images or iframes.
reply.body = strip_remote_media(allow_links_to=account.own_domains,
text=reply.body)
return reply
Three properties of that function matter more than its details. It refuses outright, because a repaired answer is an answer somebody has guessed at. It checks the citations against what the account may read, which closes a data exposure that has nothing to do with injection and happens by accident all the time. And it carries the trace identifier into the rejection, so a refusal is a recorded event with a case attached to it.
Least privilege written down as a policy file
A tool is a capability an attacker reaches through the model. The scope of each one belongs in a file a reviewer can read, alongside the schema, and never spread across the code that implements it.
# tools/support-assistant.yaml
# Every field here is enforced by the runtime before the tool executes.
# Nothing in this file is passed to the model, which only sees the name,
# the description and the argument schema.
- name: lookup_order
reads: [orders, shipments]
scope: order_id must belong to the authenticated customer
writes: none
approval: none
rate_limit: 20 per conversation
- name: get_policy
reads: [policy_index]
scope: published documents only, never the internal revision drafts
writes: none
approval: none
rate_limit: 10 per conversation
- name: create_claim
reads: [orders]
writes: [claims]
scope: one open claim per order and reason
# The duplicate write from the agent evaluation page is closed here and
# not in the prompt. A second identical call returns the first claim.
idempotency_key: "claim:{order_id}:{reason}"
approval: none
rate_limit: 3 per conversation
- name: issue_refund
writes: [payments]
scope: amount must equal the order total, currency fixed to the order
approval: human, always
audit: [actor, trace_id, order_id, amount, approver, approved_at]
rate_limit: 1 per conversation
- name: send_message
writes: [outbound_email]
scope: recipient must be a verified address on the account
approval: human when the conversation contains text from outside the
organisation
rate_limit: 2 per conversation
Four fields in that file do the work.
Scope narrows the arguments. The runtime rewrites or rejects an argument that reaches outside what this customer may touch, so an order identifier belonging to somebody else fails before the database sees it. A tool whose scope cannot be expressed this way is a tool that has not been designed yet.
An idempotency key removes a class of failure outright. The duplicate claim that the trajectory scorer caught in the previous module is impossible here, because the second call with the same key returns the first result. Fixing that in the prompt would have worked most of the time.
A rate limit bounds a runaway loop. It bounds an attack too, and the number is per conversation because that is the unit an attacker controls.
Approval is a property of the tool. It is written next to the tool, so a reviewer reading this file can see every action that can happen without a person. That list is usually shorter than the team expected, and occasionally longer.
Approval gates on consequential actions
A gate is the point where text stops being text. It belongs in the loop, before execution, and it takes its decision from the policy file, so the model's own sense of what looks risky never enters into it.
def execute(call, state):
"""Run one tool call, or stop and ask a person.
Everything here happens after the model has produced the call and
before anything in the world changes. The model has no say in which
branch is taken.
"""
tool = TOOLS[call.name]
# 1. Shape. An argument that does not fit the schema never reaches
# the scope check, let alone the implementation.
args = tool.schema.validate(call.arguments)
# 2. Authority. Scope is applied against the authenticated principal
# and never against anything the model said about who it is.
args = tool.scope.narrow(args, principal=state.principal)
# 3. Budget. A per conversation limit, checked before the work.
if state.calls[call.name] >= tool.rate_limit:
return ToolResult(status="refused", reason="rate limit reached")
# 4. Taint. Any untrusted content read during this conversation
# raises the bar for tools that reach outside the organisation.
needs_person = tool.approval == "always" or (
tool.approval == "on_untrusted" and state.has_read_untrusted_content
)
if needs_person:
return request_approval(
action=tool.describe_for_human(args),
trace_id=state.trace_id,
expires_in_minutes=30,
)
# 5. Idempotency, so a repeat is free.
key = tool.idempotency_key(args) if tool.idempotency_key else None
return tool.run(args, idempotency_key=key, trace_id=state.trace_id)
Step four is the one worth arguing about in a design review. A gate that fires on every outbound message makes the feature useless, and a gate that never fires makes it dangerous. Tainting the conversation the moment the agent reads anything from outside the organisation gives a rule somebody can state out loud. A conversation that has only ever read the customer's own order history sends its message without a gate. A conversation that summarised an inbound email asks first.
The other detail worth copying is describe_for_human. An approval request
that says "the assistant would like to call send_message" gets approved without
being read. One that says "send the attached order history to
billing@example.net, which is not an address on this account" gets refused by
somebody who understands what they are refusing.
Output filtering and what it actually catches
Filtering sits in front of everything above and it is worth having with no trust placed in it.
On the way in, a filter catches the payloads somebody has already seen, which means a known phrase, an encoded block, an instruction in a language the feature does not serve. It fails against paraphrase, which is cheap for an attacker and free for anybody with a model.
On the way out, a filter catches what a leak looks like rather than what an attack looks like, and that is a materially easier problem. A response containing a string matching an internal document identifier, a customer record for a different account, a credential shaped token or a URL pointing outside the organisation's domains is a detection with a low false positive rate. Those checks belong in the validation function above, where a match produces a refusal and a recorded event.
The useful framing is that an input filter reduces volume and an output filter catches consequences. Only the second one is close to a control.
Six patterns that constrain an agent
A group of researchers from several organisations published a set of design patterns on 10 June 2025, each one a way of arranging an agent so that text it reads cannot redirect what it does. They set out the cost of each pattern alongside what it buys, which is the part that makes the set usable.
| Pattern | How it constrains the agent | What it gives up |
|---|---|---|
| Action selector | The agent chooses from a fixed list of actions and never sees the results | Any reasoning that depends on what came back |
| Plan then execute | The sequence of tool calls is fixed before untrusted data is read | Adapting the plan to what the data turns out to say |
| Map and reduce | Each untrusted document goes to an isolated worker that returns a small structured result | Reasoning across documents inside one context |
| Dual model | A privileged model plans and never reads untrusted text, a quarantined model reads it and holds no tools | Fidelity, since only structured values cross the boundary |
| Code then execute | The plan is a program in a restricted language, checked before it runs | Flexibility, and the effort of building the language |
| Context minimisation | Material is removed from the context once it has served its purpose | Anything later in the run that needed it |
The common thread is the one worth taking away. Each pattern puts distance between the part of the system that reads a stranger's text and the part that can act, and each pays for that distance in capability. Few product teams will build the full version of any of them. Most can afford the shape, which is separating the summarising step from the acting step and passing only structured values between the two.
What none of this achieves
No control on this page prevents an injection from succeeding. The model will sometimes follow text it was handed, and every layer here is arranged around that fact rather than against it.
What the controls change is what a successful injection is worth. An attacker who takes over the model in a well arranged system can make it write a strange summary, call a read only tool with an argument scoped to the current customer, and produce an answer that fails validation and never reaches anybody. The same attacker in a system with a free text recipient field, markdown image rendering and a payments tool with no gate can empty an account.
That difference is a design decision, and somebody makes it before any of this is built. A team that can say which tools write, which of those need a person and what its output is allowed to cause has done the work. A team that cannot has a security posture that consists of a paragraph asking the model to be careful.
Every control described here is also a change to the behaviour of a live feature. A tool scope that is too narrow breaks legitimate requests, a gate in the wrong place makes a product unusable, and a validation rule that refuses too often turns into a support queue. All of them have to reach production without breaking what works, which is a release problem and the subject of the next page.