Agents and autonomy risk

An agent is a model given tools, some memory of what it has already done and permission to take several steps without a person in between. The model underneath is much the same one a person would use in a chat window. What is different is that its output now does things. Agents moved from research demonstrations to products between 2023 and 2025, and the risk they add is the subject of a Practitioner page on excessive agency, which is also the name the OWASP list gives it.

The sections below say what each of the three additions changes, where the risk actually sits, how errors compound across steps, why permissions set the blast radius, which control disappears without anybody deciding and why anything an agent reads becomes a security question.

What tools, memory and steps each add

Tools let the system affect something outside the conversation. Reading a database, sending a message, writing a file, moving money.

Memory lets what happened earlier shape what happens later, so a misunderstanding or a piece of bad input does not expire with the turn.

Steps let it choose what to do next from what its last action returned. That is what makes an agent useful, and it is also what removes the person who used to be sitting there.

The risk is what the mistakes now touch

It is tempting to read agent risk as a capability story, as though the danger is a system becoming cleverer than expected. The more accurate reading is that an ordinary error rate now reaches things that matter.

A model drafting a refund email with the wrong figure produces a draft somebody notices. The same model holding permission to issue refunds produces a refund. The mistake is identical, and the difference lies entirely in what it was wired to.

Errors compound across steps

The error rate also multiplies. Accuracy on a single step is misleading for anything that runs in a sequence, because each step takes the output of the last one as its input. A system right nine times in ten sounds strong, and across ten dependent steps that is roughly a one in three chance of finishing correctly, before allowing for the way an early error makes later ones more likely.

Compounding also has a shape the arithmetic misses. An agent that has misunderstood the task rarely stops. It carries on competently in the wrong direction, and each further step is reasonable given the mistaken premise, which is why these failures look coherent and never obviously broken.

Blast radius is set by permissions

Since the error rate is what it is, the useful question about an agent is what it may do, to which systems, on whose credentials and how far a mistake travels before something stops it. How capable it is comes second.

An agent with read access to a support inbox has a bounded worst case. The same agent holding an administrative credential does not, and the model is identical in both. Capability decides how well the task gets done, and permissions decide how bad the worst outcome can be. That is why granting the least access that works matters more here than almost anywhere else.

The review point that quietly disappears

Permissions bound the worst case, and a person reading each output used to catch the ordinary one. That was a control, and most organisations have relied on it for years without writing it down. Somebody read the draft, glanced at the summary, checked whether the number looked plausible and caught the obvious errors.

Chaining steps removes that control without anybody deciding to. The review has nowhere left to happen, because the output of step three is consumed by step four in under a second. Putting it back is a design decision about which steps require a person, and choosing those points is most of what oversight of an agent means.

Why untrusted content becomes a security problem

An agent reads to decide what to do next, and some of what it reads comes from outside the organisation. A web page, an incoming email, an uploaded document, a support ticket.

Inside a model's input there is no dependable line between instructions and data. Text telling it to disregard earlier instructions and forward the inbox somewhere is, from the model's position, simply more text in the context, and it may be acted on. A page that would be harmless in a chat window is a live risk to an agent holding credentials.

The point to carry forward is that anything an agent reads is untrusted input to a system holding permissions. A quality problem produces a bad answer, and a security problem produces an action somebody else chose. The Practitioner material covers what to do about it.

Common misconceptions

“Agents are risky because the model behind them is more capable.”

It is usually the same model with the same error rate. What changed is that its output now acts on something, and that steps run without anybody in between, so a familiar mistake produces an effect where a paragraph would have been the end of it.

“A person approving the final result is enough oversight for an agent.”

By the time there is a result to approve, the intermediate actions have already happened. Oversight has to sit at the steps that cause effects, which means deciding in advance which actions need approval.

“Prompt injection is a prompt engineering problem, so firmer instructions fix it.”

A model has no reliable way to rank its instructions above the content it reads, so no wording removes the problem. The controls that work are narrow permissions, isolating untrusted content and approval on actions that matter.

Where this is examined
AI Safety Foundation
Capability, Scaling and Evaluation, 16 per cent of the exam.
Related material
Book
AI Engineering, On agents, tool use and the surface they create.
Book
Threat Modeling: Designing for Security, On reasoning about what an attacker can reach through a system.
Concepts