Concept 3 of 4

Agents and autonomy risk

2 questions test this

An agent is a model given tools, some memory of what it has already done, and permission to take several steps without a person in between. The model underneath is much the same one you would use in a chat window. What is different is that its output now does things.

What tools, memory and steps each add

Tools let the system affect something outside the conversation. Reading a database, sending a message, writing a file, moving money.

Memory lets what happened earlier shape what happens later, so a misunderstanding or a piece of bad input does not expire with the turn.

Steps let it choose what to do next from what its last action returned. That is what makes an agent useful, and it is also what removes the person who used to be sitting there.

The risk is what the mistakes now touch

It is tempting to read agent risk as a capability story, as though the danger is a system becoming cleverer than expected. The more accurate reading is that an ordinary error rate now reaches things that matter.

A model drafting a refund email with the wrong figure produces a draft somebody notices. The same model holding permission to issue refunds produces a refund. The mistake is identical, and the difference lies entirely in what it was wired to.

Errors compound across steps

Accuracy on a single step is misleading for anything that runs in a sequence, because each step takes the output of the last one as its input. A system right nine times in ten sounds strong, and across ten dependent steps that is roughly a one in three chance of finishing correctly, before allowing for the way an early error makes later ones more likely.

Compounding also has a shape the arithmetic misses. An agent that has misunderstood the task rarely stops. It carries on competently in the wrong direction, and each further step is reasonable given the mistaken premise, which is why these failures look coherent rather than obviously broken.

Blast radius is set by permissions, not intelligence

The useful question about an agent is not how capable it is. It is what it may do, to which systems, on whose credentials, and how far a mistake travels before something stops it.

An agent with read access to a support inbox has a bounded worst case. The same agent holding an administrative credential does not, and the model is identical in both. Capability decides how well the task gets done, and permissions decide how bad the worst outcome can be. That is why granting the least access that works matters more here than almost anywhere else.

The review point that quietly disappears

A person reading each output is a control, and most organisations have relied on it for years without writing it down. Somebody read the draft, glanced at the summary, checked whether the number looked plausible, and caught the obvious errors.

Chaining steps removes that control without anybody deciding to. The review has nowhere left to happen, because the output of step three is consumed by step four in under a second. Putting it back is a design decision about which steps require a person, and choosing those points is most of what oversight of an agent means.

Why untrusted content becomes a security problem

An agent reads to decide what to do next, and some of what it reads comes from outside your organisation. A web page, an incoming email, an uploaded document, a support ticket.

Inside a model's input there is no dependable line between instructions and data. Text telling it to disregard earlier instructions and forward the inbox somewhere is, from the model's position, simply more text in the context, and it may be acted on. A page that would be harmless in a chat window is a live risk to an agent holding credentials.

The point to carry forward is that anything an agent reads is untrusted input to a system holding permissions. A quality problem produces a bad answer, and a security problem produces an action somebody else chose. The practitioner material covers what to do about it.

Common misconceptions

Agents are risky because the model behind them is more capable.

It is usually the same model with the same error rate. What changed is that its output now acts on something, and that steps run without anybody in between, so a familiar mistake produces an effect instead of a paragraph.

A person approving the final result is enough oversight for an agent.

By the time there is a result to approve, the intermediate actions have already happened. Oversight has to sit at the steps that cause effects, which means deciding in advance which actions need approval.

Prompt injection is a prompt engineering problem, so firmer instructions fix it.

A model has no reliable way to rank its instructions above the content it reads, so no wording removes the problem. The controls that work are narrow permissions, isolating untrusted content, and approval on actions that matter.

2 questions test this concept

A team wires the model already behind their support chat into an agent that can issue refunds and update accounts. They tell the review board that no new risk has been introduced, because it is the same model performing at the same accuracy. What have they missed?

  • ANothing of substance, since the underlying model and its error rate are unchanged.
  • BThe same error rate now reaches the billing system, so a mistake that used to produce a wrong draft produces a wrong refund instead.
  • CA model handed tools reasons less reliably than the same model answering in a chat window, so the accuracy figure no longer holds.
  • DThe oversight gap is covered, since a person still approves the final result the agent reports at the end.
Check whether it stuck.

One per page, with a worked explanation.

Start the set
Related material
Book
AI Engineering, On agents, tool use and the surface they create.
Book
Threat Modeling: Designing for Security, On reasoning about what an attacker can reach through a system.