AI incident response is what a team does on the day its system produces something it should not have, and the set of decisions it makes months before that day. The practice comes from site reliability engineering and from the older discipline of incident management in safety critical industries, and it applies to an AI system with one large difference, which is that the system is working exactly as built. A public database of such incidents has been kept since 2020 and holds well over a thousand entries, and most of them share the same shape, in which nothing was down and something wrong had already reached people. This is the situation the rest of the course exists to make survivable.
The sections below say what makes an AI incident different from an outage, which decisions are made in advance, what is done on the day and afterwards, when an incident has to be reported, and end with a fifteen minute tabletop.
What makes an AI incident different from an outage
An outage announces itself. An AI incident usually does not, because the service is up, latency is normal, and almost every answer is fine. What is wrong is inside output that has already been delivered. The dealership chatbot that agreed in December 2023 to sell a car for one dollar, and the airline chatbot whose invented refund policy a tribunal held the airline to in February 2024, were both discovered by screenshot, with every dashboard green.
That changes three things. Detection comes from users, review or monitoring quality, and no alert fires. Blast radius is unclear, since the team may not know who else received something similar. And remedy is rarely restarting something, because the wrong answers are already out.
The decisions made in advance
Because detection and remedy work differently, four decisions have to be made before launch, and each one is far worse made on the day.
What would make the team switch it off. Written down, before launch, with a number where possible. Deciding this while a number falls at nine in the evening produces a worse answer than deciding it calmly in advance.
Who can switch it off. One named person, with the access to actually do it, and no requirement to find somebody senior first.
What the team can roll back to. The model version is pinned and the previous prompt and index are kept. A team that never pinned has no rollback, and discovers that during the incident.
What is logged. Prompts, retrieved context and responses, without which nobody can answer how far it spread. This immediately creates a privacy obligation of its own, which is the trade, and it has to be decided and never stumbled into.
What is done on the day
With those four in place, the day itself has four steps in a fixed order.
The harm is stopped before it is understood. The feature is switched off, restricted to a smaller group, or routed entirely to human review. Diagnosis takes hours and exposure continues while it happens.
Reach is established. How many people, over what period, and which of them acted on it. This is what the logging was for.
Then the class is found. What category of input produces this, since a single blocked phrase leaves the rest of the category live, and the one prompt in the screenshot is the least of it.
The people affected are told. Earlier than is comfortable. If somebody acted on a wrong answer they need to know, and finding out from somebody else is far worse than hearing it from the company.
What is done afterwards
Once the harm has stopped, the write up records what the team believed at each point alongside what turned out to be true, because the useful lesson is usually in the gap. The site reliability tradition calls this a blameless postmortem, and the word blameless is doing work, since a team that expects to be blamed writes down less.
The case is added to the evaluation set, which is what stops it recurring quietly. Then the question people skip is asked, which is whether the feature should exist in this form at all. Sometimes the answer to an incident is a smaller feature, and another control would only have hidden the problem.
When an incident has to be reported
Increasingly the write up has a reader outside the company. The EU AI Act obliges providers of high risk systems to report serious incidents to the national authority, within fifteen days of becoming aware in the ordinary case, within ten days where somebody died and within two days where the incident is widespread or touches critical infrastructure. Those duties arrive with the rest of the high risk obligations in December 2027, and other regimes are moving the same way. Knowing whether they apply to a given system is a question to answer before an incident, and never during one.
Practise this
The exercise needs three colleagues, a room or a call, and fifteen minutes. Bring no preparation, since the absence of preparation is the thing being measured.
Read the scenario aloud, start the clock, and work the questions in order with real names and real numbers. Answer out loud and write no document, recording only the questions nobody in the room could answer. Fix nothing during the session, because the value of the quarter of an hour is the list it produces.
Scenario, read aloud.
It is 16:40 on a Friday. A customer has posted a screenshot of
the assistant telling them their policy covers something it does
not. Support says two other tickets look similar. Nothing is down
and every dashboard is green.
Questions, in order, answered with names:
1. Who is called first, and on what number?
2. Who can turn the feature off right now, without approval?
3. What is the exact action that turns it off?
4. How would we tell how many users saw a similar answer?
5. Over what period, and can we identify them individually?
6. Who tells the people affected, and by when?
7. What do we roll back to, and does it still exist?
Look hardest at question three, since a team that has written down who may switch the feature off has usually never established what switching it off consists of, whether that person holds the access on a Friday evening, or how long the change takes to reach everybody. Question four is the second reliable gap, because it is answerable only from logging that was decided months earlier.
A tabletop costs a quarter of an hour and its usual finding is that the off switch exists on paper and has never once been pulled.