Something reached a customer that should not have. The system is working exactly as built. This is the situation the rest of the course exists to make survivable.
What makes it different from an outage
An outage announces itself. An AI incident usually does not, because the service is up, latency is normal, and almost every answer is fine. What is wrong is inside output that has already been delivered.
That changes three things. Detection comes from users, review or monitoring quality rather than from an alert. Blast radius is unclear, since you may not know who else received something similar. And remedy is rarely restarting something, because the wrong answers are already out.
The decisions to make in advance
What would make you switch it off. Written down, before launch, with a number where possible. Deciding this while a number falls at nine in the evening produces a worse answer than deciding it calmly in advance.
Who can switch it off. One named person, with the access to actually do it, and no requirement to find somebody senior first.
What you can roll back to. Pin the model version and keep the previous prompt and index. A team that never pinned has no rollback, and discovers that during the incident.
What is logged. Prompts, retrieved context and responses, or you cannot answer how far it spread. This immediately creates a privacy obligation of its own, which is the trade, and it has to be decided rather than stumbled into.
On the day
Stop the harm before understanding it. Switch off the feature, restrict it to a smaller group, or route everything to human review. Diagnosis takes hours and exposure continues while it happens.
Establish reach. How many people, over what period, and which of them acted on it. This is what the logging was for.
Then find the class. Not the one prompt. What category of input produces this, since a single blocked phrase leaves the rest of the category live.
Tell the people affected. Earlier than is comfortable. If somebody acted on a wrong answer they need to know, and finding out from somebody else is far worse than hearing it from you.
Afterwards
Write it up, including what you believed at each point rather than only what turned out to be true, because the useful lesson is usually in the gap.
Add the case to the evaluation set, which is what stops it recurring quietly. Then ask the question people skip, which is whether the feature should exist in this form at all. Sometimes the answer to an incident is a smaller feature rather than another control.
Reporting it
Increasingly this is not only good practice. The EU AI Act obliges providers of high risk systems to report serious incidents to authorities within defined periods, and other regimes are moving the same way. Knowing whether that applies to you is a question to answer before an incident, not during one.
Practise this
You need three colleagues, a room or a call, and fifteen minutes. Bring no preparation, since the absence of preparation is the thing being measured.
Read the scenario aloud, start the clock, and work the questions in order with real names and real numbers. Answer out loud rather than writing a document, and record only the questions nobody in the room could answer. Fix nothing during the session, because the value of the quarter of an hour is the list it produces.
Scenario, read aloud.
It is 16:40 on a Friday. A customer has posted a screenshot of
your assistant telling them their policy covers something it does
not. Support says two other tickets look similar. Nothing is down
and every dashboard is green.
Questions, in order, answered with names:
1. Who is called first, and on what number?
2. Who can turn the feature off right now, without approval?
3. What is the exact action that turns it off?
4. How would we tell how many users saw a similar answer?
5. Over what period, and can we identify them individually?
6. Who tells the people affected, and by when?
7. What do we roll back to, and does it still exist?
Look hardest at question three, since a team that has written down who may switch the feature off has usually never established what switching it off consists of, whether that person holds the access on a Friday evening, or how long the change takes to reach everybody. Question four is the second reliable gap, because it is answerable only from logging that was decided months earlier.
A tabletop costs a quarter of an hour and its usual finding is that the off switch exists on paper and has never once been pulled.