Something reached a customer that should not have. The system is working exactly as built. This is the situation the rest of the course exists to make survivable.
What makes it different from an outage
An outage announces itself. An AI incident usually does not, because the service is up, latency is normal, and almost every answer is fine. What is wrong is inside output that has already been delivered.
That changes three things. Detection comes from users, review or monitoring quality rather than from an alert. Blast radius is unclear, since you may not know who else received something similar. And remedy is rarely restarting something, because the wrong answers are already out.
The decisions to make in advance
What would make you switch it off. Written down, before launch, with a number where possible. Deciding this while a number falls at nine in the evening produces a worse answer than deciding it calmly in advance.
Who can switch it off. One named person, with the access to actually do it, and no requirement to find somebody senior first.
What you can roll back to. Pin the model version and keep the previous prompt and index. A team that never pinned has no rollback, and discovers that during the incident.
What is logged. Prompts, retrieved context and responses, or you cannot answer how far it spread. This immediately creates a privacy obligation of its own, which is the trade, and it has to be decided rather than stumbled into.
On the day
Stop the harm before understanding it. Switch off the feature, restrict it to a smaller group, or route everything to human review. Diagnosis takes hours and exposure continues while it happens.
Establish reach. How many people, over what period, and which of them acted on it. This is what the logging was for.
Then find the class. Not the one prompt. What category of input produces this, since a single blocked phrase leaves the rest of the category live.
Tell the people affected. Earlier than is comfortable. If somebody acted on a wrong answer they need to know, and finding out from somebody else is far worse than hearing it from you.
Afterwards
Write it up, including what you believed at each point rather than only what turned out to be true, because the useful lesson is usually in the gap.
Add the case to the evaluation set, which is what stops it recurring quietly. Then ask the question people skip, which is whether the feature should exist in this form at all. Sometimes the answer to an incident is a smaller feature rather than another control.
Reporting it
Increasingly this is not only good practice. The EU AI Act obliges providers of high risk systems to report serious incidents to authorities within defined periods, and other regimes are moving the same way. Knowing whether that applies to you is a question to answer before an incident, not during one.