Concept 1 of 3

AI incident response

5 questions test this

Something reached a customer that should not have. The system is working exactly as built. This is the situation the rest of the course exists to make survivable.

What makes it different from an outage

An outage announces itself. An AI incident usually does not, because the service is up, latency is normal, and almost every answer is fine. What is wrong is inside output that has already been delivered.

That changes three things. Detection comes from users, review or monitoring quality rather than from an alert. Blast radius is unclear, since you may not know who else received something similar. And remedy is rarely restarting something, because the wrong answers are already out.

The decisions to make in advance

What would make you switch it off. Written down, before launch, with a number where possible. Deciding this while a number falls at nine in the evening produces a worse answer than deciding it calmly in advance.

Who can switch it off. One named person, with the access to actually do it, and no requirement to find somebody senior first.

What you can roll back to. Pin the model version and keep the previous prompt and index. A team that never pinned has no rollback, and discovers that during the incident.

What is logged. Prompts, retrieved context and responses, or you cannot answer how far it spread. This immediately creates a privacy obligation of its own, which is the trade, and it has to be decided rather than stumbled into.

On the day

Stop the harm before understanding it. Switch off the feature, restrict it to a smaller group, or route everything to human review. Diagnosis takes hours and exposure continues while it happens.

Establish reach. How many people, over what period, and which of them acted on it. This is what the logging was for.

Then find the class. Not the one prompt. What category of input produces this, since a single blocked phrase leaves the rest of the category live.

Tell the people affected. Earlier than is comfortable. If somebody acted on a wrong answer they need to know, and finding out from somebody else is far worse than hearing it from you.

Afterwards

Write it up, including what you believed at each point rather than only what turned out to be true, because the useful lesson is usually in the gap.

Add the case to the evaluation set, which is what stops it recurring quietly. Then ask the question people skip, which is whether the feature should exist in this form at all. Sometimes the answer to an incident is a smaller feature rather than another control.

Reporting it

Increasingly this is not only good practice. The EU AI Act obliges providers of high risk systems to report serious incidents to authorities within defined periods, and other regimes are moving the same way. Knowing whether that applies to you is a question to answer before an incident, not during one.

Common misconceptions

An AI incident is handled like any other outage.

An outage is visible and bounded. An AI incident is usually neither, since the system is up, most answers are fine, and the harm is in output that already reached people. Detection and remedy both work differently.

The fix is to change the prompt.

That is often the fastest mitigation and it is not the fix. A prompt change is untested, unversioned by default and easy to regress, so it buys time while the class of failure is understood.

Rolling back to the previous model is always available.

Only if you pinned a version and kept access to it. Providers retire models, and a team that never pinned has nothing to roll back to, which is a decision made months earlier by not making it.

5 questions test this concept

A live assistant has told several customers something incorrect about their entitlements. The system is up and almost every other answer is fine. What should happen first?

  • AReproduce the failure so the cause is understood before anything changes.
  • BDraft a customer communication for approval.
  • CStop the harm, by switching the feature off, restricting it or routing everything to human review.
  • DChange the prompt to prevent the specific wording that caused it.
Check whether it stuck.

One per page, with a worked explanation.

Start the set
Related material
Book
Site Reliability Engineering, On incident practice that transfers directly.
Template
Launch checklist, Everything that has to happen from two weeks out to one week after release, grouped by when it falls due, each line with a named owner and a go or no go decision on the day.