Every control in this course is downstream of one decision, and it is the cheapest place to prevent harm. Most of the safety work on a product is choosing correctly here.
The question that does the work
Not how accurate the system is. What a wrong answer costs, who bears it, and whether they can tell.
Those three together, because they behave differently. A wrong answer a user immediately recognises as wrong costs almost nothing. The same wrong answer, plausible enough to act on, in a domain where the user cannot judge, is a different product.
Four cases where the answer is no
When somebody cannot tell the output is wrong and cannot afford to be wrong. Medical, legal, financial and safety advice to people without the expertise to challenge it. The combination is what matters, since either alone is manageable.
When a reason is owed and cannot be given. Credit, employment, housing, insurance and anything a regulator treats similarly. If the system cannot support an explanation of an individual decision, it is unusable there regardless of accuracy, and a generated explanation is not one.
When the same job is done better by a rule. If you can write the rule down in a sentence, write it. A rule is cheaper, faster, explainable, testable and correctable, and reaching for a model there is a downgrade with better marketing.
When you would ship it anyway whatever the evaluation showed. If the decision is already made, the evaluation is theatre, and it is more honest to say so than to run it.
The cases that look like no and are not
High volume, low stakes. Ranking, suggesting, drafting, summarising for somebody who will read the source anyway. Errors are cheap and visible, which is the good quadrant.
Assisting an expert. A radiologist, an analyst or a support agent who can judge the output and is expected to. The system raises throughput and the person keeps the decision, which is a different arrangement from the person rubber stamping it.
Anything reversible and cheap to undo. A draft, a suggestion, a filter with an appeal. Reversibility does more for safety than accuracy does.
Narrowing rather than refusing
The answer is rarely all or nothing. A feature unsafe for everyone is often safe for a segment, or safe with a person in the path, or safe once it cites its sources so the reader can check.
The move is to name the specific thing that makes it unsafe, then ask what would have to be true for it not to be. Sometimes the answer is a smaller feature. Sometimes it is that the data does not exist. Either is a better outcome than launching and finding out.
Writing it down
Whatever you decide, record what you decided and why, before you build. Not for governance, though it serves that. Because in six months somebody will ask why this was allowed, and a decision reconstructed from memory is worth very little next to one written when the reasoning was fresh.
Practise this
You need one AI feature you are seriously considering, stated in a single sentence, and fifteen minutes of honesty.
Answer the six questions below in writing, in full sentences rather than notes, and give question six real numbers from your own traffic estimate rather than round ones. Do this before any model is chosen, because the point of the test is that none of it depends on which model you pick.
Feature in one sentence ..........................
Who receives the output ..........................
1. What does a wrong answer cost the person who receives it?
2. Can that person tell it is wrong, without expertise they
may not have?
3. Is a wrong answer reversible, and by whom, and how fast?
4. Is a reason owed for this decision, and could we give a
real one rather than a generated one?
5. Could a written rule do the same job?
6. At ____ answers a day and a ____ per cent error rate, that
is ____ wrong answers a day. Write the sentence that
describes what those people experience.
Stop rule
1 high and 2 no narrow the feature or stop
4 yes and we cannot stop
5 yes write the rule instead
The sixth answer is the one that changes minds. An error rate that sounds tolerable as a percentage reads very differently written out as a count of real people receiving a wrong answer every day, and that count is the number a regulator, a journalist or your own support queue will eventually use. Watch also for any question you catch yourself hedging or skipping, since the one you would rather not write down is usually the one carrying the risk.
Written out, some features fail this before anybody has chosen a model, which is the cheapest place a feature can fail.