Deciding whether to ship an AI feature at all is the decision that prevents more harm than any control does, and it is the cheapest place to prevent it. Every control on the Practitioner course is downstream of this one. Most of the safety work on a product is choosing correctly here, before a model has been chosen and before anything has been built.
The sections below give the question that does the work, the four cases where the answer is no, the cases that alarm people and are usually fine, how a feature is narrowed where it cannot be shipped whole and why the decision is written down.
The question that does the work
How accurate the system is comes second. The question that decides is what a wrong answer costs, who bears it and whether they can tell.
Those three are taken together, because they behave differently. A wrong answer a user immediately recognises as wrong costs almost nothing. The same wrong answer, plausible enough to act on, in a domain where the user cannot judge, is a different product.
Four cases where the answer is no
Four combinations of those answers settle the matter on their own.
When somebody cannot tell the output is wrong and cannot afford to be wrong. Medical, legal, financial and safety advice to people without the expertise to challenge it. The combination is what matters, since either alone is manageable.
When a reason is owed and cannot be given. Credit, employment, housing, insurance and anything a regulator treats similarly. If the system cannot support an explanation of a single decision, it is unusable there regardless of accuracy, and a generated explanation is no explanation.
When the same job is done better by a rule. Where the rule can be written down in a sentence, the rule is written. A rule is cheaper, faster, explainable, testable and correctable, and reaching for a model there is a downgrade with better marketing.
When it would ship anyway whatever the evaluation showed. If the decision is already made, the evaluation is theatre, and it is more honest to say so than to run it.
The cases that alarm people and are usually fine
Three kinds of feature alarm people and are usually fine.
High volume, low stakes. Ranking, suggesting, drafting, summarising for somebody who will read the source anyway. Errors are cheap and visible, which is the good quadrant.
Assisting an expert. A radiologist, an analyst or a support agent who can judge the output and is expected to. The system raises throughput and the person keeps the decision, which is a different arrangement from the person rubber stamping it.
Anything reversible and cheap to undo. A draft, a suggestion, a filter with an appeal. Reversibility does more for safety than accuracy does.
Narrowing the feature
Between the two lists, the answer is rarely all or nothing. A feature unsafe for everyone is often safe for a segment, or safe with a person in the path, or safe once it cites its sources so the reader can check.
The move is to name the specific thing that makes it unsafe, then ask what would have to be true for it to be safe. Sometimes the answer is a smaller feature. Sometimes it is that the data does not exist. Either is a better outcome than launching and finding out.
Writing it down
Whatever is decided, the decision and the reasoning are recorded before anything is built. Governance is served by that, and it is a side effect. The reason is that in six months somebody will ask why this was allowed, and a decision reconstructed from memory is worth very little next to one written when the reasoning was fresh.
Practise this
The exercise needs one AI feature under serious consideration, stated in a single sentence, and fifteen minutes of honesty.
Answer the six questions below in writing, in full sentences, and give question six real numbers from the team's own traffic estimate, with no rounding. Do this before any model is chosen, because the point of the test is that none of it depends on which model is picked.
Feature in one sentence ..........................
Who receives the output ..........................
1. What does a wrong answer cost the person who receives it?
2. Can that person tell it is wrong, without expertise they
may not have?
3. Is a wrong answer reversible, and by whom, and how fast?
4. Is a reason owed for this decision, and could we give a
real one, written by a person?
5. Could a written rule do the same job?
6. At ____ answers a day and a ____ per cent error rate, that
is ____ wrong answers a day. Write the sentence that
describes what those people experience.
Stop rule
1 high and 2 no narrow the feature or stop
4 yes and we cannot stop
5 yes write the rule instead
The sixth answer is the one that changes minds. An error rate that sounds tolerable as a percentage reads very differently written out as a count of real people receiving a wrong answer every day, and that count is the number a regulator, a journalist or the team's own support queue will eventually use. Watch also for any question that gets hedged or skipped, since the one nobody wants to write down is usually the one carrying the risk.
Written out, some features fail this before anybody has chosen a model, which is the cheapest place a feature can fail.