Concept 1 of 4

When not to ship an AI feature

5 questions test this

Every control in this course is downstream of one decision, and it is the cheapest place to prevent harm. Most of the safety work on a product is choosing correctly here.

The question that does the work

Not how accurate the system is. What a wrong answer costs, who bears it, and whether they can tell.

Those three together, because they behave differently. A wrong answer a user immediately recognises as wrong costs almost nothing. The same wrong answer, plausible enough to act on, in a domain where the user cannot judge, is a different product.

Four cases where the answer is no

When somebody cannot tell the output is wrong and cannot afford to be wrong. Medical, legal, financial and safety advice to people without the expertise to challenge it. The combination is what matters, since either alone is manageable.

When a reason is owed and cannot be given. Credit, employment, housing, insurance and anything a regulator treats similarly. If the system cannot support an explanation of an individual decision, it is unusable there regardless of accuracy, and a generated explanation is not one.

When the same job is done better by a rule. If you can write the rule down in a sentence, write it. A rule is cheaper, faster, explainable, testable and correctable, and reaching for a model there is a downgrade with better marketing.

When you would ship it anyway whatever the evaluation showed. If the decision is already made, the evaluation is theatre, and it is more honest to say so than to run it.

The cases that look like no and are not

High volume, low stakes. Ranking, suggesting, drafting, summarising for somebody who will read the source anyway. Errors are cheap and visible, which is the good quadrant.

Assisting an expert. A radiologist, an analyst or a support agent who can judge the output and is expected to. The system raises throughput and the person keeps the decision, which is a different arrangement from the person rubber stamping it.

Anything reversible and cheap to undo. A draft, a suggestion, a filter with an appeal. Reversibility does more for safety than accuracy does.

Narrowing rather than refusing

The answer is rarely all or nothing. A feature unsafe for everyone is often safe for a segment, or safe with a person in the path, or safe once it cites its sources so the reader can check.

The move is to name the specific thing that makes it unsafe, then ask what would have to be true for it not to be. Sometimes the answer is a smaller feature. Sometimes it is that the data does not exist. Either is a better outcome than launching and finding out.

Writing it down

Whatever you decide, record what you decided and why, before you build. Not for governance, though it serves that. Because in six months somebody will ask why this was allowed, and a decision reconstructed from memory is worth very little next to one written when the reasoning was fresh.

Common misconceptions

A good enough model makes any use case safe.

Some use cases are unsafe at any accuracy, because the harm comes from the wrong answers rather than the average. A system that is right ninety nine times and refuses somebody medication on the hundredth has not been fixed by a better model.

Adding a human reviewer resolves the risk.

Only if the reviewer can actually judge the output, has time to, and is expected to disagree. A person clicking approve on four hundred suggestions a day is not oversight, they are a signature on somebody else's decision.

If a competitor has shipped it, the risk is acceptable.

You cannot see their incident rate, their review load or their legal advice, and their regulator may not be yours. A competitor's launch is evidence about their appetite, not about your exposure.

5 questions test this concept

A team proposes an assistant that tells customers whether their insurance policy covers a given medical procedure. Customers cannot readily judge whether an answer is right. Which consideration should decide whether this is built as described?

  • AThat the reader cannot tell the output is wrong and cannot afford to be wrong, which is the combination that makes it unsafe.
  • BWhether the model achieves above 95 per cent accuracy on the policy documents.
  • CWhether a competitor already offers something similar.
  • DWhether the answers can be generated within an acceptable latency.
Check whether it stuck.

One per page, with a worked explanation.

Start the set
Related material
Book
Prediction Machines, On where cheap prediction changes a decision and where it does not.
Book
Weapons of Math Destruction, On decisions that should not be automated.
Template
Experiment brief, The belief being tested, a hypothesis with a threshold and a date, how long the test runs and on how much traffic, and a table saying in advance what you will do with each possible result.
Template
One page product strategy, The customer you are serving, the problem worth solving, the approach you are betting on, what you are deliberately not doing, and the conditions that have to hold for any of it to work.