Concept 3 of 4

Transparency and explainability

1 question test this

The two words are used together and mean different things.

Transparency is about the system. What data it learned from, how it was built, who is accountable, what its known limits are. It is a documentation and disclosure property, and any model can have it if somebody does the work.

Explainability is about a decision. Why this input produced this output. It is a technical property, and some model families have it while others largely do not.

A model can be transparent and unexplainable, which describes most published foundation models. It can also be explainable and opaque, if a well understood model is deployed with nothing documented.

The gradient

Linear and logistic regression are directly interpretable, since each coefficient states a contribution. A shallow decision tree can be read as a sequence of rules. Ensembles such as random forests and gradient boosting lose that directness and can be probed. Deep neural networks and foundation models are not interpretable in any faithful sense, and the field's best tools give approximations.

The tradeoff, and when it is worth taking

More flexible models usually perform better on hard problems and explain themselves worse. That is the tradeoff the syllabus asks about.

The gap is smaller than assumed on tabular data, where boosted trees and regression are frequently close. Where it is real, the question is what the decision touches.

Regulation may simply settle it. Credit, employment and insurance decisions carry a duty to give reasons in many jurisdictions, and a model that cannot support one is unusable regardless of accuracy.

Where somebody has to act on the output, an explanation is what makes acting possible. A fraud analyst given a score and no reason cannot investigate efficiently. Where the stakes are low and the volume high, such as ranking recommendations, accuracy can reasonably win.

Post hoc tools

For models that cannot explain themselves, feature attribution methods approximate an explanation by measuring how much each input contributed. SHAP is the best known, and the managed implementations follow it. SageMaker Clarify on AWS, Vertex Explainable AI on Google Cloud and the responsible AI dashboard in Azure Machine Learning all produce attributions of this kind.

They are approximations. They can be unstable, they can mislead when inputs are correlated, and they describe the model rather than the world. They are much better than nothing and they are not a faithful account.

Model cards

A model card is the transparency artefact. It records what the model does, what it was trained on, how it performs overall and by subgroup, its known limitations, and the uses it is not suitable for.

Every provider has a managed version, SageMaker Model Cards on AWS, Model Cards in Vertex AI on Google Cloud and responsible AI scorecards in Azure. What makes them worth using is that the record sits next to the model rather than in a document somebody wrote once and never revisited. Open models, open data and clear licensing contribute to the same end, since a model whose training data and terms are published can be assessed by somebody outside the team that built it.

Human centred design

The syllabus asks for the principles, and they are about the person receiving the explanation rather than the technique producing it.

Explain in terms the audience can act on, which usually means naming the factors that mattered rather than presenting a score. Say how confident the system is and be honest when it is not. Give a route to a person, because an automated decision with no appeal is a design failure whatever its accuracy. And set expectations before the answer, since somebody told a system is assisting them reads its output differently from somebody told it is deciding.

Practise this

You need one automated decision your system makes about a person, fifteen minutes, and a willingness to take your own sentences apart.

Write the explanation you would actually send, addressed to that person and in the words they would read, in about four sentences. Then go through it claim by claim. Underline every statement about why the outcome happened and write beside it where that comes from, whether a logged input value, a rule in code, an attribution score, or nothing you could produce if asked.

Decision ...........................................

The explanation as the person would receive it

  ..................................................
  ..................................................
  ..................................................

  Claim in the explanation      Evidence I could produce    Y / N

  ..........................    ........................    Y  N
  ..........................    ........................    Y  N
  ..........................    ........................    Y  N

Would it survive the person replying to ask which of their
details produced the outcome?    Y / N

The claims marked N are the story. They are almost always the sentences about why rather than about what, since a system can usually evidence the inputs it held and rarely the reason, and a paragraph produced by asking the model to explain itself supplies exactly those sentences with none of the evidence. Check as well whether the explanation names anything the person could change, because one that does not has told them a fact about the machine rather than about their case.

Fifteen minutes of this shows you the gap between an explanation and a story about the answer, which is the distinction the whole subject rests on.

Common misconceptions

Asking a language model to explain its answer gives you an explanation.

It gives you generated text about the answer, produced the same way the answer was. It can be fluent and unrelated to what actually drove the output, so it is a readable story rather than an audit trail.

Explainable models are less accurate, so accuracy should decide.

The gap is often small and sometimes zero, and where it is real the decision belongs to the situation. Two points of accuracy is a poor trade for being unable to tell somebody why they were refused.

Open weights make a model explainable.

Open weights make it inspectable, which is transparency. Being able to read billions of parameters does not tell you why one input produced one output, and that is a separate problem.

1 question test this concept

A customer whose request an assistant refused asks the team why. The team holds a thorough model card covering training data, evaluation results and known limitations, and sends the customer a link to it. What has the team provided?

  • AAn explanation, since the evaluation results describe how the model behaves on this class of input.
  • BEverything the customer is owed, because the refusal falls inside the limitations the card already documents.
  • CTransparency about the system, which is not an account of why this input produced this output.
  • DNothing of value, since a model card describes a model rather than the product built on it.
Check whether it stuck.

One per page, with a worked explanation.

Start the set
Related material
Book
Weapons of Math Destruction, On opacity as a property that protects the system rather than the subject.
Book
Data Science for Business, On model choice and what each family lets you see.