Concept 2 of 2

Transparency and explainability

3 questions test this

The two words are used together and mean different things.

Transparency is about the system. What data it learned from, how it was built, who is accountable, what its known limits are. It is a documentation and disclosure property, and any model can have it if somebody does the work.

Explainability is about a decision. Why this input produced this output. It is a technical property, and some model families have it while others largely do not.

A model can be transparent and unexplainable, which describes most published foundation models. It can also be explainable and opaque, if a well understood model is deployed with nothing documented.

The gradient

Linear and logistic regression are directly interpretable, since each coefficient states a contribution. A shallow decision tree can be read as a sequence of rules. Ensembles such as random forests and gradient boosting lose that directness and can be probed. Deep neural networks and foundation models are not interpretable in any faithful sense, and the field's best tools give approximations.

The tradeoff, and when it is worth taking

More flexible models usually perform better on hard problems and explain themselves worse. That is the tradeoff the syllabus asks about.

The gap is smaller than assumed on tabular data, where boosted trees and regression are frequently close. Where it is real, the question is what the decision touches.

Regulation may simply settle it. Credit, employment and insurance decisions carry a duty to give reasons in many jurisdictions, and a model that cannot support one is unusable regardless of accuracy.

Where somebody has to act on the output, an explanation is what makes acting possible. A fraud analyst given a score and no reason cannot investigate efficiently. Where the stakes are low and the volume high, such as ranking recommendations, accuracy can reasonably win.

Post hoc tools

For models that cannot explain themselves, feature attribution methods approximate an explanation by measuring how much each input contributed. SHAP is the best known and it is what SageMaker Clarify implements.

They are approximations. They can be unstable, they can mislead when inputs are correlated, and they describe the model rather than the world. They are much better than nothing and they are not a faithful account.

Model cards

A model card is the transparency artefact. It records what the model does, what it was trained on, how it performs overall and by subgroup, its known limitations, and the uses it is not suitable for.

SageMaker Model Cards is the AWS implementation, and it matters because it puts that record next to the model rather than in a document somebody wrote once. The exam also names open source models, open data and clear licensing as contributors to transparency, since a model whose training data and terms are published can be assessed by somebody outside the team that built it.

Human centred design

The syllabus asks for the principles, and they are about the person receiving the explanation rather than the technique producing it.

Explain in terms the audience can act on, which usually means naming the factors that mattered rather than presenting a score. Say how confident the system is and be honest when it is not. Give a route to a person, because an automated decision with no appeal is a design failure whatever its accuracy. And set expectations before the answer, since somebody told a system is assisting them reads its output differently from somebody told it is deciding.

Common misconceptions

Asking a language model to explain its answer gives you an explanation.

It gives you generated text about the answer, produced the same way the answer was. It can be fluent and unrelated to what actually drove the output, so it is a readable story rather than an audit trail.

Explainable models are less accurate, so accuracy should decide.

The gap is often small and sometimes zero, and where it is real the decision belongs to the situation. Two points of accuracy is a poor trade for being unable to tell somebody why they were refused.

Open weights make a model explainable.

Open weights make it inspectable, which is transparency. Being able to read billions of parameters does not tell you why one input produced one output, and that is a separate problem.

3 questions test this concept

A regulator requires that any customer refused credit be given the reasons. The data science team proposes a deep neural network that is two points more accurate than a logistic regression model. How should this be assessed?

  • AAdopt the interpretable model, because a duty to give reasons makes explainability a requirement rather than a preference.
  • BAdopt the neural network, since accuracy is the primary obligation to customers.
  • CAdopt the neural network and have it generate an explanation of each refusal.
  • DAdopt the neural network and publish a model card, which satisfies the requirement.
Check whether it stuck.

One per page, with a worked explanation.

Start the set
Related material
Book
Weapons of Math Destruction, On opacity as a property that protects the system rather than the subject.
Book
Data Science for Business, On model choice and what each family lets you see.