Transparency and explainability are two words used together that mean different things, and the difference decides whether an accurate model is usable at all. Transparency is about the system, meaning what data it learned from, how it was built, who is accountable and what its known limits are. It is a documentation and disclosure property, and any model can have it if somebody does the work. Explainability is about a decision, meaning why this input produced this output. It is a technical property, and some model families have it while others largely do not.
A model can be transparent and unexplainable, which describes most published foundation models. It can also be explainable and opaque, if a well understood model is deployed with nothing documented. The sections below run along the gradient from interpretable to opaque models, set out the tradeoff and when it is worth taking, describe the post hoc tools and the model card, and end with the design principles for explaining a decision to the person it affects.
The gradient from interpretable to opaque
Linear and logistic regression are directly interpretable, since each coefficient states a contribution. A shallow decision tree can be read as a sequence of rules. Ensembles such as random forests and gradient boosting lose that directness and can be probed. Deep neural networks and foundation models are not interpretable in any faithful sense, and the field's best tools give approximations.
When the tradeoff between performance and explanation is worth taking
More flexible models usually perform better on hard problems and explain themselves worse. That is the tradeoff both vendor exam guides ask about.
The gap is smaller than assumed on tabular data, where boosted trees and regression are frequently close. Where it is real, the question is what the decision touches.
Regulation may simply settle it. Credit, employment and insurance decisions carry a duty to give reasons in many jurisdictions. In the United States a lender refusing credit has been obliged since 1974 to state the principal reasons, and European data protection law gives a person subject to a solely automated decision a right to meaningful information about the logic involved. A model that cannot support a reason is unusable in those settings regardless of accuracy.
Where somebody has to act on the output, an explanation is what makes acting possible. A fraud analyst given a score and no reason cannot investigate efficiently. Where the stakes are low and the volume high, such as ranking recommendations, accuracy can reasonably win.
Post hoc tools
For models that cannot explain themselves, feature attribution methods approximate an explanation by measuring how much each input contributed. SHAP is the best known, and the managed implementations follow it. SageMaker Clarify on AWS, Vertex Explainable AI on Google Cloud and the responsible AI dashboard in Azure Machine Learning all produce attributions of this kind.
They are approximations. They can be unstable, they can mislead when inputs are correlated, and they describe the model and never the world. They are much better than nothing and they are no faithful account.
Model cards
Where a decision cannot be explained, the system can still be documented, and a model card is the transparency artefact. The format was proposed in 2019 and records what the model does, what it was trained on, how it performs overall and by subgroup, its known limitations and the uses it is not suitable for.
Every provider has a managed version, SageMaker Model Cards on AWS, Model Cards in Vertex AI on Google Cloud and responsible AI scorecards in Azure. What makes them worth using is that the record sits next to the model, where a document somebody wrote once and never revisited does not. Open models, open data and clear licensing contribute to the same end, since a model whose training data and terms are published can be assessed by somebody outside the team that built it.
Human centred design
Human centred design for explainable AI is the set of principles about the person receiving the explanation, and both vendor exam guides ask for it by that name.
The explanation is given in terms the audience can act on, which usually means naming the factors that mattered where a score would say nothing. It says how confident the system is and is honest when it is not. It gives a route to a person, because an automated decision with no appeal is a design failure whatever its accuracy. And it sets expectations before the answer, since somebody told a system is assisting them reads its output differently from somebody told it is deciding.
Practise this
The exercise needs one automated decision the system makes about a person, fifteen minutes and a willingness to take one's own sentences apart.
Write the explanation that would actually be sent, addressed to that person and in the words they would read, in about four sentences. Then go through it claim by claim. Underline every statement about why the outcome happened and write beside it where that comes from, whether a logged input value, a rule in code, an attribution score or nothing that could be produced if asked.
Decision ...........................................
The explanation as the person would receive it
..................................................
..................................................
..................................................
Claim in the explanation Evidence I could produce Y / N
.......................... ........................ Y N
.......................... ........................ Y N
.......................... ........................ Y N
Would it survive the person replying to ask which of their
details produced the outcome? Y / N
The claims marked N are the story. They are almost always the sentences about why, since a system can usually evidence the inputs it held and rarely the reason, and a paragraph produced by asking the model to explain itself supplies exactly those sentences with none of the evidence. Check as well whether the explanation names anything the person could change, because one that does not has told them a fact about the machine and nothing about their case.
Fifteen minutes of this shows the gap between an explanation and a story about the answer, which is the distinction the whole subject rests on.