What is agent explainability?
The ability to give a defensible account of why a decision was justified, to somebody who needs to act on it. That is a different thing from being able to show what the agent did, which is why agent estates with excellent tracing frequently cannot explain anything.
Three capabilities get conflated, and separating them is the first useful move because they are built differently and satisfy different obligations.
| Capability | The question it answers | Who it is for | What it cannot do |
|---|---|---|---|
| Observability | What steps occurred, in what order, with what arguments | Engineers debugging | Say whether any step was justified |
| Audit trail | What happened, completely, and on whose authority | A third party establishing accountability | Convey the basis in terms a layperson can use |
| Explainability | On what basis was this the right outcome, and what would change it | A reviewer, a regulator, an affected person | Substitute for either of the above |
The relationship is one of dependency rather than overlap. You cannot explain a decision you did not record, so observability is the substrate and the audit trail is what makes the record defensible. Explainability is the layer that turns either into something a person outside the engineering team can use, and it does not come free with the other two.
Agents made this harder by appearing to make it easier. With a classical model you got a score and no visibility, so the gap was obvious and the field built attribution methods to close it. An agent hands you a full narrative of steps plus its own stated reasoning, which reads like an explanation and satisfies nobody who examines it. The richness of the output is what stops teams noticing they still have nothing.
Why is an agent's stated reasoning not an explanation?
Because it is produced by the same process that produced the answer, and research shows it frequently is not what drove the answer. There is a direct test: remove or alter the stated reasoning, and if the conclusion does not change, the reasoning was not load-bearing.
This is the finding that should change how you treat chain-of-thought output, and it is well established rather than speculative.
Work by Lanham and colleagues in 2023 demonstrated that chain-of-thought explanations can be systematically unfaithful: truncating or perturbing the stated reasoning often leaves the final answer unchanged, which indicates the explanation was not doing the work. A related failure documented in clinical settings, sometimes called chain-of-thought mismatch, has a model's stated rationale supporting one conclusion while the model selects a different one.
The mechanism is not mysterious. An agent asked to explain itself generates text that plausibly accompanies its answer, using the same next-token machinery that produced the answer. Nothing in the architecture connects the narrative to the computation. A fluent, coherent, internally consistent rationale is exactly what you would expect from a system optimised to produce fluent, coherent, internally consistent text, whether or not it reflects the actual basis.
Who needs an explanation, and what does each need?
Three audiences with genuinely different requirements, and most implementations build one artifact and offer it to all three. An engineer needs the mechanism, a reviewer needs the decisive factors, and an affected person needs a reason and a route to challenge it.
Giving the wrong audience the wrong artifact fails in a specific way each time, which is worth knowing before you build.
- 01
The engineer, debuggingNeeds the full span tree: every model call, tool call, argument and retrieved document. Fails when: given a summary, which removes exactly the detail required to locate the fault. This audience is well served in most estates.
- 02
The reviewer, deciding whether to overrideNeeds the small number of factors that actually determined the outcome, and what would have to differ to change it. Fails when: handed a raw trace, because reading forty spans under time pressure is not a review, and the reviewer approves rather than reads.
- 03
The affected person, contesting itNeeds a plain statement of the decision, the reason, the consequence, and how to challenge it. Fails when: given anything technical at all. A span tree is not an explanation to somebody who has been declined a loan, and offering one reads as evasion.
The reviewer case is the one most worth designing for, because it is where explainability stops being a compliance exercise and starts affecting outcomes. A reviewer who cannot see why in the time available will rubber-stamp, and a rubber-stamping reviewer is an oversight control that exists on paper only. That connects directly to whether your approval gates do anything.
What does the law actually require?
A reason, not a trace. The recurring standard across regimes is meaningful information about the logic of a decision, expressed so the affected person can understand it and contest it. Producing a complete technical record, however thorough, does not on its own satisfy that.
Two obligations are worth distinguishing because their timing differs and one is already live.
- Data protection law already applies. The GDPR is long interpreted as requiring meaningful information about the logic involved in automated decisions, and the Court of Justice of the European Union confirmed a right to explanation under Article 15(1)(h) in Case C-203/22. That is settled and current, independent of any AI-specific timetable.
- The EU AI Act adds a specific right. Article 86 establishes a right to explanation of individual decision-making for high-risk systems, alongside Article 13's requirement that providers supply instructions enabling users to interpret outputs. Article 86's individual right tracks the deferred high-risk timeline rather than the earlier obligations, and that timing has moved more than once, so verify the current date rather than relying on any secondary summary including this one.
What an explanation has to contain is more consistent than the timing. Across the guidance the same four elements recur: the origin of the decision, the rationale for it, its consequence, and the route to redress. Available, understandable and concrete, interrogable by a lay person as well as a regulator. Not a white paper about how the system works in general.
The practical implication for an agentic system is uncomfortable. Your trace can establish origin and consequence well. Rationale is the element you probably cannot produce faithfully, and redress is a process question your engineering team does not own. Whoever sets the obligation should be setting it explicitly, which is covered at AI governance.
Faithful, or merely plausible?
The distinction that decides whether an explanation is worth anything at all. A faithful explanation reflects the actual basis for the decision. A plausible one merely convinces. Optimising for user satisfaction optimises for the second, which raises trust without raising correctness.
This is the same trap as citations on a hallucinated answer, and it is more dangerous here because the artifact is being offered as evidence.
- Plausibility is easy and measurable. Ask users whether an explanation was clear and helpful, and you can improve that score indefinitely without the explanation becoming any more accurate about your system.
- Faithfulness is hard and rarely measured. It requires checking whether the stated basis actually determined the outcome, which means perturbation tests, counterfactual checks, or a grading step, none of which appear in a satisfaction survey.
- Post-hoc attribution methods are not a clean escape. Established techniques such as local surrogate models and feature attribution are operationally available and do not guarantee faithfulness, stability, or comprehensibility, and they are themselves susceptible to adversarial manipulation. Useful tools rather than proofs.
- Current regulation does not yet distinguish the two. Neither the main frameworks nor most guidance separates a faithful explanation from a merely plausible one, which is a gap several researchers expect to close.
The exposure worth understanding before it arrives. As faithfulness research matures, a documented explanation that can later be shown to misrepresent the system's actual basis may fail a meaningful-explanation standard regardless of how fluent it was. Which means an organization that has industrialised the production of persuasive, unvalidated explanations has been manufacturing liability rather than compliance. Grading a sample of your explanations for faithfulness now is considerably cheaper than discovering the gap in a proceeding.
What can you actually produce today?
More than nothing and less than people assume. The workable approach is to build explanations from things you can verify rather than from the agent's narrative, and to grade whatever narrative you do use before anybody outside the team sees it.
Five practices, ordered so the verifiable ones come first.
-
Build the explanation from the trace, not from the narrative
Which documents were retrieved, which policy version applied, which tool returned what, which threshold was crossed. These are facts you recorded rather than text the model generated, and they carry the origin and consequence elements the law asks for. Assemble them into a statement rather than asking the agent to describe itself.
-
Use counterfactuals, which are both faithful and comprehensible
What would have needed to differ for a different outcome, established by actually varying the input and observing the result rather than by asking. This is the most underused technique available: it is verifiable, it is what an affected person actually wants to know, and it doubles as a fairness probe.
-
If you use the agent's rationale, grade it
Prompt for an explanation, then check whether it logically supports the answer and whether the answer survives the explanation being altered. Without that grading step, self-generated explanations are unreliable, and the grading is itself an evaluation problem with its own caveats. See agent evaluation.
-
Write the three artifacts separately
An engineer view, a reviewer view carrying the decisive factors, and a person-facing statement with reason, consequence and redress. They are different documents from the same underlying record, and attempting one artifact for three audiences produces something that works for none.
-
Prefer a simpler component where the explanation matters most
The oldest answer in the field and still the strongest. If a step must be explained to a regulator, a deterministic rule or an interpretable model at that step is explainable by construction and needs no faithfulness argument. Reserve the agent's discretion for the steps where an explanation is less load-bearing.
One expectation to set with stakeholders. Full mechanistic explanation of why a model produced a particular output is an open research problem, not a procurement question, and no platform resolves it. What is achievable is a defensible account built from recorded facts and verified counterfactuals, delivered to the right audience in usable form. Promising more than that internally is how explainability commitments become undeliverable.