Key Concepts
- 1When an agent makes a consequential decision — a recommendation, a decline, a price — someone will eventually ask 'why?', and the system must be able to answer
- 2Explainability serves three audiences: the user (trust), support/ops (debugging), and regulators/auditors (accountability) — each needs a different level of detail
- 3An audit trail records the inputs, the reasoning path, the tools called, and the outcome, so any decision can be reconstructed after the fact
- 4LLM decisions are probabilistic and can vary run-to-run, which makes logging the actual inputs and outputs essential — you can't re-derive what happened from the prompt alone
- 5Explanations shown to users should be honest and grounded in the real decision factors, not post-hoc rationalizations that sound good but aren't why the agent acted
- 6Regulated decisions (credit, pricing, eligibility) may carry a legal right to explanation; the architecture must capture enough to satisfy that, not just marketing-friendly summaries
- 7There's a tension: full transparency can leak signals (fraud logic) or overwhelm users — PMs design layered explanations that reveal appropriately by audience
Why It Matters for PMs
Trust and adoption: Users accept agent decisions they understand. Honest, grounded explanations are what convert a black box into a tool people rely on.
Debuggability and accountability: When something goes wrong, audit trails determine whether you can diagnose and defend the decision — critical for both ops and legal.
Regulatory readiness: For consequential decisions, a right to explanation may apply. Building audit trails now avoids a scramble when a regulator or auditor asks.
Test Your Understanding
Question 1 (Multiple Choice)
Why are audit trails especially important for LLM-based decisions?
Question 2 (Multiple Choice)
What's the risk of showing users a 'post-hoc rationalization' explanation?
Question 3 (Multiple Choice)
Why can't explanations always be maximally transparent?
Question 4 (Short Answer)
What would you capture in an audit trail so any agent decision can be reconstructed later?
Think about your answer before revealing the explanation.
Question 5 (Short Answer)
How would you tailor the level of explanation differently for a user vs an auditor?
Think about your answer before revealing the explanation.