Key Concepts
- 1Not all agent errors are equal — classifying them by type and severity is what turns a vague 'the agent is flaky' into a prioritized, fixable backlog
- 2Key categories: comprehension errors (misread intent), retrieval errors (wrong candidates), reasoning errors (bad tradeoff logic), tool errors (API failed), and hallucination (invented facts)
- 3Severity depends on consequence: a wrong product suggestion is recoverable; a wrong price, wrong charge, or fabricated policy is a trust/financial failure
- 4Each error class has a different owner and fix: comprehension -> prompt/capture, retrieval -> index/catalog, tool -> integration/reliability, hallucination -> grounding/guardrails
- 5Recoverability matters: errors the user can easily correct in-conversation are lower priority than silent errors the user acts on without noticing
- 6Measuring error rates requires ground truth — human review, user corrections, and outcome data — because the agent won't reliably flag its own mistakes
- 7A shared taxonomy lets the team track error trends over time and target the highest-severity, highest-frequency classes first
Why It Matters for PMs
Prioritization: A shared error taxonomy converts flakiness complaints into a ranked backlog. PMs can then target the highest-severity, highest-frequency failures first.
Right owner, right fix: Different error classes have different root causes and owners. Classification routes each failure to the fix that actually addresses it.
Severity over frequency: Rare but catastrophic errors (wrong charge, fabricated policy) can outweigh common minor ones. Classification surfaces this so priorities aren't just frequency-driven.
Test Your Understanding
Question 1 (Multiple Choice)
Why classify agent errors by type and severity?
Question 2 (Multiple Choice)
Which error class maps to a grounding/guardrail fix?
Question 3 (Multiple Choice)
Why can a rare error sometimes outrank a common one in priority?
Question 4 (Short Answer)
How would you build a taxonomy of agent errors that routes each class to the right fix?
Think about your answer before revealing the explanation.
Question 5 (Short Answer)
Why can't you rely on the agent to flag its own errors, and what would you use instead?
Think about your answer before revealing the explanation.