Incident Response for Agent Outages

Key Concepts

• 1Agent incidents include not just downtime but degradation — the agent is up but giving wrong, harmful, or off-brand answers, which is harder to detect than a hard outage
• 2Detection needs behavioral monitoring, not just uptime checks: spikes in handoffs, corrections, hallucination signals, or cost can indicate a silent quality incident
• 3Dependencies multiply failure modes: LLM provider outages, tool/API failures, and retrieval degradation each need graceful fallbacks, not a dead conversation
• 4Fallback strategy is a design decision: degrade to a simpler model, a scripted flow, or a human handoff — anything but an unhandled error at the point of sale
• 5The kill-switch and feature flags from the ops toolkit are your fastest mitigation; roll back the suspect change before deep diagnosis
• 6Communication matters: during an incident, honest in-product messaging ('we're having trouble, here's a human') preserves trust better than broken confidence
• 7Post-incident review should treat agent quality regressions like outages — root cause, timeline, and prevention — because a confidently-wrong agent can do real damage

Why It Matters for PMs

Test Your Understanding