Key Concepts
- 1A/B testing agents is harder than testing static UI: outputs are probabilistic and multi-turn, so the same variant produces different conversations each time
- 2Non-determinism means you need larger samples and careful metric definitions to detect a real effect through the noise of varied responses
- 3The unit of analysis shifts: a conversation (multi-turn) is the experimental unit, not a page view, and metrics like resolution and satisfaction span turns
- 4Guardrail metrics are essential in agent experiments: a variant that lifts conversion but raises hallucination, cost, or handoff rate may be a bad trade
- 5Prompt and model changes can have broad, non-local effects — improving one flow can regress another — so holistic evaluation beats single-metric optimization
- 6Offline evaluation (replaying cases against a variant) complements online A/B tests, letting you catch regressions before exposing real users
- 7Novelty and drift confound results: agent behavior and user adaptation change over time, so experiment windows and monitoring must account for it
Why It Matters for PMs
Valid experimentation: Agent non-determinism breaks naive A/B assumptions. PMs must adapt sample sizes, units, and metrics to draw trustworthy conclusions.
Guardrails prevent bad wins: A variant can lift the headline metric while quietly worsening cost, hallucination, or handoff. Multi-metric evaluation is essential.
Offline plus online: Replay-based offline evaluation catches regressions cheaply before live exposure, a practice unique to and important for agent systems.
Test Your Understanding
Question 1 (Multiple Choice)
Why is A/B testing agents harder than testing static UI?
Question 2 (Multiple Choice)
What is the appropriate experimental unit for an agent test?
Question 3 (Multiple Choice)
Why include guardrail metrics in agent experiments?
Question 4 (Short Answer)
How would you design an A/B test for an agent change given output non-determinism?
Think about your answer before revealing the explanation.
Question 5 (Short Answer)
Why is offline replay evaluation a useful complement to online A/B testing for agents?
Think about your answer before revealing the explanation.