Key Concepts
- 1Latency is a product feature in conversational commerce: every second of agent response time measurably increases abandonment, just like page-load time in classic ecommerce
- 2The response is a chain — retrieval, LLM reasoning, tool calls, re-ranking — and total latency is the sum, so PMs must budget each stage, not just the model
- 3Perceived latency differs from actual: streaming the response and showing progress ('finding options...') makes a 3-second answer feel faster and holds attention
- 4There's a quality/latency tradeoff: bigger models and deeper retrieval improve answers but cost time; the PM sets the budget that maximizes conversion, not pure quality
- 5Tail latency matters more than average: the p95/p99 slow responses are where users bail, so budgets and SLAs should target tails, not means
- 6Tool-call latency (inventory, pricing, tax services) is often the hidden bottleneck; parallelizing independent calls and caching stable data reclaims seconds
- 7Latency budgets should be per-interaction-type: a quick clarifying answer needs sub-second feel, while a complex multi-constraint search can justify a bit more time
Why It Matters for PMs
Direct conversion impact: Response latency drives abandonment as reliably as page load. Treating it as a first-class metric protects conversion at every turn.
Quality/speed tradeoff is a PM decision: Bigger models and deeper search cost time. PMs must set the budget that maximizes conversion, not chase maximum answer quality.
Tails, not averages: Users bail on the slow p99 responses. Focusing SLAs on tail latency prevents a good average from hiding a conversion-killing tail.
Test Your Understanding
Question 1 (Multiple Choice)
Why should PMs budget latency per stage, not just for the model?
Question 2 (Multiple Choice)
Why does tail latency (p95/p99) matter more than average?
Question 3 (Multiple Choice)
How can perceived latency be improved without reducing actual compute time?
Question 4 (Short Answer)
How would you set different latency budgets for a quick clarifying reply vs a complex multi-constraint search?
Think about your answer before revealing the explanation.
Question 5 (Short Answer)
Where would you look first to cut latency in an agent response chain, and why?
Think about your answer before revealing the explanation.