Key Concepts
- 1You can't improve what you don't instrument — agent flows need event tracking at every step: intent captured, results shown, option selected, checkout started, order completed
- 2The agent funnel differs from a traditional one: measure clarifying-question depth, recommendation-acceptance rate, and correction rate, not just page-view-to-purchase
- 3Assist vs last-click attribution: the agent often influences a purchase that completes elsewhere; naive last-click undercounts its true contribution
- 4Guardrail metrics matter as much as goal metrics: track containment (resolved without human handoff) alongside satisfaction so you don't optimize conversion at the cost of experience
- 5Cohort by intent quality: sessions with clear intent convert differently than exploratory ones, so a single blended conversion rate hides the real story
- 6Instrument cost per conversation (tokens, tool calls) next to conversion, because a feature that lifts conversion but triples cost may be net-negative
- 7Close the loop to revenue: tie agent events to actual order value and returns, not just clicks, so PMs report contribution margin rather than vanity engagement
Why It Matters for PMs
Decision-making foundation: Without proper instrumentation, every agent iteration is guesswork. Good telemetry is what turns the agent from a black box into a tunable product.
Attribution credibility: Executives fund what they can measure. Correct assist/incrementality attribution is how PMs prove the agent's revenue contribution and secure roadmap investment.
Balancing quality and cost: Tracking conversion alongside cost-per-conversation and satisfaction prevents optimizing one metric into a business-negative outcome.
Test Your Understanding
Question 1 (Multiple Choice)
Why is naive last-click attribution a problem for agent flows?
Question 2 (Multiple Choice)
Which is a guardrail metric a PM should track alongside conversion?
Question 3 (Multiple Choice)
Why track cost-per-conversation next to conversion?
Question 4 (Short Answer)
What events would you instrument across an agent shopping session, and why those?
Think about your answer before revealing the explanation.
Question 5 (Short Answer)
Why is a single blended conversion rate misleading, and how would you segment it?
Think about your answer before revealing the explanation.