Key Concepts
- 1You can't improve what you don't measure. For agentic systems, measure: accuracy, clarity, trust, and conversion.
- 2**Accuracy**: Did the agent recommend a product that the user actually wanted? Measure: (1) User bought it → accurate, (2) User rejected it and bought something else → inaccurate.
- 3**Clarity**: Did the agent explain its reasoning well? Measure: (1) User understands the recommendation (survey), (2) User can explain why this product fits them.
- 4**Trust**: Does the user believe the agent is acting in their interest? Measure: (1) Would you recommend this agent to a friend? (2) Do you trust the agent's recommendations? (NPS-style survey).
- 5**Conversion**: Did the recommendation drive a purchase? Measure: (1) Recommendation acceptance rate (% of users who buy what the agent recommended), (2) AOV impact (average order value with agent vs. without).
- 6A/B testing: Compare agent recommendations vs. human recommendations or simple search. Measure which drives higher conversion and customer satisfaction.
- 7Confidence vs. accuracy trade-off: An agent that's 90% confident and 95% accurate is great. An agent that's 90% confident but only 60% accurate is terrible (overconfident). Track this 2x2.
Why It Matters for PMs
Data-driven decisions: Without metrics, you're guessing. You don't know if your agent is actually helping. Metrics tell you the truth.
Continuous improvement: Metrics identify problems. 'Users are rejecting 40% of recommendations' → investigate why → improve. Metrics drive product iteration.
ROI calculation: You can calculate the ROI of building an agent. Example: agent increases AOV by 5% → $X incremental revenue per month → did the agent build cost pay for itself?
Test Your Understanding
Question 1 (Multiple Choice)
What's the most important metric for an agentic commerce system?
Question 2 (Multiple Choice)
You have two agents: Agent A is 90% confident and 85% accurate. Agent B is 80% confident and 90% accurate. Which is better?
Question 3 (Multiple Choice)
How would you measure if an agent actually improved conversion?
Question 4 (Short Answer)
Design a metric to measure 'clarity' of agent explanations.
Think about your answer before revealing the explanation.
Question 5 (Short Answer)
A/B test reveals that agent recommendations have a 25% acceptance rate, while simple search has a 20% acceptance rate. Is the agent worth building?
Think about your answer before revealing the explanation.