Key Concepts
- 1Guardrails are hard constraints the agent cannot cross regardless of how it reasons — the non-negotiable 'never' list that protects users and the business
- 2Financial guardrails: never charge without explicit confirmation, never exceed a spend limit, never complete a purchase the user didn't approve
- 3Data guardrails: never reveal another customer's data, never echo payment or credential data, never act on instructions embedded in untrusted content (prompt injection)
- 4Safety guardrails: never sell restricted/illegal items to ineligible users, never give regulated advice (medical, legal, financial) it isn't authorized to give
- 5Guardrails should be enforced in code/policy layers around the LLM, not just requested in the prompt — because prompts can be jailbroken and models drift
- 6Fail-safe defaults: when uncertain, the agent should stop and hand off to a human rather than guess on a high-stakes action — safe failure over confident error
- 7Guardrails must be tested adversarially: red-team the agent to find where it can be coaxed past a 'never', because attackers and edge cases will
Why It Matters for PMs
Catastrophic-risk prevention: The worst agent failures — unauthorized charges, data leaks, selling restricted goods — are guardrail failures. Hard limits prevent low-probability, high-severity disasters.
Prompt-level rules aren't enough: Because models can be jailbroken, enforcing guardrails in code is what makes them reliable. This is a core architectural principle PMs must insist on.
Trust and license to operate: A single guardrail breach can end customer trust or draw regulatory action. Investing in guardrails protects the business's ability to run the agent at all.
Test Your Understanding
Question 1 (Multiple Choice)
Why should critical guardrails be enforced in code/policy rather than only in the prompt?
Question 2 (Multiple Choice)
What is the fail-safe default when an agent is uncertain about a high-stakes action?
Question 3 (Multiple Choice)
Which is a legitimate hard guardrail for a commerce agent?
Question 4 (Short Answer)
How would you decide what belongs on an agent's 'never' list, and where to enforce each rule?
Think about your answer before revealing the explanation.
Question 5 (Short Answer)
Why is adversarial (red-team) testing essential for guardrails, and what would you test for?
Think about your answer before revealing the explanation.