Key Concepts
- 1LLM capacity is constrained differently than web servers: provider rate limits, token throughput, and GPU availability cap how many conversations you can serve concurrently
- 2Demand is spiky in commerce (launches, promotions, seasonal peaks), so capacity planning must handle bursts, not just average load
- 3Rate limits are a hard external ceiling; hitting them causes failures, so PMs need headroom, backoff strategies, and possibly reserved/provisioned throughput
- 4Cost scales with capacity: provisioning for peak is expensive, so the plan balances reserved capacity against burst/on-demand and graceful degradation under overload
- 5Queueing and load-shedding decisions matter: under extreme load, is it better to slow everyone slightly, prioritize checkout flows, or shed low-value traffic?
- 6Multi-provider or fallback-model strategies add resilience against a single provider's outage or throttling, at the cost of complexity
- 7Capacity planning ties directly to the launch calendar: a promotion without capacity headroom turns a marketing win into an outage
Why It Matters for PMs
Peak readiness: Commerce demand spikes at exactly the high-value moments (launches, promos). Capacity planning turns those peaks into revenue instead of outages.
Cost vs resilience tradeoff: Provisioning for peak is expensive; under-provisioning fails. PMs own the balance of reserved capacity, burst, and graceful degradation.
Prioritization under load: When capacity is scarce, choosing to protect checkout over browsing is a product decision that directly affects revenue during incidents.
Test Your Understanding
Question 1 (Multiple Choice)
How is LLM capacity constrained differently from typical web servers?
Question 2 (Multiple Choice)
Why must commerce capacity planning handle bursts, not just averages?
Question 3 (Multiple Choice)
Under extreme load, which is a sensible load-shedding priority?
Question 4 (Short Answer)
How would you plan capacity for a major promotion given provider rate limits?
Think about your answer before revealing the explanation.
Question 5 (Short Answer)
What's the tradeoff between provisioning for peak and relying on graceful degradation under overload?
Think about your answer before revealing the explanation.