Topic 4
Degrade gracefully, shed load
Not every bad day is an outage. Sometimes the system is up but overwhelmed: a launch, a viral post, a batch job sharing your rate limit. Two techniques from Google's SRE book, "Addressing Cascading Failures", apply.
Graceful degradation does less work per request: lower-quality, cheaper results. The book's example is a search that only looks through the subset of data in an in-memory cache. For an AI application, the equivalents are a smaller model, skipping the reranker, fewer retrieved passages, a shorter maximum output, or switching off an expensive optional feature.
Load shedding refuses some requests outright, so the rest can be served. The SRE book describes returning HTTP 503 (service unavailable) to new requests once too many are in flight, or throttling based on CPU, memory or queue length.
Predict first
Scroll sideways to see the whole diagram
Sometimes the problem isn't an outage but too much demand.
Text version
- Sometimes the problem isn't an outage but too much demand.
- Google's SRE book describes load shedding: drop a share of traffic as a server nears overload, for example returning 503 to new requests once too many are in flight, so it can still do as much useful work as possible.
- The alternative, queueing everything, makes every request slow until they all time out. A fast, clear refusal is a form of graceful degradation.
The SRE book adds two cautions for degraded modes. They should rarely trigger, typically only when capacity planning fails or load shifts unexpectedly. And they must be exercised regularly, because the code path you never use is often the code path that doesn't work. A fallback you've never run is a hypothesis, not a plan.
Quick check