# Designing for Failure (Retries, Timeouts & Circuit Breakers) > How to build distributed systems that bend instead of break: assume failure everywhere, add timeouts and backoff retries and circuit breakers, then fail soft with degradation, fallbacks, and bulkheads so one sick dependency can't sink the whole ship. --- # Designing for Failure (Retries, Timeouts & Circuit Breakers) You built the happy path. The request comes in, you call the database, you call the payment service, you call the recommendations API, you stitch the results together, you return a response. It works on your laptop. It works in the demo. Then it goes to production, and one Tuesday afternoon a single downstream service gets slow - not down, just *slow* - and within ninety seconds your entire app is unresponsive and your phone is buzzing. Nothing in your code "broke." And yet everything is on fire. Here's the shift that fixes this: in a system made of many moving parts talking over a network, failure is not an exception you handle at the edges. It's the normal weather. Networks drop packets, services get slow, dependencies fall over, machines reboot without asking. The question was never *if* - it's *when*, and *how gracefully*. This guide is about building systems that **bend instead of break**: that give a little when a part fails, isolate the damage, and keep serving what they still can - instead of toppling like dominoes. This is the architecture that means you *don't* get the 2am page. ## How to read this - **Want the patterns right now?** Jump to [Phase 2: The Core Patterns](02-the-core-patterns.md) - timeouts, retries with backoff and jitter, and circuit breakers, each with a mental model and an ASCII diagram. That's the toolkit. - **Want it to finally make sense?** Read in order. Phase 1 installs the mindset (why one slow dependency cascades), Phase 2 gives you the three core defenses, and Phase 3 shows you how to fail *soft* - degrade, fall back, isolate - so an outage shrinks instead of spreads. ## The phases 1. **[Everything Fails](01-everything-fails.md)** - the mindset shift. In a distributed system, networks drop, services slow, and dependencies die - not *if* but *when*. The fallacies of distributed computing, and why a single slow dependency can drag your whole system down (cascading failure). 2. **[The Core Patterns](02-the-core-patterns.md)** - the three defenses you reach for constantly: **timeouts** (never wait forever - the most commonly missing safeguard there is), **retries** with backoff and jitter (and only for safe-to-repeat operations), and **circuit breakers** (stop hammering a dead dependency; fail fast, then recover). 3. **[Failing Soft: Degradation & Redundancy](03-failing-soft.md)** - when a part fails anyway, lose a feature instead of the product. Graceful degradation, fallbacks, bulkheads (isolate failures so one drowning feature can't drown the rest), and redundancy - plus the retry storm that turns a small outage into a big one. > Deep dives into specific tooling - service meshes, distributed tracing, chaos engineering, and > SLO/error-budget math - are deliberately left to follow-up guides. This one is about the patterns and > the mindset that work no matter what your stack is. For scaling the system you're protecting, see > [Designing for Scale](/guides/designing-for-scale); for the human side of an outage in progress, see > [When Prod Is Down](/guides/when-prod-is-down). --- # Everything Fails There's a moment, the first time you operate a real distributed system, when you realize the rules you learned writing single-process programs don't hold anymore. On your laptop, calling a function returns - maybe it throws, but it *returns*, instantly and reliably, every time. Carry that instinct into a world where "calling a function" now means sending bytes across a network to a different machine that might be overloaded, mid-deploy, or gone entirely, and the instinct quietly betrays you. The mindset this phase installs is uncomfortable at first and freeing once it lands: **assume every remote call can fail or hang, and design as if it will.** Not out of pessimism - out of accuracy. Failure isn't the edge case in a distributed system. It's the baseline. Once you build *expecting* it, the chaos stops being scary and starts being something you planned for. ## What "distributed" really changes **What it actually is.** A *distributed system* is any system whose parts run on different machines and talk over a network: your app calling a database on another host, a payment API, a cache, a queue, another team's service. The instant a call leaves your process and crosses the network, you've traded a guarantee for a gamble. πŸ“ **Terminology.** A *dependency* is anything your code calls and waits on to do its job - a database, an internal service, a third-party API. When people say "a dependency failed," they mean one of those calls didn't come back the way you needed. **Why people get this wrong.** The wrong picture is that a network call is like a local call, only a bit slower. A local call has two outcomes: it returns, or it throws. A network call has *three* - it returns, it fails, or, the one that gets everybody, **it hangs**, neither succeeding nor failing, just leaving you waiting. That third outcome is where most outages are born, and we'll come back to it hard in [Phase 2](02-the-core-patterns.md). ## The fallacies of distributed computing There's a famous list, originally from engineers at Sun Microsystems, called the **fallacies of distributed computing** - the false assumptions almost everyone makes when they first build systems that span machines. You don't need to memorize all of them, but it helps to know the shape of the lies your intuition tells you (source: the canonical list is widely documented, e.g. ): ```text The comforting lie The reality you design for ───────────────────── ───────────────────────────────────────── "The network is reliable" β†’ packets drop, connections reset, links flap "Latency is zero" β†’ every call costs real time; it adds up "Bandwidth is infinite" β†’ big payloads clog; throughput has a ceiling "The network is secure" β†’ assume it isn't; things in transit are exposed "Topology doesn't change" β†’ hosts come and go; IPs move; nodes restart "There's one administrator" β†’ many owners, many deploys, no single hand "Transport cost is zero" β†’ serializing/moving data isn't free "The network is homogeneous"β†’ mixed hardware, versions, and conditions ``` **Why this matters.** Every one of these is an assumption that's *fine* on your laptop and *false* in production. Each guide phase after this one directly answers one of these fallacies: timeouts answer "latency is zero," retries answer "the network is reliable," circuit breakers and bulkheads answer "topology doesn't change." The fallacies are the disease; the patterns are the treatment. πŸ’‘ **Key point.** You will never make the network reliable. The whole game is building systems that stay *useful* on top of an unreliable network - not eliminating failure, but containing it. ## The slow dependency that takes down everything Now the heart of it - the failure mode that surprises people most, because nothing actually *crashed*. Picture a request to your web service. To answer it, your service calls a downstream dependency - say, a recommendations service. Normally that call returns in a few milliseconds, and each incoming request ties up one worker (a thread, a connection, a slot - pick your stack's word) while it waits, then frees it. Now the recommendations service gets *slow* - not down, just slow, taking several seconds to respond. Here's what happens, step by step: ```mermaid flowchart LR Slow[slow dependency] --> Stuck[workers stuck waiting] Stuck --> Fill[worker pool fills] Fill --> Down[no free workers β†’ whole service down] ``` *What just happened:* each slow call holds a worker hostage for seconds instead of milliseconds. Requests keep arriving, keep grabbing workers, and those workers keep getting stuck - your finite pool fills up. Once every worker is parked waiting on the slow dependency, your service has no capacity left to serve *anything*, including requests that have nothing to do with recommendations. From the outside, your service is down. From the inside, nothing crashed; everyone is just *waiting*. πŸ“ **Terminology.** This is a **cascading failure** (also called *resource exhaustion* when it's a pool that fills up): a failure in one component spreads to others because they share a finite resource - here, your worker pool. One slow part starved everything else. ⚠️ **Gotcha - slow is more dangerous than down.** A dependency that's cleanly *down* often fails fast: the connection is refused, you get an error in milliseconds, and you can react. A dependency that's *slow* fails in the worst possible way - it holds your resources hostage. This is why "is it up?" is the wrong question. The question is "is it *answering in time*?" - and that question only has meaning if you've set a deadline, which is exactly the next phase. πŸͺ– **War story.** A classic version of this: a minor, non-critical feature (say, fetching a user's profile avatar from a third party) shares the same connection pool as everything else. The avatar provider has a bad day and starts hanging. Within minutes, the *checkout* page - which doesn't show avatars at all - stops loading, because every connection is tied up waiting on avatars. A feature nobody would miss took down the feature that makes the money. The fix isn't "make avatars more reliable." It's isolation, and we'll get there in [Phase 3](03-failing-soft.md). ## Why this mindset saves you later Everything in the next two phases follows from one decision you make here: **stop treating failure as exceptional.** Once you accept that any remote call can hang, fail, or lie, you start asking the right questions about every dependency you add: - How long am I willing to wait for this? (timeout) - If it fails transiently, is it safe to try again - and how? (retries) - If it's clearly broken, how do I stop making it worse? (circuit breaker) - If it's gone, what can I still give the user? (graceful degradation) - How do I keep this one failure from becoming everyone's failure? (bulkheads) You don't bolt resilience on at the end. You design for it from the first dependency, because the slow Tuesday afternoon is coming whether you planned for it or not. The only choice is whether it's a shrug or a 2am page. ## Recap 1. A **distributed system** spans machines and talks over a network; the moment a call leaves your process, its reliability becomes a gamble, not a guarantee. 2. A network call has **three** outcomes, not two: succeed, fail, or **hang** - and the hang is where most outages start. 3. The **fallacies of distributed computing** are the false assumptions (network is reliable, latency is zero, …) your single-machine instincts whisper; the resilience patterns are the antidotes. 4. A single **slow** dependency can exhaust a shared resource (your worker/connection pool) and take down your *whole* service through **cascading failure** - even parts unrelated to the slow one. 5. **Slow is more dangerous than down.** Design assuming failure, and ask "is it answering in time?", not just "is it up?". --- # The Core Patterns [Phase 1](01-everything-fails.md) showed a single slow dependency take down a whole service while nothing technically crashed. This phase hands you the three tools that would have stopped it cold - simple enough to hold in your head, and they work together: - **Timeouts** decide *how long you're willing to wait* before giving up. (This is the one most systems are missing.) - **Retries** decide *whether and how to try again* after a transient failure - carefully, with backoff and jitter, and only for operations that are safe to repeat. - **Circuit breakers** decide *when to stop trying altogether* because the dependency is clearly broken - so you fail fast and let it recover instead of pounding on it. Learn these three and you've covered the overwhelming majority of "my service fell over because of something downstream" situations. ## Timeouts - never wait forever **What it actually is.** A *timeout* is a deadline on a call: "if I don't get a response within N seconds, stop waiting and treat it as a failure." It's the answer to the third outcome from Phase 1 - the *hang* - by refusing to hang. **Why people get this wrong.** Most libraries and HTTP clients ship with **no timeout, or a wildly generous default** (sometimes effectively infinite). So the code that looks innocent - `http.get(recommendations_url)` - is quietly a promise to wait *forever* if the other side never answers. That's the exact mechanism that exhausted the worker pool in Phase 1. **What it does in real life.** With a timeout, a slow dependency can hold a worker for *at most* the timeout duration, not indefinitely - the worker comes back, free to serve someone else, and the slow dependency degrades that one feature instead of starving the entire service. ```text No timeout: With a 1s timeout: call ──────────────────────▢ ??? call ───▢ (1s) ──▢ βœ— give up (worker stuck forever) worker freed, handle the failure ``` **A real example.** A timeout being hit, shown with `curl`'s max-time flag standing in for whatever your client does: ```console $ curl --max-time 2 https://recs.internal/recommend curl: (28) Operation timed out after 2003 milliseconds with 0 bytes received ``` *What just happened:* `curl` waited two seconds, got nothing, and gave up with error 28 instead of hanging indefinitely. In your service that translates to "the recommendations call failed; move on" - maybe you skip recommendations and return the rest of the page. The worker is free again, and the slow dependency cost you one feature for one request, not your whole service. ⚠️ **Gotcha - set timeouts at every layer, and budget them.** A timeout only protects the layer that has one. If your outer request has a 30-second timeout but the database call inside it has none, the database hang still wins. Inner timeouts should also *add up to less* than the outer one - if the overall request must answer in 3 seconds, you can't give a sub-call 5. This is a **timeout budget**: the parent deadline is divided among the children, not handed to each in full. πŸ’‘ **Key point.** If you do exactly one thing from this entire guide, set timeouts on every remote call. It's the single highest-leverage line of defense, and the one most commonly missing. ## Retries - try again, but carefully **What it actually is.** A *retry* is trying a failed call again, on the bet that the failure was **transient** - a momentary blip, a packet drop, a brief overload - rather than permanent. Many network failures genuinely are transient, so a retry often turns a glitch the user never sees into a non-event. **Why people get this wrong - two ways, both bad.** First, **the naive retry loop**: fail, immediately try again, fail, immediately try again. If the dependency is struggling because it's *overloaded*, hammering it with instant retries pours gasoline on the fire. Worse, if *every* client retries at the same instant, they all hit the recovering service together and knock it back down - a **thundering herd**. The fix is two ideas stacked: - **Exponential backoff** - wait longer between each attempt: 1s, then 2s, then 4s, then 8s. You give a struggling dependency room to breathe instead of crowding it. - **Jitter** - add randomness to each wait so clients *don't* line up and retry in sync. Backoff spreads retries out in time; jitter spreads them out across *clients* (see AWS's ["Exponential Backoff And Jitter"](https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/)). ```text Naive retry (everyone in lockstep): Backoff + jitter (spread out): t=0 βœ—βœ—βœ—βœ—βœ— all clients retry at once tβ‰ˆ1Β±r βœ— βœ— βœ— t=0 βœ—βœ—βœ—βœ—βœ— …again, in sync tβ‰ˆ2Β±r βœ— βœ— βœ— t=0 βœ—βœ—βœ—βœ—βœ— …herd keeps stampeding tβ‰ˆ4Β±r βœ— βœ— βœ— recovering service gets crushed load is smeared across time ``` **A real example.** ```console attempt 1: GET /recommend β†’ 503 Service Unavailable (wait ~1s + jitter) attempt 2: GET /recommend β†’ 503 Service Unavailable (wait ~2s + jitter) attempt 3: GET /recommend β†’ 200 OK ``` *What just happened:* the first two attempts hit a temporarily overloaded service and got `503`. Instead of retrying instantly, the client waited about a second (plus jitter), then about two, giving the service room to recover. By the third attempt it was healthy again, and the user never saw an error. The second way people get retries wrong is the dangerous one: ⚠️ **Gotcha - only retry idempotent operations.** An operation is **idempotent** if doing it twice has the same effect as doing it once. Reading data is idempotent; setting a value to `X` is idempotent. *Charging a credit card is not* - retry that blindly and you may bill the customer twice. The trap is the *hang*: the request may have actually *succeeded* on the server, but the response got lost on the way back, so the client thinks it failed and retries. πŸ“ **Terminology.** *Idempotent* = safe to repeat; the end state is the same no matter how many times you do it. The safe pattern for non-idempotent actions (payments, "create order") is an **idempotency key** - a unique ID the client sends so the server can recognize "I've already processed this one" and skip it. This is the heart of reliable webhook delivery, covered in [Webhooks & Message Queues](/guides/webhooks-and-message-queues) - read that before retrying anything that *changes state*. πŸ’‘ **Key point.** Retries are for *transient* failures on *idempotent* operations. A `503` or a timeout on a read? Retry with backoff and jitter. A `400 Bad Request` (your request is malformed) or a `404`? Don't retry - the answer won't change, and you're just adding load. Retry the *retryable*, not everything. ## Circuit breakers - stop hammering a corpse Timeouts cap each call; retries handle the occasional blip. But what about when a dependency isn't blipping - it's just *down*, and has been for a while? Retrying every single request, each waiting out its full timeout first, is pure waste: you're spending your own resources waiting on something you already have strong evidence is broken. That's where the circuit breaker comes in. **What it actually is.** A *circuit breaker* is exactly the electrical metaphor: a switch that **trips open** when it sees too much trouble, cutting the connection so the trouble can't spread. Once tripped, calls **fail fast** - instantly, without even attempting the call - until enough time passes to check whether it has recovered. It lives in three states: ```mermaid flowchart LR Closed["CLOSED
normal calls, count failures"] Open["OPEN
fail fast for a cooldown"] HalfOpen["HALF-OPEN
let ONE test call through"] Closed -- "many failures in a row" --> Open Open -- "after a cooldown" --> HalfOpen HalfOpen -- "test succeeds" --> Closed HalfOpen -- "test fails (reopen)" --> Open ``` **What it does in real life.** When the dependency is healthy, the breaker is **closed** and traffic flows normally while it quietly counts failures. When failures cross a threshold, it trips **open**: for the next stretch of time (the cooldown), every call returns an error *immediately* without trying. After the cooldown, it goes **half-open** and lets a single test call through - if that succeeds, the breaker closes and normal traffic resumes; if it fails, the breaker opens again and waits another round. **Why this saves you.** First, **fail fast**: when something's down, your users get a quick, clear error (or a fallback - see [Phase 3](03-failing-soft.md)) instead of waiting out a timeout on every request. Second, **let it recover**: a dependency struggling under load can't recover if you keep slamming it. The open breaker takes the pressure off and protects *both* sides. ⚠️ **Gotcha - tune the thresholds, or the breaker lies.** A breaker that trips on one stray error flaps open and closed, rejecting perfectly good traffic. One that needs a thousand failures won't protect you in time. The right thresholds (how many failures, over what window, how long the cooldown) depend on your traffic and take observation to get right - start conservative and adjust from what you see in production. πŸͺ– **War story.** The combination that bites people: a circuit breaker wrapped around a call that *also* has aggressive retries *inside* the breaker. Each "attempt" is really three retried calls, so the breaker's failure count and timing math are off by 3x and it never behaves as expected. Decide where retries live relative to the breaker (usually: breaker outside, limited retries within a single logical attempt) and keep it consistent. ## How the three fit together These aren't three separate tools you pick between - they're layers on the *same* call: ```mermaid flowchart LR Call([outbound call]) --> CB{Circuit breaker open?} CB -- "open" --> Fast[fail fast, don't even try] CB -- "closed" --> Retry["Retry (idempotent only)
backoff + jitter"] Retry --> Timeout["Timeout
cap each attempt"] Timeout --> Net[the actual network call] ``` A call goes out; the **timeout** caps how long each attempt can take; **retries** handle a transient failure with backoff and jitter; and the **circuit breaker** sits over the whole thing, ready to trip and fail fast if the dependency is clearly broken. Timeouts make retries safe - a hung call can't block a retry. Retries make blips invisible. The breaker stops the bleeding when blips become an outage. ## Recap 1. **Timeouts** cap how long you'll wait - set them on *every* remote call, at every layer, with inner timeouts adding up to less than outer ones (a timeout budget). The most commonly missing safeguard. 2. **Retries** handle *transient* failures, but only with **exponential backoff** and **jitter**, so you don't stampede a recovering service. 3. **Retry only idempotent operations** - anything that changes state needs an **idempotency key**; see [Webhooks & Message Queues](/guides/webhooks-and-message-queues). 4. A **circuit breaker** trips **open** after too many failures, **fails fast** during a cooldown, then tests recovery in **half-open** - protecting both your resources and the struggling dependency. 5. The three layer onto the same call: breaker outside, retries within an attempt, timeout on each call. --- # Failing Soft: Degradation & Redundancy The patterns in [Phase 2](02-the-core-patterns.md) decide *how you call* a dependency safely. This phase is about what you do when, despite all of that, the dependency is gone: the timeout fired, retries ran out, the breaker is open - and now you have to return *something* to the user. Here's the mindset that separates resilient systems from fragile ones: **a failure should cost you a feature, not the product.** When the recommendations service is down, the store should still take orders. When the avatar service hangs, the page should still load - just without avatars. Failing *soft* means the user notices less, or nothing; failing *hard* means a 500 error page and a lost customer. Same underlying outage, wildly different outcome - and more than anything, this is how you avoid the 2am page from [When Prod Is Down](/guides/when-prod-is-down). ## Graceful degradation - serve something useful **What it actually is.** *Graceful degradation* means that when a dependency fails, you return a **reduced but still useful** result instead of an error. Not the best answer - a *good-enough* answer. The feature quietly steps down a level rather than falling off a cliff. **What it does in real life.** You decide, ahead of time, what "less" looks like for each feature when its dependency is unavailable: ```text Feature Full result Degraded (dependency down) ────────────────────── ──────────────────── ────────────────────────────── Personalized recs tailored picks generic "popular items" list User avatar their photo a default silhouette Live inventory count "3 left in stock" hide the count, still sell Search ranking service smart ranking plain newest-first ordering Currency conversion live FX rate last cached rate (with a note) ``` **A real example.** A page-render path that degrades instead of erroring: ```console [req 8c2f] rendering /product/42 [req 8c2f] recommendations: circuit OPEN β†’ using fallback: top-sellers [req 8c2f] avatar service: timeout β†’ using fallback: default silhouette [req 8c2f] response 200 OK (degraded: recs, avatar) ``` *What just happened:* two dependencies were unavailable - the recommendations breaker was open and the avatar call timed out. Instead of returning a `500`, the request fell back to top-sellers and a default avatar and returned a perfectly usable `200`. The user got a slightly less personalized page and almost certainly didn't notice. Logging *which* parts degraded means you can still see the problem and fix it - soft failure shouldn't mean *silent*. πŸ’‘ **Key point.** Degradation is a *product* decision as much as an engineering one. For every dependency, ask: "if this is down, what's the least-bad thing we can still show?" The answer is rarely "an error page." Decide it before the outage, not during. ## Fallbacks - the answer you keep in your pocket A *fallback* is the specific source of that degraded answer - the pre-arranged Plan B you reach for when Plan A fails. The common ones: - **Cached / stale data.** Serve the last good value you saw. A slightly old exchange rate or product list beats no page at all. (Be upfront where it matters - "prices may be delayed.") - **A default or static value.** The silhouette avatar, the generic "popular items," an empty-but-valid result. - **A secondary source.** A second provider, a read replica, a different region. ⚠️ **Gotcha - your fallback must not depend on the thing that's failing.** A cache that lives inside the same down service isn't a fallback. A "backup" provider you call through the same exhausted connection pool isn't isolated. Test that your Plan B actually works *while Plan A is broken* - that's the only condition under which you'll ever need it, and it's exactly the condition people forget to test. ## Bulkheads - isolate the flooding Back in [Phase 1](01-everything-fails.md), one slow feature (avatars) drowned an unrelated one (checkout) because they shared a single resource pool. The bulkhead pattern is the direct cure. **What it actually is.** The name comes from ships: a hull is divided into sealed **bulkhead** compartments, so a breach in one floods only that compartment instead of sinking the whole vessel. In software, a *bulkhead* means giving each dependency its **own isolated pool** of resources - connections, threads, whatever's finite - so one dependency saturating its pool can't starve the others. *One shared pool - a hang anywhere starves everything:* ```mermaid flowchart LR Recs[recs] --> Pool[shared pool] Av["avatars (hang)"] --> Pool Co[checkout] --> Pool Pool --> Starve[fills β†’ all starve] ``` *Bulkheads - each gets its own pool, so a hang is contained:* ```mermaid flowchart LR Av2["avatars (hang)"] --> PA[avatars pool] Recs2[recs] --> PR[recs pool] Co2[checkout] --> PC[checkout pool] ``` **What it does in real life.** Give the avatar calls, say, their own small connection pool. When avatars hang, *those* connections fill up and avatar requests start failing (which your timeout and fallback already handle) - but checkout and recommendations are drawing from entirely different pools, untouched. The breach is sealed in one compartment. πŸ’‘ **Key point.** Bulkheads turn "one dependency is sick" into "one *feature* is sick" instead of "the whole service is down." They're the structural backstop behind everything else in this guide: even if a timeout is misconfigured or a breaker is slow to trip, isolation limits the blast radius. ## Redundancy - more than one of the thing **What it actually is.** *Redundancy* is having more than one of something critical so that losing one doesn't lose the capability: multiple instances of a service behind a load balancer, a database with replicas, deployments across more than one availability zone or region. If a single thing failing can take you down, that thing is a **single point of failure**, and redundancy is how you remove it. πŸ“ **Terminology.** A *single point of failure* (SPOF) is any one component whose failure takes the whole system down. Hunting down SPOFs - "what's the one box / one service / one zone that, if it died right now, would end us?" - is a core resilience exercise. ⚠️ **Gotcha - redundancy you've never failed over to is a guess, not a guarantee.** A standby replica that's never been promoted, a second region that's never taken live traffic - you don't actually know it works until you try, and discovering it's misconfigured *during* a real outage is the worst possible time. Deliberately failing over (the gentle end of *chaos engineering*) turns "we have a backup" into "we know the backup works." ## The retry storm - when your own defenses attack you One last warning: this is how well-intentioned resilience turns into a self-inflicted outage. We added retries in [Phase 2](02-the-core-patterns.md) to ride out transient failures. But picture a dependency that briefly hiccups under load. Every client retries - and those retries *are extra load*, so the dependency, already struggling, now gets the normal traffic *plus* a flood of retries on top. That pushes it further down, which causes more failures, which triggers *more* retries. The system attacks itself. ```mermaid flowchart TD Hiccup[dependency hiccups] --> Retry[clients retry] Retry --> Load[extra load on a struggling service] Load --> Fail[more failures] Fail --> Retry ``` *What just happened:* This is a **retry storm** (a flavor of the **thundering herd** from Phase 2): retries meant to *recover* from an overload instead *amplify* it into a full outage. The cure is the whole toolkit working together: - **Backoff and jitter** (Phase 2) so retries are spread out in time and across clients, not synchronized. - **A retry budget / cap** - limit retries per request *and* limit the overall fraction of traffic that may be retries, so retries can never become the majority of your load. - **Circuit breakers** (Phase 2) so that once a dependency is clearly down, you stop retrying entirely and fail fast instead of feeding the storm. ⚠️ **Gotcha - resilience features can amplify failures.** Retries, especially, are double-edged: the same mechanism that hides a blip can multiply an outage. Always pair retries with backoff, jitter, a cap, and a breaker. Resilience added carelessly is just a new failure mode with good intentions. ## Why this is how you avoid the 2am page A system built this way doesn't experience the Phase 1 nightmare. The slow dependency hits a timeout. A couple of capped, jittered retries either succeed or give up cleanly. The breaker trips and stops the pile-on. The feature degrades to a cached or default result. The bulkhead keeps the rest of the product healthy. From the user's side: a slightly plainer page for a few minutes. From your side: a graph that dips and recovers - not a phone that rings at 2am. On the rare day something *does* get through all of this, you'll want the human procedure in [When Prod Is Down](/guides/when-prod-is-down) - but you'll be walking into that calm, because you designed the system to bend. ## Your turn: recommendations is failing and taking checkout with it Reading the patterns is the easy part. Choosing between them while checkout is actually breaking is the job. There's no single right answer below and nothing is scored right or wrong - but the clock is real, and every minute belongs to a customer who can't check out. Isolate it, then read the debrief. ```scenario { "title": "Recommendations is failing and dragging checkout down with it", "brief": "You're on call, twelve minutes before your highest-traffic hour. The recommendations service starts timing out. It shares a connection pool with checkout, and checkout requests are now queuing behind the hung recommendations calls too. Nobody ever built a bulkhead. There's a feature flag that can turn recommendations off and fall back to a generic list.", "prompt": "What do you do first?", "clock": { "unit": "min", "running": "checkout stuck behind recs", "resolved": "checkout free of recs" }, "resolvedHeading": "Checkout is taking orders again. Here's how it went.", "actions": [ { "id": "check-pool", "label": "Check the connection pool metrics", "cost": 2, "reveals": "$ pool-stats checkout-api\nconnections: 200/200 in use\n held by recommendations calls: 188\n held by checkout calls: 12 (queued)\navg wait: 4.8s", "note": "Confirms it: checkout isn't broken, it's queued behind recommendations in the same pool. Two minutes well spent, and it tells you exactly what to isolate." }, { "id": "restart-recs", "label": "Restart the recommendations service", "cost": 3, "reveals": "$ kubectl rollout restart deployment/recommendations\ndeployment.apps/recommendations restarted\n[90s later]\nrecommendations: p99 latency 6200ms (climbing again)", "note": "Healthy for ninety seconds, then slow again. You changed something on a system whose root cause you hadn't found, and checkout stayed queued behind the pool the whole time regardless." }, { "id": "more-retries", "label": "Turn up retries and the timeout on the recs call so requests get through", "cost": 4, "reveals": "config: recs.max_retries 1 -> 5, recs.timeout 500ms -> 3000ms\n...\nrecommendations error rate: 41% -> 68%\ncheckout p99 latency: 5.1s -> 9.4s", "note": "You fed a struggling dependency more load. Its error rate got worse, and checkout - which was never waiting on recommendations to succeed, only waiting on the pool - got worse too." }, { "id": "scale-recs", "label": "Scale up the recommendations service", "cost": 6, "reveals": "$ kubectl scale deployment/recommendations --replicas=12\ndeployment.apps/recommendations scaled\n[4 min later]\nrecommendations: p99 still 6100ms\nrecommendations-db: cpu 97%, slow queries: 340", "note": "More pods didn't help - the bottleneck is the database behind recommendations, not its replica count. Checkout doesn't care how many recommendations pods exist; it only cares that it shares their pool." }, { "id": "page-recs-team", "label": "Ping the team that owns recommendations and wait for their fix", "cost": 8, "reveals": "you: recommendations is timing out and it's taking checkout down with it - can someone look?\nrecs-oncall: just paged, digging into the db now, give me ~15\nyou: checkout is failing right now, need something sooner than that", "note": "Asking for help isn't the mistake. Waiting on their fix before doing anything yourself is - that fix runs on their clock, and checkout is bleeding on yours." }, { "id": "deploy-bulkhead", "label": "Write and deploy a dedicated connection pool for checkout", "cost": 10, "reveals": "$ git commit -m \"give checkout its own connection pool\"\n$ deploy checkout-api\nrunning tests... 3m40s\ndeploying... 4m10s\n[10 min later] checkout-api: dedicated pool live, 0/50 in use", "note": "This is the real fix, and it's real work: tests, a deploy, a rollout. Exactly right for next week's design review. Not a first move in an incident that's costing you checkout traffic right now." }, { "id": "flip-flag", "label": "Flip the flag to disable recommendations and serve the fallback", "cost": 1, "resolves": true, "reveals": "$ feature-flag set recommendations.enabled false\nflag updated\n[req 91a2] recommendations: disabled by flag -> fallback: top-sellers\n[req 91a2] checkout: 200 OK\ncheckout-api pool: 12/200 in use, 0 queued", "note": "Recommendations is still broken. Checkout doesn't care anymore - it was never actually about recommendations." } ], "debrief": { "ideal": 3, "text": "The move that frees checkout has nothing to do with fixing recommendations - isolate it, serve the fallback, and let recommendations be someone else's Tuesday. A failure should cost you a feature, not the product, and the flag is just how fast you can make that true when it wasn't designed in ahead of time.", "notes": [ { "when": "if-taken", "action": "more-retries", "text": "Turning the retries up is this guide's retry storm in miniature: you fed a struggling dependency more load, its error rate got worse, and checkout - stuck on the pool, not on whether recommendations succeeded - stayed broken for every one of those minutes." }, { "when": "if-taken", "action": "restart-recs", "text": "The restart bought ninety seconds of healthy metrics and nothing for checkout, which was never actually waiting on recommendations to be healthy - just waiting on the shared pool." }, { "when": "if-taken", "action": "scale-recs", "text": "More capacity for a service that isn't capacity-constrained buys you nothing. The bottleneck was the database behind recommendations, and checkout was stuck on the pool the entire four minutes it took to learn that." }, { "when": "if-taken", "action": "page-recs-team", "text": "The team's fix runs on their clock, not yours. Whatever they find, checkout doesn't get any faster by waiting on it - only by no longer depending on it." }, { "when": "if-taken", "action": "deploy-bulkhead", "text": "The dedicated pool is the correct permanent fix, and it's also the slowest action on this list. That's the argument for building it in a design review before the outage, not live during one." }, { "when": "if-not-taken", "action": "check-pool", "text": "You never confirmed checkout was queued behind recommendations in the shared pool - you flipped the flag on a strong guess, and it happened to be right. The bulkhead you build afterward is what turns that guess into something you never have to make twice." } ] } } ``` ## Recap 1. **Fail soft, not hard:** a failure should cost a *feature*, not the product. Decide the least-bad degraded result for each dependency *before* the outage. 2. **Graceful degradation** returns a reduced-but-useful result; **fallbacks** are where that result comes from (cache, default, secondary source) - and a fallback must not depend on the failing thing. 3. **Bulkheads** give each dependency its own isolated resource pool, so one saturated pool can't starve the rest - turning "the service is down" into "one feature is down." 4. **Redundancy** removes single points of failure - but untested failover is a guess; verify it works *before* you need it. 5. Beware the **retry storm**: retries can amplify an outage into a bigger one. Always pair them with backoff, jitter, a retry cap, and a circuit breaker.