Designing for Failure (Retries, Timeouts & Circuit Breakers)

How to build distributed systems that bend instead of break: assume failure everywhere, add timeouts and backoff retries and circuit breakers, then fail soft with degradation, fallbacks, and bulkheads so one sick dependency can't sink the whole ship.

1. Everything Fails In a distributed system, networks drop, services slow, and dependencies die - not if but when; the fallacies of distributed computing explain why, and a single slow dependency can take your whole system down through cascading failure.
2. The Core Patterns The three defenses you reach for constantly: timeouts so you never wait forever, retries with exponential backoff and jitter for transient failures on idempotent operations only, and circuit breakers that stop hammering a dead dependency and fail fast until it recovers.
3. Failing Soft: Degradation & Redundancy When a part fails anyway, lose a feature instead of the product: serve cached, partial, or default results via graceful degradation and fallbacks, isolate failures with bulkheads so one drowning feature can't sink the rest, add redundancy, and avoid the retry storm that turns a small outage into a big one.