# When Prod Is Down: Staying Calm
> How to handle a production outage without panicking: assess the blast radius, stop the bleeding before you diagnose, run the incident with clear comms, and turn the wreckage into a blameless postmortem that prevents the next one.
---
# When Prod Is Down: Staying Calm
The alert fires. The dashboard goes red. Someone types "is anyone else seeing this?" in the channel and
your stomach drops through the floor. Maybe you shipped the last deploy. Maybe you have no idea what
happened. Either way, your hands are a little cold and your brain wants to either freeze or start frantically
typing commands - and both of those instincts will make it worse.
Here's the thing nobody tells you: handling an outage well is not about being a genius who instantly knows
the fix. It's a *procedure*. Calm under fire isn't a personality trait - it's a checklist you follow when
your judgment is compromised by adrenaline. This guide gives you that procedure: what to do in the first
five minutes, how to stop the damage before you understand it, how to run the response without chaos, and
how to make sure the outage buys you something instead of just costing you a night.
You don't need to be the most senior person in the room to use this. You need to be the calmest, and calm is
learnable.
## How to read this
- **In a panic right now?** Go straight to [Phase 1: The First Five Minutes](01-the-first-five-minutes.md)
and use the **PROD-DOWN CHECKLIST** at the very top. Do those things in order. Read the explanations
later, when it's over.
- **Want it to finally make sense?** Read in order. Each phase builds the mindset that makes the next one
feel obvious - first how to stabilize, then how to coordinate, then how to learn.
## The phases
1. **[The First Five Minutes](01-the-first-five-minutes.md)** - don't panic, don't start randomly changing
things. Assess the blast radius (who and what is affected, how bad), then *stop the bleeding* before you
diagnose. The whole mindset: restore service first, understand later.
2. **[Triage & Mitigate](02-triage-and-mitigate.md)** - the fastest paths back to green: roll back the last
deploy, flip the feature flag off, scale up, fail over. Plus how to actually *run* an incident - one
coordinator, a clear channel, a timeline written as you go - and why the silent hero is the most
dangerous person on the call.
3. **[After: the Blameless Postmortem](03-after-the-blameless-postmortem.md)** - once it's over: build the
timeline, separate root cause from contributing factors, and protect the *blameless* rule that keeps
people forthcoming. Then turn the incident into prevention. Every outage is tuition; make it buy something.
> Deep dives into specific tooling - distributed tracing, chaos engineering, SLO/error-budget math, on-call
> rotation design - are deliberately left to follow-up guides. This one is about the human procedure that
> works no matter what your stack is.
---
# The First Five Minutes
The first five minutes of an outage decide how the next two hours go - this is when panic does its damage:
typing commands you can't explain, restarting things at random, freezing at a dashboard hoping it turns green.
The site being down is bad; *you* making it worse by acting faster than you can think is the danger you
control. Here's exactly what to do, in order - use the card while it's happening, read the rest when you can.
## The PROD-DOWN CHECKLIST
> **Alert just fired? Don't touch anything yet. Do these, in order.**
| # | Do this | Why |
|---|---|---|
| 1 | **Don't panic.** | Panic-actions are the #1 way a small outage becomes a big one. |
| 2 | **Confirm it's real.** Is it the service, or your laptop/VPN/the monitoring? | You can't fix an outage that isn't happening - and you'll cause one chasing a ghost. |
| 3 | **Declare the incident out loud.** Post in the incident channel: *"Declaring an incident: checkout 500s. I'm coordinating."* | Makes it official, pulls in help, starts the clock and the record. |
| 4 | **Assess the blast radius.** *Who* is affected, *what* can't they do, *how bad* (all users or some?). | Severity drives everything: who you wake, how fast you act, what risks are acceptable. |
| 5 | **Note the start time and the last change.** When did it begin? What deployed/changed just before? | The last change is the prime suspect, every single time. |
| 6 | **Stop the bleeding.** Mitigate first - roll back, flag off, scale, fail over. *Do not diagnose yet.* | Restore service first, understand later. (Full menu in [Phase 2](02-triage-and-mitigate.md).) |
| 7 | **Only now, start investigating** - with service restored or at least stabilized. | Calm debugging beats frantic debugging every time. |
---
## Don't panic, and don't start randomly changing things
The alert dumps adrenaline into your system - useful for running from a predator, terrible for debugging. It
makes "do *something*" feel like progress even when it's harmful; the urge to type is chemical, not a plan.
Minute one isn't for *fixing* the outage, it's for not making it worse - the fix comes after.
**The cardinal rule:** *don't change anything you can't explain and can't undo.* Restarting a service you
don't understand, clearing a cache "just to see," bouncing a database - these feel like action, but might turn
a degraded service into a dead one, or destroy the evidence you need.
β οΈ **The "let me try a few things" trap.** The most expensive outages are the ones where someone fixed the
real problem in minute two but kept tinkering, and the third "let me try this" broke something new. Once
service is restored, *stop touching it.*
πͺ **War story.** A teammate paged for high latency once restarted the app servers, flushed the cache, and
failed over the database all at once, "to be safe." Latency went away, but so did the ability to tell which
change mattered - the cache flush caused a thundering herd that took the site down ten more minutes. One at a
time, observed, is faster *and* safer.
## Confirm it's real before you respond to it
Plenty of "outages" are the responder's own VPN, an expired cert, a broken monitoring check, or one bad node
the load balancer already routes around - fixing a problem that was never there is its own outage. Confirm
fast, from the *user's* angle, not your laptop's:
```console
$ curl -s -o /dev/null -w "%{http_code} %{time_total}s\n" https://api.example.com/health
503 0.42s
$ curl -s -o /dev/null -w "%{http_code} %{time_total}s\n" https://api.example.com/health
503 0.39s
```
*What just happened:* Hitting the health endpoint directly, twice, from outside your own environment returned
`503` both times in under half a second - consistent failures from a clean path, almost certainly real and
server-side. Two `200`s instead would point at your own connection or a single bad instance.
π‘ **Key point.** "Is it real?" and "is it *everywhere*?" are different questions - you want both answered. A
503 from one region but 200s from another is a smaller incident than a 503 everywhere.
## Assess the blast radius
The most important judgment call in the first five minutes: it sets the size of your response. You wouldn't
wake the whole on-call tree for a cosmetic bug in a beta feature, but you *would* for "no one can check out."
π **Terminology.** *Blast radius*, from explosives, means how far the damage reaches: **who's affected, what
they can't do, how widely** - real-world impact, not technical symptom. "A queue is backed up" is a symptom;
"orders aren't being confirmed" is blast radius. Three questions, in plain language:
```text
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β BLAST RADIUS = WHO Γ WHAT Γ HOW BAD β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β WHO β all users? a region? logged-in only? one customer? β
β WHAT β core flow (checkout, login) or an edge feature? β
β HOW BAD β fully broken, degraded/slow, or cosmetic? β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
big radius β act fast, pull in people, accept bolder fixes
small radius β calmer pace, fewer people, careful fixes
```
The combination, not any single answer, sets the severity: "slow" for "everyone" on "checkout" is an all-hands
emergency; "fully broken" for "one internal admin report" is a ticket for tomorrow. Say it out loud in the
channel so everyone shares the same picture:
> *"Blast radius: all logged-in users, checkout returns 500, started ~14:03. This is customer-facing and
> revenue-impacting - treating as high severity."*
β οΈ **Don't under-call it to avoid the fuss.** Resist labeling things minor to skip waking people - it's far
cheaper to spin down an over-declared incident ("false alarm, go back to bed") than discover an hour later the
"minor" thing was quietly losing orders. When unsure, round *up*.
## Note the start time and the last change
Two facts, written down in the first minute, shape the whole investigation: *when did it start* and *what
changed right before*. The overwhelmingly common cause of a sudden outage is a recent change - a deploy, a
config flip, a feature flag, a migration, an infra change. Graph turned red at 14:03, deploy went out at
14:01? Prime suspect, before reading a single log line.
```console
$ kubectl rollout history deployment/checkout-api
REVISION CHANGE-CAUSE
6 release v2.31.0
7 release v2.32.0 # shipped 14:01, two minutes before the alerts
```
*What just happened:* You checked the deployment's recent revisions and saw `v2.32.0` rolled out at 14:01 -
moments before the 14:03 alert. That timing doesn't *prove* the deploy caused it, but makes it the first thing
to mitigate. Correlation is a lead, not a verdict - and that's exactly what you want right now.
π‘ **Key point.** "What changed?" beats "what's wrong?" as your opening question - *what's wrong* can take an
hour to understand, *what changed* you can often answer in thirty seconds, and changing it back is often the
whole fix.
## Stop the bleeding - restore service first, understand later
The mindset that separates calm responders from heroes-who-make-it-worse:
> **Restore service first. Understand later.**
When you cut yourself badly, you apply pressure *before* investigating which blood vessel you nicked. Same
here: if you can get users working again - rolling back, flipping a flag, failing over - *do that now*, even
without understanding root cause. A restored service buys you the most valuable thing in an incident: time to
think clearly instead of under fire.
Engineers love understanding first - correct instinct for a normal bug, dangerous during an outage. Every
minute spent on root cause while users are broken is impact you could have stopped. Mitigation and diagnosis
are separate jobs, and in the first five minutes you only have one: mitigation.
```mermaid
flowchart LR
alert["alert"] --> confirm["confirm
real?"]
confirm --> assess["assess
blast radius"]
assess --> mitigate["MITIGATE
stop the bleeding"]
mitigate --> diagnose["diagnose
(now calm, service restored)"]
diagnose --> fix["permanent fix
(later, reviewed)"]
alert -. minimize this gap = time users are in pain .-> mitigate
```
The whole game in the early phase is shrinking that shaded gap - the time users are hurting - with mitigation,
not understanding. Phase 2 is the menu for stopping the bleeding.
πͺ **War story.** A senior engineer once rolled back a deploy within four minutes, the site came back, and
*then* she said: "Okay, it's stable. Now let's figure out what that deploy did." Nobody had to debug a live
fire. The bug was subtle and took an hour to fully understand - an hour that, thanks to the rollback, cost
zero customer impact.
> βοΈ Once service is breathing again and you've got room to investigate, the skills that carry you are
> log-reading and stack-trace reading. See [Reading Logs Without Drowning](/guides/reading-logs-without-drowning)
> and [Reading a Stack Trace](/guides/reading-a-stack-trace) for the calm-investigation half of the job.
## Your turn: it's 14:03 and checkout is down
Reading the checklist is the easy part. Doing it while a clock runs is the job. There is no single right
answer below and nothing is scored right or wrong - but the clock is real, and every minute you spend is a
minute nobody can check out. Restore service, then read the debrief.
```scenario
{
"title": "14:03 - checkout is returning 500s",
"brief": "You're on call. The alert fired sixty seconds ago: checkout is throwing 500s. You do not know why. Nothing else has been touched yet, and every minute on this clock is a minute customers cannot buy anything.",
"actions": [
{
"id": "curl",
"label": "Hit the health endpoint from outside, twice",
"minutes": 1,
"reveals": "$ curl -s -o /dev/null -w \"%{http_code} %{time_total}s\\n\" https://api.example.com/health\n503 0.42s\n$ curl -s -o /dev/null -w \"%{http_code} %{time_total}s\\n\" https://api.example.com/health\n503 0.39s",
"note": "Consistent 503s from a clean path, in under half a second. It's real, it's server-side, and it isn't your VPN."
},
{
"id": "deploys",
"label": "Check what shipped recently",
"minutes": 1,
"reveals": "$ kubectl rollout history deployment/checkout-api\nREVISION CHANGE-CAUSE\n6 release v2.31.0\n7 release v2.32.0 # shipped 14:01",
"note": "v2.32.0 went out at 14:01. The alert fired at 14:03. That is not proof, but it is the prime suspect, and you got it in under a minute."
},
{
"id": "declare",
"label": "Declare the incident in the channel",
"minutes": 1,
"reveals": "you: Declaring an incident: checkout 500s since ~14:03, all users. I'm coordinating.\nrota-bot: paged @secondary\nmaya: on it - I shipped v2.32.0 at 14:01, want me to prep a rollback?",
"note": "Costs a minute, buys you a second pair of hands and a record. Notice who answered, and what she just told you."
},
{
"id": "logs",
"label": "Tail the application logs and read the stack traces",
"minutes": 4,
"reveals": "checkout-api ERROR NullPointerException: cart.discount is null\n at CheckoutService.total(CheckoutService.java:88)\n ... 2,140 more in the last 60s",
"note": "Genuinely interesting: a null discount is blowing up the total. It is also still true after you mitigate, and it cost four minutes with the site down to learn it."
},
{
"id": "db",
"label": "Check the database for load or locks",
"minutes": 4,
"reveals": "connections: 41/200 slow queries: 0 replication lag: 0.1s\ncpu: 12%",
"note": "Healthy, and boring. A reasonable hypothesis, ruled out - that is what investigation is. It cost four minutes."
},
{
"id": "restart",
"label": "Restart the app servers to clear it",
"minutes": 3,
"reveals": "deployment.apps/checkout-api restarted\n$ curl -s -o /dev/null -w \"%{http_code}\\n\" https://api.example.com/health\n503",
"note": "Still 503. You changed something you could not explain on a system you had not diagnosed, and got nothing for it. If it HAD worked you would never know why."
},
{
"id": "rollback",
"label": "Roll back to v2.31.0",
"minutes": 2,
"resolves": true,
"reveals": "$ kubectl rollout undo deployment/checkout-api\ndeployment.apps/checkout-api rolled back\n$ curl -s -o /dev/null -w \"%{http_code}\\n\" https://api.example.com/health\n200",
"note": "Green. You still do not know exactly what v2.32.0 did - and you did not need to."
}
],
"debrief": {
"idealMinutes": 4,
"text": "The move that ends this outage is the one you can make before you understand it. Confirm it's real, ask what changed, put it back. Root cause is a question for a calm afternoon with the site up - and the null discount will still be in the logs when you get there.",
"notes": [
{ "when": "if-taken", "action": "logs", "text": "You read the logs before mitigating. That instinct is correct for a normal bug and expensive during an outage: the stack trace was still waiting for you after the rollback, but those four minutes of downtime were not refundable." },
{ "when": "if-taken", "action": "db", "text": "Checking the database was a sensible hypothesis, and ruling it out is real work. Notice only what it cost: four minutes, while the answer was one line of deploy history away." },
{ "when": "if-taken", "action": "restart", "text": "The restart is this phase's war story in miniature - an action you could not explain, on a system you had not diagnosed. It happened to be harmless. The version that flushes a cache and causes a thundering herd is the same decision with worse luck." },
{ "when": "if-not-taken", "action": "declare", "text": "You never declared the incident. It worked, because you were right and quick. It also meant nobody else knew, nobody checked your thinking, and there is no record - and Maya, who shipped v2.32.0 and could have told you in one line, never got asked." },
{ "when": "if-not-taken", "action": "curl", "text": "You never confirmed it from the user's side. It was real this time. The times it isn't, you have just rolled back production to chase your own VPN." }
]
}
}
```
## Recap
1. **Don't panic, don't randomly change things.** The adrenaline urge to "do something" is the danger. Change
nothing you can't explain and can't undo.
2. **Confirm it's real** from the user's angle before you respond - not your laptop, VPN, or a broken check.
3. **Assess the blast radius** - *who Γ what Γ how bad* - and say it out loud. When unsure, round up.
4. **Note the start time and the last change.** "What changed?" beats "what's wrong?" and is usually the prime
suspect.
5. **Stop the bleeding before you diagnose.** Restore service first, understand later - only one of those is
your job in minute one.
---
# Triage & Mitigate
You've confirmed it's real, know the blast radius, and spotted the prime suspect. Now make the pain stop: fast
mitigations that get users working before you fully understand what broke, plus how to *run* the incident so
five people helping doesn't become five people colliding.
Carry this mindset throughout: **in the moment, mitigation beats root cause.** You're not trying to be right,
you're trying to make the graph green. Clever diagnosis waits until the bleeding stops.
## The mitigation menu - reach for these first
Ordered roughly by how often they work and how fast they are - start at the top. The unifying idea: a sudden
outage usually means *something changed*, so the fastest fixes *undo a change*.
### 1. Roll back the last deploy (the most common fix, full stop)
**Why this is first.** If the outage started right after a deploy, rolling it back is the single most likely
fix in incident response - it directly undoes the prime suspect, is fast, and is usually reversible. Reach for
it before anything cleverer.
```console
$ kubectl rollout undo deployment/checkout-api
deployment.apps/checkout-api rolled back
$ kubectl rollout status deployment/checkout-api
Waiting for deployment "checkout-api" rollout to finish: 2 of 4 updated replicas are available...
deployment "checkout-api" successfully rolled out
```
*What just happened:* `rollout undo` told Kubernetes to redeploy the *previous* known-good revision;
`rollout status` confirmed the old pods came up healthy. If `v2.32.0` was the culprit, checkout recovers within
a minute or two. You haven't found the bug, but you've removed it from production - that's the job right now.
> βοΈ A rollback is, at heart, a Git operation - putting production back on a known-good commit. The deeper
> mechanics of reverting safely live in [Git Disaster Recovery](/guides/git-disaster-recovery); during an
> incident, your deploy tool's "rollback" or "redeploy previous" button is usually the fastest front door.
β οΈ **Rollback gotcha: the irreversible migration.** Rolling back code is safe; a *database migration* often
isn't - if the deploy added a column, backfilled data, or changed a schema, old code may not run against the
new database, and reversing it can lose data. Ask: *"Did this release include a database change?"* If yes,
pause and get the person who wrote it on the call first - the one place "just roll it back" digs the hole
deeper.
### 2. Turn off the feature flag
**When it beats a rollback.** If the broken behavior sits behind a flag, flipping it off is faster and more
surgical than a rollback - no redeploy, no waiting for pods, and it disables *only* the bad thing, leaving the
rest of the release in place.
```console
$ flagctl set checkout_new_pricing --off
flag "checkout_new_pricing" β OFF (effective immediately, all environments)
```
*What just happened:* You disabled the new code path at runtime. Requests fall back to old, known-good
behavior with nothing redeployed - if the new pricing logic was the problem, checkout recovers in seconds.
π‘ **Key point.** This is *why* mature teams put risky changes behind flags: a flag turns "emergency rollback
under pressure" into "flip a switch." No flag on risky changes yet? Postmortem action item
([Phase 3](03-after-the-blameless-postmortem.md)).
### 3. Scale up / give it more resources
**When this is the move.** If nothing changed in code but load spiked - a traffic surge, a viral moment, a
batch job hammering the database - give the system more room while you find the source.
```console
$ kubectl scale deployment/checkout-api --replicas=12
deployment.apps/checkout-api scaled
```
*What just happened:* You scaled to 12 replicas, adding capacity to absorb the load. This is a *mitigation*,
not a cure - if a slow query or runaway client is the real cause, more replicas may only buy minutes, but
that's exactly what you're shopping for.
β οΈ **Scaling can move the bottleneck, not remove it.** Tripling app servers when the *database* is the
bottleneck makes things *worse* - more app servers means more connections hammering the same overwhelmed
database. Scale the tier that's actually saturated, and watch downstream effects.
### 4. Fail over to a healthy replica/region
**When this saves you.** If a database replica, availability zone, or region is unhealthy and you have a
standby, failing over restores service without fixing the sick component - you isolate the damage and route
around it.
```console
$ ./failover.sh --promote db-replica-2 --region us-west-2
Promoting db-replica-2 to primary...
Health check passed. Traffic now routing to us-west-2.
```
*What just happened:* You promoted a healthy standby and shifted traffic to it. The primary is still broken,
but users are now served by the healthy one, buying time to investigate the failed component out of the
critical path.
β οΈ **Know your failover before the fire.** Failover is the mitigation most likely to go wrong *if you've never
practiced it* - a stale replica can serve old data, a half-configured standby can fail under real traffic. Not
confident the standby is healthy and current? A rollback or flag-flip is safer. (Practicing failover before
the fire is, you guessed it, a postmortem action item.)
---
```mermaid
flowchart TD
q1{"Did a deploy go out
right before?"} -->|yes| rb["ROLL BACK the deploy"]
q1 -->|no| q2{"Is the bad behavior
behind a flag?"}
q2 -->|yes| ff["FLAG IT OFF"]
q2 -->|no| q3{"Did load spike with
no code change?"}
q3 -->|yes| sc["SCALE UP the saturated tier"]
q3 -->|no| q4{"Is one replica/zone/
region unhealthy?"}
q4 -->|yes| fo["FAIL OVER to the healthy one"]
q4 -->|no| none["Stabilize what you can,
then diagnose (logs/traces)"]
```
π **Terminology.** *Mitigation* = anything that reduces or stops user impact, whether or not it fixes the
cause. *Remediation* (the permanent fix) = fixing root cause so it can't recur. In the moment you want
mitigation; the postmortem produces remediation. "The pain stopped" β "the problem is fixed" - different
milestones.
## Running the incident - so help doesn't become chaos
Once more than one or two people are involved, *coordination* matters as much as the fix. The classic failure
mode isn't too few people - it's several skilled engineers debugging in parallel, stepping on each other, making
simultaneous changes nobody knows about, while leadership has no idea what's happening. A well-run incident
with three beats a chaotic one with ten.
### One coordinator (the incident commander)
One person - the **incident commander (IC)** - runs the response. Crucially, *the IC usually isn't the one
with hands on the keyboard*: their job is to coordinate, not fix - track what's being tried, decide what to
try next, keep comms flowing, pull in the right people.
π **Terminology.** *Incident Commander (IC)* - the single person accountable for coordinating the response.
Not necessarily the most senior or knowledgeable, just the one holding the overall picture so responders can
focus on their piece. The role can (and during long incidents, should) be handed off, but exactly one person
holds it at any moment.
**Why "exactly one" matters.** With no coordinator, everyone assumes someone else is watching the whole board,
and nobody is; with two, they give conflicting directions. One named IC means there's always an answer to
"what are we doing and who's deciding?" First on scene with nobody else stepped up? *You're* IC until you hand
it off - say so: *"I'm IC for this. [Name], can you investigate the database? I'll handle comms."*
### A clear comms channel and status updates
Spin up (or use a standing) incident channel and put *everything* there - what's being tried, what's ruled
out, status updates. Anyone joining can scroll up and catch the story instead of asking people to re-explain,
and it becomes your timeline (next section) for free.
**Two audiences, two cadences.** Responders need detail. *Everyone else* - leadership, support, the rest of the
company - needs a short, regular heartbeat so they stop interrupting to ask "any update?":
> *14:18 - Incident update: Checkout is failing for all users (started ~14:03). Suspected cause: the 14:01
> release. Rolling it back now. Next update in 15 min or when status changes.*
π‘ **Key point.** "Next update at [time]" is the single most calming sentence in incident comms - it frees
responders from answering "any update?" Send it even if it's just "still working, no change, next update in
15."
### Write the timeline as you go
Don't wait and reconstruct the timeline afterward - adrenaline scrambles memory and you *will* misremember the
order and times. Drop one-line, timestamped notes into the channel *as things happen* for an accurate timeline,
zero memory required.
```text
14:03 alerts fire - checkout 500s, all regions
14:05 declared incident, IC = Maria
14:07 blast radius: all logged-in users, checkout fully down
14:08 noticed deploy v2.32.0 went out 14:01 - prime suspect
14:12 rolling back v2.32.0
14:15 rollback complete, 500s dropping
14:17 checkout confirmed working - bleeding stopped
14:18 posted status update; starting root-cause investigation
```
*What this gives you:* a clear, minute-by-minute record while it's fresh, for mining later - *time to detect*
(alert to declared) and *time to mitigate* (declared to bleeding stopped) - numbers only trustworthy if
written down live.
### β οΈ The most dangerous person on the call: the silent hero
There's an anti-pattern that feels like heroism and is actually sabotage: the engineer who goes quiet, fixes
(or "fixes") something alone, and tells no one. Even when they're *right*, they've broken the incident:
- Nobody else knows a change was made, so when the graph moves, the team can't tell what caused it.
- If their change makes things worse, others may simultaneously make *other* changes - now two uncontrolled
variables, no idea which did what.
- The timeline gets a hole in it, so the postmortem can't learn from what actually happened.
πͺ **War story.** Mid-incident, the graphs suddenly recovered - then, a minute later, crashed harder. One
engineer had quietly restarted a service at the same moment another had quietly cleared a cache; each assumed
*their* change worked, and neither had announced it. Untangling who did what cost more time than the original
bug. The rule that prevents this is simple and absolute:
> **Announce every change before you make it, in the channel.** *"I'm about to restart the worker pool -
> objections? Going in 30 seconds."* No silent fixes. No exceptions. The hero who saves the day silently turns
> a 20-minute incident into a 2-hour mystery.
The discipline keeps the response a *coordinated* effort instead of several people gambling in the dark.
## Recap
1. **Mitigation beats root cause in the moment.** Make the graph green first; be clever later.
2. **The mitigation menu, top to bottom:** roll back the deploy (most common fix) β flag it off β scale the
saturated tier β fail over to a healthy replica/region.
3. **Watch the gotchas:** irreversible DB migrations make rollbacks dangerous; scaling can move the bottleneck;
never fail over to an unverified standby.
4. **One incident commander**, coordinating rather than typing, so there's always a single answer to "what are
we doing?"
5. **One comms channel, regular heartbeat updates** ("next update at [time]"), and a **timeline written live** -
not reconstructed from memory.
6. **No silent heroes.** Announce every change before you make it - the quiet fixer is the most dangerous
person on the call.
---
# After: the Blameless Postmortem
The site is back and every instinct says close the laptop and never speak of this again. That instinct is the
single most expensive mistake in the incident - an outage you don't learn from is one you've *prepaid for next
time.*
> **Every outage is tuition. The only question is whether you let it buy something.**
You already paid the cost - stress, downtime, lost sleep. The postmortem is how you collect what you bought:
understanding and prevention. Skip it and you paid full price for nothing. This phase covers running that
process so it's clear, useful, and safe to participate in.
## Start while it's fresh: the timeline
Before any analysis, write down *what happened, in order, with timestamps.* A live timeline kept during the
incident ([Phase 2](02-triage-and-mitigate.md)) is mostly assembled already. Otherwise, reconstruct it today
while memories are warm - every day you wait, details blur and order scrambles.
A good timeline is purely factual - no blame, no analysis yet, just events:
```text
13:55 release v2.32.0 merged and auto-deployed to prod
14:01 v2.32.0 finished rolling out to all pods
14:03 error-rate alert fires (checkout 500s)
14:05 incident declared, IC assigned
14:08 v2.32.0 identified as prime suspect (timing)
14:12 rollback initiated
14:17 checkout recovered - user impact ends
14:40 root cause confirmed: null-pointer on missing promo field
```
From this you can read the two numbers that measure your response:
- **Time to detect** - incident start to alert firing (here, ~2 minutes: 14:01 β 14:03).
- **Time to mitigate** - declared to user-impact-ended (here, ~12 minutes: 14:05 β 14:17).
π **Terminology.** **MTTR** - *mean time to recovery/resolve* - is the average recovery time across many
incidents. One incident gives a data point; MTTR is the trend. Read these off the real timeline, don't
fabricate or eyeball them.
π‘ **Key point.** These numbers show *where to invest*: slow to detect means an alerting gap, fast to detect
but slow to mitigate means a tooling or runbook gap. The timeline diagnoses your *response*, not just the bug.
## Root cause vs. contributing factors
The seductive trap of a postmortem is hunting for *the one cause* - a line of code, a person, a bad command -
and going home. Real outages almost never have one cause; they have a *chain* of individually-survivable
things that lined up to let the failure through. Separate the two:
- **Root cause** - the technical trigger. *"A null-pointer exception when an order arrived without a promo-code
field."* True, but incomplete by itself.
- **Contributing factors** - everything that let that trigger become a customer-facing outage: no canary
stage, no test for the missing-field case, no alert until customers were already failing, and a rollback that
took twelve minutes because nobody had practiced it.
```text
ROOT CAUSE βββΊ the technical trigger that fired
(null on promo field)
CONTRIBUTING FACTORS βββΊ why the trigger reached users
β’ no canary / staged rollout caught it
β’ no test covered the missing-field case
β’ no alert until customers were already failing
β’ rollback was slow (never rehearsed)
The lesson lives in the contributing factors,
because that's where you have the most leverage to prevent the NEXT,
different, outage.
```
π‘ **Key point.** The contributing factors hold the *real* value: fixing the null-pointer prevents that exact
bug, but fixing "no canary stage" prevents a whole *class* of outages you haven't hit yet. Push past "what
broke?" to "why did it reach our users?"
## Blameless: systems fail, not people
> **A blameless postmortem treats the failure as a property of the system, not a fault of a person.**
π **Terminology.** *Blameless* doesn't mean "no accountability" or "nobody made a mistake" - it means the
postmortem fixes the *system that allowed the mistake to cause an outage*, not identify a human to punish. The
question is never "who screwed up?" but "why did our system let a normal human error become a customer-facing
outage?"
This isn't just kindness - it's practical: **blame destroys the information you need to prevent the next
outage.** In a culture that hunts for someone to fire:
- People hide what they actually did, so your timeline is fiction and you can't learn from it.
- People stop volunteering for risky-but-important work, so the most fragile systems get the least attention.
- The deepest, most useful insights - "I didn't understand what that flag did" - never get spoken,
because saying them is dangerous.
A blameless culture is what makes people tell the truth, the only raw material a postmortem has. **Punish
truth-telling and get silence; reward it and get the information that prevents the next outage.**
**In practice:** "Maria deployed the bad code" is blame and teaches nothing - anyone could deploy that code
tomorrow. "Our pipeline let a change with no test for a common input go to 100% of production with no canary
and no fast alert" is blameless and *actionable.* Same event, different hunt: one for a culprit, one for a
fix. The person who pushed the button is almost never the cause - just the last visible step in a chain the
system should have caught.
πͺ **War story.** The most psychologically safe team I've seen had a ritual: whoever was "closest to" an
incident often *volunteered* to write the postmortem, and the group reframed any "I messed up" into "what
about the system made that mistake so easy, and so costly?" New engineers, braced to be blamed, were stunned
to get helped instead. That team shipped faster *because* people weren't afraid - fear makes people slow,
defensive, and quiet, and quiet is fatal to learning.
β οΈ **Watch for blame in disguise.** "Why didn't you test it?" is blame wearing a process costume; "what would
have made it easy to catch this in testing?" is genuinely blameless. Same concern, different result - one
makes the person defend themselves, the other makes the team think about the system. Listen for the accusatory
"you" and reframe toward the system.
## Turn the incident into prevention
A postmortem that ends in understanding but no *changes* is a diary entry, not an investment. What matters is
a short list of **action items**, concrete, owned, and tracked like any other work:
- **Specific and verifiable** - "add a canary stage that holds at 5% for 10 minutes before full rollout," not
"be more careful with deploys."
- **Owned** - a named person, not "the team" (which means no one).
- **Tracked** - a real ticket with a due date, reviewed like any other work, not a bullet in a doc nobody
reopens.
The strongest action items remove the *contributing factors*, since each defuses a whole class of future
outages. They fall into three families:
```mermaid
flowchart TD
w["weakest: 'be more careful'
relies on humans not being human. Don't."] --> a["better alert / runbook
catch it faster, recover faster"]
a --> t["test / canary / staged
catch this class of bug BEFORE users do"]
t --> g["strongest: guardrail
make the failure structurally impossible"]
```
- **Better alerts** - if you were slow to detect, add or tune an alert so next time you know in seconds, not
from customers. (Detection gaps come straight off your "time to detect" number.)
- **Tests & canaries** - a test for the input that broke; a canary/staged rollout so a bad deploy hits 5% of
traffic, trips an alert, and auto-rolls-back before it reaches everyone.
- **Guardrails** - the strongest of all: make the failure *structurally impossible*. A schema constraint that
rejects the bad data, a type that can't be null, a deploy gate that blocks releases without canary coverage -
it beats "remember to be careful" because it doesn't depend on anyone remembering.
π‘ **Key point.** Prefer guardrails over vigilance. "We'll remember to check this next time" is the weakest
action item - it relies on humans being more reliable than they are. Convert "remember toβ¦" into "the system
won't let youβ¦" and one outage becomes permanent protection.
β οΈ **The graveyard of good intentions.** The most common postmortem failure isn't bad analysis - it's great
analysis whose action items never get done. Unowned, undated, untracked items quietly die, and six months later
the *same outage* recurs. Put them in the same backlog as your feature work, with owners and dates, and review
them. An action item that isn't tracked didn't happen.
## Every outage is tuition - make it buy something
Look at the arc of this guide: calm in the first five minutes, bleeding stopped before diagnosing, the
incident run with one coordinator, clear comms, and a live timeline. Now you close the loop - a clear-eyed,
blameless postmortem that converts the wreckage into prevention.
That last step compounds. A team that does this turns each outage into permanent improvements - better
alerts, real tests, structural guardrails - so the system gets *more* resilient and incidents get rarer and
shorter. A team that skips it pays the same tuition over and over, reliving the same outage with different
dates.
You already paid for the lesson - in stress, downtime, sleep. The postmortem is how you collect what you
bought. Don't leave it on the table.
> **Make the outage buy something. That's the difference between a team that gets paged at 2am forever and a
> team that, slowly, stops.**
## Recap
1. **Timeline first, while it's fresh** - purely factual, with timestamps; read your *time to detect* and *time
to mitigate* off it (and don't fabricate the numbers).
2. **Root cause vs. contributing factors** - the trigger is rarely the whole story; the leverage lives in the
factors that let the trigger reach users.
3. **Blameless means systems fail, not people** - not out of niceness but because blame destroys the clear
information a postmortem runs on. Punish candor, get silence.
4. **Watch for blame in disguise** - reframe "why didn't you�" into "what would have made this easy to catch?"
5. **Turn it into prevention** - specific, owned, tracked action items; prefer guardrails (make it impossible)
over vigilance (remember to be careful).
6. **Every outage is tuition.** You've already paid. The postmortem is how you collect what it bought.
Watch it animated: [root-cause analysis](/explainers/RootCause.dc.html)