The First Five Minutes

The first five minutes of an outage decide how the next two hours go - this is when panic does its damage: typing commands you can't explain, restarting things at random, freezing at a dashboard hoping it turns green. The site being down is bad; you making it worse by acting faster than you can think is the danger you control. Here's exactly what to do, in order - use the card while it's happening, read the rest when you can.

The PROD-DOWN CHECKLIST

Alert just fired? Don't touch anything yet. Do these, in order.

Don't panic, and don't start randomly changing things

The alert dumps adrenaline into your system - useful for running from a predator, terrible for debugging. It makes "do something" feel like progress even when it's harmful; the urge to type is chemical, not a plan. Minute one isn't for fixing the outage, it's for not making it worse - the fix comes after.

The cardinal rule: don't change anything you can't explain and can't undo. Restarting a service you don't understand, clearing a cache "just to see," bouncing a database - these feel like action, but might turn a degraded service into a dead one, or destroy the evidence you need.

⚠️ The "let me try a few things" trap. The most expensive outages are the ones where someone fixed the real problem in minute two but kept tinkering, and the third "let me try this" broke something new. Once service is restored, stop touching it.

🪖 War story. A teammate paged for high latency once restarted the app servers, flushed the cache, and failed over the database all at once, "to be safe." Latency went away, but so did the ability to tell which change mattered - the cache flush caused a thundering herd that took the site down ten more minutes. One at a time, observed, is faster and safer.

Confirm it's real before you respond to it

Plenty of "outages" are the responder's own VPN, an expired cert, a broken monitoring check, or one bad node the load balancer already routes around - fixing a problem that was never there is its own outage. Confirm fast, from the user's angle, not your laptop's:

$ curl -s -o /dev/null -w "%{http_code} %{time_total}s\n" https://api.example.com/health
503 0.42s
$ curl -s -o /dev/null -w "%{http_code} %{time_total}s\n" https://api.example.com/health
503 0.39s

What just happened: Hitting the health endpoint directly, twice, from outside your own environment returned 503 both times in under half a second - consistent failures from a clean path, almost certainly real and server-side. Two 200s instead would point at your own connection or a single bad instance.

💡 Key point. "Is it real?" and "is it everywhere?" are different questions - you want both answered. A 503 from one region but 200s from another is a smaller incident than a 503 everywhere.

Assess the blast radius

The most important judgment call in the first five minutes: it sets the size of your response. You wouldn't wake the whole on-call tree for a cosmetic bug in a beta feature, but you would for "no one can check out."

📝 Terminology. Blast radius, from explosives, means how far the damage reaches: who's affected, what they can't do, how widely - real-world impact, not technical symptom. "A queue is backed up" is a symptom; "orders aren't being confirmed" is blast radius. Three questions, in plain language:

   ┌─────────────────────────────────────────────────────────────┐
   │  BLAST RADIUS  =  WHO  ×  WHAT  ×  HOW BAD                     │
   ├─────────────────────────────────────────────────────────────┤
   │  WHO     → all users? a region? logged-in only? one customer? │
   │  WHAT    → core flow (checkout, login) or an edge feature?    │
   │  HOW BAD → fully broken, degraded/slow, or cosmetic?          │
   └─────────────────────────────────────────────────────────────┘
        big radius  → act fast, pull in people, accept bolder fixes
        small radius → calmer pace, fewer people, careful fixes

The combination, not any single answer, sets the severity: "slow" for "everyone" on "checkout" is an all-hands emergency; "fully broken" for "one internal admin report" is a ticket for tomorrow. Say it out loud in the channel so everyone shares the same picture:

"Blast radius: all logged-in users, checkout returns 500, started ~14:03. This is customer-facing and revenue-impacting - treating as high severity."

⚠️ Don't under-call it to avoid the fuss. Resist labeling things minor to skip waking people - it's far cheaper to spin down an over-declared incident ("false alarm, go back to bed") than discover an hour later the "minor" thing was quietly losing orders. When unsure, round up.

Note the start time and the last change

Two facts, written down in the first minute, shape the whole investigation: when did it start and what changed right before. The overwhelmingly common cause of a sudden outage is a recent change - a deploy, a config flip, a feature flag, a migration, an infra change. Graph turned red at 14:03, deploy went out at 14:01? Prime suspect, before reading a single log line.

$ kubectl rollout history deployment/checkout-api
REVISION  CHANGE-CAUSE
6         release v2.31.0
7         release v2.32.0      # shipped 14:01, two minutes before the alerts

What just happened: You checked the deployment's recent revisions and saw v2.32.0 rolled out at 14:01 - moments before the 14:03 alert. That timing doesn't prove the deploy caused it, but makes it the first thing to mitigate. Correlation is a lead, not a verdict - and that's exactly what you want right now.

💡 Key point. "What changed?" beats "what's wrong?" as your opening question - what's wrong can take an hour to understand, what changed you can often answer in thirty seconds, and changing it back is often the whole fix.

Stop the bleeding - restore service first, understand later

The mindset that separates calm responders from heroes-who-make-it-worse:

Restore service first. Understand later.

When you cut yourself badly, you apply pressure before investigating which blood vessel you nicked. Same here: if you can get users working again - rolling back, flipping a flag, failing over - do that now, even without understanding root cause. A restored service buys you the most valuable thing in an incident: time to think clearly instead of under fire.

Engineers love understanding first - correct instinct for a normal bug, dangerous during an outage. Every minute spent on root cause while users are broken is impact you could have stopped. Mitigation and diagnosis are separate jobs, and in the first five minutes you only have one: mitigation.

The whole game in the early phase is shrinking that shaded gap - the time users are hurting - with mitigation, not understanding. Phase 2 is the menu for stopping the bleeding.

🪖 War story. A senior engineer once rolled back a deploy within four minutes, the site came back, and then she said: "Okay, it's stable. Now let's figure out what that deploy did." Nobody had to debug a live fire. The bug was subtle and took an hour to fully understand - an hour that, thanks to the rollback, cost zero customer impact.

⏭️ Once service is breathing again and you've got room to investigate, the skills that carry you are log-reading and stack-trace reading. See Reading Logs Without Drowning and Reading a Stack Trace for the calm-investigation half of the job.

Your turn: it's 14:03 and checkout is down

Reading the checklist is the easy part. Doing it while a clock runs is the job. There is no single right answer below and nothing is scored right or wrong - but the clock is real, and every minute you spend is a minute nobody can check out. Restore service, then read the debrief.

{
  "title": "14:03 - checkout is returning 500s",
  "brief": "You're on call. The alert fired sixty seconds ago: checkout is throwing 500s. You do not know why. Nothing else has been touched yet, and every minute on this clock is a minute customers cannot buy anything.",
  "actions": [
    {
      "id": "curl",
      "label": "Hit the health endpoint from outside, twice",
      "minutes": 1,
      "reveals": "$ curl -s -o /dev/null -w \"%{http_code} %{time_total}s\\n\" https://api.example.com/health\n503 0.42s\n$ curl -s -o /dev/null -w \"%{http_code} %{time_total}s\\n\" https://api.example.com/health\n503 0.39s",
      "note": "Consistent 503s from a clean path, in under half a second. It's real, it's server-side, and it isn't your VPN."
    },
    {
      "id": "deploys",
      "label": "Check what shipped recently",
      "minutes": 1,
      "reveals": "$ kubectl rollout history deployment/checkout-api\nREVISION  CHANGE-CAUSE\n6         release v2.31.0\n7         release v2.32.0      # shipped 14:01",
      "note": "v2.32.0 went out at 14:01. The alert fired at 14:03. That is not proof, but it is the prime suspect, and you got it in under a minute."
    },
    {
      "id": "declare",
      "label": "Declare the incident in the channel",
      "minutes": 1,
      "reveals": "you: Declaring an incident: checkout 500s since ~14:03, all users. I'm coordinating.\nrota-bot: paged @secondary\nmaya: on it - I shipped v2.32.0 at 14:01, want me to prep a rollback?",
      "note": "Costs a minute, buys you a second pair of hands and a record. Notice who answered, and what she just told you."
    },
    {
      "id": "logs",
      "label": "Tail the application logs and read the stack traces",
      "minutes": 4,
      "reveals": "checkout-api  ERROR  NullPointerException: cart.discount is null\n  at CheckoutService.total(CheckoutService.java:88)\n  ... 2,140 more in the last 60s",
      "note": "Genuinely interesting: a null discount is blowing up the total. It is also still true after you mitigate, and it cost four minutes with the site down to learn it."
    },
    {
      "id": "db",
      "label": "Check the database for load or locks",
      "minutes": 4,
      "reveals": "connections: 41/200   slow queries: 0   replication lag: 0.1s\ncpu: 12%",
      "note": "Healthy, and boring. A reasonable hypothesis, ruled out - that is what investigation is. It cost four minutes."
    },
    {
      "id": "restart",
      "label": "Restart the app servers to clear it",
      "minutes": 3,
      "reveals": "deployment.apps/checkout-api restarted\n$ curl -s -o /dev/null -w \"%{http_code}\\n\" https://api.example.com/health\n503",
      "note": "Still 503. You changed something you could not explain on a system you had not diagnosed, and got nothing for it. If it HAD worked you would never know why."
    },
    {
      "id": "rollback",
      "label": "Roll back to v2.31.0",
      "minutes": 2,
      "resolves": true,
      "reveals": "$ kubectl rollout undo deployment/checkout-api\ndeployment.apps/checkout-api rolled back\n$ curl -s -o /dev/null -w \"%{http_code}\\n\" https://api.example.com/health\n200",
      "note": "Green. You still do not know exactly what v2.32.0 did - and you did not need to."
    }
  ],
  "debrief": {
    "idealMinutes": 4,
    "text": "The move that ends this outage is the one you can make before you understand it. Confirm it's real, ask what changed, put it back. Root cause is a question for a calm afternoon with the site up - and the null discount will still be in the logs when you get there.",
    "notes": [
      { "when": "if-taken", "action": "logs", "text": "You read the logs before mitigating. That instinct is correct for a normal bug and expensive during an outage: the stack trace was still waiting for you after the rollback, but those four minutes of downtime were not refundable." },
      { "when": "if-taken", "action": "db", "text": "Checking the database was a sensible hypothesis, and ruling it out is real work. Notice only what it cost: four minutes, while the answer was one line of deploy history away." },
      { "when": "if-taken", "action": "restart", "text": "The restart is this phase's war story in miniature - an action you could not explain, on a system you had not diagnosed. It happened to be harmless. The version that flushes a cache and causes a thundering herd is the same decision with worse luck." },
      { "when": "if-not-taken", "action": "declare", "text": "You never declared the incident. It worked, because you were right and quick. It also meant nobody else knew, nobody checked your thinking, and there is no record - and Maya, who shipped v2.32.0 and could have told you in one line, never got asked." },
      { "when": "if-not-taken", "action": "curl", "text": "You never confirmed it from the user's side. It was real this time. The times it isn't, you have just rolled back production to chase your own VPN." }
    ]
  }
}

Recap

1. Don't panic, don't randomly change things. The adrenaline urge to "do something" is the danger. Change nothing you can't explain and can't undo.
2. Confirm it's real from the user's angle before you respond - not your laptop, VPN, or a broken check.
3. Assess the blast radius - who × what × how bad - and say it out loud. When unsure, round up.
4. Note the start time and the last change. "What changed?" beats "what's wrong?" and is usually the prime suspect.
5. Stop the bleeding before you diagnose. Restore service first, understand later - only one of those is your job in minute one.

← Guide overview · Phase 2: Triage & Mitigate →