The 3am Page: A Calm Playbook

Your phone buzzes. You're awake, heart already going, before you've read a single word. That physical jolt is normal - everyone gets it, even the engineer who's been doing this for a decade. What separates a calm response from a panicked one isn't the absence of adrenaline. It's having a sequence to follow so the adrenaline doesn't have to make decisions for you.

This phase is that sequence. For the technical mechanics of diagnosing and fixing what's actually broken, When Prod Is Down goes deep on triage and mitigation. Here, the focus is the order of operations and the headspace that makes following it possible at 3am.

Step 1: Triage before you touch anything

Before you fix, before you even fully understand - answer three questions:

• Is this actually urgent? Some alerts are noisy or low-stakes and can wait for morning. Check the alert's severity and what it's actually measuring before assuming the worst.
• Who's affected, and how many? All users, or one customer's edge case? Payment flow, or a background job nobody's watching tonight? This determines how fast you need to move and who else needs to know.
• How bad is "bad"? Fully down is different from slow. Data loss risk is different from a cosmetic bug. Match your urgency to the actual severity, not to how scared you feel.

Five minutes spent triaging is never wasted, even on something that turns out to be nothing. Rushing straight to "fix it" without knowing what "it" is wastes far more time than it saves.

Step 2: Stop the bleeding, not the mystery

The instinct at 3am is to understand why before you act. Resist it. Your job in the first stretch of an incident is to reduce user pain, not to solve the puzzle.

That usually means one of: roll back the last deploy, restart the failing service, fail over to a healthy replica, or turn off the feature flag behind the bad behavior. None of these require knowing the root cause

• they remove the most likely culprit from production. When Prod Is Down walks through each of these mitigations in detail, with the tradeoffs of each.

💡 The mantra worth memorizing. Mitigate first, root-cause later. You are not being paid at 3am to be clever. You're being paged to make the graph green. The clever part - understanding exactly what broke and why - can happen in daylight, with coffee, and other people awake to help.

Step 3: Know when to stop struggling alone

This is the part beginners get wrong most often, almost always in the same direction: they wait too long to escalate because asking for help feels like admitting failure.

It isn't. Escalating is the job working correctly. Here's the honest test:

• You've spent 15-20 minutes and don't have a mitigation in sight. That's long enough. Escalate.
• The blast radius is bigger than you expected once you looked closer. Get someone with more context on the call - don't quietly absorb a bigger incident than you signed up for.
• You're about to do something you're not fully sure about (a database change, a forced failover, anything hard to undo). A second opinion costs two minutes and prevents the incidents that turn into legendary war stories for the wrong reasons.
• You don't know what's happening, full stop. Not knowing isn't the failure. Sitting with "I don't know" alone for an hour, when someone else could've told you in five minutes, is.

🪖 The math that should convince you. Waking up a senior engineer costs them twenty minutes of disrupted sleep. An extra ninety minutes of a payment outage while you quietly try to figure it out solo costs the company real money and costs you a much worse morning. The senior engineer would rather be woken up. Every time.

What escalating actually sounds like

It's not a distress signal, it's a normal status update: "Hey, I'm on-call for checkout, seeing 500s since 2:47, tried a rollback and it didn't help, I think I need another set of eyes - can you hop on?" That's it. No apology required, no explanation of why you couldn't solve it yourself. You're doing exactly what on-call is designed for: contain what you can, pull in help for what you can't.

Once things are stable, don't let the story of the night end at "and then it was fixed." What you learned in that window - the fix, the near-misses, the runbook that didn't exist - becomes the material for the next phase.

Quick check before you move on:

[
  {
    "q": "During an incident, what should you generally do first?",
    "choices": [
      "Find the root cause before making any changes",
      "Stop the bleeding with a fast mitigation, then investigate the cause",
      "Wait for a senior engineer to take over"
    ],
    "answer": 1,
    "explain": "Mitigation (rollback, restart, failover, flag-off) reduces user pain fast without requiring you to understand the root cause yet."
  },
  {
    "q": "You've been stuck for 20 minutes with no mitigation working. What should you do?",
    "choices": [
      "Keep trying alone until you find it, escalating looks like failure",
      "Escalate - this is exactly what the escalation path is for",
      "Wait until morning to ask for help"
    ],
    "answer": 1,
    "explain": "Escalating after a reasonable attempt is the process working correctly, not a personal failure."
  },
  {
    "q": "Why triage (urgency, blast radius, severity) before fixing anything?",
    "choices": [
      "It's a formality most teams skip",
      "It ensures your response matches the actual severity instead of your adrenaline level",
      "It's only necessary for outages affecting all users"
    ],
    "answer": 1,
    "explain": "Triage keeps you from either overreacting to something minor or underreacting to something severe."
  }
]

← Phase 1: Before Your First Shift · Guide overview · Phase 3: After the Fire →