# Your First On-Call > What it's actually like to carry the pager for the first time: getting set up before your shift starts, staying calm through the 3am page, and turning the aftermath into a blameless postmortem instead of a blame session. --- # Your First On-Call The first time you carry the pager, the fear isn't the outage. It's not knowing what an outage even looks like yet, and worrying you'll freeze, page the wrong person, or somehow make it worse. That fear is normal, and it goes away fast once you know three things: what to check before your shift starts, what to actually do when the phone buzzes at 3am, and what happens afterward. This guide is about the human side of on-call - the setup, the nerves, the etiquette, the way a bad night turns into a team asset instead of a personal scar. For the technical playbook of diagnosing and fixing a live outage, see [When Prod Is Down](/guides/when-prod-is-down) - that guide is the deep dive on triage and mitigation; this one is about surviving your first rotation as a human being. ## The phases 1. **[Before Your First Shift](01-before-your-first-shift.md)** - what to verify before you're on the hook: alerts actually reach you, you know where the runbooks live (and that missing ones are normal), and you know exactly who to call at 3am if you're stuck. 2. **[The 3am Page: A Calm Playbook](02-the-3am-page-a-calm-playbook.md)** - triage before you fix anything, stop the bleeding before you understand it, and how to escalate without feeling like you failed. 3. **[After the Fire: the Postmortem](03-after-the-fire-the-postmortem.md)** - why blameless postmortems work better than finding someone to blame, why "human error" is rarely the real story, and writing up what happened so the next on-call engineer has it easier than you did. Read them in order before your first shift. If you're already on-call and the pager is going off right now, jump straight to Phase 2. --- # Before Your First Shift Most on-call anxiety comes from the unknowns you could have checked beforehand. Do it now, while there's no pressure, not during your first page. ## Make sure the alert can actually reach you An alert that fires into the void is worse than no alert - the team thinks someone's on it, and no one is. Before your shift starts: - **Send yourself a test page.** Most tools (PagerDuty, Opsgenie, VictorOps) have a "test notification" button. Use it. Confirm it hits your phone, not only your laptop that's closed in a bag by 11pm. - **Check the escalation delay.** If you don't acknowledge within N minutes, who gets paged next? Know that number - it tells you how much slack you have to wake up and open your laptop. - **Turn off Do Not Disturb overrides that block it.** iOS and Android both let critical apps bypass silent mode, but only if you've granted that permission. Check it, don't assume it. - **Test at low volume, not zero.** A silent phone under a pillow doesn't wake anyone. Set the alert tone loud enough to survive being in another room. A missed page because a phone setting was wrong is a process failure, not a personal one - but it's one you can eliminate in five minutes tonight instead of learning about it during an actual outage. ## Find the runbooks - and expect gaps A runbook is a written procedure for a known problem: "if the checkout queue backs up, do X, then Y." Good teams keep them in a wiki, a repo folder, or pinned in the on-call tool itself. Find that location before your shift, not during it. Then expect this: **for a lot of what pages you, there will be no runbook.** New failure modes don't get a runbook until after they've happened once. That's not a gap in your training - it's a gap in the system's history, and it's completely normal on any team, including ones that have been running for years. 📝 **What this means in practice.** If you're paged for something with no documented fix, you're not failing by not knowing it instantly. You're doing exactly what the first responder to a new problem is supposed to do: investigate, mitigate, and (per [Phase 3](03-after-the-fire-the-postmortem.md)) write the runbook that didn't exist yet, so the next person has it. Skim the runbooks that do exist even if none apply to your service tonight - you'll absorb what "on-call documentation" looks like at your company, which makes writing your own during a real incident far less intimidating. ## Know your escalation path cold "Escalation path" means: when you're stuck, who do you call, in what order, and how. Before your shift, know the answers to these without having to look them up: - **Who's the secondary/backup on-call?** There should always be a second person for when you're unreachable or out of your depth. - **How do you actually reach them at 3am?** A Slack DM they won't see until morning doesn't count. Know the tool's "escalate" button, or the phone number, whichever your team actually uses. - **Who's the "break glass" contact** - the senior engineer or manager you page when it's bad enough to need someone with more context than the backup on-call has? - **Is there a status page or customer-comms process**, and whose job is it to update it? You don't want to be figuring that out mid-incident. Write these down somewhere you can find fast and half-asleep - a pinned note, a bookmark, not a memory you're trusting to survive an adrenaline spike. ## The mindset shift The goal of this prep isn't to make you capable of fixing everything solo. It's to remove every excuse for hesitating to ask for help. A new on-call engineer who escalates fast and often is doing it right. One who sits alone at 3am afraid to wake someone up, because they never checked who that someone was, is the failure mode this phase exists to prevent. Ready for the page itself? [When Prod Is Down](/guides/when-prod-is-down) covers the technical triage steps in depth - Phase 2 here covers the calmer, human version of the same moment. Quick check before you move on: ```quiz [ { "q": "You're paged for a failure mode with no existing runbook. What does that mean?", "choices": [ "You should have known the fix already", "This is normal - write the runbook after, for the next person", "The on-call process is broken and needs to be escalated immediately" ], "answer": 1, "explain": "Runbooks get written after a failure happens once, not before. A missing runbook is a gap in the system's history, not a personal shortcoming." }, { "q": "What's the best way to verify your alerts will actually wake you up?", "choices": [ "Assume the on-call tool handles it correctly", "Send yourself a real test page before your shift starts", "Check that your laptop is charged" ], "answer": 1, "explain": "A test page confirms the full path - tool to phone to a sound loud enough to wake you - instead of hoping it works." }, { "q": "Why write down your escalation path in advance instead of looking it up during an incident?", "choices": [ "It's required by most incident tools", "Adrenaline and half-asleep brains are bad at finding things quickly", "It's only needed for very large outages" ], "answer": 1, "explain": "The whole point of prep is removing lookups from the moment you're least equipped to do them." } ] ``` --- # The 3am Page: A Calm Playbook Your phone buzzes. You're awake, heart already going, before you've read a single word. That physical jolt is normal - everyone gets it, even the engineer who's been doing this for a decade. What separates a calm response from a panicked one isn't the absence of adrenaline. It's having a sequence to follow so the adrenaline doesn't have to make decisions for you. This phase is that sequence. For the technical mechanics of diagnosing and fixing what's actually broken, [When Prod Is Down](/guides/when-prod-is-down) goes deep on triage and mitigation. Here, the focus is the order of operations and the headspace that makes following it possible at 3am. ## Step 1: Triage before you touch anything Before you fix, before you even fully understand - answer three questions: - **Is this actually urgent?** Some alerts are noisy or low-stakes and can wait for morning. Check the alert's severity and what it's actually measuring before assuming the worst. - **Who's affected, and how many?** All users, or one customer's edge case? Payment flow, or a background job nobody's watching tonight? This determines how fast you need to move and who else needs to know. - **How bad is "bad"?** Fully down is different from slow. Data loss risk is different from a cosmetic bug. Match your urgency to the actual severity, not to how scared you feel. Five minutes spent triaging is never wasted, even on something that turns out to be nothing. Rushing straight to "fix it" without knowing what "it" is wastes far more time than it saves. ## Step 2: Stop the bleeding, not the mystery The instinct at 3am is to understand *why* before you act. Resist it. Your job in the first stretch of an incident is to reduce user pain, not to solve the puzzle. That usually means one of: roll back the last deploy, restart the failing service, fail over to a healthy replica, or turn off the feature flag behind the bad behavior. None of these require knowing the root cause - they remove the most likely culprit from production. [When Prod Is Down](/guides/when-prod-is-down) walks through each of these mitigations in detail, with the tradeoffs of each. 💡 **The mantra worth memorizing.** *Mitigate first, root-cause later.* You are not being paid at 3am to be clever. You're being paged to make the graph green. The clever part - understanding exactly what broke and why - can happen in daylight, with coffee, and other people awake to help. ## Step 3: Know when to stop struggling alone This is the part beginners get wrong most often, almost always in the same direction: they wait too long to escalate because asking for help feels like admitting failure. It isn't. Escalating is the job working correctly. Here's the real test: - **You've spent 15-20 minutes and don't have a mitigation in sight.** That's long enough. Escalate. - **The blast radius is bigger than you expected once you looked closer.** Get someone with more context on the call - don't quietly absorb a bigger incident than you signed up for. - **You're about to do something you're not fully sure about** (a database change, a forced failover, anything hard to undo). A second opinion costs two minutes and prevents the incidents that turn into legendary war stories for the wrong reasons. - **You don't know what's happening, full stop.** Not knowing isn't the failure. Sitting with "I don't know" alone for an hour, when someone else could've told you in five minutes, is. 🪖 **The math that should convince you.** Waking up a senior engineer costs them twenty minutes of disrupted sleep. An extra ninety minutes of a payment outage while you quietly try to figure it out solo costs the company real money and costs you a much worse morning. The senior engineer would rather be woken up. Every time. ## What escalating actually sounds like It's not a distress signal, it's a normal status update: *"Hey, I'm on-call for checkout, seeing 500s since 2:47, tried a rollback and it didn't help, I think I need another set of eyes - can you hop on?"* That's it. No apology required, no explanation of why you couldn't solve it yourself. You're doing exactly what on-call is designed for: contain what you can, pull in help for what you can't. Once things are stable, don't let the story of the night end at "and then it was fixed." What you learned in that window - the fix, the near-misses, the runbook that didn't exist - becomes the material for the next phase. ## Your turn: it's 2:47am and you're alone with it Reading "escalate at 15-20 minutes" is easy in daylight. Deciding it live, on your first page, when asking for help feels like admitting you couldn't handle it, is the actual job. There is no single right answer below and nothing is scored right or wrong - but the clock is real, and every minute on it is a minute you carry this by yourself. ```scenario { "title": "2:47am - checkout is down and you're alone", "brief": "First rotation, first real page. It's 2:47am, checkout is throwing 500s on the payment flow, and nobody else on the team knows yet. Every minute you spend deciding is a minute you're carrying this by yourself.", "prompt": "What do you do first?", "clock": { "unit": "min", "running": "carrying it alone", "resolved": "you're not alone anymore" }, "resolvedHeading": "Help is on the way. Here's how it went.", "actions": [ { "id": "triage", "label": "Check severity: who's affected, how bad, is it urgent", "cost": 3, "reveals": "checkout-api error rate: 94% (500s)\naffected: all logged-in users, payment flow\nstatus: fully down, not degraded", "note": "Three minutes well spent. You now know this is real, it's the payment flow, and it's fully down - that's what decides how fast you move next." }, { "id": "rollback", "label": "Roll back the last deploy", "cost": 5, "reveals": "$ kubectl rollout undo deployment/checkout-api\ndeployment.apps/checkout-api rolled back\n$ curl -s -o /dev/null -w \"%{http_code}\\n\" https://api.example.com/health\n500", "note": "Sensible first move. It just didn't work, and you're several minutes further into this." }, { "id": "restart", "label": "Restart the checkout service, in case it's stuck", "cost": 5, "reveals": "deployment.apps/checkout-api restarted\n$ curl -s -o /dev/null -w \"%{http_code}\\n\" https://api.example.com/health\n500", "note": "Also reasonable, also no change. Two real attempts, still down." }, { "id": "logs", "label": "Dig into the stack traces yourself until you find it", "cost": 12, "reveals": "checkout-api ERROR NullPointerException: paymentToken.expiry is null\n at PaymentService.charge(PaymentService.java:112)\n ... 3,800 more in the last 90s", "note": "You found something real - a null token expiry - and it is genuinely interesting. You found it alone, twelve minutes in, on your first-ever page, before anyone else knew you were struggling." }, { "id": "one-more-try", "label": "Try one more fix before you wake anyone up", "cost": 10, "reveals": "$ kubectl scale deployment/checkout-api --replicas=6\ndeployment.apps/checkout-api scaled\n$ curl -s -o /dev/null -w \"%{http_code}\\n\" https://api.example.com/health\n500", "note": "It didn't work. You're now well past the point where a second opinion would have been cheap." }, { "id": "wait-and-watch", "label": "Wait a few minutes to see if it recovers on its own", "cost": 6, "reveals": "checkout-api error rate: 94% (500s)\nno change", "note": "Nothing changed except the clock." }, { "id": "escalate", "label": "Page the secondary on-call for a second set of eyes", "cost": 2, "resolves": true, "reveals": "you: Hey, I'm on-call for checkout, seeing 500s since 2:47, tried a rollback and a restart, neither helped - I think I need another set of eyes, can you hop on?\nmaya: on my way, give me 2 min\nmaya: joined the call", "note": "Two minutes, and you're no longer the only person who knows this is happening." } ], "debrief": { "ideal": 10, "text": "The win here isn't the fix - it's not carrying this alone. Triage told you it was real and it was bad, one real mitigation attempt told you it wasn't a quick one, and paging your secondary at that point is the process working, not you failing. Root cause can wait for daylight and a second set of eyes.", "notes": [ { "when": "if-taken", "action": "logs", "text": "You found something real - a null payment token expiry - and it's worth knowing. But you found it alone, twelve minutes into your first-ever page, before anyone else knew you were struggling. Interesting isn't the same as urgent." }, { "when": "if-taken", "action": "one-more-try", "text": "This is the trap the guide names directly: not escalating because you feel like you should be able to handle it. One more idea, alone, past the point where a second opinion would have cost two minutes and saved ten." }, { "when": "if-taken", "action": "wait-and-watch", "text": "Waiting cost six minutes and taught you nothing you didn't already know. It isn't triage and it isn't mitigation - it's the appearance of doing something." }, { "when": "if-not-taken", "action": "triage", "text": "You skipped straight to fixing without knowing what you were fixing or how bad it was. It worked out this time - but you were guessing at urgency instead of knowing it." } ] } } ``` Quick check before you move on: ```quiz [ { "q": "During an incident, what should you generally do first?", "choices": [ "Find the root cause before making any changes", "Stop the bleeding with a fast mitigation, then investigate the cause", "Wait for a senior engineer to take over" ], "answer": 1, "explain": "Mitigation (rollback, restart, failover, flag-off) reduces user pain fast without requiring you to understand the root cause yet." }, { "q": "You've been stuck for 20 minutes with no mitigation working. What should you do?", "choices": [ "Keep trying alone until you find it, escalating looks like failure", "Escalate - this is exactly what the escalation path is for", "Wait until morning to ask for help" ], "answer": 1, "explain": "Escalating after a reasonable attempt is the process working correctly, not a personal failure." }, { "q": "Why triage (urgency, blast radius, severity) before fixing anything?", "choices": [ "It's a formality most teams skip", "It ensures your response matches the actual severity instead of your adrenaline level", "It's only necessary for outages affecting all users" ], "answer": 1, "explain": "Triage keeps you from either overreacting to something minor or underreacting to something severe." } ] ``` --- # After the Fire: the Postmortem The outage is over, the graphs are green, and you finally get to sleep. The instinct is to close the laptop and never think about it again. Don't - the postmortem is where a bad night turns into something the whole team gets to benefit from, including future-you on your next rotation. ## Blameless, not because it's nice - because it works A blameless postmortem assumes everyone involved did a reasonable thing given what they knew at the time. It asks "what let this happen?" instead of "who caused this?" That's not a soft HR preference - it's the approach that actually surfaces the truth. Here's why. If postmortems hunt for someone to blame, people learn fast to protect themselves instead of the process: they leave out the part where they almost pushed a bad config, they downplay how confused they were, they stay quiet about the workaround they didn't fully understand. None of that is deceit for its own sake - it's a rational response to a process that punishes candor. Take away the blame, and the same person tells you exactly what happened, including the embarrassing parts, because nothing bad happens to them for saying it. Those embarrassing parts are usually where the real fix lives. ## "Human error" is where investigations stop too early "An engineer ran the wrong command" or "someone forgot to update the config" reads like a root cause. It isn't - it's where a lazy investigation stops. People don't fail randomly; they fail in exactly the conditions a system sets up for that failure. Push one level further, every time: - "An engineer ran the wrong command" → *why was it possible to run that command against production without a confirmation step or a safeguard?* - "Someone forgot to update the config" → *why did updating it depend on one person's memory instead of being automated or checked in CI?* - "The on-call engineer didn't know about the edge case" → *why did that knowledge live only in one senior engineer's head instead of a runbook?* Each real root cause is a process or system gap that made the human error possible, or made it matter once it happened. Fix the gap, and the *next* person - who will absolutely also be human and also capable of a typo - doesn't cause the same outage. 📝 **Key point.** "Human error" is a symptom description, not a diagnosis. If a postmortem's action item is "be more careful," the investigation stopped one question too early. ## Writing it up for the next on-call engineer You're not writing this for your manager or for a compliance checkbox. You're writing it for the version of someone else - maybe you in six months - who gets paged for the same thing at 3am and needs it to go faster than it did for you. Good postmortems are short and specific: - **Timeline.** What happened, in order, with real timestamps. Pull it from the incident channel if you wrote one live - don't reconstruct from memory, adrenaline scrambles that. - **Impact.** Who was affected, for how long, and how badly. Concrete numbers beat vague severity. - **What worked.** The mitigation that actually stopped the bleeding - so it becomes the new runbook entry. - **Contributing factors, not a villain.** The process or system gaps that let this happen, from the "human error" digging above. - **Action items with owners.** Vague resolutions ("improve monitoring") don't get done. Specific ones ("add an alert on queue depth > 5,000, owner: Priya, due Friday") do. If there was no runbook for what paged you, this is where you write the one that didn't exist before - the gap you found in [Phase 1](01-before-your-first-shift.md) is exactly the gap this document closes. ## The upside nobody mentions A rough first on-call shift feels like a personal ordeal in the moment. Written up clearly, it becomes institutional memory - the next new hire on this rotation gets a runbook you didn't have, an alert that didn't exist before your outage, or a documented escalation contact instead of a guess. That's the actual payoff of on-call: not that you survived it, but that the team is measurably better prepared because you did. Quick check before you finish: ```quiz [ { "q": "Why do blameless postmortems tend to produce more accurate accounts of what happened?", "choices": [ "They make the incident feel less serious", "People share embarrassing but important details when they aren't at risk of blame", "They skip the timeline to save time" ], "answer": 1, "explain": "Removing blame removes the incentive to hide the details, which are often exactly where the real cause lives." }, { "q": "Why is 'human error' usually not the true root cause?", "choices": [ "Humans never actually make mistakes", "It's typically a symptom of a system or process gap that let the error matter", "It's always a training problem" ], "answer": 1, "explain": "The deeper question is why the system allowed that mistake to happen or to cause damage - that gap is the fixable root cause." }, { "q": "Who is the primary audience for a postmortem write-up?", "choices": [ "Whoever caused the incident, as a record of what went wrong", "The next on-call engineer who might face the same problem", "No one - it's mostly a formality" ], "answer": 1, "explain": "A good postmortem is documentation aimed at making the next person's response faster and easier, not a report card." } ] ``` ## Where to go next You've made it through a first on-call rotation, from setup to a 3am page to writing it up afterward. That same instinct for staying calm under pressure and reasoning clearly about what happened, shows up again in a different high-stakes setting: [Technical Interviews](/guides/technical-interviews) - another situation where the pressure feels bigger than the actual problem in front of you.