When Prod Is Down: Staying Calm

How to handle a production outage without panicking: assess the blast radius, stop the bleeding before you diagnose, run the incident with clear comms, and turn the wreckage into a blameless postmortem that prevents the next one.

1. The First Five Minutes When prod is down, don't panic and don't start randomly changing things: assess the blast radius (who and what is affected, how bad), then stop the bleeding before you diagnose. Restore service first, understand later.
2. Triage & Mitigate The fastest paths back to green during an outage - roll back the last deploy, flip the feature flag off, scale up, fail over - and how to actually run the incident: one coordinator, a clear comms channel, and a timeline written as you go. Mitigation over root cause, in the moment.
3. After: the Blameless Postmortem Once the outage is over: build the timeline, separate root cause from contributing factors, protect the blameless rule (systems fail, not people), and turn the incident into prevention with concrete action items, alerts, and guardrails. Every outage is tuition - make it buy something.