The Untested Backup, and Ransomware

You've got the model, the rule, and the numbers. Now the two ways teams with all of that still lose everything — and both are avoidable. The first is quiet and self-inflicted: a backup that was never tested and turns out not to work. The second is loud and adversarial: ransomware that reaches for your backups on purpose. This is the phase that turns a backup plan into something that actually saves you.

The untested backup is the classic trap

Here is the most expensive sentence in this entire topic: a backup you have never restored from is not a backup — it's a hope.

It feels paranoid until you've lived it. Backup jobs fail silently all the time: a path changed, a credential expired, a disk filled, a config typo meant the job dumped an empty file every night for six months. The job kept reporting "success" because it ran without crashing — it wasn't backing up anything. Nobody noticed, because nobody ever tried to use the output. The dashboard was green the entire time.

What the dashboard shows:   backup job: ✅ SUCCESS (ran every night)
What's actually in the file: 0 rows. Empty dump. Six months of nothing.
When you find out:          during the disaster, when you go to restore.

What just happened: "the job ran" and "the backup works" are different claims, and only one of them gets checked by default. The gap between them is where companies die. A successful run proves the script executed; it proves nothing about whether the bytes it produced can rebuild your system.

The only proof: the restore drill

There is exactly one way to know a backup works: restore from it and check the result. Not read the logs. Not confirm the file exists and has a plausible size. Actually pull the backup into a clean environment, bring the data up, and verify it's real.

# the restore drill, in spirit:
# 1. take last night's backup (the real artifact you'd use in a disaster)
# 2. restore it into a fresh, isolated environment
createdb restore_test
gunzip -c mydb-2026-06-29.sql.gz | psql restore_test

# 3. verify it's actually your data, not an empty shell
psql restore_test -c "SELECT count(*) FROM orders;"   # expect a real, sane number
psql restore_test -c "SELECT max(created_at) FROM orders;"  # expect ~last night

What just happened: you proved the backup is restorable and recent, in a sandbox, on a calm Tuesday — not at 3am during a real outage. The row count catches the empty-dump failure; the latest timestamp catches a stale or wrong-source backup. Do this on a schedule (a quarterly drill at minimum, automated if you can), and treat a failed drill as a real incident, because it is one — you've discovered you were unprotected.

A restore drill also quietly validates your RTO: time the drill. If "promise: back in 4 hours" but the drill takes 9, your RTO is fiction and now you know — while it's cheap to fix.

Schrödinger's backup: the condition of any backup is unknown until you attempt a restore. Don't leave the box closed.

Ransomware changes the threat model

Classic backups assume accidents — a dead disk, a fat-fingered delete. Ransomware is different: it's an intelligent adversary who wants your recovery to fail. Modern ransomware doesn't immediately encrypt and announce itself. It often sits quietly, finds your backups, and encrypts or deletes those first — because a victim who can restore doesn't pay the ransom.

This breaks an assumption hiding inside ordinary 3-2-1. If your "offsite" backup is a cloud bucket your production server can write to and delete from, then anyone — or any malware — with your server's credentials can wipe that backup too. Reachable and deletable means destroyable. Same failure domain, a logical one instead of a physical one.

Immutable and offline: the copy they can't reach

The defense is a copy the attacker cannot alter or delete, even with full control of your servers. Two forms, often combined:

IMMUTABLE  — write-once, read-many. The storage itself refuses deletion or
             overwrite until a retention period expires. (Object-lock /
             "WORM" on cloud object storage.) Even with your keys, an
             attacker can't erase what's locked.

OFFLINE    — physically disconnected. Tapes in a vault, or a drive that's
             unplugged between backup runs. An air gap is unhackable over the
             network because there's no network to it.

What just happened: both close the loophole. Immutability means even a fully compromised account can't delete the locked copy until its retention window passes; offline (the "air gap") means there's no live path to the copy at all. This is the modern upgrade to the "1" in 3-2-1: not enough that the offsite copy is elsewhere — at least one copy must be unreachable by a live, compromised system.

A practical small-team version: keep your normal offsite backups, and enable object-lock on the bucket (or pull periodic copies to a drive you disconnect) so there's always one copy that today's compromised credentials cannot touch.

For builders

If you run your own infrastructure (see /guides/what-a-server-is), bake two non-negotiables into the plan from day one. One: a recurring restore drill, with a row-count-and-timestamp check, treated as an incident when it fails. Two: at least one immutable or offline copy, so a stolen credential or a ransomware run can't take your recovery down with your production. Everything earlier in this guide is preparation; these two are what make it real when the bad morning comes — and it does come.

[
  {
    "q": "Why is a backup job that reports 'success' every night still not proof you're protected?",
    "choices": [
      "Success messages are often delayed by an hour",
      "'The job ran' only proves the script executed, not that the bytes can rebuild your system",
      "Nightly is too infrequent to count as a real backup",
      "Success only counts if the job runs on a weekend"
    ],
    "answer": 1,
    "explain": "Jobs fail silently — an expired credential or config typo can dump an empty file while still 'succeeding.' Only an actual restore proves the output is usable."
  },
  {
    "q": "What is the only reliable way to know a backup actually works?",
    "choices": [
      "Confirm the backup file exists and has a plausible size",
      "Read the backup job logs for errors",
      "Restore it into a clean environment and verify the data is real and recent",
      "Check that the job has run successfully 30 days in a row"
    ],
    "answer": 2,
    "explain": "The restore drill is the only proof: pull the real artifact into a sandbox, bring the data up, and check it (e.g. row counts and the latest timestamp). It also validates your RTO if you time it."
  },
  {
    "q": "Why can ordinary 3-2-1 backups still fall to ransomware?",
    "choices": [
      "Ransomware encrypts faster than backups can run",
      "If your offsite copy is reachable and deletable by your server, compromised credentials can wipe it too",
      "3-2-1 was never designed for cloud storage",
      "Ransomware only targets databases, which 3-2-1 ignores"
    ],
    "answer": 1,
    "explain": "An attacker with your server's credentials can destroy any backup that server can write to or delete. The fix is an immutable (object-locked) or offline (air-gapped) copy they can't reach."
  }
]

← Phase 2 | Overview