# Automating the Boring Stuff (Ops Scripting) > Turn the manual tasks you keep redoing by hand into scripts that are faster, repeatable, and self-documenting - from the when-to-automate mindset to a real bash backup script to knowing when to reach for Python and cron. --- # Automating the Boring Stuff (Ops Scripting) You know the task. The one you do by hand every few days: copy these files, restart that service, run these four commands in this exact order, check the output, hope you didn't fat-finger step three. It works - until the day you're tired, or in a hurry, or someone else has to do it without you. Then it doesn't. This guide is about turning those repeated manual chores into scripts. Not because scripting is fashionable, but because a script is faster, doesn't forget a step, and - the part nobody tells you - is *documentation that actually runs*. By the end you'll have a clear sense of when to automate (and when not to bother), a real bash script you can adapt, and a straight answer to "should this be Python instead?" This is an intermediate guide. It assumes you're comfortable in a terminal - running commands, moving around directories, editing files. If that's still shaky, start with [The Terminal and Shell](/guides/the-terminal-and-shell) and come back. ## How to read this - **Not sure whether something is even worth automating?** Read [Phase 1: If You've Done It Twice, Script It](01-if-youve-done-it-twice.md) - it's the decision, not the code. - **Ready to write a real script?** Go straight to [Phase 2: Shell Scripting Essentials](02-shell-scripting-essentials.md) for an annotated, working example. - **Want it to finally make sense?** Read in order - each phase builds on the last. ## The phases 1. **[If You've Done It Twice, Script It](01-if-youve-done-it-twice.md)** - the mindset. Why manual steps are slow, error-prone, and unrepeatable, why a script is documentation that runs, and the clear-eyed test for when automating is worth it (and when it isn't). 2. **[Shell Scripting Essentials](02-shell-scripting-essentials.md)** - a real bash script, built up piece by piece: variables, arguments, conditionals, loops, exit codes, and the one line (`set -euo pipefail`) that turns a silent disaster into a loud, safe failure. 3. **[When to Reach for Python](03-when-to-reach-for-python.md)** - the point where bash starts hurting (data structures, parsing, anything cross-platform), how to tell, and how to *schedule* your automation with cron so it runs without you - safely, because it's idempotent, logged, and dry-runnable. > Deliberately deferred to follow-up guides: full configuration management (Ansible, etc.), CI/CD pipelines, and infrastructure-as-code. This guide is about *your* repeated tasks - the scripts you write to save your own afternoons. Once those are second nature, the bigger tools make a lot more sense. Related reading: [The Terminal and Shell](/guides/the-terminal-and-shell) for the shell fundamentals these scripts run on, and [Linux for Servers](/guides/linux-for-servers) for running and scheduling automation on a real machine. --- # If You've Done It Twice, Script It Before any code, the mindset - because this is where automation actually pays off or quietly wastes your time. The skill isn't writing scripts. Plenty of people can write a script. The skill is *noticing* which boring task deserves one, and being clear-eyed about which doesn't. Picture the task you do by hand right now. You open a terminal, you run a handful of commands, you eyeball the output, you move on. It feels fine. It's only a couple of minutes. But that feeling is hiding three real costs. ## What a manual task actually costs **It's slow - and the slowness compounds.** Two minutes, three times a week, is over five hours a year on one chore. That math alone rarely justifies a script. But the *time* is the smallest cost. The other two are the ones that hurt. **It's error-prone, and the errors are silent.** Every manual run is a fresh chance to skip a step, run them out of order, copy the wrong path, or do step four before step three finished. Most of the time you get away with it. The one time you don't is the one you'll remember - the deleted directory, the service restarted in prod instead of staging, the backup that overwrote the thing it was supposed to protect. **It's unrepeatable - it lives in your head.** This is the cost almost nobody counts. The "process" is a sequence of commands that exists only in your memory and your shell history. When you're on vacation, or you leave, or a teammate has to do it at 2am, the knowledge isn't *anywhere*. They reconstruct it from guesswork. ```text A manual task lives here: A script lives here: ┌──────────────┐ ┌──────────────┐ │ your memory │ │ a file, in │ │ + shell │ │ version │ │ history │ │ control │ └──────────────┘ └──────────────┘ │ │ gone when you are anyone can read it, different every run run it, and trust it ``` ## The reframe: a script is documentation that runs When you write the task down as a script, you haven't just automated it - you've *documented* it. And unlike a wiki page or a README, this documentation can't go stale, because it's the same thing that does the work. A README that says "to deploy, run these five commands" drifts out of date the moment someone changes step three and forgets to update the doc. A script *is* step three - change the process, change the script, and the documentation updates itself. Reading it tells you, and the next person, and future-you, exactly what happens, in order, with the real paths and real flags. 💡 **Key point.** The best reason to automate a boring task usually isn't speed. It's that the script becomes the single, truthful, runnable record of how the task is *actually* done - so it survives you forgetting, leaving, or having a bad day. ## The clear-eyed test: when to automate Automation has a cost too. Writing the script, testing it, and maintaining it as things change is real work. So don't automate reflexively. Run it through a few questions: - **Have you done it more than twice - and will you do it again?** Once is a one-off; do it by hand. Twice is a coincidence. Three times is a pattern, and patterns are what scripts are for. The old rule of thumb is "if you've done it twice, the third time should be a script." - **Is it the same every time?** Automation loves *repeatable* tasks. If every run is genuinely different and needs judgment, a script will fight you. (You can often still automate the boring 80% and leave the judgment to a human.) - **Does getting it wrong hurt?** A task that's error-prone *and* consequential - touching backups, production, money, or data you can't recreate - earns a script even at low frequency, precisely because the script removes the chance to fat-finger it. - **Will more than one person need to run it?** The moment a task has to outlive your memory, "it's in my head" stops being acceptable. The script is the handoff. And the plain counter-cases - when *not* to automate: ⚠️ **Don't automate the genuinely rare one-off.** If you'll do it once, ever, the time spent scripting (and testing it carefully) is pure loss. Do it by hand, carefully. ⚠️ **Don't automate a task you don't yet understand.** A script that encodes a process you only half-grasp just makes your confusion fast and repeatable. Do it manually until you understand *why* each step is there. Then automate it. ⚠️ **Beware the trap.** There's a well-known cartoon-shaped truth here: it is very easy to spend three days automating a task that took you ten minutes a month, and never break even. Automate to remove *real* recurring pain, not because automating is more fun than the chore. (For a consequential, error-prone task, "breaking even" includes the disasters you didn't have - which is harder to see but very real.) ## 🪖 A short war story A teammate "knew how" to refresh the staging database from a backup - four commands, done by hand every couple of weeks for a year. Then they were out sick the week before a big demo, staging was stale, and nobody else could do it: the process had never been written down anywhere but their muscle memory. We rebuilt it from shell history, got it wrong twice, and ended up with thirty lines of bash that should have existed eleven months earlier. It wasn't worth automating for *speed* - it was worth automating so it didn't live in one person's head. ## Recap 1. A manual task has three costs: it's **slow** (and that compounds), **error-prone** (silently), and **unrepeatable** (it lives in your head). 2. A script is **documentation that runs** - the one record of the process that can't drift out of date. 3. Automate when the task is **repeated**, **repeatable**, **consequential**, or **shared** - and *don't* automate true one-offs or processes you don't yet understand. 4. The point isn't usually speed. It's removing the chance to get it wrong, and making the knowledge survive you. You've got the judgment. Next, let's turn one of those tasks into a real, safe bash script. --- # Shell Scripting Essentials A bash script is the exact same commands you'd type at the prompt, saved in a file so you can run them all at once, the same way, every time. This phase fills in the pieces that turn a list of commands into something robust enough to trust - including the one line that separates "a script" from "a safe script." We'll build one real example end to end: a script that backs up a directory and rotates out old backups so they don't fill the disk. Type it in and run it as you go - the pieces click fastest when you watch them work. ## The shebang and your first run Create a file called `backup.sh`: ```bash #!/usr/bin/env bash echo "Hello from a script" ``` That first line is the **shebang** (`#!`) - it tells the system which program runs the file. `/usr/bin/env bash` finds bash wherever it lives, more portable than hardcoding a path. Make it runnable and run it: ```console $ chmod +x backup.sh $ ./backup.sh Hello from a script ``` *What just happened:* `chmod +x` flips the file's permission so the system can run it as a program; `./backup.sh` then runs it. Bash reads the file top to bottom and executes each line as if you'd typed it. **A script is just typed-ahead terminal commands.** 📝 **Terminology.** A `#` starts a **comment** (bash ignores the rest of the line) except on the first line, where `#!` is the shebang. Use comments for *why*, not *what*. ## The safety line: `set -euo pipefail` Put this line right after the shebang in almost every script you write - it's the most important one in this guide: ```bash #!/usr/bin/env bash set -euo pipefail ``` By default, bash is alarmingly forgiving: a failed command doesn't stop the script, and an unset variable is silently treated as empty. For a backup script, that's exactly how you end up deleting good backups because the step that made the new one silently failed. `set -euo pipefail` makes bash strict instead - three flags, plus one: - **`-e`** - **exit on error.** A failed command (non-zero exit) stops the script immediately instead of blundering onward. - **`-u`** - **error on unset variables.** Using a variable you forgot to set is now a loud error, not a silent empty string - catches typos like `$BACKUP_DIRR` before they erase the wrong thing. - **`-o pipefail`** - in a pipeline like `a | b`, if `a` fails, the whole pipeline is considered failed. Without this, only the *last* command's success counts, so failures hide mid-pipe. ⚠️ **Gotcha.** Without `set -e`, a script that fails on line 3 happily runs lines 4 through 40 against a broken state - how "the backup script" becomes "the script that quietly stopped backing up six weeks ago and nobody noticed." Adding this one line is the cheapest reliability you will ever buy. (Sometimes you *expect* a command to fail and want to handle it yourself - use `if ! some_command; then ...`, as below. It doesn't trip `-e` because the failure is part of an `if` test.) ## Variables - name the things that might change Hardcoding paths all over a script means changing them in ten places later. Pull them into **variables** at the top, where they're easy to find and edit: ```bash SOURCE_DIR="$HOME/projects/myapp" BACKUP_ROOT="$HOME/backups" KEEP=5 ``` *What just happened:* `NAME="value"` defines a variable - **no spaces around the `=`** (`NAME = "value"` is an error, and trips up everyone at least once). Read it back with `$SOURCE_DIR`. `$HOME` is one bash gives you free: your home directory. ⚠️ **Gotcha - always quote your variables.** Write `"$SOURCE_DIR"`, not `$SOURCE_DIR`. If a path contains a space (and one day it will - `My Documents`), an unquoted variable splits into two arguments and your command does something wild. Quoting keeps it one value - this single habit prevents a whole category of "it worked on my machine" bugs. ## Arguments - let the caller pass values in You don't want to edit the script every time you back up a different folder. Let the caller pass the source directory as an **argument**: ```console $ ./backup.sh /home/ada/projects/myapp ``` Arguments arrive as `$1`, `$2`, and so on (`$1` is first). Check the caller actually provided one: ```bash if [ -z "${1:-}" ]; then echo "Usage: $0 " >&2 exit 1 fi SOURCE_DIR="$1" ``` *What just happened:* `[ -z "${1:-}" ]` tests whether the first argument is empty (`-z` = "zero length"). The `${1:-}` is a small dance for `set -u` - "`$1`, or empty if unset" - so checking a missing argument doesn't itself trip the unset-variable error. If it's empty, we print a usage message to **standard error** (`>&2`) and `exit 1`. ## Exit codes - how a script says "it worked" or "it didn't" `exit 1` above matters: every command and script ends with an **exit code** - `0` for success, anything else for failure. It's how scripts talk to `set -e` and to schedulers like cron. See the last one in `$?`: ```console $ ls /tmp >/dev/null; echo $? 0 $ ls /does-not-exist >/dev/null 2>&1; echo $? 2 ``` *What just happened:* `ls` on a real directory succeeded, so `$?` is `0`. On a missing one it failed, so `$?` is non-zero (`2`, ls's code for "serious trouble"). `>/dev/null` and `2>&1` just discard the output - we only wanted to see the code. The rule to internalize: **end your script with `exit 0` on success, `exit 1` (or another non-zero) on failure**, so whatever runs it knows what happened. Cron, in particular, decides whether to alert you based on this number. ## Conditionals - check before you act We've already used `if`. The full shape is worth seeing once: ```bash if [ ! -d "$SOURCE_DIR" ]; then echo "Source directory does not exist: $SOURCE_DIR" >&2 exit 1 fi mkdir -p "$BACKUP_ROOT" ``` *What just happened:* `[ ! -d "$SOURCE_DIR" ]` reads as "if NOT (`!`) a directory (`-d`) exists here." A missing source fails loudly and early rather than backing up nothing. `mkdir -p` creates the backup folder - `-p` means it won't complain if the folder already exists, making the script safe to re-run. (That property is *idempotency* - Phase 3 covers it.) ## Putting it together: backup + rotate Now the whole script. Read it top to bottom - every piece is something we just covered: ```bash #!/usr/bin/env bash set -euo pipefail # --- configuration (the things you might change) --- BACKUP_ROOT="$HOME/backups" KEEP=5 # how many recent backups to keep # --- argument check --- if [ -z "${1:-}" ]; then echo "Usage: $0 " >&2 exit 1 fi SOURCE_DIR="$1" if [ ! -d "$SOURCE_DIR" ]; then echo "Source directory does not exist: $SOURCE_DIR" >&2 exit 1 fi # --- make the backup --- mkdir -p "$BACKUP_ROOT" TIMESTAMP="$(date +%Y%m%d-%H%M%S)" # e.g. 20260619-142530 ARCHIVE="$BACKUP_ROOT/backup-$TIMESTAMP.tar.gz" echo "Backing up $SOURCE_DIR -> $ARCHIVE" tar -czf "$ARCHIVE" "$SOURCE_DIR" # --- rotate: delete all but the newest $KEEP archives --- echo "Keeping the $KEEP most recent backups; removing older ones." ls -1t "$BACKUP_ROOT"/backup-*.tar.gz | tail -n +$((KEEP + 1)) | while read -r old; do echo "Removing old backup: $old" rm -f "$old" done echo "Done." exit 0 ``` A few lines deserve a closer look: - `TIMESTAMP="$(date +%Y%m%d-%H%M%S)"` - `$(...)` is **command substitution**: it runs the command and drops its output into the variable. Putting the timestamp in the filename means each run makes a *new* archive instead of clobbering the last one. - `tar -czf "$ARCHIVE" "$SOURCE_DIR"` - **c**reate a **z** (gzip) archive into the named **f**ile. This is the actual backup. - The rotation line is a pipeline: `ls -1t` lists archives **newest first**, `tail -n +$((KEEP + 1))` skips the first `KEEP` and prints the rest, and `while read` **loops** over deleting each one. If `KEEP` is 5, lines 1–5 stay and deletion starts at line 6. Here's a real run, the second time (so there's already an old backup to clean up): ```console $ ./backup.sh ~/projects/myapp Backing up /home/ada/projects/myapp -> /home/ada/backups/backup-20260619-142530.tar.gz Keeping the 5 most recent backups; removing older ones. Removing old backup: /home/ada/backups/backup-20260612-090210.tar.gz Done. $ echo $? 0 ``` *What just happened:* The script created a fresh timestamped archive, found it now had six backups, kept the five newest, and removed the oldest - exiting `0` to say it worked. Run it again tomorrow and it does the same thing correctly, without you remembering a single flag. That's the payoff from Phase 1: the process is now a file, not a memory. 💡 **Key point.** A reliable script is mostly *guard rails*: `set -euo pipefail`, quote every variable, check arguments and paths before acting, end with a clear exit code. `tar` and `rm` are the easy part - the guard rails are what make it safe to run unattended, which Phase 3 needs. ## Recap 1. A script is typed-ahead terminal commands; the **shebang** (`#!/usr/bin/env bash`) says what runs it, and `chmod +x` lets you run it. 2. Put **`set -euo pipefail`** right after the shebang - exit on error, error on unset variables, catch failures inside pipes. The single best line for reliability. 3. **Variables** (`NAME="value"`, no spaces; read as `$NAME`) - and always **quote** them. 4. **Arguments** arrive as `$1`, `$2`; check they exist before using them. 5. **Exit codes**: `0` is success, non-zero is failure - end with `exit 0`/`exit 1` so other tools know what happened. 6. **Conditionals** (`if [ ... ]`) check before acting; **loops** (`while read`) handle many items. You can now write a script that does real work and fails safely. Next: how to tell when bash is the wrong tool, and how to schedule a script so it runs without you. ## Try it yourself Decode any cron schedule - edit it and see the plain-English meaning plus the next run times: ```playground-cron */15 9-17 * * 1-5 ``` --- # When to Reach for Python Bash is wonderful glue - it's on every Unix machine and pipes commands together effortlessly. But there's a point where a script stops feeling clever and starts feeling like wrestling. Recognizing that point, and switching tools instead of pushing through, is part of the craft. Then: *scheduling* automation to run without you, and making it safe to leave alone. ## The signs bash is the wrong tool You don't abandon bash because someone said Python is better. You switch when bash is actively fighting you. The usual three signals: **You need real data structures.** Bash has strings and clumsy arrays, and that's about it. Track a list of records with several fields, build a lookup table, or nest data, and bash turns into string-splitting hacks that break the first time a value has a space or comma. Python just has lists and dictionaries that hold your data. **You're parsing structured text - JSON, CSV, XML, an API response.** This is the big one. People do parse JSON in bash (usually shelling out to `jq`), but once the logic gets real - "for each user, if their plan is expired, call this endpoint" - you're hand-rolling a parser out of `grep`, `cut`, and `sed`, and it's fragile. Python reads JSON in one line and gives you real objects: ```python import json with open("users.json") as f: users = json.load(f) for user in users: if user["plan"] == "expired": print(f"Would notify {user['email']}") ``` *What just happened:* `json.load` turns the file into a real Python list of dictionaries - no parsing by hand - and a plain loop walks it, reading fields by name. The same job in bash would be a tangle of text-slicing that breaks on the first unusual character. When input is structured, reach for a language that understands structure. **It needs to run on Windows too.** Bash scripts assume a Unix world - `tar`, `rm`, `/`-style paths. If your automation has to run on Windows as well as Linux/macOS, bash is the wrong foundation. Python runs the same code across all three, with cross-platform helpers like `pathlib` for file paths built in. ⚠️ **Gotcha - don't over-correct.** This isn't "Python good, bash bad." For gluing a few commands together, bash is *less* ceremony and more upfront about what it's doing - reaching for Python to run three commands in a row is its own kind of overkill. The rule: **bash to orchestrate commands; Python when there's real logic or real data.** Many good setups are a short bash script that calls Python for the gnarly middle bit. ```text the task is mostly... reach for... running commands in order ──► bash simple checks + loops ──► bash ───────────────────────────────────────── structured data (JSON/CSV) ──► Python nested logic / many fields ──► Python must run on Windows too ──► Python ``` ## Scheduling: making it run without you A script you still remember to run by hand has only solved half the problem. The other half is **scheduling** - having the machine run it for you, on time, forever. On Linux and macOS, the classic tool is **cron**. 📝 **Terminology.** **cron** is a background service that runs commands on a schedule. A single scheduled entry is a **cron job**, and the list of them is your **crontab** ("cron table"). You edit your schedule with `crontab -e`, and each line is *five time fields plus the command*: ```text ┌───────── minute (0–59) │ ┌─────── hour (0–23) │ │ ┌───── day of month (1–31) │ │ │ ┌─── month (1–12) │ │ │ │ ┌─ day of week (0–6, Sun=0) │ │ │ │ │ 0 2 * * * /home/ada/backup.sh /home/ada/projects/myapp ``` That line means: at **minute 0 of hour 2** (2am), **every day** (the `*`s mean "any"), run the backup script. A `*` is "every," so `0 2 * * *` is "once a day at 2am." (If cron's five fields make your eyes cross, you're in good company - most people keep a reference handy.) ⚠️ **Gotcha - cron runs with almost no environment.** This bites everyone exactly once. Cron doesn't load your shell profile, so your usual `PATH` and environment variables may be missing - a script that runs perfectly in your terminal mysteriously does nothing under cron. Defend against it: **use absolute paths** everywhere (`/home/ada/backup.sh`, not `./backup.sh`; `/usr/bin/tar`, not `tar` if unsure), and **capture output to a log** (next section). For scheduling on an actual server - including the systemd timers many modern systems prefer over cron - see [Linux for Servers](/guides/linux-for-servers). ## The three rules for automation you can walk away from Once a script runs *unattended*, you can't be there to catch it - it has to be safe to run on its own, and safe to run *again*. Three properties make that true; build them in from the start. ### 1. Idempotent - safe to run more than once 📝 **Terminology.** **Idempotent** means running the script twice has the same result as running it once. No duplicates, no damage, no "it only works the first time." This matters because scheduled scripts *will* re-run - cron fires again, a job gets retried, you run it manually to test. A non-idempotent script that *appends* a line to a config file every run will have added it fifty times by next month. Make operations safe to repeat: use `mkdir -p` (no error if the folder exists), check whether work is already done before redoing it, and prefer "make the end state correct" over "blindly perform an action." ```bash # NOT idempotent - adds the line every single run: echo "127.0.0.1 myapp.local" >> /etc/hosts # Idempotent - only adds it if it's not already there: if ! grep -q "myapp.local" /etc/hosts; then echo "127.0.0.1 myapp.local" >> /etc/hosts fi ``` *What just happened:* The first version appends unconditionally - run it ten times, get ten copies. The second checks first with `grep -q` ("quietly look for the line") and only adds it if missing. Run *that* ten times and the file is identical to running it once. (The `if !` also sidesteps `set -e`: a "not found" from grep is expected here, not a failure.) ### 2. Dry-run - let it tell you what it *would* do Before you trust a script to delete files or hit an API on its own, you want to see its plan *without* it doing anything. A **dry-run** mode does exactly that: it prints every action it would take and changes nothing. ```bash DRY_RUN="${DRY_RUN:-false}" remove_file() { if [ "$DRY_RUN" = "true" ]; then echo "[dry-run] would remove: $1" else rm -f "$1" fi } ``` *What just happened:* The `remove_file` helper checks a `DRY_RUN` flag: run the script normally and it deletes; run it with `DRY_RUN=true ./backup.sh ...` and it just *narrates* what it would delete. This is how you safely test a destructive script against real data - turn on dry-run, read the plan, and only then let it loose. For anything that deletes, moves, or overwrites, dry-run pays for itself the first time it stops you from wiping the wrong directory. ### 3. Logging - so you know what happened while you weren't looking An unattended script that prints to a screen nobody is watching might as well be silent. **Logging** means recording what it did, when, and whether it worked, so there's a trail when you check in or something breaks. At its simplest, redirect output to a file in your cron line: ```text 0 2 * * * /home/ada/backup.sh /home/ada/projects/myapp >> /home/ada/logs/backup.log 2>&1 ``` *What just happened:* `>> .../backup.log` *appends* the script's output to a log file (so each run adds rather than wipes it), and `2>&1` sends error messages to the same place. Now every nightly run leaves a dated record. A tiny timestamp helper inside the script makes the log readable: ```bash log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*"; } log "Starting backup of $SOURCE_DIR" ``` *What just happened:* `log` prefixes whatever you pass it with a timestamp, so the file reads like `[2026-06-19 02:00:01] Starting backup of ...`. When a teammate asks "did the backup run last night?", you have an answer instead of a shrug. Combined with exit codes, this is how cron-driven automation stays trustworthy: it tells you when it worked, and it's loud when it didn't. 💡 **Key point.** Unattended automation lives or dies on three properties: **idempotent** (safe to re-run), **dry-runnable** (shows its plan before acting), and **logged** (leaves a trail). A fast script without these is a fast way to cause an incident at 2am while you sleep. ## Recap 1. Switch from bash to **Python** when you need real **data structures**, you're **parsing structured text** (JSON/CSV/XML/APIs), or it must run **cross-platform** - but keep bash for gluing commands together. 2. **cron** schedules scripts to run on their own; remember its five time fields and that it runs with **almost no environment** - use absolute paths and log the output. 3. Make unattended scripts **idempotent** (safe to run twice), give them a **dry-run** mode (narrate, don't act), and add **logging** (a timestamped trail). 4. These safety properties matter *more* the less you're watching - which is the whole point of scheduling. That's the arc: notice the repeated task (Phase 1), turn it into a safe bash script (Phase 2), and know when to upgrade to Python and how to schedule it safely (here). You now have what you need to start deleting boring tasks from your week - carefully.