Managing Services with systemd

In Phase 1 we said a server's whole job is keeping long-running services alive. This phase is about the thing that actually does that keeping: systemd. If you only learn one server tool deeply, learn this one - it's how you start, stop, supervise, and debug every service on a modern Linux box, and how you read what they're saying.

What systemd actually is

What it actually is. systemd is two things wearing one name. First, it's PID 1 - the very first process the kernel starts at boot, the ancestor of everything else (you'll see it at the top of ps as /sbin/init, which is a symlink to systemd). Second, it's the service manager: the program that decides what should run, starts those things in the right order, restarts them if they crash, and tracks whether each one is healthy.

Each one is a "unit" systemd keeps watch over.

Why people get this wrong. Coming from running programs by hand, the temptation is to cd somewhere and launch the binary yourself - ./nginx & or similar. That program dies when your SSH session ends (the Phase 1 war story), nothing restarts it if it crashes, and nothing starts it after a reboot. systemd exists to take that off you: you describe what should run, and it babysits it forever, whether or not you're logged in.

📝 Terminology. A unit is anything systemd manages, described by a small text file. The kind you'll touch most is a .service unit (a daemon to run). Files live in /lib/systemd/system/ (shipped by packages) and /etc/systemd/system/ (your overrides). You drive it with systemctl; you read its logs with journalctl.

systemctl status - your first move, always

Before you start or stop anything, ask systemd what it already thinks. status is read-only and tells you almost everything you need in one screen:

$ systemctl status nginx
● nginx.service - A high performance web server and a reverse proxy server
     Loaded: loaded (/lib/systemd/system/nginx.service; enabled; preset: enabled)
     Active: active (running) since Wed 2026-06-18 09:15:22 UTC; 2h 31min ago
       Docs: man:nginx(8)
   Main PID: 8123 (nginx)
      Tasks: 3 (limit: 4915)
     Memory: 8.1M
        CPU: 412ms
     CGroup: /system.slice/nginx.service
             ├─8123 "nginx: master process /usr/sbin/nginx"
             ├─8124 "nginx: worker process"
             └─8125 "nginx: worker process"

Jun 18 09:15:22 web-prod-1 systemd[1]: Starting nginx.service...
Jun 18 09:15:22 web-prod-1 systemd[1]: Started nginx.service.

What just happened: the green ● and Active: active (running) mean it's up right now, and since … 2h 31min ago is its uptime. Loaded: … enabled is the other half: enabled means it's set to start automatically at boot (enabled and running are different things, and confusing them bites everyone). Main PID: 8123 is the actual process, and the CGroup tree shows every process systemd tracks as part of this service. The last lines are recent log entries - status hands you a peek at the journal for free, often all you need to see what went wrong.

💡 Key point. Two independent questions live in that output. Is it running now? → the Active: line. Will it come back after a reboot? → the Loaded: line (enabled vs disabled). A service can be running but not enabled (works until the next reboot, then vanishes) or enabled but not running (will start next boot, but isn't up now). Keep them separate in your head.

start / stop / restart - controlling a running service

These three do exactly what they say, take effect immediately, and don't survive a reboot on their own (that's what enable is for). They almost always need sudo, because changing what's running on the machine is privileged.

$ sudo systemctl stop nginx
$ systemctl is-active nginx
inactive
$ sudo systemctl start nginx
$ systemctl is-active nginx
active

What just happened: stop told systemd to shut the service down cleanly (signals the process, waits for it to exit); is-active is a quick, scriptable check that prints one word instead of a whole status screen. Then start brought it back. Neither command printed anything on success - on Unix, silence is success. A failed start won't be silent: it'll point you at systemctl status and the journal.

restart is stop-then-start in one step, and it's what you reach for after editing a service's config:

$ sudo systemctl restart nginx

What just happened: systemd stopped nginx and started it again, so the running process re-reads its config. There's a gentler cousin worth knowing:

$ sudo systemctl reload nginx

What just happened: reload asks the service to re-read its config without dropping its running process - for nginx that means no dropped connections. Not every service supports it; when in doubt, restart always works but causes a brief blip. There's also reload-or-restart for "reload if you can, otherwise restart."

⚠️ Gotcha. Editing a config file changes nothing by itself. The running process loaded its config once, at start, and won't notice your edit until you reload or restart it. "I changed the file and it didn't take effect" is an enormous source of outages - the edit and the reload are two separate steps.

enable / disable - surviving a reboot

This is the half that catches people. enable and disable control boot behavior, and by default they don't touch the current state at all:

$ sudo systemctl enable nginx
Created symlink /etc/systemd/system/multi-user.target.wants/nginx.service → /lib/systemd/system/nginx.service.

What just happened: enable created a symlink that wires nginx into the boot sequence - it'll start automatically every future boot. It did not start nginx right now; if it was stopped, it's still stopped. When you want both at once, say so explicitly:

$ sudo systemctl enable --now nginx

What just happened: --now means "and also do it immediately" - enables nginx for future boots and starts it now, the combination you usually want on a fresh install. (disable --now is the mirror image.)

💡 Key point. Memorize the pairing: start/stop = right now, enable/disable = at boot, --now = both. Almost every "the service is gone after I rebooted the box" mystery is a service that was started but never enabled.

journalctl - reading what a service actually said

systemd captures the standard output and standard error of every service into one central, indexed log called the journal. journalctl is how you read it, and where you go the moment something misbehaves.

The most useful invocation: "show me this one service's logs, newest stuff visible":

$ journalctl -u nginx -n 20 --no-pager
Jun 18 09:15:22 web-prod-1 systemd[1]: Starting nginx.service...
Jun 18 09:15:22 web-prod-1 systemd[1]: Started nginx.service.
Jun 18 11:42:07 web-prod-1 nginx[8124]: 2026/06/18 11:42:07 [error] 8124#8124: *15 open() "/var/www/html/favicon.ico" failed (2: No such file or directory)

What just happened: -u nginx filters the journal down to the nginx unit (without it you'd get logs from the entire system interleaved - occasionally useful, usually overwhelming). -n 20 limits it to the last 20 lines, and --no-pager dumps it straight to your terminal instead of a scrollable pager. You can immediately read that nginx is up and that someone requested a missing favicon.

When you're actively debugging - restart the service in one window, watch the logs stream in another - the indispensable flag is -f, follow:

$ journalctl -u nginx -f
Jun 18 12:03:55 web-prod-1 nginx[8124]: 2026/06/18 12:03:55 [error] ...
^C

What just happened: -f tails the journal live, printing new lines as they're written, like tail -f on a plain log file. Ctrl-C stops following. This is the single most useful debugging loop on a server: trigger the thing, watch what the service says about it in real time.

A few more you'll lean on constantly:

$ journalctl -u sshd --since "1 hour ago" -p err

What just happened: --since takes human-friendly time ("1 hour ago", "today", "2026-06-18 09:00"), and -p err filters by priority to errors and worse - "SSH daemon errors from the last hour," a great first query when logins are failing. (-p understands the syslog levels: emerg, alert, crit, err, warning, notice, info, debug.)

⚠️ Gotcha. By default the journal may be stored only in memory and lost on reboot, depending on the distro - check Storage= in /etc/systemd/journald.conf (auto keeps it on disk only if /var/log/journal/ exists; persistent always does). If journalctl --list-boots shows only the current boot, make the journal persistent before you need it to investigate a crash that caused the reboot.

🪖 War story. A service "won't start" and the panic begins. Nine times out of ten the answer is sitting right there: systemctl status the-service shows failed, and the last few log lines name the exact reason

• a config syntax error, a port already in use, a missing permission. The discipline: status first, journal second, fix third. You almost never have to guess.

Recap

1. systemd is PID 1 (the first process) and the service manager that starts, supervises, and restarts your long-running services.
2. A unit (usually a .service) is a text file describing something systemd manages; you drive it with systemctl.
3. systemctl status <svc> is your first move - it shows running-or-not (Active:), start-at-boot-or-not (Loaded: enabled), the PID, and recent logs in one view.
4. start/stop/restart act now; enable/disable act at boot; --now does both. Don't confuse "running" with "enabled."
5. Edit a config, then reload/restart - the edit alone changes nothing.
6. journalctl -u <svc> reads a service's logs; -f follows them live, -n limits lines, --since and -p filter by time and priority.

Next, the discipline that keeps all of this from becoming a liability: running the box safely - users, sudo, scheduled jobs, the firewall, and SSH hardening.

← Guide overview · Phase 3: Running It Safely →