# Docker Without the Magic > What a container actually is, how a Dockerfile builds an image layer by layer, and the gotchas that bite everyone - so 'works on my machine' stops being a thing that happens to you. --- # Docker Without the Magic You've run `docker run` and watched a thing start. You've copied a `Dockerfile` off a blog and it worked, mostly. But if someone asked you "what *is* a container, really?" - you'd hesitate. And when the container runs fine but the port "isn't there," or the data vanishes on restart, or your image is somehow two gigabytes, it all feels like magic that turned against you. It isn't magic. Docker rests on a small handful of plain ideas, and once you have them, the weird behavior stops being weird. This guide installs those ideas first, then shows you the everyday commands and the traps - so the next time someone says "but it works on my machine," you can hand them a container and end the conversation. ## How to read this - **Hit a wall right now?** Jump to [Phase 3: Volumes & the Gotchas](03-volumes-and-the-gotchas.md) and use the cheat-card at the top - it maps the classic "why is it doing that" symptoms to calm fixes. - **Want Docker to finally make sense?** Read in order. Each phase builds on the last: the mental model, then building images, then keeping data alive and dodging the traps. ## The phases 1. **[Image vs Container (and Why Not a VM)](01-image-vs-container.md)** - the one mental model the whole tool rests on: an image is a frozen snapshot, a container is a running instance of it, and both are far lighter than a virtual machine because they share the host's kernel. 2. **[Building an Image: the Dockerfile & Layers](02-the-dockerfile-and-layers.md)** - a Dockerfile is a recipe, each instruction is a cached *layer*, and once you see the layers you'll understand why build order makes the difference between a 2-second rebuild and a 2-minute one. Plus `build`, `run`, ports, and the registry. 3. **[Volumes & the Gotchas](03-volumes-and-the-gotchas.md)** - containers are throwaway, so your data needs somewhere safe to live (volumes), your config belongs in environment variables, and a short list of traps - huge images, secrets baked into layers, the unpublished port - are named before they bite. > This guide gets you fluent with a single container. Running several containers together - a web app plus > its database plus a cache, wired up and started with one command - is its own skill, and it has its own > guide: [Docker Compose for Real Projects](/guides/docker-compose-for-real-projects). --- # Image vs Container (and Why Not a VM) Before any commands, let's install the idea the entire tool stands on. Almost every Docker confusion - "why didn't my change show up?", "where did the container go?", "is this a tiny computer?" - comes from blurring two words that mean genuinely different things: **image** and **container**. Get the difference clear and most of Docker stops being mysterious. ## The one analogy: a class and an object If you've written any code, you already know this shape: ```mermaid flowchart LR IMG["Image
(class)"] -->|docker run| C1["container 1"] IMG -->|docker run| C2["container 2"] IMG -->|docker run| C3["container 3"] ``` *One image → many containers: run it five times, you get five independent containers.* An **image** is a **read-only, packaged snapshot of a filesystem, plus the metadata for how to run it** - which command to start, which port the program listens on, what environment it expects. It's a frozen template. It doesn't *do* anything on its own, the same way a class definition doesn't run until you instantiate it. It sits on disk, and you can copy it, share it, and start it as many times as you like. A **container** is a **running instance of an image** - the image brought to life as a live process, with its own slice of memory, its own writable scratch space on top of the image's frozen files, and its own lifecycle (it starts, runs, and stops). One image, many containers: start the same image three times and you have three independent containers that can't see each other's running state, exactly like three objects built from one class. 📝 **Terminology.** *Image* = the frozen template on disk. *Container* = a live process running from that template. You **build** images and you **run** containers. People say "Docker image" and "Docker container" interchangeably in casual speech, but the moment something breaks, the distinction is the first thing to get straight. ## Why people get this wrong The common wrong picture is "a container is a little computer." It feels like one - its own filesystem, processes, network address - so calling it a tiny machine seems fair. But that picture leads you astray fast: - It makes you expect the container to *remember* things, like a computer does. It doesn't, by default - when a container stops, the writable scratch layer it was using is thrown away. (That's Phase 3's whole topic.) - It makes you think editing a file on your laptop should change what's inside a running container. It won't - the container is running from the frozen image, not from your project folder. You have to **rebuild** the image or mount your folder in deliberately. The accurate picture is smaller and more useful: a container is **one (or a few) isolated processes, wrapped so they think they have the machine to themselves.** That's it. The isolation is a costume, not a second computer. ## Containers vs virtual machines: where the weight goes The question everyone asks next: isn't this just a virtual machine? It is not, and the difference is the reason Docker took over. It comes down to one thing - **the kernel**. > 📝 **Terminology.** The *kernel* is the core of an operating system: the part that actually talks to the > hardware and shares it among programs. If that's fuzzy, the > [What an Operating System Is](/guides/what-an-operating-system-is) guide explains it from scratch. *Virtual machines - each carries a full guest OS (heavy):* ```mermaid flowchart LR VApps["app + guest OS ×3"] --> Hyp[hypervisor] --> VHost[host OS] --> VHW[hardware] ``` *Containers - share the host's one kernel (light):* ```mermaid flowchart LR CApps["app + libs ×3"] --> Eng[Docker Engine] --> CHost["host OS
one shared kernel"] --> CHW[hardware] ``` **The VM way.** A virtual machine emulates a whole computer. On top of your real OS sits a *hypervisor*, and on top of that, each VM runs its **own complete guest operating system** - its own kernel, its own boot process - before your app even starts. Powerful (you can run Windows on Linux) but heavy: each VM carries gigabytes of OS and takes the better part of a minute to boot, because it really is booting one. **The container way.** Containers don't bring their own OS kernel. They **share the host's kernel** and ask it - through the same system calls every normal program uses - for their own isolated view of the filesystem, processes, and network. The container holds your app and its libraries, nothing more. No guest OS to boot, so a container starts in a fraction of a second and weighs megabytes, not gigabytes. 💡 **Key point.** A VM virtualizes the *hardware* and runs a full OS on top. A container virtualizes the *operating system* - an isolated process group sharing the one kernel that's already running. That single design choice is why containers are small and fast, and it's the trade-off too: because they share the host kernel, Linux containers need a Linux kernel. (On macOS and Windows, Docker quietly runs a small Linux VM in the background to provide one - why "it's not a VM" has a footnote on those machines.) **The gotcha this dissolves.** People expect a container to be a *security* boundary as strong as a VM. It's strong, but not the same: a container shares the host kernel, so a kernel-level exploit isn't walled off the way it is between VMs. Fine for most work; for running fully untrusted code, it's a real distinction. Knowing *where* the isolation comes from tells you exactly how far to trust it. ## Why this saves you later Holding "image = frozen template, container = running instance, both share the host kernel" in your head defuses a whole row of future headaches: - **"My code change didn't take effect."** Of course - the container is running the *old image*. You rebuild the image (Phase 2) or mount your source in (Phase 3). - **"The container disappeared / my data is gone."** Containers are instances; stopping one discards its scratch layer. Persistence is a thing you add on purpose (Phase 3). - **"Why is this so much faster than the VM we used to use?"** No guest OS to boot. Now you know exactly what you're *not* paying for. ## Recap 1. An **image** is a read-only, layered snapshot of a filesystem plus how to run it - a frozen template, like a class. 2. A **container** is a running instance of an image - a live process with its own scratch space, like an object. One image can spawn many containers. 3. You **build** images; you **run** containers. 4. Containers are not little computers and not VMs: they're **isolated processes sharing the host's kernel**, which is why they're small and start in moments. 5. A **VM** carries its own full OS and kernel (heavy, strong isolation); a **container** shares the host's kernel (light, fast, isolation with a kernel-shaped footnote). Next, we'll build an image ourselves and watch it come together one cached layer at a time. Watch it animated: [containers vs. VMs](/explainers/ContainersVMs.dc.html) --- # Building an Image: the Dockerfile & Layers Now you'll make an image of your own. The thing that turns a frozen template from Phase 1 into something you can actually build is a file called a **Dockerfile** - a plain-text recipe Docker reads top to bottom. The single most useful thing to understand here isn't any one command; it's **layers**, because layers explain why two Dockerfiles that produce the identical image can rebuild in 2 seconds or 2 minutes. ## The Dockerfile: a recipe Docker reads top to bottom A Dockerfile is an ordered list of instructions that describe how to assemble an image: start from some base, copy your files in, install dependencies, and declare how to run the program. Each line is an instruction in `CAPITALS` followed by its arguments. Here's a realistic one for a small Node.js app, annotated: ```dockerfile # Start from an official image that already has Node installed. # This becomes our base layer - we build on top of it. FROM node:20-slim # Set the working directory inside the image. Commands below run here, # and it's created if it doesn't exist. WORKDIR /app # Copy ONLY the dependency manifests first (see "why order matters" below). COPY package.json package-lock.json ./ # Install dependencies. This is the slow step we want to cache. RUN npm ci # Now copy the rest of the source code. COPY . . # Document the port the app listens on (this is metadata, not a published port). EXPOSE 3000 # The default command to run when a container starts from this image. CMD ["node", "server.js"] ``` 📝 **Terminology.** *Base image* - the image named in `FROM` that you build on top of. You almost never start from nothing; you start from an official image (like `node`, `python`, `nginx`) that someone else already assembled. *`RUN`* executes a command *while building the image*. *`CMD`* is the command that runs *when a container starts*. Confusing those two is one of the most common Dockerfile mistakes - `RUN` happens at build time, `CMD` at run time. ## Layers: every instruction is a cached step Each instruction in the Dockerfile produces a **layer** - a saved diff of what changed in the filesystem at that step. The final image is those layers stacked, read-only, one on top of the next: ```mermaid flowchart TD L6["CMD [node, server.js] - metadata, tiny"] L5["EXPOSE 3000 - metadata, tiny"] L4["COPY . . - your source code"] L3["RUN npm ci - installed node_modules (BIG)"] L2["COPY package.json ... - the manifests"] L1["WORKDIR /app"] L0["FROM node:20-slim - the base image"] L6 --> L5 --> L4 --> L3 --> L2 --> L1 --> L0 ``` *Each instruction is one read-only layer; the final image is them stacked.* When you rebuild, Docker walks the instructions in order and **reuses the cached layer for any step whose inputs haven't changed.** The instant it hits a step whose inputs *did* change, that layer and **every layer after it** are rebuilt from scratch. The cache is a streak that breaks the moment something upstream changes. That single rule is why the Dockerfile above copies `package.json` *before* the source code. Your dependencies change rarely; your source changes constantly. By installing dependencies before copying source, an ordinary code edit leaves the `npm ci` layer untouched - Docker reuses it from cache, and the rebuild is fast. ⚠️ **Gotcha.** Flip those two lines - `COPY . .` *before* `RUN npm ci` - and every code change invalidates the copy layer, which forces `npm ci` to re-run on every single build, reinstalling every dependency. The image is identical; the build is agonizing. **Order your Dockerfile from least-frequently-changed to most-frequently-changed.** That's the whole art of a fast Dockerfile. ## `docker build`: turn the recipe into an image ```console $ docker build -t my-app:1.0 . [+] Building 18.4s (10/10) FINISHED => [internal] load build definition from Dockerfile 0.0s => [1/5] FROM docker.io/library/node:20-slim 2.1s => [2/5] WORKDIR /app 0.1s => [3/5] COPY package.json package-lock.json ./ 0.0s => [4/5] RUN npm ci 14.8s => [5/5] COPY . . 0.2s => exporting to image 1.1s => => naming to docker.io/library/my-app:1.0 ``` *What just happened:* `docker build` read the Dockerfile and built each layer in order. `-t my-app:1.0` **tags** the result - `my-app` is the name, `1.0` the *tag* (usually a version). The `.` at the end is the **build context**: the folder Docker hands to the build so `COPY` has something to copy from. Notice `[4/5] RUN npm ci` took 14.8s - the slow step. Now change one line of `server.js` and build again: ```console $ docker build -t my-app:1.1 . [+] Building 1.6s (10/10) FINISHED => [1/5] FROM docker.io/library/node:20-slim 0.0s => CACHED [2/5] WORKDIR /app 0.0s => CACHED [3/5] COPY package.json package-lock.json ./ 0.0s => CACHED [4/5] RUN npm ci 0.0s => [5/5] COPY . . 0.2s => exporting to image 0.4s ``` *What just happened:* this is the payoff for ordering the Dockerfile well. Steps 1–4 say `CACHED` - your dependencies didn't change, so Docker reused those layers, including the expensive `npm ci`. Only `COPY . .` re-ran, because only your source changed. The build dropped from ~18s to under 2s. Same image, fraction of the time. 📝 **Terminology.** A *tag* (`my-app:1.0`) is a human-readable label for an image version. If you leave it off, Docker assigns `latest` - which is a name, not a promise of newness, and is a common source of "wait, which version is this?" confusion later. ## `docker run`: bring the image to life You have an image. Phase 1's lesson: running it creates a container. ```console $ docker run -d -p 8080:3000 --name web my-app:1.0 3f9a2c1b7e4d8a6f0c2e1d9b4a7c6e5f0a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d ``` *What just happened:* `docker run` started a container from `my-app:1.0`. The flags carry the meaning: - `-d` - **detached**: run in the background and hand you back the terminal (it printed the container's long ID). - `-p 8080:3000` - **publish a port**: forward port `8080` on your machine to port `3000` inside the container. The format is `HOST:CONTAINER`, and getting it backwards is a classic mistake. - `--name web` - give the container a friendly name instead of a random one, so you can refer to it later. ⚠️ **Gotcha.** `EXPOSE 3000` in the Dockerfile **does not** open the port to your machine - it's only documentation of what the app listens on. The port is only reachable from your laptop because of `-p 8080:3000` on `docker run`. Forgetting `-p` is the single most common "it works in the container but I can't reach it" mistake - we'll return to it in [Phase 3](03-volumes-and-the-gotchas.md). Check it's running and reach it: ```console $ docker ps CONTAINER ID IMAGE COMMAND STATUS PORTS NAMES 3f9a2c1b7e4d my-app:1.0 "node server.js" Up 6 seconds 0.0.0.0:8080->3000/tcp web $ curl http://localhost:8080 Hello from inside the container! ``` *What just happened:* `docker ps` lists *running* containers (add `-a` to also see stopped ones). The `PORTS` column confirms the mapping `0.0.0.0:8080->3000/tcp` - traffic to your port 8080 reaches the app's port 3000. The `curl` proves it end to end. When you're done, `docker stop web` halts it and `docker rm web` removes it. ## The registry: where images live so others can pull them An image on your laptop helps only you. A **registry** is a server that stores images so anyone (or any deploy server) can download them - Docker Hub is the default public one, companies run private ones. 📝 **Terminology.** *Registry* = the image store. *`pull`* = download an image from it. *`push`* = upload one to it. This is the actual mechanism behind "works on my machine" finally being true everywhere: you push the exact image you tested, and the server pulls the exact same bytes. ```console $ docker pull nginx:latest latest: Pulling from library/nginx a2abf6c4d29d: Pull complete ... Status: Downloaded newer image for nginx:latest $ docker tag my-app:1.0 yourname/my-app:1.0 $ docker push yourname/my-app:1.0 The push refers to repository [docker.io/yourname/my-app] 5f70bf18a086: Pushed ... 1.0: digest: sha256:9c1e... size: 1986 ``` *What just happened:* `pull` downloaded `nginx` layer by layer (notice it pulls *layers* - if you already have a layer, it's skipped, which is why related images share storage). To `push` your own image you first `tag` it with your registry username (`yourname/my-app`), then `push` sends up each layer Docker Hub doesn't already have. The deploy target later runs `docker pull yourname/my-app:1.0` and gets a byte-for-byte identical environment to the one you built. ## Recap 1. A **Dockerfile** is an ordered recipe; each instruction builds a read-only **layer**. 2. The **build cache** reuses unchanged layers and rebuilds everything from the first changed step onward - so order instructions least-changed first (dependencies before source). 3. **`docker build -t name:tag .`** turns the recipe into a tagged image. 4. **`docker run -d -p HOST:CONTAINER --name … image`** starts a container; `-p` is what actually opens a port to your machine (`EXPOSE` alone does not). 5. A **registry** stores images: **`pull`** to download, **`push`** to upload - the real machinery that makes an environment reproducible everywhere. Next, the part that surprises everyone: where your data goes when the container stops, and the traps that catch every newcomer. --- # Volumes & the Gotchas Here's where the calm picture from Phase 1 - "a container is a running instance, not a little computer" - turns into something you have to act on. Because containers are *instances*, anything they wrote vanishes when they're removed. That's not a bug; it's the design. This phase shows you where to keep data that must survive, how to feed config in cleanly, and the traps that catch nearly everyone - each named before it can ruin your afternoon. ## Cheat-card: symptom → calm fix Arrived here mid-problem? Find your symptom, then read the matching section below. | Symptom | Likely cause | Calm fix | |---|---|---| | Data gone after `docker rm` / restart | Wrote to the container's throwaway layer | Mount a **volume** for that path - see *Containers are ephemeral* | | "It works in the container but I can't reach the port" | Forgot `-p`, or wrong `HOST:CONTAINER` order, or app bound to `127.0.0.1` | See *The unpublished port* | | Image is enormous (hundreds of MB / GBs) | Heavy base image, build tools left in, files copied in | See *The huge image* | | A secret you set is still in the image | Baked in with `ENV` / `COPY` at build time - it's in a layer forever | See *Secrets in layers* | | Config differs per environment but it's hardcoded | Values baked into the image | Pass **environment variables** at run time - see *Config goes in env vars* | ## Containers are ephemeral - volumes are how data survives Recall from Phase 1 that a running container gets a thin **writable layer** on top of the frozen image. Every file the app creates - uploaded images, a database's data, logs - lands in that writable layer. And that layer belongs to the container, so when the container is removed, the layer is removed with it. The data is *gone*. ```mermaid flowchart LR subgraph WO["Without a volume"] C1["container
writable layer
(data here - dies on rm)"] end subgraph W["With a volume"] C2["container
/data"] end V["volume on host
outlives the container"] C2 -- "/data" --> V ``` A **volume** is storage that lives *outside* the container's lifecycle - managed by Docker on the host - and is mounted into the container at a path you choose. The app writes to that path as normal, but the bytes land in the volume, which survives the container being stopped, removed, and replaced. 📝 **Terminology.** *Volume* - Docker-managed persistent storage you attach to a container. There's also a *bind mount*, which maps a specific folder from your host machine into the container (handy in development, so editing a file on your laptop shows up live inside the container - the exact thing Phase 1 said wouldn't happen *unless you mount it in*). Run a database with a named volume so its data outlives the container: ```console $ docker run -d --name db \ -v pgdata:/var/lib/postgresql/data \ -e POSTGRES_PASSWORD=secret \ postgres:16 a7c3e9f1b2d4... ``` *What just happened:* `-v pgdata:/var/lib/postgresql/data` created (or reused) a named volume `pgdata` and mounted it at the path Postgres stores its data. Now you can `docker rm -f db` and start a fresh `postgres` container pointing at the same `pgdata` volume, and your tables are still there. Confirm the volume exists independently: ```console $ docker volume ls DRIVER VOLUME NAME local pgdata ``` *What just happened:* the volume is listed as its own object, not tied to the container. That separation is the whole point - the container is disposable, the volume is not. 💡 **Key point.** Treat containers as throwaway. Any data that must survive a restart belongs in a **volume**, never in the container's writable layer. If you ever think "I'll just keep it inside the container," that's the moment the data is one `docker rm` away from gone. ## Config goes in environment variables, not into the image You want the *same image* to run in development, staging, and production - that's the reproducibility promise. But those environments need different database URLs, API keys, and feature flags. Bake those values into the image and you've made three different images and lost the promise. **The fix:** pass configuration in at *run time* with `-e` (you already saw `-e POSTGRES_PASSWORD=secret` above). The image stays generic; the environment is supplied when the container starts. ```console $ docker run -d --name web \ -e DATABASE_URL=postgres://db:5432/app \ -e LOG_LEVEL=info \ -p 8080:3000 \ my-app:1.0 ``` *What just happened:* the app reads `DATABASE_URL` and `LOG_LEVEL` from its environment at startup. The same `my-app:1.0` image runs everywhere; only the `-e` values change. (Many of these? Graduate to an `--env-file` or to Compose - the [next guide](/guides/docker-compose-for-real-projects).) ## The gotchas everyone hits ### The unpublished port This is the most common "but it works in the container!" moment, so it earns first place. Three distinct causes wear the same disguise - the app runs fine inside, but you can't reach it from your machine: 1. **You forgot `-p` entirely.** `EXPOSE` in the Dockerfile is only documentation (Phase 2). Without `-p HOST:CONTAINER` on `docker run`, no port is published. Check `docker ps` - an empty `PORTS` column is the tell. 2. **You reversed the mapping.** `-p` is `HOST:CONTAINER`. If your app listens on 3000 inside, it's `-p 8080:3000`, not `-p 3000:8080`. Backwards, and you'll connect to a port nothing is listening on. 3. **The app bound to `127.0.0.1` inside the container.** A program listening on `localhost` *inside* the container is only reachable from inside that container - Docker's port forwarding can't reach it. The app must listen on `0.0.0.0` (all interfaces) for `-p` to work. This one is sneaky because the Docker command is correct; the app's own config is the problem. ```console $ docker ps CONTAINER ID IMAGE STATUS PORTS NAMES 3f9a2c1b7e4d my-app:1.0 Up 4 seconds web ``` *What just happened:* the `PORTS` column is empty - nothing is published. That's cause #1. Stop the container and re-run it with `-p 8080:3000`, and the mapping will appear. ### The huge image ⚠️ **Gotcha.** Reach for a full `ubuntu` or default `node` base "to be safe" and your image balloons to hundreds of megabytes or more - slow to push, pull, and deploy. Two habits keep it lean: - **Start from a slim base.** `node:20-slim` or `-alpine` variants, `python:3.12-slim`, etc., ship far less than the full image. (Verify a variant's contents before committing to it - `alpine` uses a different C library that occasionally trips up native dependencies.) - **Don't ship your build tools.** Compilers, dev dependencies, and caches don't belong in the final image. The standard fix is a **multi-stage build** (build in one stage, copy only the finished artifact into a clean final stage) - worth looking up once your image matters. A `.dockerignore` file (like `.gitignore`) keeps `node_modules`, `.git`, and local junk out of the build context entirely, so they're never copied in. ### Secrets in layers ⚠️ **Gotcha - this one is genuinely dangerous.** Anything you put into the image at *build time* becomes a permanent layer, and **layers are not erased by deleting the file in a later step.** `COPY` a private key in, or hardcode a password with `ENV` in the Dockerfile, and that secret is baked into the image's history. Anyone who can pull the image can extract it from the layers - even if a later instruction "removes" it, the earlier layer still holds it. The rule that keeps you safe: **secrets are run-time input, never build-time content.** Pass them with `-e` (or a secrets manager / Compose secrets) when the container *starts* - exactly like the config above - so they live in the running container's environment, not frozen into a distributable image. ## Recap 1. Containers are **ephemeral**: the writable layer dies with the container. Use a **volume** (`-v`) for any data that must survive. 2. A **volume** is host-managed storage mounted into the container; it outlives the container and shows up in `docker volume ls`. 3. **Config and secrets go in at run time** via `-e` environment variables - keeps one image usable everywhere, and keeps secrets out of the image. 4. The traps: the **unpublished port** (forgot `-p`, reversed `HOST:CONTAINER`, or app bound to `127.0.0.1`), the **huge image** (slim base + don't ship build tools), and **secrets in layers** (never bake them in). You can now reason about a single container end to end - build it, run it, reach it, persist it, and avoid the traps. The next step up is orchestrating *several* containers together - a web app, its database, and a cache, defined in one file and started with one command: [Docker Compose for Real Projects](/guides/docker-compose-for-real-projects). When it's time to put a container on a real server, see [Deploying to a VPS](/guides/deploying-to-a-vps).