# Testing in CI (What Runs on Every Push)

> CI is a server that automatically runs your test suite on every push and pull request, so broken code can't quietly reach main - here's the mental model, what a run actually does, and how to keep it trustworthy.


---

# Testing in CI (What Runs on Every Push)

You wrote a test. It passes on your machine. You push, open a pull request - and a few minutes later
a little check turns green (or red) next to your branch, run by a computer you've never logged into.
If you've ever wondered *what that check actually is*, who runs it, and why "it passed locally" stops
being good enough on a real team, this guide is for you.

This is the **testing** side of CI: tests as an automatic gate that stands between your code and the
shared `main` branch. We're staying on the test step on purpose - the wider world of building,
packaging, and deploying (CI/CD as a whole) is a separate topic for another day.

## How to read this

- **Just need the gist of that red/green check?** Read [Phase 1](01-what-ci-testing-actually-is.md) -
  it's the whole mental model in one phase.
- **Want it to finally make sense?** Read in order. Phase 1 is the idea, Phase 2 is what a run does
  step by step, Phase 3 is how teams keep the gate trustworthy.

## The phases

1. **[What CI Testing Actually Is](01-what-ci-testing-actually-is.md)** - the mental model: a server runs
   your whole test suite automatically on every push and pull request, so nobody merges broken code.
   Why this beats "it passed on my machine."
2. **[Inside the Pipeline](02-inside-the-pipeline.md)** - what a single CI run actually does: check out
   the code, install dependencies, run lint and tests, report pass or fail. An annotated GitHub Actions
   config and a real run log, focused on the test step.
3. **[Keeping CI Trustworthy](03-keeping-ci-trustworthy.md)** - the one thing that destroys CI's value:
   flaky tests. Why flakiness happens, how to keep the suite fast and reliable, and how required checks
   protect `main`.

> The mechanics of *deploying* what CI builds - environments, release pipelines, rollbacks - are
> deliberately out of scope here. This guide is about tests as a gate, not the full CI/CD machine.

**Related guides:** [Unit, Integration & E2E](/guides/unit-integration-e2e) (the test pyramid that keeps
CI fast) · [Git With Other People](/guides/git-with-other-people) (pull requests, the place these checks
show up).


---

# What CI Testing Actually Is

Picture the moment you open a pull request. You push your branch, the page loads, and there's a small
spinner next to your commit that turns into a green check or a red X. Most people learn to read that
check long before they understand it - green good, red bad, merge when green. That's fine for surviving,
but the moment a check goes red for a reason you can't reproduce, the mystery becomes a problem.

## The mental model: a tireless teammate who runs your tests

**What it actually is.** *CI* - continuous integration - is, for our purposes, a **server that
automatically runs your test suite every time anyone pushes code.** That's the whole core idea. Not a
person. Not your laptop. A machine that wakes up on every push, grabs your code, runs the tests, and
reports back: passed or failed.

📝 **Terminology.** *CI* stands for *continuous integration* - the practice of merging everyone's work
together often and checking it automatically each time. The *server* that does the checking is the
**CI server** (or *CI runner*). Common ones: GitHub Actions, GitLab CI, CircleCI, Jenkins. They differ
in setup but do the same job.

**Why people get this wrong.** The common picture is that CI is some elaborate, mysterious deployment
robot. It can grow into that - but the heart of it is humble: it runs the same `npm test` or `pytest`
or `cargo test` you already run, just on a fresh machine, automatically, every time. If you can run
your tests in a terminal, you already understand 90% of what CI does. The other 10% is *where* and
*when* it runs.

**What it does in real life.** Every push triggers a fresh run. You don't click anything. You don't
remember to do it. The robot doesn't get tired, doesn't skip the slow test because it's late, and
doesn't forget. That reliability is the entire point - humans forget to run tests; the server never does.

```mermaid
flowchart TD
  Push[You push a branch] --> Wake[CI server wakes up]
  Wake --> Clean[1. grab a clean machine]
  Clean --> Code[2. download your code]
  Code --> Deps[3. install dependencies]
  Deps --> Test[4. run the test suite]
  Test --> Report[Report back to your PR: green or red]
```

## The green check is the gate

**What it actually is.** That check next to your pull request is the CI server's verdict, posted right
where the team makes the merge decision. Green means *every test passed on a clean machine.* Red means
*at least one failed* - and you can click in to see exactly which one.

**Why this matters more than it looks.** The check isn't just information; on most teams it's a **gate.**
A repository can be configured so the merge button stays disabled until the check is green. That single
setting changes the social contract of the team: nobody can merge broken code into `main`, not even by
accident, not even the person who set up the project. The rule is enforced by a machine, so it applies
to everyone equally.

```text
   Pull request: "Add cart subtotal"

      ✓  Tests passed (142 passed)            ← green: merge button is enabled
   ─────────────────────────────────────
      ✗  Tests failed (1 failed, 141 passed)  ← red: merge button is blocked
```

💡 **Key point.** CI testing turns "we *should* run the tests before merging" (a hope) into "you
*cannot* merge unless the tests pass" (a guarantee). The value isn't the tests - you already had those.
The value is that they now run automatically and block bad merges.

## Why this beats "it passed on my machine"

You've heard the phrase, maybe said it. The tests pass for you, fail for someone else, and the argument
goes in circles. CI ends that argument, because it removes *your machine* from the equation entirely.

Here's why your machine lies to you, gently and constantly:

- **You have things installed that you forgot about.** A global tool, a specific language version, an
  environment variable set months ago. Your tests quietly depend on it. The new teammate doesn't have
  it, and neither does the CI server.
- **You didn't commit everything.** A new file you forgot to `git add`, a dependency you installed
  locally but never wrote into `package.json`. It works for you because it's *on your disk* - not
  because it's *in the repository.*
- **Your machine is in a particular state.** A database left running from yesterday, a cache, a built
  artifact. The next person starts from nothing.

**What CI does about it.** The CI server starts from a **clean, empty machine every single time.** It
has nothing installed that you didn't declare. It has nothing on disk except exactly what's committed to
the repository. So if your tests pass in CI, they pass *from a clean checkout of what's actually in the
repo* - which is the only state that matters, because it's the state every teammate and every deploy
starts from.

🪖 **War story.** A teammate once spent a morning convinced a colleague had broken the build "out of
nowhere." The tests passed on his machine, failed in CI. The culprit: a config file he'd created weeks
earlier, used in every test, and never committed - it lived on his laptop and nowhere else. For him the
suite was green forever; for the clean CI machine, and for every new hire, it had never passed once. CI
didn't break anything. It told the truth his laptop had been hiding.

⚠️ **Gotcha.** "It passed on my machine" and "it passed in CI" are answers to *different questions.*
Yours answers "does it work in my exact setup?" CI answers "does it work from a clean copy of the
repo?" When they disagree, **CI is almost always the one to trust** - because a clean checkout is what
production gets, not your laptop.

**Why this saves you later.** Once you internalize that CI runs from a clean room, red checks stop
feeling like personal attacks and start being useful clues. A test green for you and red in CI is rarely
the universe being unfair - it's almost always *something on your machine that isn't in the repo.*

## Recap

1. **CI is a server that automatically runs your test suite** on every push and pull request - the same
   tests you run locally, just on a fresh machine, every time, without anyone remembering to.
2. **The green/red check on a PR is the server's verdict**, posted where the merge decision happens.
3. Teams turn that check into a **gate**: the merge button stays blocked until tests pass, so broken
   code can't reach `main`.
4. CI beats "it passed on my machine" because it **starts from a clean checkout** - nothing installed,
   nothing on disk except what's committed - which is the only state that actually matters.


---

# Inside the Pipeline

You know *what* CI is now: a server that runs your tests on a clean machine. This phase opens the hood
and walks through *what a single run actually does*, step by step, then shows you the file that tells it
to do that. The goal isn't to make you a CI-config expert; it's so that when a run fails, you can read
the log and know which step broke and why.

## What one run does, in order

**What it actually is.** A CI run is a short, ordered checklist the server works through. Every system
calls the pieces something slightly different, but the shape is almost always the same:

```mermaid
flowchart TD
  Checkout[1. Check out - download the exact commit you pushed]
  Setup[2. Set up - install the language/runtime version]
  Install[3. Install - fetch the project's dependencies]
  Lint[4. Lint - check style/formatting, optional]
  Test[5. Test - run the suite - the part we care about most]
  Report[6. Report - pass = green, any failure = red]
  Checkout --> Setup --> Install --> Lint --> Test --> Report
```

📝 **Terminology.** The whole sequence is often called a **pipeline** or a **workflow**. One unit of
work inside it (a self-contained run on one machine) is a **job**. Each line a job runs is a **step**.
In GitHub Actions specifically, the file that defines all this is a **workflow**.

**Why the early steps matter for testing.** Steps 1–3 are setup, but they're where a surprising number
of "test failures" actually come from. If *install* fails - a dependency that no longer exists, a
version conflict - the run goes red before a single test executes. That's a broken environment, not a
broken test. Reading the log top to bottom tells you which it is.

## Failing the build, and fast feedback

**What it actually is.** Each step either succeeds or fails. The moment a step fails, the job stops and
the whole run is marked failed - this is called **failing the build.** "The build is red" just means
some step (often the test step) failed.

**Why CI stops on the first failure.** There's no point running the deploy steps if the tests failed -
you'd be shipping broken code. So the run halts at the first failure and reports immediately. This is
the **fast feedback** loop: you push, and within minutes you know whether your change is safe, while the
change is still fresh in your head. The longer feedback takes, the more you've moved on and the more it
costs to come back.

💡 **Key point.** "Failing the build" doesn't mean you broke something catastrophic. It means a check
that's *supposed* to catch problems caught one. A red build is the system working, not the system
breaking.

## A real (small) CI config

Here's a complete, minimal GitHub Actions workflow for a Node.js project. It lives in the repository at
`.github/workflows/ci.yml`, so it travels with the code. Read the comments - they map each line to the
checklist above.

```yaml
# .github/workflows/ci.yml
name: CI

# WHEN to run: on every push, and on every pull request.
on: [push, pull_request]

jobs:
  test:
    # WHERE to run: a fresh, clean Ubuntu machine, provided by GitHub.
    runs-on: ubuntu-latest

    steps:
      # 1. Check out - download the exact commit being tested.
      - uses: actions/checkout@v4

      # 2. Set up - install Node 20 on the clean machine.
      - uses: actions/setup-node@v4
        with:
          node-version: '20'

      # 3. Install - fetch dependencies from package-lock.json.
      - run: npm ci

      # 4. Lint - check formatting/style (optional but common).
      - run: npm run lint

      # 5. Test - THE part this guide is about.
      - run: npm test
```

*What just happened:* This file tells GitHub, "on every push or pull request, spin up a clean Ubuntu
machine, put Node 20 on it, install exactly the dependencies the lockfile pins, check the code style,
then run the test suite." The `test` job goes green only if **every** step exits successfully. If
`npm test` reports a single failing test, that step fails, the job fails, and the PR's check turns red.

⚠️ **Gotcha.** Notice `npm ci`, not `npm install`. `npm ci` installs *exactly* what's pinned in
`package-lock.json` and errors if the lockfile is out of sync - which is precisely what you want on a
clean CI machine, where "install whatever's newest" would let untested dependency versions sneak in.
(`ci` here is npm's "clean install" command - an unrelated coincidence of letters with *continuous
integration*.)

## Reading the run log

When a run finishes, you get a log - each step with a result and its output. Here's what the test step
looks like when everything passes:

```console
Run npm test

> myapp@1.0.0 test
> jest

 PASS  src/cart.test.js
 PASS  src/pricing.test.js
 PASS  src/checkout.test.js

Test Suites: 3 passed, 3 total
Tests:       142 passed, 142 total
Time:        6.4 s
```

*What just happened:* The server ran `npm test`, which ran Jest, which executed every test file. All 142
tests passed, so the step exited successfully and contributed a green check. The numbers here are
illustrative - your own run prints your real suite's counts and timing.

And here's the same step when one test fails:

```console
Run npm test

> myapp@1.0.0 test
> jest

 PASS  src/cart.test.js
 FAIL  src/pricing.test.js
   ● applies promo code to subtotal

     expect(received).toBe(expected)

     Expected: 90
     Received: 100

       at Object.<anonymous> (src/pricing.test.js:24:31)

 PASS  src/checkout.test.js

Test Suites: 1 failed, 2 passed, 3 total
Tests:       1 failed, 141 passed, 142 total
Time:        6.6 s
Error: Process completed with exit code 1.
```

*What just happened:* One test - `applies promo code to subtotal` - expected `90` but got `100`. Jest
reported it, then exited with code `1` (the universal "I failed" signal). CI saw the non-zero exit code,
marked the step failed, failed the job, and turned the PR red. **The log tells you exactly which test,
what it expected, what it got, and the file and line** - you don't have to guess, just re-run that one
test locally.

📝 **Terminology.** An **exit code** is the number a command returns when it finishes: `0` means
success, anything else means failure. This is how CI knows whether a step passed without understanding
anything about your test framework - it just checks the exit code. Your test runner exits `0` when all
tests pass and non-zero when any fail; that's the entire contract.

## Testing across versions and OSes (the matrix)

Sometimes "it passes" needs to mean "it passes *everywhere we support*." If your project must work on
several language versions or operating systems, you can ask CI to run the same job across a grid of
combinations - called a **build matrix.**

```yaml
jobs:
  test:
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        os: [ubuntu-latest, windows-latest]
        node-version: ['18', '20']
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: ${{ matrix.node-version }}
      - run: npm ci
      - run: npm test
```

*What just happened:* This runs the test job **four** times - Node 18 and Node 20, each on Ubuntu and
Windows (2 × 2). Each combination gets its own clean machine and its own green/red result; the overall
check passes only if all four pass. This catches the classic "works on my Node version / my OS but not
yours" bug before it reaches anyone.

⚠️ **Gotcha.** A matrix multiplies your CI time and cost - four combinations means four full test runs
per push. Add OSes and versions only where you genuinely support them. A library shipped to the world
needs a broad matrix; an internal app that only ever runs on one Node version on Linux does not. More on
keeping the suite fast in [Phase 3](03-keeping-ci-trustworthy.md).

**Why this saves you later.** When a matrix run is red on `windows-latest` but green on `ubuntu-latest`,
you've learned something specific and valuable for free: your code has an OS-specific assumption (a file
path, a line ending, a case-sensitive import) - a bug a single-machine setup would have shipped to a
Windows user to discover the hard way.

## Recap

1. A CI run is an **ordered checklist**: check out the code, set up the runtime, install dependencies,
   lint, **run the tests**, report.
2. Any step failing **fails the build** - the run stops and reports red. That's fast feedback, not
   catastrophe.
3. The config is a file in the repo (`.github/workflows/ci.yml` for GitHub Actions) that says *when*,
   *where*, and *what steps* to run.
4. CI reads **exit codes**: your test runner exits `0` on success, non-zero on failure, and the log
   names the exact failing test, expected vs. actual, file and line.
5. A **build matrix** runs the same tests across versions/OSes to catch "works on mine, not yours"
   bugs - at the cost of more CI time.


---

# Keeping CI Trustworthy

CI only works if people believe it. A green check is a promise - "this code is safe to merge" - and the
entire value of CI rests on that promise being true. The fastest way to destroy a CI setup isn't a broken
server or a bad config. It's something quieter and far more corrosive: tests that lie. Once the team
stops trusting red, you've lost the gate, with no warning.

## The cheat-card: red check, calm response

When a check goes red, work down this list before assuming you broke something:

| Symptom | Likely cause | Calm first move |
|---|---|---|
| Red in CI, green locally | Something on your machine isn't in the repo | Check for uncommitted files / undeclared deps (see [Phase 1](01-what-ci-testing-actually-is.md)) |
| Red on *install*, before tests run | Dependency / lockfile problem, not a test | Read the log - fix the environment, not the test |
| Red on one matrix cell only | OS- or version-specific bug | Reproduce on *that* version/OS |
| **Passed, then re-ran and passed** (was red) | **Flaky test** | Don't shrug - quarantine and fix it (below) |
| Red on a clear assertion failure | A real bug your test caught | Read expected vs. actual; fix the code |

The dangerous row is the flaky one. Let's give it the attention it deserves.

## ⚠️ Flaky tests: the thing that kills CI

**What it actually is.** A *flaky test* is a test that **passes or fails randomly on the exact same
code.** You change nothing, re-run the build, and the result flips. It's not testing your code anymore -
it's flipping a coin.

📝 **Terminology.** "Flaky" is the standard industry word for *non-deterministic* test results: the same
input doesn't always give the same output. A reliable test is *deterministic* - same code, same result,
every time.

**Why this is so much worse than a normal failure.** A real failure is straight with you: it tells you something is
broken, you fix it, it goes green. A flaky failure teaches the team a poison lesson - *red doesn't
necessarily mean broken.* Once people learn that, here's what happens, every time:

```text
   Day 1:   Red build.  "Probably just flaky." → click re-run → green → merge.
   Day 30:  Red build.  "Just re-run it." → green → merge.  (Nobody reads it.)
   Day 90:  Red build that is a REAL bug.  → "just re-run it" → still red →
            "ugh, the CI is flaky again" → merge anyway → ships the bug.
```

That last line is the catastrophe: the flaky tests trained everyone to ignore red, so when red finally
*meant* something, nobody listened. **A flaky suite is worse than no suite** - no suite at least doesn't
lull you into false confidence.

💡 **Key point.** Flakiness doesn't cost you one test. It slowly costs you the team's trust in *every*
test. Treat a flaky test as a real bug - a bug in your test suite - not as background noise.

## Why tests go flaky

Flakiness almost always traces back to one of three sources. Knowing them lets you spot a flaky test by
its shape:

**1. Timing.** The test assumes something finished by now, when sometimes it hasn't.

```javascript
// Flaky: hopes the data loads within 100ms. On a busy CI machine, sometimes it doesn't.
await sleep(100);
expect(screen.getText()).toBe('Loaded');

// Reliable: waits for the actual condition, however long it takes.
await waitFor(() => expect(screen.getText()).toBe('Loaded'));
```

*What just happened:* The first version races the clock - a `sleep` is a guess, and a busy CI machine is
slower than your laptop, so the guess sometimes loses. The second waits for the *thing you actually care
about* instead of a fixed delay. **Fixed sleeps are the single most common source of flakiness.** Wait
for conditions, not for time.

**2. Test order / shared state.** A test passes alone but fails when run after another, because they
share something - a database row, a global variable, a file, a clock.

```text
   test A:  creates user "alice", does NOT clean up
   test B:  asserts "no users exist"   ← passes alone, FAILS after A

   Run order [A, B] → B fails.   Run order [B, A] → B passes.
   CI shuffles or parallelizes order → result flips → "flaky."
```

*What just happened:* Neither test is wrong in isolation, but they leak state into each other. The fix is
**isolation**: each test sets up and tears down its own data, assuming nothing about what ran before.

**3. External dependencies.** The test reaches out to a real network service, a live API, a real clock,
or randomness. Anything outside your control can hiccup, and your test fails for a reason that has
nothing to do with your code. The fix is to *control* those edges - fake the network, fix the clock,
seed the randomness - so the test only fails when *your* code is actually wrong.

⚠️ **Gotcha.** The instinct when a test flakes is to add a retry or a longer `sleep` and move on. That
doesn't fix flakiness - it *hides* it, makes the suite slower, and lets the underlying race survive to
bite again. A retry around a genuinely flaky test is sweeping the coin-flip under the rug.

## When you can't fix it right now: quarantine

Sometimes you find a flaky test and can't fix it this minute. The right move isn't to leave it failing
randomly (poisoning trust) or to delete it (losing coverage) - it's to **quarantine** it: mark it as
skipped, with a tracking ticket, so the build goes reliably green again while the flaky test waits to be
fixed.

```javascript
// Skip until fixed - keeps the build trustworthy without losing the test.
test.skip('reconnects after network drop - FLAKY, see issue #482', () => {
  // ...
});
```

*What just happened:* The test is temporarily removed from the gate, with a breadcrumb (`#482`) so it's
tracked, not forgotten. The build's green means something again - quarantine is a holding cell, not a graveyard.

## Keeping the suite fast (so people run it)

A slow suite causes flakiness's cousin: people stop waiting for it. If CI takes 40 minutes, developers
merge before it finishes, context-switch away, and the feedback loop you built for safety becomes a
formality. Speed is a reliability feature.

The biggest lever is the **shape** of your suite - the test pyramid. Many fast, isolated unit tests at
the base; fewer integration tests in the middle; a handful of slow end-to-end tests at the top. A suite
that's mostly slow E2E tests is both *slow* and *flaky-prone* (E2E tests touch the most moving parts).
The pyramid keeps CI fast *and* reliable at once.

> ⏭️ The pyramid - what each layer tests and why the proportions matter - is its own topic. See
> [Unit, Integration & E2E](/guides/unit-integration-e2e) for the full picture. Here, the takeaway:
> **the pyramid is what keeps CI fast enough to trust.**

A few other practical levers, in rough order of payoff:

- **Run tests in parallel** across CPUs or machines - most runners support it.
- **Cache dependencies** between runs so `install` doesn't re-download the world every time.
- **Fix or quarantine flaky tests promptly** - a flaky suite wastes time on re-runs, which is its own
  slowness tax.

## Required checks: making the gate real

**What it actually is.** A *required status check* is a repository setting that says: *this check must be
green before the merge button is enabled.* It's what turns "we have CI" into "CI actually protects
`main`."

**Why you need it.** Without it, CI is advisory - a red check is just a suggestion someone can merge past.
With required checks, the platform refuses the merge until the gate is green. On GitHub this lives under
**branch protection rules** for `main`:

```text
   Branch protection on `main`:
     ☑ Require status checks to pass before merging
         └ required check:  CI / test
     ☑ Require branches to be up to date before merging
     ☑ Require a pull request before merging
```

*What just happened:* With these on, nobody - including admins, unless they explicitly override - can
merge into `main` until the `CI / test` check is green and the branch includes the latest `main`. The
gate is no longer a social norm you hope people honor; it's enforced by the platform for everyone.

📝 **Terminology.** **Branch protection** (GitHub's name; GitLab calls them *protected branches* with
*merge request approval rules*) is the set of rules guarding a branch - required checks, required
reviews, "must be up to date." Required *status checks* are the CI piece of that.

🪖 **War story.** A team added CI, felt safe, and kept shipping bugs to `main` for weeks - because the
checks ran but were never marked *required.* People glanced at red, judged it "probably flaky," and
merged anyway. The fix was one checkbox: make the check required. Suddenly red meant *blocked*, not
*ignorable*, and the bugs stopped reaching `main`. CI you can merge past isn't a gate; it's a decoration.

## Recap

1. CI's whole value is **trust in the green check** - protect it ruthlessly.
2. **Flaky tests** (pass/fail randomly on the same code) are the top threat: they teach the team that
   red is ignorable, so real failures get ignored too. A flaky suite is worse than no suite.
3. Flakiness comes from **timing** (fixed sleeps), **shared state / order**, and **external
   dependencies** - fix by waiting on conditions, isolating tests, and controlling the edges.
4. Can't fix now? **Quarantine** the test (skip + tracking ticket) so the build stays reliably green.
5. Keep the suite **fast** - mostly via the test pyramid ([Unit, Integration & E2E](/guides/unit-integration-e2e)),
   plus parallelism and caching - so people actually wait for it.
6. **Required status checks / branch protection** make the gate real: the merge button stays blocked
   until CI is green. CI you can merge past is decoration, not protection.