# Load & Performance Testing > Your tests prove the app is correct; load tests prove it survives a thousand people showing up at once. Here's the mental model, the metrics that actually matter (percentiles, not averages), and how to run one test and read where it breaks. --- # Load & Performance Testing You've got green tests. Every unit test passes, the integration suite is clean, you clicked through the app yourself and it worked. So why does the launch announcement still tighten your stomach? Because every test you've written so far asked one question - *is it correct?* - and not the one that takes prod down at the worst possible moment: *does it still work when a thousand people show up at the same second?* Code that is perfectly correct for one user can fall apart under a crowd: the database connection pool runs dry, a query that was fine at ten rows crawls at ten million, memory creeps up over six hours until the process is killed. None of that shows up in a passing test suite - it shows up on launch day, unless you go looking for it first. This guide is about going looking for it first. It won't make your app faster (that's profiling, a different skill - we'll point you there). It answers a narrower, more urgent question: **will it hold?** ## How to read this - **Want it to finally make sense?** Read in order. Each phase builds on the last: first *why* load testing is its own discipline, then *what to measure*, then *how to run one and read the result*. - **Need a specific answer fast?** Phase 2 defines every metric and test type (throughput, latency percentiles, error rate; load vs. stress vs. soak vs. spike). Phase 3 is the hands-on workflow and how to spot the breaking point. ## The phases 1. **[Why Load-Test](01-why-load-test.md)** - the mental model: correctness tests ask "is it right?", load tests ask "does it survive a crowd?" The launch-day scenario, and what you're trying to learn before your users learn it for you. 2. **[The Metrics That Matter](02-the-metrics-that-matter.md)** - throughput, latency and why you measure *percentiles* not averages (the slow tail is what users feel), error rate under load - plus the four test types: load, stress, soak, spike. 3. **[Running One & Reading It](03-running-one-and-reading-it.md)** - the workflow end to end: pick a realistic scenario, ramp up virtual users, watch the numbers, and find the *knee* where latency spikes and errors climb. With an illustrative readout, and the traps that make the numbers lie. > Deliberately deferred: *why* a given endpoint is slow - flame graphs, query plans, CPU profiles, distributed tracing. That's a future **performance** category (profiling and observability). This guide stops at finding the *symptom* under load; the cause is a separate hunt. --- # Why Load-Test Here's the trap, and almost everyone falls into it once. The test suite is green, you've used the app yourself, a teammate clicked through it. By every signal you have, *it works*. So when someone asks "is it ready for launch?" you say yes - because the only question you know how to answer is *does it work?* Then launch day arrives, a few thousand people hit it in the same ten minutes, and it falls over. The confusing part: *nothing changed*. The code failing right now is the exact same code that passed every test this morning. Nothing broke. You just answered the wrong question. Once you see the two questions clearly, everything about load testing follows. ## Two completely different questions **What it actually is.** There are two separate things you can ask about a system, and they need different tools to answer: - **"Is it correct?"** - Given this input, do I get the right output? Does the login button log you in? Does the total add up? This is what your unit tests, integration tests, and end-to-end tests check. They run one path at a time and check the answer. - **"Does it hold?"** - When a *crowd* arrives at once, does it stay fast and keep answering, or does it slow to a crawl, start throwing errors, and tip over? This is what a **load test** checks. A correctness test runs your code. A load test runs your code *the way a crowd would* - many requests at the same time, sustained - and watches what happens to speed and stability. ```text CORRECTNESS TEST LOAD TEST ────────────────────────────── ────────────────────────────── one request at a time │ hundreds/thousands at once asks: is the output right? │ asks: is it still fast & up? passes or fails │ produces a curve (speed vs. load) green = the logic is correct │ green = it survives the crowd runs in CI on every commit │ runs before launch / big changes ``` **Why people get this wrong.** The instinct is that a green test suite means "ready." But correctness and capacity are independent. Perfectly correct code can be catastrophically slow under load, and - more confusingly - *fast* code can still collapse under a crowd for reasons that have nothing to do with the logic. The logic was never the bottleneck. Once you hold these as two separate questions, "it passed all the tests but died on launch day" stops being a mystery. The tests answered *correct?* They were never asked *holds?* Load testing is asking the second question on purpose, in private, before the crowd asks it for you in public. ## Why correct code falls over under a crowd If the code is right, what actually breaks? It's almost never the logic - it's a **shared, finite resource** that one user never touches the edge of, but a thousand users exhaust together. A few of the usual suspects: - **The database connection pool.** Your app keeps a small fixed set of connections to the database - say 20. One user borrows one for a few milliseconds and returns it; you'd never notice. But when 500 requests arrive at once and each wants a connection, 480 of them *wait in line* for one to free up. Requests that took 50 ms now take seconds, purely from queueing. - **Memory.** Each in-flight request holds some memory. One at a time, trivial. Thousands at once, and the process balloons - and if it crosses the limit, the operating system kills it (the "out of memory" story from the [Processes, Memory & CPU](/guides/processes-memory-and-cpu) guide). - **A query that doesn't scale.** A query with no index is fine at ten rows and a disaster at ten million. Your test database had ten rows. Production has ten million. The logic is identical; the behavior is not. - **An external dependency.** You call a payment API or a third-party service. It's fast for one call and rate-limits or queues you under a flood. 📝 **Terminology.** A *resource* here means anything the system has a limited amount of and must share across requests: CPU time, memory, database connections, file handles, network sockets, threads. Load problems are almost always a story about one of these running out. The pattern is the same every time: a resource that is effectively infinite for one user is sharply finite for a crowd. Correctness tests use one user, so they never see the edge. A load test exists to walk you up to that edge deliberately. 🪖 **War story.** A team launched a sign-up flow that passed every test and ran beautifully in the demo. The launch tweet went out; within minutes sign-ups were timing out. The code was fine - the cause was a connection pool of 10 against a sudden burst of hundreds of concurrent sign-ups, so 10 requests worked and the rest queued until they timed out. A thirty-minute load test the week before would have shown the same queue forming. Nobody had thought to ask the second question. ## What you're actually trying to learn A load test isn't a pass/fail gate like a unit test. You're not looking for a green checkmark - you're trying to *learn three numbers about your system* before reality teaches them to you: 1. **How much can it take?** At what level of traffic does it stop being fast and reliable? This is your **capacity** - the real ceiling, not the hopeful one. 2. **What happens *at* the edge?** When you push past comfortable, does it degrade gracefully (gets a bit slower, keeps serving) or fall off a cliff (errors spike, everything times out at once)? The difference decides whether a traffic spike is a slow afternoon or a full outage. 3. **Does it stay healthy over time?** Run it for hours, not minutes - does it hold steady, or does something slowly leak (memory, connections, disk) until it dies at hour six? A short test can't see this; a sustained one can. 📝 **Terminology.** A *virtual user* (VU) is the load tool's stand-in for one person using your app - it sends requests, waits a realistic moment ("think time"), then sends the next, just like a human would. You run a test by simulating many virtual users at once. We'll use the term throughout. ⚠️ **Gotcha - don't confuse "will it hold" with "why is it slow."** Load testing tells you *that* the system slows down or breaks at, say, 800 concurrent users - the symptom. It does **not** tell you *which line of code or query* caused it - the cause. Finding the cause is profiling and observability (flame graphs, query plans, traces), a separate discipline living in the future **performance** category. Load testing finds the breaking point; profiling explains it - trying to do both at once is how a clear afternoon turns into a confused one. ## Recap 1. There are **two different questions**: *is it correct?* (unit/integration/[e2e tests](/guides/unit-integration-e2e)) and *does it hold under a crowd?* (load tests). Green correctness tests say nothing about capacity. 2. Correct code falls over because a **shared finite resource** - connections, memory, a slow query, an external API - is fine for one user and exhausted by a crowd. 3. A load test isn't pass/fail; it's how you **learn your real capacity, how it behaves at the edge, and whether it stays healthy over time** - before your users find out for you. 4. Load testing finds the **symptom** (it breaks at N users); **profiling** finds the cause (why). This guide stays on the first. Now you know *why* you're running one. Next is what to actually watch while it runs - three numbers that tell the whole story, one of which is measured in a way that trips up almost everyone the first time. --- # The Metrics That Matter A load test throws a wall of numbers at you - graphs, tables, counters ticking up. It's tempting to fixate on the one big number ("we did 5,000 requests a second!") and miss the one quietly telling you the system is in trouble. There are only three measurements you truly need, and one of them - latency - is measured in a way that feels strange until you understand *why*, at which point it becomes the most important number on the screen. ## Throughput - how much work it's getting done **What it actually is.** Throughput is the **rate of work**: how many requests your system completes per second. You'll see it written as **req/s** (requests per second), sometimes **RPS** or, for full multi-step user journeys, transactions per second. It answers "how much is flowing through right now?" **What it does in real life.** As you add virtual users, throughput climbs - more users, more requests completed per second - until, at some point, it *stops climbing* even though you keep adding users. That flat ceiling means the system is completing work as fast as it possibly can. Pile on more users past that and they don't get served faster - they get served *slower*, queueing for a system already maxed out. ```text throughput (req/s) │ ┌────────────── ← ceiling: it can't go faster │ ┌───┘ │ ┌───┘ │ ┌───┘ │ ┌───┘ └──┴────────────────────────────▶ virtual users more users → more throughput … until it flattens. that flat line is your capacity. ``` ⚠️ **Gotcha.** High throughput is not the same as *healthy*. A system can report a big req/s number while half those requests are fast error responses (an error page returns quickly). Throughput tells you *how much* is flowing, never whether it's *good*. Always read it alongside latency and errors - never alone. ## Latency - how long each user waits **What it actually is.** Latency is the time between a request going out and the response coming back - **how long one user waits**. Throughput is the system's view ("work per second"); latency is the *user's* view ("how long did *I* sit there?"). It's measured in milliseconds (ms). **Why people get this wrong: the average lies.** The instinct is to track *average* latency, and it's almost always misleading. Imagine 100 requests: ninety-nine come back in 50 ms, one takes 5,000 ms because it got stuck behind a full connection pool. The average is about 100 ms - looks great, and tells you *nothing* about the user who waited five seconds. Average latency hides the disaster by drowning it in good results, and the slow ones aren't random noise - they're real people hitting the exact contention load testing exists to find. **What it does in real life: read percentiles instead.** A **percentile** answers "what was the experience for the slowest X% of requests?" It's the clear way to see the *tail* - the slow requests that an average smooths away. 📝 **Terminology.** A *percentile* (written **pN**) is the value below which N% of your measurements fall. **p50** (the *median*) = half of requests were faster than this, half slower - your typical experience. **p95** = 95% were faster, so this is roughly your "unlucky but not rare" user. **p99** = 99% were faster; this is the worst 1% - and on a busy site, 1% of requests is a *lot* of real, annoyed people. ```text 100 requests, sorted slowest-last: p50 ──────────────────┐ "typical user" (half are faster) p95 ──────────────────────────────┐ "unlucky user" (1 in 20) p99 ──────────────────────────────────────┐ "worst 1%" │ [50ms ········ 55ms · 60ms ···· 120ms · 480ms · 5000ms] └── the long slow tail is real users, and it's what they remember ──┘ ``` 💡 **Key point.** Users don't experience your *average* - they experience their *own* request. Watch **p95 and p99**: they are the felt experience of your unluckiest real users, and the first thing to spike when the system gets into trouble. The tail is the truth. ⚠️ **Gotcha.** Don't average percentiles together or across machines - a p99 of two servers is not the average of their two p99s. Percentiles have to be computed from the raw measurements pooled together. Most load tools do this for you; the point is to never hand-compute an "average p99," because the result is meaningless. ## Error rate - is it still actually working? **What it actually is.** Error rate is the share of requests that *failed* rather than returning a correct response - timeouts, dropped connections, and server errors (HTTP 5xx status codes). Under light load it should be effectively zero. Watching it climb as load rises is the clearest signal that you've pushed past what the system can handle. 📝 **Terminology.** A *5xx* is an HTTP status code in the 500–599 range - the server admitting *it* failed (500 Internal Server Error, 502 Bad Gateway, 503 Service Unavailable). Distinct from *4xx* codes (like 404), which mean the *client* asked for something wrong. Under load, the code you dread is 5xx: the server buckling. **What it does in real life.** Error rate is usually the *last* thing to go and the most decisive. The typical failure story: throughput flattens (maxed out) → latency climbs, especially p95/p99 (requests queueing) → errors begin (timeouts, outright rejections). By the time errors climb, users aren't just waiting - they're getting failures. That's the line you don't want to cross in production. 💡 **Key point.** Read all three together as one story. **Throughput** = how much work is getting done. **Latency (percentiles)** = how it feels to each user. **Error rate** = whether it's working at all. One number alone always misleads; the three together tell you exactly where on the curve you are. ## The four test types - same tool, four questions You drive all four with the same load tool and the same three metrics. What changes is the *shape* of the load, because each shape answers a different question. ```text virtual users over time: LOAD STRESS SOAK SPIKE ┌────┐ ┌──────┐ keep ┌──────────┐ │┐ │ │ │ │ pushing │ │ ││ sudden ┌┘ └ ┌┘ └→ till it │ steady, │ ─┘│ jump, then ┘ (hold ┘ breaks │ for hours│ ───── └─ drop at peak) └──────────┘ "expected "where's the "does it leak "can it survive peak - fine?" breaking point?" over time?" a sudden surge?" ``` - 📝 **Load test** - apply your **expected peak** traffic and hold it. The question: *at the busiest level we realistically expect, does it stay fast and error-free?* This is your baseline confidence check, the one you run most often. - 📝 **Stress test** - keep **increasing** load past expected peak until the system degrades or breaks. The question: *where's the ceiling, and how does it behave when we cross it - graceful slowdown or sudden collapse?* This finds the breaking point on purpose (Phase 3 is built around it). - 📝 **Soak test** (also called *endurance*) - apply a **moderate, sustained** load for a long time - hours, sometimes a full day. The question: *does anything slowly degrade?* Soak tests catch the bugs short tests can't see: memory leaks, connection leaks, disks filling with logs, caches growing without bound. The system looks perfect for ten minutes and dies at hour six. Only a soak finds that. - 📝 **Spike test** - jump from low to very high load **suddenly**, then drop back. The question: *can it absorb a flash crowd?* - the launch tweet, the TV mention, the flash sale at noon. Not just whether it survives the spike, but whether it *recovers* cleanly afterward or stays wedged. 💡 **Key point.** These aren't four different tools - they're four traffic *shapes* you apply with the same setup, each answering a question the others can't. "Will our normal Tuesday hold?" is a load test. "What's our actual ceiling?" is stress. "Will it survive the night?" is soak. "Can it take a sudden flood?" is spike. ## Recap 1. **Throughput** (req/s) = how much work gets done. It climbs with users, then flattens - that flat ceiling is your capacity. High throughput alone doesn't mean healthy. 2. **Latency** = how long each user waits. **Read percentiles (p50/p95/p99), never the average** - the average hides the slow tail, and the slow tail is exactly what real users feel. 3. **Error rate** (timeouts, 5xx) should be near zero and is the decisive signal you've gone too far. Read the three metrics together as one story. 4. **Four test types, one tool, four questions:** **load** (expected peak - fine?), **stress** (where's the breaking point?), **soak** (does it leak over hours?), **spike** (can it absorb a sudden surge?). Next, the hands-on part: pick a realistic scenario, ramp up the virtual users, and find the exact point where the curve turns - the breaking point. --- # Running One & Reading It You know why you're testing and what the numbers mean. Now the actual loop. A load test isn't a button you press for a verdict - it's a small experiment you design, run, and *read*. Done right, it ends with you pointing at one spot on a graph and saying "that's where we break, and it's at a level we won't hit for months" - the calm, boring outcome you want. The whole thing is four moves: pick a realistic scenario, ramp the load up, watch the three metrics, and find the knee. ## Step 1 - Pick a realistic scenario **What it actually is.** A scenario is the *story* of what a virtual user does - the sequence of requests that mimics a real person. Not just hammering one URL, but the journey: load the home page, search, view an item, add to cart, check out, each with a realistic pause ("think time") in between. **Why this matters.** If you blast a single trivial endpoint - say a `/health` check that returns "ok" and touches nothing - you'll get a gorgeous, enormous throughput number that means *nothing*, because no real user spends their day hitting your health check. The endpoints that break under load are the expensive ones: the search that runs a heavy query, the checkout that writes to the database and calls a payment API. Test the journeys that actually cost something, weighted roughly the way real traffic is. ⚠️ **Gotcha - the model is only as real as the inputs.** Vary your test data. If all ten thousand virtual users search for the same word and request the same product, your database serves it all from cache and reports dazzling numbers production will never reproduce, because real users search for *different* things and blow past that cache. Same for logins: reusing one account behaves nothing like thousands of distinct sessions. Realistic, *varied* data is the difference between a test that warns you and one that flatters you. ## Step 2 - Ramp up (don't slam) **What it actually is.** Ramping means adding virtual users *gradually* over time - start at a handful, climb steadily to your target - rather than launching all of them in the first instant. You configure a *ramp profile*: e.g. add 50 users every 30 seconds up to 1,000. **Why you ramp instead of slamming.** A gradual ramp lets you *see the breaking point coming* - as load rises smoothly, you watch latency and errors rise with it and read the exact level where things turn. Drop all 1,000 users in at once and everything degrades simultaneously, so you learn only "1,000 was too many," not *whether the trouble started at 400 or 900* - the number you actually came for. (Slamming is its own test, the **spike test** from Phase 2 - but for *finding capacity*, you ramp.) ```text a ramp profile: users 1000 │ ┌─────── ← hold at target │ ┌─────┘ 500 │ ┌─────┘ │ ┌─────┘ │ ┌─────┘ 0 └──────┴────────────────────────────────▶ time each step up = a new data point on your capacity curve ``` ## Step 3 - Watch the three metrics live As the ramp climbs, keep all three numbers from Phase 2 in view at once - they move as a story: - **Throughput** rises with the user count… until it flattens. Note the flat ceiling. - **Latency percentiles** (p95, p99) stay low and steady… until they start curving upward. Watch the tail first; p99 bends before p50 does. - **Error rate** sits at zero… until it lifts off. This is usually the last and most decisive move. The moment these three turn together is the whole point of the exercise. It has a name. ## Step 4 - Find the knee (the breaking point) **What it actually is.** The **knee** (also called the *breaking point* or *saturation point*) is the spot on the curve where the system stops scaling gracefully and starts falling apart: throughput flattens, latency turns sharply upward, and errors begin to climb, all around the same load level. Below the knee, more users get served fine; above it, more users just make everyone slower and then start failing. 📝 **Terminology.** The *knee* is the bend in the latency-vs-load curve - named because the line, flat-then-sharply-up, looks like a bent knee. It marks the capacity ceiling: the real number for "how much can this take?" ```text latency (p99) │ ╱ ← past the knee: latency explodes, │ ╱ errors climbing - the cliff │ ╱ │ ╱ │ ─────────────────╱ ← THE KNEE: capacity ceiling │ ──────────────── (flat & healthy below it) └──────────────────────────────────▶ virtual users comfortable breaking ``` **A real example - reading a stress-test readout.** Here's a ramp result table. The shape of these numbers is **illustrative - not a measurement of any real system** - to show you how to *read* one, not what your server will do. ```console $ k6 run --vus-max 1000 ramp-checkout.js scenarios: ramping from 0 to 1000 VUs over 10m, then hold 5m VUs throughput p50 p95 p99 error% ---- ----------- ----- ------ ------ ------ 100 480 r/s 42ms 88ms 120ms 0.00% 300 1,410 r/s 45ms 96ms 140ms 0.00% 500 2,300 r/s 51ms 120ms 210ms 0.01% 700 2,950 r/s 68ms 240ms 680ms 0.04% 800 3,050 r/s 110ms 520ms 1,900ms 0.20% 900 3,040 r/s 280ms 2,100ms 6,400ms 3.10% 1000 2,780 r/s 640ms 5,800ms 14,000ms 11.40% ``` *What just happened:* (illustrative figures) Read it top to bottom as a story. From 100 to 500 users everything is healthy - throughput climbs in step with users, latency is calm (p99 around 120–210 ms), errors essentially zero. At **700** the first cracks show: throughput growth is slowing and p99 has jumped to 680 ms, the tail stretching even though the typical user (p50, 68 ms) still feels fine. At **800** throughput has basically *stopped climbing* (3,050 r/s, the ceiling) while p99 crosses past a second and errors tick up. By **900–1000** it's a cliff: adding users no longer adds throughput (it's *dropping*), p99 blows out to many seconds, errors hit double digits - real users getting failures, not just waits. **The knee is right around 700–800 users.** That's your real capacity: comfortable headroom if you expect 300 concurrent users at launch, a problem to fix *now* if you expect 900. ⚠️ **Gotcha - test like production, or the numbers lie.** This is the single biggest way load tests betray you. The result above is only meaningful if the test ran against an environment that *matches production* in the ways that bite: - **Data volume.** A query against 10,000 rows behaves nothing like the same query against 50 million. Test on production-scale data, or your latency numbers are fiction. - **Environment shape.** A laptop, or a "staging" box with a quarter of prod's memory and one CPU, will find a fake breaking point far below the real one - or hide a real one you'd hit in prod. - **Realistic, varied inputs** (from Step 1) - distinct users, varied searches, so caches behave the way they will in real life. A load test against a tiny, empty, single-core environment produces confident, precise, *wrong* numbers - arguably worse than no test, because it tells you you're safe when you aren't. Can't test against true production scale? Say so out loud and treat the result as a rough floor, not a guarantee. ## When it breaks: symptom, not cause You found the knee at ~800 users and decided that's not enough headroom. Now what? Here's the boundary of this guide, stated plainly: The load test told you the **symptom** - *"it saturates around 800 concurrent users; p99 and errors explode there."* It did **not** tell you the **cause** - *which* resource ran out, *which* query went quadratic, *where* the time actually went. Those are different questions answered by different tools: profilers, query plan analysis, flame graphs, distributed tracing - the **performance** category (profiling and observability), a separate hunt with its own guide. ⚠️ **Gotcha.** Don't try to *guess* the cause from the load test alone. The temptation is to eyeball the knee, declare "must be the database," and start adding indexes. Sometimes you're right; often you're not, and you burn a day optimizing the wrong thing. The disciplined order: **load test to find the breaking point → profile/observe to find the cause → fix → load test again to confirm the knee moved.** Load testing is how you *measure*; profiling is how you *diagnose*; re-running the load test proves the fix worked. 🪖 **War story.** A team load-tested a reporting endpoint, found a knee at a few hundred users, and "knew" it was the database - so they spent a sprint adding indexes and a read replica. The knee barely moved. When they finally profiled it, the time was going to JSON serialization of an enormous response payload in the app layer; the database was never the bottleneck. The load test had correctly found *where* it broke - only the profiler found *why*. ## Recap 1. **Pick a realistic scenario** - model real user *journeys* with varied data, not one trivial endpoint, or the numbers flatter you. 2. **Ramp up gradually**, don't slam - a smooth ramp lets you *see the breaking point arrive* and read the exact level where it turns. 3. **Watch throughput, latency percentiles, and error rate together** as the load climbs. 4. **Find the knee** - where throughput flattens, p95/p99 curve sharply up, and errors lift off. That's your real capacity; compare it to the traffic you actually expect. 5. **Test like production** (data volume, environment, varied inputs) or the result is confident fiction. A load test finds the **symptom**; **profiling** (a future performance guide) finds the cause - keep those two jobs, and that order, separate. That's the full loop. You can now answer the question that started this guide - *will it hold?* - with a number and a graph instead of a launch-day stomach-drop.