Latency vs Throughput

Here's where most performance confusion starts. Two people look at the same system. One says "it's slow" - they clicked a button and waited four seconds. The other says "it's fine" - the server is handling thousands of requests a minute without breaking a sweat. They're both right, because they're talking about two completely different numbers. Once you can tell those two numbers apart, half the fog clears.

The two numbers

What they actually are.

• Latency is how long one thing takes, start to finish. You click; how long until you see the result? That's latency. It's a duration - measured in milliseconds or seconds.
• Throughput is how many things the system finishes in a given time. How many requests per second can the server handle? How many photos per minute can the pipeline process? That's throughput. It's a rate - measured in things-per-second.

Latency is about waiting. Throughput is about volume. They feel like the same idea - "fast" - but they answer different questions, and a system can be great at one while terrible at the other.

📝 Terminology. Latency = the time for a single operation to complete (a duration). Throughput = the number of operations completed per unit of time (a rate). When someone says "it's slow," your first job is to find out which one they mean.

The highway analogy

This is the picture that makes it stick. Think of work flowing through your system like cars driving down a highway.

                     ┌─────────────────────────────────────────┐
   a car enters ───► │  ═══════  the highway  ═══════           │ ───► a car exits
                     └─────────────────────────────────────────┘

   LATENCY    = how long ONE car takes to drive end to end
                (does NOT change if you add more lanes)

   THROUGHPUT = how many cars exit per minute
                (MORE lanes  ->  more cars per minute,
                 even though each car's drive takes the same time)

• Latency is how long it takes a single car to drive from the on-ramp to the off-ramp. Adding lanes doesn't make that one car arrive any sooner - its trip is the same length.
• Throughput is how many cars get off the highway per minute. Add lanes and you move far more cars per minute, even though each individual car's drive took exactly as long as before.

That's the whole insight: widening the road raises throughput without touching latency. More lanes (more servers, more worker threads, more parallel processing) lets you serve more users at once - but it does nothing for the user who's stuck behind a slow operation. Their one car still takes four seconds.

Why improving one can hurt the other

Here's the part that surprises people, and it's worth slowing down for. These two numbers aren't just independent - pushing on one can actively drag down the other.

The classic example is batching. Imagine your system sends data somewhere. You can send each item the instant it arrives, or you can wait, collect a hundred items, and send them all in one trip.

   SEND IMMEDIATELY            BATCH OF 100
   (low latency)              (high throughput)

   item ─► send                item ─┐
   item ─► send                item ─┤  wait... collect...
   item ─► send                item ─┤  ...then send all 100
                                ...  ─┘  in one trip

   each item leaves fast       each item waits for the batch,
   but you pay the per-trip    but one trip carries 100 items,
   cost 100 times              so far more move per second

Batching is great for throughput: one trip carries a hundred items instead of one, so the per-item overhead (the cost of opening a connection, the round-trip across the network) gets paid once instead of a hundred times. More items move per second.

But every item now waits for the batch to fill before it goes anywhere. The first item you collected sat there while ninety-nine more arrived. You raised throughput by adding latency. That's not a bug - it's a deliberate trade, and engineers make it on purpose all the time. The point is to make it on purpose, knowing what you're giving up.

⚠️ The trap: optimizing the number nobody cares about. A team proudly "improves performance" by batching aggressively - throughput goes way up, the dashboards look great. Meanwhile every user's action now takes an extra half-second because their request is waiting in a batch. They optimized throughput and degraded the thing users actually feel. Always know which number you're moving, and which number your users care about. They're often not the same number.

Why this saves you later. The next time a ticket says "make it faster," you won't start coding. You'll ask one question: do you mean each operation should take less time, or do you mean the system should handle more at once? Those lead to completely different fixes. Reducing latency might mean removing a slow step from a single request. Raising throughput might mean running more copies in parallel. Confuse them and you'll add four servers to a problem that needed one slow database query fixed - or rewrite a query to help a system that just needed more lanes.

Recap

1. Latency = how long one operation takes (a duration). Throughput = how many operations finish per second (a rate). Different numbers, different questions.
2. The highway: latency is one car's drive time; throughput is cars-per-minute. Adding lanes raises throughput without changing any single car's latency.
3. Improving one can hurt the other - batching raises throughput by adding latency. Trade on purpose, not by accident.
4. When someone says "slow," find out which number they mean before you touch anything.

Next: even once you know which number to chase, you still don't know where the slowness lives. That's the next rule - and it's the one that saves the most wasted effort.

← Guide overview · Phase 2: Measure Before You Optimize →