# Latency vs Throughput

Here's where most performance confusion starts. Two people look at the same system. One says "it's slow" - they clicked a button and waited four seconds. The other says "it's fine" - the server is handling thousands of requests a minute without breaking a sweat. They're both right, because they're talking about two completely different numbers. Once you can tell those two numbers apart, half the fog clears.

## The two numbers

**What they actually are.**

- **Latency** is how long *one* thing takes, start to finish. You click; how long until you see the result? That's latency. It's a *duration* - measured in milliseconds or seconds.
- **Throughput** is how *many* things the system finishes in a given time. How many requests per second can the server handle? How many photos per minute can the pipeline process? That's throughput. It's a *rate* - measured in things-per-second.

Latency is about *waiting*. Throughput is about *volume*. They feel like the same idea - "fast" - but they answer different questions, and a system can be great at one while terrible at the other.

📝 **Terminology.** *Latency* = the time for a single operation to complete (a duration). *Throughput* = the number of operations completed per unit of time (a rate). When someone says "it's slow," your first job is to find out which one they mean.

## The highway analogy

This is the picture that makes it stick. Think of work flowing through your system like cars driving down a highway.

```text
                     ┌─────────────────────────────────────────┐
   a car enters ───► │  ═══════  the highway  ═══════           │ ───► a car exits
                     └─────────────────────────────────────────┘

   LATENCY    = how long ONE car takes to drive end to end
                (does NOT change if you add more lanes)

   THROUGHPUT = how many cars exit per minute
                (MORE lanes  ->  more cars per minute,
                 even though each car's drive takes the same time)
```

- **Latency** is how long it takes a single car to drive from the on-ramp to the off-ramp. Adding lanes doesn't make that one car arrive any sooner - its trip is the same length.
- **Throughput** is how many cars get off the highway per minute. Add lanes and you move *far* more cars per minute, even though each individual car's drive took exactly as long as before.

That's the whole insight: **widening the road raises throughput without touching latency.** More lanes (more servers, more worker threads, more parallel processing) lets you serve more users at once - but it does nothing for the user who's stuck behind a slow operation. Their one car still takes four seconds.

## Why improving one can hurt the other

Here's the part that surprises people, and it's worth slowing down for. These two numbers aren't just independent - pushing on one can actively *drag down* the other.

The classic example is **batching**. Imagine your system sends data somewhere. You can send each item the instant it arrives, or you can wait, collect a hundred items, and send them all in one trip.

```text
   SEND IMMEDIATELY            BATCH OF 100
   (low latency)              (high throughput)

   item ─► send                item ─┐
   item ─► send                item ─┤  wait... collect...
   item ─► send                item ─┤  ...then send all 100
                                ...  ─┘  in one trip

   each item leaves fast       each item waits for the batch,
   but you pay the per-trip    but one trip carries 100 items,
   cost 100 times              so far more move per second
```

Batching is great for throughput: one trip carries a hundred items instead of one, so the per-item overhead (the cost of opening a connection, the round-trip across the network) gets paid once instead of a hundred times. More items move per second.

But every item now *waits* for the batch to fill before it goes anywhere. The first item you collected sat there while ninety-nine more arrived. You raised throughput by adding latency. That's not a bug - it's a deliberate trade, and engineers make it on purpose all the time. The point is to make it *on purpose*, knowing what you're giving up.

⚠️ **The trap: optimizing the number nobody cares about.** A team proudly "improves performance" by batching aggressively - throughput goes way up, the dashboards look great. Meanwhile every user's action now takes an extra half-second because their request is waiting in a batch. They optimized throughput and degraded the thing users actually feel. Always know *which* number you're moving, and *which* number your users care about. They're often not the same number.

**Why this saves you later.** The next time a ticket says "make it faster," you won't start coding. You'll ask one question: *do you mean each operation should take less time, or do you mean the system should handle more at once?* Those lead to completely different fixes. Reducing latency might mean removing a slow step from a single request. Raising throughput might mean running more copies in parallel. Confuse them and you'll add four servers to a problem that needed one slow database query fixed - or rewrite a query to help a system that just needed more lanes.

## Recap

1. **Latency** = how long *one* operation takes (a duration). **Throughput** = how many operations finish *per second* (a rate). Different numbers, different questions.
2. The **highway**: latency is one car's drive time; throughput is cars-per-minute. Adding lanes raises throughput without changing any single car's latency.
3. Improving one can hurt the other - **batching** raises throughput by *adding* latency. Trade on purpose, not by accident.
4. When someone says "slow," find out which number they mean *before* you touch anything.

Next: even once you know *which* number to chase, you still don't know *where* the slowness lives. That's the next rule - and it's the one that saves the most wasted effort.