# Metrics & a Taste of PromQL PromQL queries look intimidating the first time - `rate(http_requests_total{status="500"}[5m])` is a lot of punctuation to meet at once. But almost every query you'll read is built from a few small ideas stacked together. Learn the ideas and the punctuation stops being scary. The single most important one, the thing that confuses *everybody* at first: **why you almost never graph a counter directly, and reach for `rate()` instead.** ## The three metric types Prometheus metrics come in a few types, and the type tells you how to *read* the number. Three cover the vast majority of what you'll see. **Counter - a number that only ever goes up.** Total requests served, total errors, total bytes sent. A counter starts at zero when the process starts and climbs. It resets to zero only when the process restarts. - The wrong picture: "the counter is the current value, like a speedometer." It isn't. `http_requests_total = 18342` doesn't mean "18,342 requests per second" - it means "18,342 requests *ever*, since this process started." On its own that number is nearly useless. What you care about is *how fast it's climbing*, which is the whole reason `rate()` exists (next section). **Gauge - a number that goes up and down.** Current memory in use, current temperature, number of items in a queue, active connections. A gauge is a snapshot of "right now," and reading it directly *does* make sense - `process_resident_memory_bytes = 58000000` means "using 58 MB right now." **Histogram - counts bucketed by size.** A histogram answers "how were these values distributed?" - most often for request durations. Instead of one number, it records counts in buckets: "how many requests finished under 0.1s, under 0.5s, under 1s, …". This is what lets you ask for a **percentile** later - "95% of requests finished faster than X" - which is how you actually talk about latency. (A single average latency hides the slow tail; percentiles don't.) > 📝 There's a fourth type, **summary**, a close cousin of histogram that computes percentiles service-side. Histograms are the more common, more flexible choice - treat summary as histogram's relative and don't lose sleep over it yet. ⚠️ **Gotcha - reading a counter like a gauge.** This is the classic beginner mistake. You put `http_requests_total` on a graph, see a line marching steadily up and to the right, and panic that traffic is exploding. It isn't - a counter *always* goes up and to the right, by definition. A straight diagonal line means *steady* traffic. To see the actual traffic, you need its rate of change. ## Labels - the same metric, sliced many ways **What they actually are.** Labels are key-value tags attached to a metric that let one metric name cover many related streams. Look back at the `/metrics` page from Phase 1: ```text http_requests_total{method="GET",status="200"} 18342 http_requests_total{method="GET",status="500"} 12 http_requests_total{method="POST",status="200"} 4071 ``` *What just happened:* That's *one* metric name, `http_requests_total`, split into three separate time series by its labels. Each unique combination of label values is its own counter. So you can ask "all requests," or narrow to "just the 500s," or "just POSTs" - all from the same metric, by filtering on labels. **Why this is the heart of PromQL.** Filtering and grouping by labels is how you turn a firehose of numbers into an answer to a specific question. "How many errors?" → filter to `status="500"`. "Errors per endpoint?" → group by the `path` label. We'll see both below. ⚠️ **Gotcha (the one that bites hardest, covered fully in Phase 3).** Every distinct combination of label values creates a *new* time series Prometheus must store. Put something with unbounded values in a label - a user ID, a full URL with query strings, a request ID - and you generate millions of series. That's a **cardinality** explosion, and it's the most common way people accidentally take Prometheus down. Labels are for things with a *small, bounded* set of values (method, status code, endpoint name), never for unique identifiers. ## Reading your first real query Let's take that scary-looking query apart, piece by piece: ```promql rate(http_requests_total{status="500"}[5m]) ``` Reading it from the inside out: - `http_requests_total` - the metric: total HTTP requests, a **counter**. - `{status="500"}` - a **label filter**: narrow to just the series where the status code is 500 (server errors). - `[5m]` - a **range**: "give me the last 5 minutes of samples for each matching series," not just the latest point. (This is called a *range vector* - a window of values, which `rate()` needs to do its math.) - `rate(...)` - the function: take that 5-minute window and compute the **per-second average rate of increase** of the counter. *What it's showing you:* "Over the last 5 minutes, how many server errors per second, on average?" The output isn't a total - it's a *speed*: errors/sec, calculated freshly at each point on the graph. A flat line at `0` means no errors; a line that jumps to `3` means you're suddenly taking three 500s every second. ## Why you graph rates, not raw counters This is the payoff. Here's the contrast, drawn out: ```text RAW COUNTER: http_requests_total rate(http_requests_total[5m]) ("everything ever") ("requests per second, now") count req/s │ ____/ │ ╱╲ ___ │ ____/ │ ___╱ ╲____╱ ╲__ │ ____/ │ ╱ │___/ │╱ └──────────────────────► time └──────────────────────► time always climbs; slope = the the actual traffic shape: real signal, but hard to read spikes and dips you can see ``` *What just happened:* The raw counter only ever rises, so the *information* you want - how busy the service is right now - is hidden in the line's slope, which your eyes are bad at reading. `rate()` does the differencing for you: it turns "total ever" into "per second now," so a traffic spike becomes a visible bump and a quiet period becomes a dip. That's why almost every counter you ever graph is wrapped in `rate()`. `rate()` is also smart about counter **resets**: when a process restarts and the counter drops back to zero, `rate()` recognizes that as a restart rather than a giant negative blip, and accounts for it. That's another reason to use it instead of subtracting values by hand. 💡 **Key point - the rule of thumb.** *Counters get wrapped in `rate()`. Gauges you read directly.* If you find yourself plotting a bare counter and wondering why every line goes up and to the right, that's the rule reminding you. (A close relative, `increase(...)[1h]`, answers "how many total in the last hour?" - same idea, expressed as a count rather than a per-second rate.) > The numbers and shapes above are illustrative - drawn to show the *behavior*, not measured from a real system. ## Why this saves you later When you're staring at a dashboard during an incident and someone asks "is the error rate climbing?", you'll know the panel's query is a per-second rate over a window, not a raw total - and trust the line for the right reasons. In the next phase you'll reach for `rate()` on a counter without thinking, instead of plotting a useless diagonal. ## Recap 1. **Type tells you how to read it.** Counters only go up (read their *rate*); gauges go up and down (read them *directly*); histograms bucket values so you can ask for percentiles. 2. **Labels slice one metric into many series** - filter and group by them to ask specific questions. 3. **High-cardinality labels are dangerous** - never put unique IDs in a label (more in Phase 3). 4. **`rate(counter[window])`** turns "total ever" into "per second now," and handles counter resets for you. 5. **The rule:** wrap counters in `rate()`; read gauges raw. Now you can read what's stored. Let's put it on a screen and make it page someone when it matters.