# Observability: Logs, Metrics & Traces > What observability actually is, how logs, metrics, and traces each see a different slice of your running system, and how to use all three together to find out why something is slow. --- # Observability: Logs, Metrics & Traces Your service is misbehaving. The dashboard says response times are up, but it doesn't say *why*. You SSH into a box and start tailing logs, but you don't even know which of your six services is the slow one. You're poking at a black box, hoping to bump into the problem. That feeling - knowing something is wrong but having no way to ask the system what - is what observability exists to fix. Here's the reframe that makes the whole topic click: a running system is constantly throwing off three different *kinds* of evidence about itself. **Logs** are the diary of individual events. **Metrics** are the numbers it counts over time. **Traces** are the story of one request's whole journey. None of them is "the right one" - each sees something the others can't. Once you know what each kind is best at, you stop guessing and start asking precise questions: *which* service, *how* slow, and *why*. This guide gets you there. ## How to read this - **Already drowning in a slowdown right now?** Jump to [Phase 3: Putting Them Together](03-putting-them-together.md) - it walks the exact metric → trace → log path from "something's slow" to "here's the line of code." - **Want observability to finally make sense?** Read in order. We start with the mental model (Phase 1), then meet the three kinds of evidence (Phase 2), then use them together (Phase 3). ## The phases 1. **[Monitoring vs Observability](01-monitoring-vs-observability.md)** - the core distinction: monitoring watches the things you already knew to watch; observability lets you ask *new* questions about a misbehaving system without shipping new code to answer them. 2. **[The Three Pillars](02-the-three-pillars.md)** - logs (discrete events), metrics (numbers over time: counters, gauges, histograms), and traces (one request across services, broken into spans). What each is genuinely best at, and where each falls down. 3. **[Putting Them Together](03-putting-them-together.md)** - debugging a real slowdown end to end (metric alert → trace finds the slow service → logs explain why), a quick map of the tool landscape, and the two traps that bite teams: cardinality explosions and alert fatigue. > This guide stays at the level of *concepts and how they fit together*. The hands-on, tool-specific deep > dives live in their own guides: [reading logs line by line](/guides/reading-logs-without-drowning), > [Prometheus and Grafana for metrics](/guides/prometheus-and-grafana), and > [reading a Dynatrace trace](/guides/reading-dynatrace). Read this first to get the map; read those when > you're in a specific tool. --- # Monitoring vs Observability People throw these two words around as if they're the same thing, and that fuzziness is exactly why "we have monitoring" so often fails to help when prod is actually on fire. You had dashboards. You had alerts. And yet when the weird thing happened, you were back to guessing. The words name two genuinely different capabilities, and seeing the line between them is the whole foundation for everything else in this guide. So before any tools, let's install the one idea the rest of this rests on. ## The mental model: questions you prepared for vs questions you didn't **What monitoring actually is.** Monitoring is watching a fixed set of things you decided, *in advance*, were worth watching. Is the service up? Is CPU under 80%? Is the error rate below some line? You picked those questions ahead of time, wired up a dashboard or an alert for each, and now the system tells you when one of those known measures crosses a known threshold. Monitoring answers **questions you already knew to ask.** That's its strength and its ceiling in one sentence. **What observability actually is.** Observability is a property of your system: how well you can understand its internal state *from the outside*, using the data it already emits - including for questions you never thought to ask in advance. A system is observable when, faced with surprising behavior, you can keep asking *"okay, but why?"* and the data has the answer waiting, without you having to edit code, add a new log line, redeploy, and wait for the problem to happen again. Observability answers **questions you didn't know you'd need to ask.** ```mermaid flowchart LR mon["Monitoring
known checks, known thresholds
CPU · errors · uptime · disk"] -->|known alarm fires| alert{{"alert"}} q(["a new, unplanned question"]) --> data["rich data already emitted
logs · metrics · traces"] data -->|slice by customer, region,
endpoint, version| answer["answer already there"] ``` **Why people get this wrong.** The common wrong picture is "observability is just monitoring with nicer dashboards" or "it's the three tools you buy." Neither holds up. You can own every observability tool on the market and still have an unobservable system if it only emits "request handled, 200 OK." And you can have deep observability with humble tools if your system emits rich, queryable detail. The difference isn't the logo on the dashboard - it's whether the data can answer a question nobody pre-planned. ## Known unknowns vs unknown unknowns The cleanest way to hold the distinction is the language of *knowns* and *unknowns*. - A **known unknown** is a problem you anticipated: "the disk might fill up." You don't know *when*, but you knew to watch for it. Monitoring is built for these - one alert per known unknown. - An **unknown unknown** is the problem nobody saw coming: "checkout is slow, but only for users in one region, only on the new app version, only when they have more than 50 items in the cart." You can't pre-build an alert for a combination you never imagined. Observability is what lets you *discover* it after the fact, by slicing the data along dimensions you choose in the moment. 📝 **Known unknown / unknown unknown** - a known unknown is a risk you can name in advance (so you can monitor it); an unknown unknown is one you can only recognize once it shows up (so you need to be able to investigate it). Most outages that hurt are unknown unknowns. If they were known, you'd have alerted on them and probably prevented them. So the painful 2am incidents are, almost by definition, the ones monitoring alone can't explain - which is exactly why observability matters. ## What "without shipping new code" really means Here's the test that separates an observable system from a merely-monitored one, and it's worth burning into memory. When the surprising thing happens, do you say: 1. *"Let me query the data we already have and slice it by region and version"* - **observable**, or 2. *"Let me add a log line, open a PR, wait for it to deploy, and hope the problem happens again so I can catch it"* - **not observable enough.** Option 2 is the slow, miserable loop. Every "let me add some logging and redeploy" is a confession that the system couldn't answer the question with what it already emitted. The goal of observability is to make option 1 the normal case: the data is rich enough, and queryable enough, that you debug by *asking*, not by *editing*. ⚠️ **The gotcha: a green dashboard is not proof you're fine.** Monitoring can only ever be green on the checks you thought to add. The most dangerous outage is the one your dashboards are silent about because nobody predicted it. "All systems green" means "none of my *known* alarms is firing" - not "nothing is wrong." Treat a quiet dashboard as the absence of *expected* bad news, not the presence of health. **Why this saves you later.** Once this distinction is in your head, you'll stop expecting your dashboards to explain novel problems - that was never their job - and you'll start judging your system by a sharper question: *"When something weird happens, can I find out why with what I already have?"* That question drives every practical decision in the next two phases: what to log, which metrics to keep, and when to reach for a trace. ## Recap 1. **Monitoring** watches a fixed set of known things against known thresholds - it answers questions you prepared for in advance. 2. **Observability** is a property of the system: how well you can understand its internals from the outside and ask *new* questions of the data it already emits. 3. **Known unknowns** (anticipated risks) are monitoring's territory; **unknown unknowns** (surprises) are why observability exists. 4. The practical test: when something surprises you, can you investigate by *querying existing data*, or are you stuck *adding code and redeploying*? 5. A green dashboard means "no known alarm is firing," not "everything is healthy." Watch it animated: [monitoring vs. observability](/explainers/Monitoring.dc.html) --- # The Three Pillars You've heard "the three pillars of observability" recited like a slogan, but nobody told you *why* there are three or what each one is genuinely for. So you reach for logs for everything - including questions logs are terrible at - and wonder why debugging feels like archaeology. The fix isn't memorizing three definitions. It's understanding that each pillar answers a fundamentally different *shape* of question, so you learn to reach for the right one on instinct. Here's the shape of each, in one line, before we go deep: - **Logs** tell you *what happened* - individual events, in detail. - **Metrics** tell you *how much / how many / how often* - numbers, trended over time. - **Traces** tell you *where the time went* - one request's path across all your services. ## Logs - the discrete events **What a log actually is.** A log is a timestamped record of one thing that happened: a request arrived, a payment failed, a cache missed. It's the system's diary, written one event at a time, usually with human-readable (or machine-parseable) detail attached. **What logs are best at.** *Detail about a specific event.* When you've narrowed a problem down to "this request, this moment," a log line can tell you the exact error, the exact input, the exact stack trace. Nothing else carries that much context per event. **A real example.** ```console $ tail -n 1 /var/log/checkout/app.log {"ts":"2026-06-19T14:02:11.481Z","level":"ERROR","service":"checkout","trace_id":"4bf92f3577b34da6","msg":"payment declined","order_id":"ord_8812","provider":"stripe","code":"card_declined"} ``` *What just happened:* One event, fully described - an order failed at 14:02:11 because the card was declined, with the order id and provider right there. Notice the `trace_id`: that single field is the thread that will let us tie this log line back to the exact request it belongs to in Phase 3. **Where logs fall down.** Logs are *expensive at scale* and *bad at "how many."* Counting "how many payments were declined this hour" from log lines means scanning a mountain of text, and a busy system emits so much of it that storing and searching gets slow and costly. Logs answer *what happened in this one case*, not *what's the overall trend*. > ⏭️ Reading log lines well - levels, filtering, following one request through the flood - is its own > skill. We cover it properly in [Reading Logs Without Drowning](/guides/reading-logs-without-drowning). ## Metrics - numbers over time **What a metric actually is.** A metric is a number, sampled or accumulated over time, usually tagged with a few labels (service, endpoint, region). Instead of recording *every* event in full, it records an *aggregate* - "Requests handled: 1,204,883" is one cheap number, not 1.2 million log lines. **What metrics are best at.** *Trends, rates, and alerting.* A metric is just numbers, so it's cheap to store for a long time and cheap to chart. "Is error rate climbing?" "Is p99 latency above target?" - these are metric questions, and metrics answer them instantly. 📝 **The three metric types** (worth knowing by name - they behave differently): - **Counter** - a number that only ever goes *up* (until it resets), like a turnstile click. Total requests, total errors, total bytes sent. You rarely read the raw counter; you read its *rate of change* ("requests per second"). - **Gauge** - a number that goes *up and down*, a snapshot of "right now." Current memory in use, current queue depth, current temperature. A gauge is a thermometer; a counter is an odometer. - **Histogram** - buckets that count how many observations fell into each range, so you can ask about the *distribution*, not just the average. This is how you get percentiles like p50, p95, p99 ("95% of requests finished under 300ms"). Averages lie - a few very slow requests hide behind a healthy-looking mean - so latency almost always wants a histogram, not a gauge. **A real example.** ```console $ curl -s localhost:9090/metrics | grep http_request # HELP http_requests_total Total HTTP requests handled. # TYPE http_requests_total counter http_requests_total{service="checkout",status="200"} 1204883 http_requests_total{service="checkout",status="500"} 312 # TYPE http_request_duration_seconds histogram http_request_duration_seconds_bucket{le="0.1"} 1180004 http_request_duration_seconds_bucket{le="0.3"} 1203991 http_request_duration_seconds_bucket{le="1.0"} 1204870 http_request_duration_seconds_bucket{le="+Inf"} 1205195 ``` *What just happened:* A Prometheus-format metrics endpoint. The counter shows checkout handled ~1.2M successful requests and 312 with a 500 error. The histogram buckets are cumulative (`le` means "less than or equal to"): ~1.18M requests finished under 100ms, nearly all under 300ms, and a small tail took longer - that's your p99. One scrape gives you the rate *and* the shape of latency, no individual request needed. **Where metrics fall down.** Metrics tell you *that* something is happening, not *which case* or *why*. The histogram above shows a slow tail exists; it can't tell you *which order* was slow or what went wrong in it. For that, you follow the thread to a trace and then a log. > ⏭️ Collecting, querying, and charting metrics is a craft of its own. The hands-on version lives in > [Prometheus and Grafana](/guides/prometheus-and-grafana). ## Traces - one request's journey **What a trace actually is.** A trace is the complete story of *one request* as it travels through your system - the missing piece once you have more than one service. It's made of **spans**: each span is one unit of work (a service call, a DB query, an API call), nested parent-to-child into a tree. Every span records when it started and how long it took. 📝 **Span** - a single timed operation within a trace, with a name, a start time, a duration, and a parent span. The top span (the whole request) is the **root span**; everything it triggers hangs beneath it. **What traces are best at.** *Showing where the time went across services.* When a request touches five services and one is slow, a trace lays it out as a waterfall so the slow span is visually obvious - the question metrics and logs both struggle to answer: "which *part* was slow?" **A real example.** Here's one trace (`trace_id: 4bf92f3577b34da6`, 812 ms total) drawn as the request crossing services, each span's duration noted: ```mermaid sequenceDiagram Client->>api-gateway: GET /checkout (root span, 812 ms) api-gateway->>auth-service: authorize (38 ms) api-gateway->>checkout-svc: process (740 ms) checkout-svc->>payment-db: query (690 ms) Note over payment-db: the slow span checkout-svc->>cache-get: lookup (9 ms) api-gateway->>api-gateway: render-response (24 ms) ``` *What just happened:* The 812ms request spent 690ms of that in a single `payment-db` query nested under `checkout-svc` - auth, cache, and rendering were all fine. Without the trace you'd know only "the request was slow"; with it, you know exactly which span to investigate, plus the `trace_id` to pull matching logs. That's the same `trace_id` from the log line at the top of this phase - how the three pillars connect. **Where traces fall down.** A trace is one request. It won't tell you "is this slow for *everyone*?" (a metric question) or carry the full error detail inside the slow span (a log question). Traces are also usually *sampled* - storing every trace from a busy system is costly - so the exact request you want may not have been kept. > ⏭️ Reading a real trace in a commercial tool, including the waterfall view and span attributes, is > covered in [Reading Dynatrace](/guides/reading-dynatrace). ## Three lenses on the same system The reason there are three pillars is that each looks at the system from a different angle, and the gaps in one are covered by the others: ```text metric: "p99 latency on /checkout jumped at 14:00" → THAT it's wrong, for everyone trace: "in this request, payment-db took 690 of 812ms" → WHERE the time went, one request log: "payment declined / slow query: full table scan" → WHY, the exact event detail ``` Notice the natural order: a metric tells you *that* there's a problem and how widespread, a trace tells you *where* in the request it lives, and a log tells you *why* at the level of a single event. Holding all three lets you zoom from "something's wrong" all the way down to one line of detail - which is exactly the journey we walk in Phase 3. ## Recap 1. **Logs** are discrete, detailed events - best for *what happened in this specific case*; expensive at scale and poor at "how many." 2. **Metrics** are numbers over time - **counters** (only go up), **gauges** (up and down, "right now"), and **histograms** (distributions, for percentiles). Cheap to store and chart; best for *trends and alerting*; can't tell you *which case* or *why*. 3. **Traces** follow one request across services as a tree of **spans** - best for *where the time went*; one request at a time, and usually sampled. 4. The three are lenses on the same system: metric = *that / how widespread*, trace = *where*, log = *why*. A shared `trace_id` is the thread that stitches them together. --- # Putting Them Together Three pillars are interesting on their own, but the payoff is what happens when you use them *as a chain*. The skill that separates someone who flails during an incident from someone who looks calm isn't knowing more tools - it's knowing the *order* to reach for them. You go from "something's slow" to "here's the line of code" by zooming in deliberately, one pillar handing off to the next. This phase walks that exact path through one realistic slowdown, then maps the tools lightly so the product names stop being intimidating, then names the two traps that quietly ruin observability setups. ## The cheat-card: the debugging chain When something is slow or broken and you don't yet know why, walk it in this order: | Step | Pillar | The question it answers | What you learn | |---|---|---|---| | 1 | **Metric** | Is something actually wrong, and how widespread? | "p99 latency on `/checkout` doubled at 14:00, for everyone." | | 2 | **Trace** | *Where* in the request is the time going? | "In a sample slow request, one DB span ate 690 of 812ms." | | 3 | **Log** | *Why* did that specific step misbehave? | "That query did a full table scan - a missing index." | Metric → trace → log. *That it's wrong* → *where it's wrong* → *why it's wrong.* Keep reading for the full walkthrough underneath. ## Walking a real slowdown Let's debug it for real. The setup: a checkout flow spanning an API gateway, an auth service, a checkout service, and a database. Users are complaining that checkout "feels slow." ### Step 1 - the metric tells you *that* it's wrong You don't start by reading logs. With six services, you don't yet know *where* to read. You start with the metric that's closest to the symptom: checkout latency. ```console $ # p99 latency for the checkout endpoint, last 2 hours (PromQL) $ histogram_quantile(0.99, rate(http_request_duration_seconds_bucket{route="/checkout"}[5m])) 13:55 0.31 14:00 0.74 14:05 0.79 14:10 0.81 ``` *What just happened:* p99 for `/checkout` jumped from ~0.31s to ~0.8s at 14:00 and stayed there. The metric confirmed the problem is real (not one user's bad wifi), showed it affects many requests, and pinned the start time. What it can't tell you is *which* of the four services got slow - for that, follow a single request. ### Step 2 - the trace tells you *where* Now you pull up a trace for one of the slow `/checkout` requests. This is the waterfall from Phase 2, and the slow span jumps out: ```text trace_id: 4bf92f3577b34da6 total: 812 ms api-gateway ████████████████████████████████████████ 812 ms └─ auth-service ██ 38 ms └─ checkout-svc ████████████████████████████████████ 740 ms └─ payment-db ██████████████████████████████████ 690 ms ◄── here ``` *What just happened:* The trace collapsed "checkout is slow" into "`payment-db` took 690 of 812ms." Auth, the gateway, everything else - fine. You know exactly which service and operation to investigate without guessing or reading four services' worth of logs, plus the `trace_id` as your key into them. ### Step 3 - the log tells you *why* Finally, you filter logs to that one `trace_id` and look at what the database service recorded for it: ```console $ grep 4bf92f3577b34da6 /var/log/checkout/db.log {"ts":"2026-06-19T14:02:11.4Z","level":"WARN","service":"payment-db","trace_id":"4bf92f3577b34da6","msg":"slow query","duration_ms":690,"query":"SELECT * FROM payments WHERE customer_id = ?","rows_scanned":4210567,"plan":"Seq Scan"} ``` *What just happened:* There's the *why*: the query scanned 4.2 million rows with a `Seq Scan` (a full table scan, no index). Someone shipped a query hitting `payments` by `customer_id` with no supporting index, so it reads the entire table every time. The fix: add the index. Notice what made this fast and calm: you used each pillar for the question it's actually good at, and the shared `trace_id` carried you across the handoffs. That's the whole game. ```mermaid flowchart LR M["METRIC
p99 doubled at 14:00
(that + how widespread)"] --> T["TRACE
payment-db span
ate 690/812 ms
(where)"] T --> L["LOG
Seq Scan, 4.2M rows,
missing index
(why)"] ``` ## The tool landscape, lightly The product names are just implementations of these three pillars. You don't need to memorize them - you need to know which pillar each one lives in. - **Prometheus + Grafana** - the common open-source pairing for **metrics**. Prometheus scrapes and stores the numbers; Grafana charts them and drives alerts. (The metric query in Step 1 is PromQL, Prometheus's query language.) → [Prometheus and Grafana](/guides/prometheus-and-grafana). - **Dynatrace** (and peers like Datadog, New Relic, Honeycomb) - commercial, all-in-one platforms that cover **all three pillars** in one place, with strong **tracing** waterfalls and automatic correlation between them. → [Reading Dynatrace](/guides/reading-dynatrace). - **OpenTelemetry (OTel)** - *not* a backend you look at; it's the **open standard and toolkit for producing** logs, metrics, and traces in a vendor-neutral way. Your code emits OTel data; you then ship it to *any* backend (Prometheus, Dynatrace, Grafana, whatever). This is the big shift of the last few years: instrument once with OTel, and you're not locked into one vendor's agent. 💡 **The one thing to hold onto:** tools come and go, but every one of them is collecting, storing, or displaying logs, metrics, or traces. Learn the three pillars and you can pick up any tool by asking "which pillar is this, and which question is it for?" ## The two traps Observability has two failure modes that don't announce themselves until they hurt. Name them now so you spot them coming. ### Cardinality explosions 📝 **Cardinality** - the number of distinct values a label (a tag on a metric) can take. `status` has low cardinality (a handful of HTTP codes). `user_id` has *enormous* cardinality (one value per user). ⚠️ **The gotcha.** Metric systems store a separate time series for *every unique combination of label values*. Attach a high-cardinality label like `user_id`, `order_id`, or a raw URL with embedded IDs, and you don't get one series - you get one *per user*, per *order*, per URL, multiplying across every other label. Backend memory and cost can blow up fast, sometimes taking the whole monitoring system down - one of the most common ways teams accidentally break their own observability. The rule: **metrics labels should be low-cardinality** (service, endpoint, status, region - a small, bounded set of values). Per-request detail like a user or order id belongs on a **log or a trace**, not a metric label. The pillars have different cost shapes for a reason - respect them and you stay out of this hole. ### Alert fatigue ⚠️ **The gotcha.** If every metric has an alert and every alert pages a human, people stop reading the alerts. After the tenth false alarm at 3am, the on-call's brain learns to swipe the notification away without looking - and that's precisely the night the alert was real. An alert nobody trusts is worse than no alert, because it costs attention *and* delivers nothing. The fixes are mostly about restraint, not technology: - **Alert on symptoms users feel, not on every internal number.** "Checkout p99 is above target" is worth waking someone. "CPU touched 81% for 30 seconds" usually isn't. - **Page only on things a human must act on *now*.** Everything else is a dashboard or a low-priority ticket, not a page. - **Tune thresholds and add a bit of duration** ("above target for 5 minutes") so a brief blip doesn't fire. The goal is that when a page arrives, the on-call *believes* it. That trust is the actual product of a good alerting setup - and the thing alert fatigue quietly destroys. ## Recap 1. Debug in the order **metric → trace → log**: *that* it's wrong (and how widespread) → *where* in the request → *why* in that one step. 2. A shared **`trace_id`** is what carries you across the handoffs from metric to trace to log. 3. The tools are just pillars wearing brand names: **Prometheus/Grafana** for metrics, **Dynatrace/Datadog/etc.** for all three with strong tracing, and **OpenTelemetry** as the vendor-neutral standard for *producing* the data. 4. **Cardinality explosion** - never put high-cardinality values (user/order ids, raw URLs) on metric labels; that detail belongs on logs and traces. 5. **Alert fatigue** - alert on symptoms users feel, page only on what needs action now, so that when a page fires, people still trust it.