# Reading Graylog (Log Search & Streams) > What centralized logging actually is, how to search across dozens of servers from one box, and how streams, dashboards, and alerts turn a flood of logs into something you can stand on. --- # Reading Graylog (Log Search & Streams) On your laptop, when something breaks, you open one file and `grep` for the error. That works because there's one file. Now picture the same problem in production: a dozen containers, three app servers, two load balancers, a queue worker - and the one log line that explains the outage is sitting on whichever box happened to handle that one unlucky request. You can't `ssh` into all of them and `grep` in parallel while the pager is going off. That's the moment people discover they're drowning, not because there are too many logs, but because the logs are *scattered*. Here's the relief: tools like Graylog (and the very similar ELK / OpenSearch + Kibana stacks) do one profound thing - they ship every log line from every machine into one place and put a search box on top. The skill you already have for reading a single log file still applies; what changes is that now your `grep` reaches across the entire fleet at once, you can scope it to a five-minute window, and you can follow a single request as it bounced between services. This guide gives you that skill, and the mental model underneath it so the search box stops feeling like a slot machine. > ⏭️ New to reading logs at all? Start with [Reading Logs Without Drowning](/guides/reading-logs-without-drowning) > - what a log line *is*, what the levels mean, how to follow one request. This guide is the > centralized, many-servers version of that skill. ## How to read this - **Mid-incident, need to find the failing request right now?** Jump to [Phase 2: Searching Effectively](02-searching-effectively.md) and use the cheat-card at the top. - **Want centralized logging to finally make sense?** Read in order - Phase 1 installs the mental model (one search box over everything, structured fields vs. raw text), and the rest builds on it. ## The phases 1. **[Why Centralized Logs](01-why-centralized-logs.md)** - why `grep` on one box stops working across a fleet, what Graylog/ELK actually collect and where, and the two ideas everything rests on: one search box over everything, and structured fields vs. raw text. 2. **[Searching Effectively](02-searching-effectively.md)** - the query model: `field:value` searches, time-range scoping (your #1 lever), boolean operators, following one request by its correlation id, and reading the histogram to find the spike. 3. **[Streams, Dashboards & Alerts](03-streams-dashboards-alerts.md)** - routing subsets of logs into streams (e.g. just prod ERRORs), saving dashboards you can glance at, and alerting on log conditions so the system pages you instead of you discovering the fire by accident. > This guide stays at the level of *reading and searching* centralized logs. The deeper operational > side - running the cluster, designing index/retention policies, parsing pipelines, and wiring > logs together with metrics and traces - is its own topic. For where logs sit in the bigger picture, > see [Observability: Logs, Metrics & Traces](/guides/observability-logs-metrics-traces). --- # Why Centralized Logs You already know how to read logs on one machine: `tail -f` the file, `grep` for the error, scroll to the moment it broke. That skill is real and it still matters. The problem isn't the skill - it's that the file you need isn't on the machine you're logged into. Modern systems spread one request across many processes, and any of them could be where it went wrong. Centralized logging exists for exactly this discomfort. Once you see *what* it's doing and *why*, the rest of this guide is mostly learning where the buttons are. ## Why `grep` on one box stops working **What changed.** A single server with one app is a single diary. You SSH in, you read the diary. But a typical production system isn't one diary - it's dozens, scattered across machines that come and go. ```mermaid flowchart TD req["a request comes in"] --> lb["load balancer
(its own log, own box)"] lb --> app1["app-1"] lb --> app2["app-2"] lb --> app3["app-3"] app1 --> db[("database
logs slow queries, another box")] app2 --> db app3 --> db ``` When the pager goes off, you don't know which `app-N` handled the failing request. SSHing into each one and `grep`-ing by hand is slow, and it gets worse with every box you add. Containers make it sharper still: a crashed container can be *gone*, and its log file with it, before you ever log in. ⚠️ **Containers don't keep your logs for you.** A container's filesystem is usually thrown away when it restarts. If logs only live inside the container, a crash-loop erases exactly the evidence you need. Shipping logs *off* the box, somewhere durable, is the whole point. ## The first idea: one search box over everything **What it actually is.** Graylog (and ELK, and OpenSearch) is a place that *receives* log lines from every machine and stores them together, with a single search interface on top. Instead of "which box do I SSH into," the question becomes "what am I searching for." One box. One search. The whole fleet. **How the logs get there.** A small agent runs on each machine - or each container's output is collected by the platform - and forwards every log line to the central server over the network. ```mermaid flowchart LR app1["app-1"] --> ship["log shippers
forward every line
over the network"] app2["app-2"] --> ship app3["app-3"] --> ship lb["lb"] --> ship db["db"] --> ship ship --> central["Graylog / ELK / etc.
stores all logs together"] central --> you(["you search here"]) ``` πŸ“ **Log shipper / agent.** The thing that reads logs on a machine and sends them onward. You'll see names like Filebeat (the ELK world), Fluentd / Fluent Bit, Vector, or Graylog's Sidecar. They all do the same job: pick up log lines and forward them to the central store. **Why this design.** The alternative - searching each machine individually - doesn't scale and doesn't survive machines disappearing. Centralizing trades a little setup and storage cost for the ability to ask one question and have it answered across everything, even for boxes that no longer exist. πŸ“ **Graylog, ELK, OpenSearch - same shape.** **ELK** is Elasticsearch (the store/search engine) + Logstash (ingest) + Kibana (the UI). **OpenSearch** is a fork of Elasticsearch with its own dashboards UI. **Graylog** is its own web app that sits on top of an Elasticsearch/OpenSearch store. The vendor logos differ, but the mental model in this guide - ship everything in, search by field, scope by time, route with streams, alert on conditions - applies to all of them. Where the query *syntax* differs, we'll say so. ## The second idea: structured fields, not just raw text This is the one that changes how you search, so it's worth slowing down on. **The wrong picture.** Many people imagine the central store as one giant text file you `grep`. That picture works for a while, but it sells the tool short and makes your searches clumsier than they need to be. **What it actually is.** Each log line arrives as a little record with *named fields*, not just a blob of text. A line that prints like this on the box: ```text 2026-06-19T14:02:11Z level=error service=checkout request_id=a1b2c3 status=500 msg="payment gateway timeout" ``` is stored centrally as something more like a labeled card: ```text timestamp : 2026-06-19T14:02:11Z level : error service : checkout request_id : a1b2c3 status : 500 message : payment gateway timeout ``` **Why this matters.** Because the pieces are *named*, you can search by them precisely: `service:checkout AND status:500` instead of hoping the right text happens to appear next to the right other text. Time, level, service, and host are almost always fields you can lean on. The richer your log lines (i.e. the more they were written as structured key/value data rather than free prose), the sharper your searches get - which is a habit your own services benefit from. **Where fields come from.** Some are added automatically (timestamp, the source host, which stream it landed in). Some come from how your app logged the line - if you logged JSON or `key=value` pairs, those keys *become* fields. If you logged a wall of prose, you get fewer named fields and you'll lean more on full-text search of the `message`. You can still search prose; you just have fewer handles to grab. πŸ’‘ **Key point.** A centralized log isn't a giant text file - it's a pile of labeled cards. The search box lets you say "show me the cards where `service` is `checkout` and `status` is `500`, in the last 15 minutes." That sentence is the entire job. ## The gotcha that shapes everything later ⚠️ **Logs are only as good as what you logged.** Centralized logging can't show you a field your app never emitted. If `checkout` never logs a `request_id`, you can't follow a request by id no matter how good Graylog is. The search tool is a magnifying glass, not a microscope that invents detail. This is why Phase 3 ends on the trade-offs of *what* and *how much* to log - and why you must **never log secrets** (passwords, tokens, full card numbers), because a central store is searchable by lots of people and often retained for a long time. (See [Secrets Management](/guides/secrets-management).) ## Recap 1. One box β†’ `grep` a file. A fleet β†’ the file you need is on a machine you're not logged into (or one that no longer exists). 2. Graylog / ELK / OpenSearch ship every log line into one place and put a single search box on top. 3. A log shipper (Filebeat, Fluent Bit, Vector, Sidecar…) on each machine forwards lines to the center. 4. Logs are stored as records with *named fields*, not one big text blob - that's what makes precise `field:value` search possible. 5. The tool can only show what you logged; logs are only as good as the data your apps emit - and never log secrets. --- # Searching Effectively You're in the search box. The pager went off two minutes ago. The instinct is to type the error message and hit enter - and then you get ten thousand results spanning three days and you're drowning again, this time in a nicer UI. The fix isn't a cleverer search string. It's a small set of moves, applied in the right order, that shrink the haystack before you go looking for the needle. ## Cheat-card: drowning right now? | You want to… | Do this first | |---|---| | Stop seeing yesterday's noise | **Scope the time range** to the incident window (last 15m, or a fixed start/end). This is the #1 lever. | | Find errors in one service | `service:checkout AND level:error` | | See only failed requests | `status:500` (or `status:>=500`) | | Follow one user's broken request | `request_id:a1b2c3` (across *all* services at once) | | Find *when* it started | Look at the **histogram** above the results - the spike is the start. | | Exclude known noise | `... AND NOT message:"health check"` | Each of these is explained below. The order matters: **scope time, then filter, then drill in.** ## The query model: search by field **What it actually is.** A search is a question about the labeled cards from Phase 1. The basic unit is `field:value` - "show me cards where this field has this value." **A real example.** ```text service:checkout AND level:error ``` ```text timestamp level service status request_id message 2026-06-19 14:02:11 error checkout 500 a1b2c3 payment gateway timeout 2026-06-19 14:02:14 error checkout 500 d4e5f6 payment gateway timeout 2026-06-19 14:03:02 error checkout 500 7g8h9i payment gateway timeout … (showing 41 results in the selected time range) ``` *What just happened:* Instead of full-text searching for the word "error" everywhere, you asked for cards where the `service` field is exactly `checkout` and the `level` field is exactly `error`. Three lines in, a pattern is already obvious: same message, `status` 500, "payment gateway timeout." You're not reading ten thousand lines - you're reading the 41 that matter. ⚠️ **Field names and values must actually exist as you typed them.** `level:error` only works if your logs carry a `level` field with the value `error` (some apps log `ERROR`, some log `severity` instead of `level`). When a search returns nothing, suspect a wrong field name or value before you suspect the outage fixed itself. Click an existing log line to see its real field names. πŸ“ **Syntax differs slightly by tool.** Graylog and Kibana both use a Lucene-style `field:value` syntax, so most of what's here ports directly. Kibana also has its own **KQL** (Kibana Query Language) where you might write `service: "checkout" and level: "error"`. The *ideas* - field, value, boolean, range - are identical; only the punctuation moves. When in doubt, the UI usually shows which mode you're in. ## Time-range scoping: your single biggest lever **Why it's first.** Every other filter narrows *what*; the time range narrows *how much*. An incident happened in a window. If you search "all time," you're fighting every log the system ever produced. If you scope to the 15 minutes around the alert, you've often cut the haystack by orders of magnitude before typing a single field. **What it does in real life.** There's a time-range picker near the search box - relative ("last 15 minutes," "last 1 hour") or absolute (a fixed start and end). Set it to the incident window *first*, then search. ```text β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ Search: service:checkout AND status:500 β”‚ β”‚ Time: [ Last 15 minutes β–Ύ ] ← set this FIRST β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` πŸͺ– **War story.** A teammate spent twenty minutes convinced a bug was "intermittent and unreproducible." The search was scoped to the last 24 hours, so a handful of real failures were buried under a day of unrelated noise. Narrowing the time range to the ten minutes the user reported turned "intermittent" into "happens every single time, here it is." The bug hadn't changed. The window had. ## Boolean operators: combine and exclude **What they are.** `AND`, `OR`, and `NOT` (Graylog also accepts `&&`, `||`, and `-` as shorthands) combine field searches into one precise question. **A real example.** ```text service:checkout AND status:500 AND NOT message:"health check" ``` *What just happened:* You asked for checkout's failed requests, then subtracted the routine health-check noise that also happens to be in the window. `NOT` (or a leading `-`) is how you carve away the known, boring lines so the unfamiliar ones stand out. `OR` does the opposite - `service:checkout OR service:payments` widens to two services at once. ⚠️ **Quote multi-word values and watch your operators.** `message:payment gateway timeout` may be read as `message:payment` plus the loose words `gateway timeout`. Quote it: `message:"payment gateway timeout"`. And uppercase your booleans - most query parsers expect `AND`/`OR`/`NOT`, not `and`/`or`. **Ranges, when fields are numeric.** For numeric fields you can ask for ranges: `status:>=500` (server errors), or `took_ms:>2000` (requests slower than two seconds), if your app logs those as numbers. ## Following one request by its correlation id This is the move that pays for the whole centralized setup, so it's worth doing deliberately. **The problem it solves.** Back in Phase 1, one request's story was smeared across the load balancer, one app server, and the database - on different boxes. On a single machine you couldn't reassemble it. Centrally, you can - *if* every service stamped the request with the same id. πŸ“ **Correlation id / request id / trace id.** A unique string generated when a request first arrives (often at the load balancer or API gateway) and passed along to every service that touches it, each of which logs it. Same request, same id, everywhere. **A real example.** You found a failing request and copied its `request_id`. Now drop the service filter and search *only* the id: ```text request_id:a1b2c3 ``` ```text timestamp service message 14:02:09.812 gateway inbound POST /checkout request_id=a1b2c3 14:02:09.998 checkout creating order, calling payment provider 14:02:11.004 checkout payment gateway timeout after 1000ms 14:02:11.006 checkout returning 500 to client 14:02:11.040 gateway response 500 for /checkout request_id=a1b2c3 ``` *What just happened:* By searching the id across *all* services at once, the scattered diary entries snap back into one timeline. You can read the request's whole life in order: it came in, checkout called the payment provider, the provider didn't answer within a second, checkout gave up and returned a 500. You didn't reconstruct that by hand across five machines - the shared id and one search did it for you. ⚠️ **No id, no trail.** If a service doesn't log the correlation id, it won't appear in this timeline - there'll be a gap. That gap is a logging gap, not proof that nothing happened there. (This is the Phase 1 gotcha biting in practice: the tool can only show what was logged.) ## Reading the histogram: find *when* it started **What it's showing you.** Above the results, the search UI draws a bar chart of *how many matching log lines occurred over time* - each bar is a time bucket (per minute, per hour, depending on your range). It's not decoration; it's the shape of the problem. ```text matching lines per minute (search: service:checkout AND level:error) count 120 ─ β–ˆβ–ˆβ–ˆ β”‚ β–ˆβ–ˆβ–ˆ β–ˆβ–ˆβ–ˆ 80 ─ β–ˆβ–ˆβ–ˆ β–ˆβ–ˆβ–ˆ β–ˆβ–ˆβ–ˆ β”‚ β–ˆβ–ˆβ–ˆ β–ˆβ–ˆβ–ˆ β–ˆβ–ˆβ–ˆ 40 ─ β–ˆβ–ˆβ–ˆ β–ˆβ–ˆβ–ˆ β–ˆβ–ˆβ–ˆ β”‚ Β· Β· Β· Β· Β· Β· β–ˆβ–ˆβ–ˆ β–ˆβ–ˆβ–ˆ β–ˆβ–ˆβ–ˆ Β· 0 ┼──────────────────────────────────────────▢ time 13:50 13:55 14:00 ↑ 14:05 14:01 - errors jump from ~1/min to ~100/min ``` *What just happened:* The flat baseline on the left is the normal background rate of `checkout` errors - a stray one here and there. At 14:01 the bars shoot up. That cliff *is* the start of the incident. Now you know exactly which minute to scope to, and you can line it up against a deploy, a config change, or a dependency going down. πŸ’‘ **Key point.** Read the *shape* before the *lines*. A sudden cliff says "something changed at this moment" (a deploy, an outage). A slow ramp says "something is degrading" (a leak, a filling queue, a dying disk). The histogram tells you which kind of problem you have before you've read a word of any log. ## The order, every time 1. **Scope the time range** to the incident window. (Biggest lever.) 2. **Filter by field** - `service:`, `level:`, `status:` - to the slice you care about. 3. **Read the histogram** to find when it started and what shape it is. 4. **Grab a `request_id`** from a failing line and search it alone to follow the request across services. 5. **Subtract noise** with `NOT` / `-` until only the unfamiliar lines remain. ## Recap 1. Searches are `field:value` questions over labeled cards; combine them with `AND` / `OR` / `NOT`. 2. Time-range scoping is your #1 lever - set the window *before* you refine the query. 3. Quote multi-word values; uppercase your booleans; verify field names against a real log line. 4. Searching a correlation/request id alone reassembles one request's story across every service. 5. The histogram shows matches over time - a cliff means a change, a ramp means degradation; read the shape first. ## Try it yourself Build a pattern for the responses you care about (here: 4xx/5xx status codes): ```playground-regex \b[45]\d\d\b GET /api/users 200 12ms GET /api/order 404 3ms POST /api/pay 500 80ms GET /health 200 1ms ``` --- # Streams, Dashboards & Alerts So far you've been the one asking questions: you open the search box, you scope, you filter, you read the histogram. That's the right skill for an incident already in progress. But you can't sit in the search box all day, and the worst incidents are the ones nobody was looking at when they started. Making the system work *for* you between emergencies means pre-sorting the logs that matter (streams), keeping a view you can glance at (dashboards), and having it tap you on the shoulder when something's wrong (alerts) - then the uncomfortable truth underneath all of it: this is only ever as good as what you logged. ## Streams: pre-sorted piles of logs **What it actually is.** A stream is a *standing filter* - a rule that says "any log line matching this condition belongs in this pile." Once it exists, you (and dashboards and alerts) can search just that pile instead of the whole firehose every time. ```mermaid flowchart TD line["every incoming log line"] --> rules{"stream rules:
does this line match?"} rules --> s1["'All errors'
level:error"] rules --> s2["'Prod checkout'
env:prod AND service:checkout"] rules --> s3["'Slow DB queries'
took_ms:>1000"] ``` **What it does in real life.** Instead of typing `level:error AND env:prod` for the hundredth time, you define it once as a stream like "Production Errors." Now anyone can click that stream and immediately be looking at only prod errors - no query, no scoping the firehose. Streams are also the unit other features point at: you build a dashboard *over* a stream, and you attach an alert *to* a stream. πŸ“ **Stream vs. search.** A *search* is a one-off question you type. A *stream* is a saved, always-on routing rule that keeps a named subset continuously populated. (Kibana's nearest equivalents are *saved searches* and *data views* - same idea, different label: a reusable, named slice of the logs.) πŸ’‘ **Key point.** Streams turn "I'll remember to filter for prod errors" into "prod errors are already their own pile." The filtering happens at ingest, continuously, instead of in your head during an incident. ## Dashboards: the glance-able view **What it actually is.** A dashboard is a saved page of widgets - counts, the error histogram, a top-services-by-error-count list, a live tail of the latest ERRORs - usually built over one or more streams. It answers, at a glance, "is anything on fire right now?" without you typing a query. **What it does in real life.** A useful ops dashboard might show: total error count in the last hour (a single big number), the error-rate histogram (the shape from Phase 2, but always on), and a table of which services are throwing the most errors. You glance at it in the morning, after a deploy, or when someone says "is prod okay?" The dashboard is the same searches you'd run by hand, frozen into a layout so you don't have to run them. ⚠️ **A dashboard is a view, not a notifier.** It shows you the spike only if you happen to be looking at it. For "tell me even when I'm not watching," you need an alert - next. ## Alerts: let the system page you **What it actually is.** An alert is a search plus a *threshold* plus a *time window* plus a *destination*. The system runs the search on a schedule and, when the result crosses the threshold, it notifies you (email, Slack, PagerDuty, a webhook). It's the difference between finding the fire and being told about it. **A real example - an error-rate alert.** The condition reads, in plain English: ```text IF (stream: Production Errors) count of matching messages IS greater than 50 IN the last 5 minutes THEN notify #oncall and PagerDuty ``` *What just happened:* You taught the system the shape of "something is wrong." Most of the time the 5-minute error count sits well under 50, so nothing fires. When checkout starts timing out and errors jump to ~100/min (the cliff from Phase 2's histogram), the 5-minute count blows past 50 and the alert fires - and you hear about the outage from the alert, not from an angry customer twenty minutes later. ⚠️ **Tune the threshold or you'll train yourself to ignore it.** Set it too low and it fires on normal noise; people mute the channel, and then the *real* alert gets ignored too - alert fatigue. Set it too high and you find out too late. Start from the baseline you can *see* in the histogram (Phase 2): pick a threshold clearly above the normal background rate, then adjust after you've watched it for a week. πŸ’‘ **Key point.** Streams + dashboards + alerts are three views of the same searches at three levels of attention: a stream is a pile you can open, a dashboard is a glance, an alert is a tap on the shoulder. Build the search once; reuse it at whichever level fits. ## The trade-offs underneath all of this This is where centralized logging stops being free, so it's the part worth being clear about. ### Logs are only as good as what you logged ⚠️ Everything in this guide - every search, stream, dashboard, and alert - operates on data your apps chose to emit. No tool can search, route, chart, or alert on a field that was never logged. If `checkout` never logged `request_id`, no stream will ever follow a request through it. If failures are logged as bare "error" with no context, your dashboard can count them but never explain them. The leverage you get out of Graylog is set by the quality of the log lines going in. Logging structured `key=value` (or JSON) data with the fields you'll actually search on - service, level, request id, status, duration - is what makes the rest of this work. ### Never log secrets ⚠️ A central log store is searchable by many people and often kept for a long time. That makes it exactly the wrong place for passwords, API tokens, session cookies, full credit-card numbers, or personal data you have no business retaining. A secret logged once is now a secret sitting in a searchable archive, replicated and backed up, waiting. Scrub or redact sensitive fields *before* they ship - at the application or the shipper. For how to handle credentials properly, see [Secrets Management](/guides/secrets-management). ### Retention and cost are a real trade-off **The tension.** Keeping every log forever would be ideal for investigations and impossible for the budget. Centralized stores cost money to run, and that cost scales with how much you ingest and how long you keep it. So every centralized logging setup makes a *retention* choice: how long before old logs are deleted (or moved to cheaper, slower storage). πŸ“ **Retention policy.** The rule for how long logs are kept before they're aged out. Common shapes: keep high-volume DEBUG/INFO for a short window (days), keep WARN/ERROR longer (weeks), keep a thin audit trail longest. Higher-severity, lower-volume logs are cheaper to keep and more valuable later. ⚠️ **The retention window is a wall you'll hit at the worst time.** The bug you're investigating may have first appeared *before* the oldest log you still have. If retention is 7 days and the regression shipped 10 days ago, the early evidence is gone. It's worth knowing your retention window *before* an incident, so you're not surprised by an empty search that just means "older than we keep," not "never happened." The real trade-off: log enough, at the right severity, kept long enough to investigate - without paying to store noise you'll never read. ## Recap 1. **Streams** are standing filters that keep a named subset of logs (e.g. prod errors) continuously populated - define the filter once, reuse it everywhere. 2. **Dashboards** freeze your common searches into a glance-able view - but only help when someone's looking. 3. **Alerts** run a search on a schedule and notify you when it crosses a threshold - so the system finds the fire, not the customer. Tune thresholds against the visible baseline to avoid alert fatigue. 4. All of it is bounded by what you logged: structured fields in, real leverage out. 5. Never log secrets into a searchable, long-lived store; and know your retention window - cost forces a trade-off between keeping everything and keeping nothing. **Related guides:** [Reading Logs Without Drowning](/guides/reading-logs-without-drowning) Β· [Observability: Logs, Metrics & Traces](/guides/observability-logs-metrics-traces) Β· [Secrets Management](/guides/secrets-management)