# Data Quality & Pipeline Observability
> How to trust the numbers your pipelines produce: why a green job can still ship wrong data, the checks that catch silent breakage, and the observability that finds it before a human makes a decision on it.
---
# Data Quality & Pipeline Observability
Your pipeline ran. The orchestrator shows a row of green checkmarks. The job exited zero, the dashboard
refreshed, and you moved on. A week later someone in a meeting says revenue is down 12% - and acts on it.
Budgets shift, a hire gets paused. Then someone notices the number is wrong: an upstream column quietly
started arriving as `null`, the pipeline averaged over the nulls, and nobody told anyone, because nothing
*crashed*. The job was green the whole time.
That is the nightmare this guide is about. Not the loud failure that pages you at 2am - you'll catch that.
The quiet one: a pipeline that *succeeds* while producing wrong data, and stays wrong until a decision is
built on top of it. Here's the relief: this is a solvable problem. Data can be tested like code, and a
running pipeline can be watched like a production service. This guide gives you the mental model for why
green isn't enough, the specific checks that catch silent breakage, and the observability that surfaces it
before a human ever sees the bad number.
## How to read this
- **Already convinced and want the checks?** Jump to [Phase 2: Data Quality Checks](02-data-quality-checks.md) for the dimensions worth testing and where to run them.
- **Want it to actually click - why trust is the whole product?** Read in order. Phase 1 installs the mental model the rest depends on.
## The phases
1. **[Why Trust Is the Whole Product](01-why-trust-is-the-whole-product.md)** - the mental model: data can be broken even when the job is green, and a silent data bug is worse than a loud crash, because no one knows to look.
2. **[Data Quality Checks](02-data-quality-checks.md)** - the dimensions worth testing automatically (freshness, volume, schema, validity), where to run them, and an annotated check that fails the pipeline before bad data spreads.
3. **[Pipeline Observability](03-pipeline-observability.md)** - seeing the whole system: lineage, monitoring and alerting on those checks, data SLAs, and avoiding the alert fatigue that turns every signal into noise.
> This guide is about *trusting* the data a pipeline produces. How those pipelines are built - the
> extract/transform/load mechanics, batch vs. streaming, the orchestration - lives in
> [ETL & ELT Pipelines](/guides/etl-elt-pipelines). If the words "pipeline" or "transform" feel fuzzy,
> read that first, then come back here to make what they produce trustworthy.
---
# Why Trust Is the Whole Product
Before any checks or tools, let's fix one idea, because every technique later in this guide exists to
serve it: **the product of a data pipeline is not a successful job. It's a number someone can trust.**
You can have a flawless pipeline - clean code, fast runs, a wall of green in the orchestrator - and still be
doing active harm, if the numbers coming out the far end are wrong and people believe them. The job
succeeding tells you the *machinery* ran; it tells you nothing about whether the *output is true*. Those
are two completely different questions, and conflating them is the single most expensive mistake in data
work.
## The two questions a pipeline answers (and only one of them is the green checkmark)
Every pipeline run silently answers two separate questions:
1. *Did the job run to completion?* - Did the code execute, did the queries return, did the load finish
without throwing?
2. *Is the data it produced correct?* - Are the rows there, are they fresh, do the values make sense, do
the totals match reality?
The green checkmark in your orchestrator only ever answers question 1. It is a statement about the
*process*, not the *product*. Nothing about "exit code 0" inspects whether the numbers are right.
```text
What the green check means What it does NOT mean
──────────────────────────────── ────────────────────────────────
the code ran without throwing │ the rows are actually there
the queries returned │ the values are correct
the load finished │ the data is up to date
exit code 0 │ the totals match reality
─────────────────────────────────────────────────────────────────────
"the machine ran" │ "the output is true"
```
We borrow our instincts from application code, where failures are loud - a web request that breaks throws a
500, a null pointer crashes the process, you *find out*. So we assume the same of data: if something were
wrong, surely it would fail. But data pipelines mostly don't work that way. Bad data is usually *valid*
data - it's the right type, it parses, it loads. A column that should be `1000.00` but arrives as `0.00` is
a perfectly well-formed number. A table that should have a million rows but got ten thousand is a perfectly
well-formed table. The machinery has no opinion about whether the content is true; it just moves it.
Once you accept that "green" and "correct" are different questions, data quality stops feeling like
overhead and starts feeling necessary. Phase 2 exists precisely because the orchestrator can't answer
question 2 for you - *you have to add the checks that do.*
## A silent data bug is worse than a loud crash
This is the part that surprises people, so let's state it plainly: when it comes to data, **a crash is the
good outcome.** Counterintuitive, but true, and worth internalizing.
A loud failure - the job crashes, the load aborts, the orchestrator turns red - announces itself. Someone
gets paged. The bad data never reaches a dashboard, because the run didn't finish; the blast radius is
contained to "the table is stale," which is visible and fixable.
A silent failure does the opposite. The job *succeeds*, the bad data flows all the way to the dashboards
and the reports and the models, and there is no signal anywhere that something is wrong. The only way it
surfaces is when a human eventually notices the number looks off - and by then it may have already shaped a
decision.
```text
LOUD FAILURE (the good kind) SILENT FAILURE (the nightmare)
───────────────────────────── ──────────────────────────────
job goes red │ job goes green
someone is paged │ nobody is notified
bad data never ships │ bad data flows everywhere
"the table is stale" (visible) │ "the number is wrong" (invisible)
found in minutes │ found in days - after a decision
```
**A real example.** Picture a daily revenue pipeline. An upstream API changes, and the `amount` field
starts arriving as `null` for a slice of orders. Nothing crashes - `null` is a valid value. The transform
sums revenue, and `SUM` in SQL ignores nulls:
```console
$ # What the pipeline computed
SELECT SUM(amount) AS revenue FROM orders WHERE day = '2026-06-18';
revenue
-----------
842150.00
(1 row)
$ # What was actually true (nulls counted as the zeros they should have been)
SELECT SUM(COALESCE(amount, 0)) + (missing revenue, uncounted) ...
revenue
-----------
958320.00 -- the real figure
```
*What just happened:* The pipeline didn't error - it did exactly what it was told. `SUM` skipped the
`null` rows, so revenue came out lower than reality, by however much those nulled-out orders were worth.
The job was green. The dashboard updated. And the only thing wrong was the *truth* of the number, which no
exit code checks. Someone reading that dashboard has no way to know the figure is missing a chunk of real
orders.
⚠️ **Gotcha - "garbage in, garbage out" is not an excuse, it's a warning.** It's tempting to wave off bad
output as "well, the source was bad." But your pipeline is the last place that bad source data can be
*caught* before a human trusts it. If you pass garbage through silently, you've laundered it: the source's
mistake now wears your pipeline's credibility. Catching bad input is your job, not the source's.
This is why you'll spend real effort on the next two phases. You're not adding checks to make the pipeline
*run* - it already runs. You're adding them to convert silent failures into loud ones *on purpose*, so
wrong data trips a wire and turns the job red instead of slipping quietly onto a dashboard. The whole game
of data quality is **buying back the loud crash you'd normally get for free in application code.**
## Trust is fragile and slow to rebuild
One more piece of the model, because it explains why this matters beyond any single bug.
Trust in a data platform is asymmetric. It takes months of correct numbers to build, and a single bad
number - caught publicly, in a meeting, after a decision - to destroy. Once a stakeholder has been burned by
a wrong figure, they stop trusting *all* your numbers, not just the broken one, and go back to their own
spreadsheets. At that point the pipeline can be technically perfect and still be worthless, because nobody
believes it.
Every other part of this guide - the freshness checks, the lineage graphs, the alerts, the SLAs - is in
service of one outcome: that when someone looks at a number you produced, they can act on it without
second-guessing. That confidence *is* the product; the pipeline is just the machinery that delivers it. A
pipeline nobody trusts is a very expensive way to compute numbers people then ignore.
💡 **Key point.** The deliverable of data engineering is not a running pipeline. It's *justified trust* in
the numbers. Green means the machine ran; it never means the output is true. Your job in the next two
phases is to make "is the output true?" a question the pipeline answers automatically - and answers
*loudly* when the answer is no.
## Recap
1. Every pipeline run answers **two separate questions** - "did the job run?" and "is the data correct?"
The **green checkmark only ever answers the first.**
2. Bad data is usually **valid** data: right type, parses fine, loads fine. The machinery has no opinion
about whether the content is *true*, so wrong numbers don't crash anything.
3. A **silent data bug is worse than a loud crash**, because a crash contains the damage and pages someone,
while a silent failure ships wrong numbers everywhere and is only found when a human notices - often
after a decision.
4. The goal of data quality is to **convert silent failures into loud ones on purpose** - to buy back the
crash you'd get for free in application code.
5. **Trust is the product.** It's slow to build, fast to lose, and once gone the pipeline is worthless even
if it runs perfectly.
Next, the concrete part: the specific dimensions of data worth testing automatically, and where to put the
checks so they fail the run *before* bad data spreads.
---
# Data Quality Checks
Phase 1 left you with a job: make the pipeline answer "is the data correct?" automatically, and answer it
*loudly*. This phase is the how. You don't have to test the truth of every number - that's impossible. You
test a handful of **dimensions** that together catch the overwhelming majority of silent breakage, like
smoke detectors placed around the data: each watches for one kind of failure, and any one going off stops
the run. The four that earn their keep: **freshness, volume, schema, and validity** - each a question you
can answer with a query.
## Freshness - is the data actually up to date?
A freshness check asks: *when did this table last receive new data, and is that recent enough to be
useful?* It catches the most common silent failure of all - a pipeline that "succeeds" by faithfully
reprocessing yesterday's data because the *source* stopped sending new rows. The job is green; the data is
a day stale; nobody can tell from the checkmark. You find the newest timestamp in the table and compare it
to now - if the gap is bigger than it should be, you fail.
```console
$ # How old is the freshest row?
SELECT MAX(created_at) AS newest,
NOW() - MAX(created_at) AS lag
FROM orders;
newest | lag
---------------------+----------------
2026-06-17 23:58:11 | 1 day 08:31:49
(1 row)
```
*What just happened:* The newest order in the table is from late on the 17th, but it's now well into the
19th - a lag of over a day. If this pipeline is supposed to land yesterday's orders every morning, that lag
is the silent failure from Phase 1, made visible. The data isn't corrupted, it's *old* - and old data is
wrong data the moment someone reads it as current.
⚠️ **Gotcha - trust the data's own clock, not the job's.** It's tempting to mark freshness by "did the job
run today?" But a job can run on schedule and still load nothing. Always measure freshness from a timestamp
*inside the data* (`created_at`, `event_time`), not from when the pipeline executed - the point is to catch
the case where the machine ran but the data didn't arrive.
## Volume - did the row count swing wildly?
A volume check asks: *is the amount of data in the right ballpark?* Most pipelines process a fairly
predictable quantity day to day. A sudden collapse (a join silently dropped most rows) or explosion (a join
fanned out into duplicates) signals something broke upstream, even when every individual row looks fine.
You count today's rows and compare against what's normal - a fixed floor, or recent history.
```console
$ SELECT COUNT(*) FROM orders WHERE day = CURRENT_DATE;
count
-------
1284
(1 row)
```
*What just happened:* Today landed 1,284 orders. On its own that means nothing - the check only works
against expectation. If this table normally lands 40,000–60,000 rows a day, 1,284 is a five-alarm fire: a
join probably lost its match, or the source sent a partial file. The count didn't error; it just quietly
became absurd, and only a comparison reveals it.
📝 **Terminology.** *Fan-out* = a join that matches each input row to many rows on the other side,
multiplying the row count (often a sign of a missing or wrong join key). *Row-count anomaly* = a count that
deviates far enough from the historical norm to be suspicious. A simple threshold ("must be at least N")
catches the worst cases on day one; fancier setups compare against a rolling average.
## Schema - did a column change or disappear?
A schema check asks: *is the shape of the data what we agreed on?* Upstream teams rename columns, change
types, or drop fields without telling anyone - one of the most common ways a pipeline silently starts
producing nonsense. A vanished column might read as `null` everywhere; a type change from integer to string
can break a calculation while still "loading." You assert the expected columns exist with the expected
types before trusting the rest of the run.
```console
$ # Inspect the actual columns and types now arriving
SELECT column_name, data_type
FROM information_schema.columns
WHERE table_name = 'orders'
ORDER BY ordinal_position;
column_name | data_type
------------+--------------------
id | bigint
customer_id| bigint
amount | text <-- was numeric yesterday
created_at | timestamp
(4 rows)
```
*What just happened:* The `amount` column is now arriving as `text`, not the `numeric` it used to be -
someone upstream changed it. Every downstream `SUM(amount)` is about to either error or, worse, coerce
silently and produce a wrong total. A schema check that pins `amount` to `numeric` catches this *here*,
before the bad type poisons everything that reads it.
💡 **Key point.** A schema check is a *contract* with your upstream. It says "I depend on these columns
having these types," and it fails the moment that contract is broken - a loud, attributable alert instead
of a mysterious wrong number three tables downstream.
## Validity - do the values themselves make sense?
Validity is a family of checks on the *content* of individual columns. Three pull the most weight:
- **Nulls** - a column that should never be empty (a primary key, a required amount) suddenly contains
`null`. This was the exact mechanism behind the Phase 1 revenue bug.
- **Uniqueness** - a column that should have no duplicates (an `id`, an order number) suddenly has them,
usually from a fan-out join or a double-load. Duplicates inflate every count and sum built on top.
- **Ranges** - a value that should fall within sane bounds lands outside them: a negative age, an order of
`-50` items, a percentage of `1200`, a date in the year 2099.
Each is a query that counts the *violations* and fails if there are any.
```console
$ -- Validity: nulls, duplicates, and out-of-range values in one sweep
SELECT
COUNT(*) FILTER (WHERE amount IS NULL) AS null_amounts,
COUNT(*) - COUNT(DISTINCT id) AS duplicate_ids,
COUNT(*) FILTER (WHERE amount < 0) AS negative_amounts
FROM orders
WHERE day = CURRENT_DATE;
null_amounts | duplicate_ids | negative_amounts
-------------+---------------+------------------
37 | 0 | 4
(1 row)
```
*What just happened:* Today's load has 37 rows where `amount` is `null` (the silent revenue-killer from
Phase 1, caught in the act) and 4 rows where `amount` is negative (impossible for a real order - probably a
sign error or a refund leaking into the orders table). None of these rows would have crashed anything; each
is "valid" to the database. The check is what gives them an opinion: *these values don't make sense, stop
the run.*
**Try a validity sweep yourself.** Same shape of check - count the violations in one query - over a tiny
seeded library dataset. It returns zero violations today; change a threshold to watch it "fire":
```sql runnable
SELECT
SUM(CASE WHEN b.title IS NULL THEN 1 ELSE 0 END) AS null_titles,
COUNT(*) - COUNT(DISTINCT b.id) AS duplicate_ids,
SUM(CASE WHEN b.year < 1800 THEN 1 ELSE 0 END) AS impossible_years
FROM books b;
```
⚠️ **Gotcha - test the column that hurts, not every column.** It's tempting to slap a null check on all 80
columns. Don't. A null in an optional `notes` field is fine; a null in `amount` corrupts revenue. Spend your
checks where a bad value actually changes a number someone trusts - over-checking is its own failure mode,
covered as alert fatigue in [Phase 3](03-pipeline-observability.md).
## Where to run the checks: fail fast, before bad data spreads
Knowing *what* to check is half of it. The other half is *where* - **run checks as early as you can, and
make a failure stop the pipeline before bad data reaches anything downstream.** That's the difference
between catching bad data in one staging table and chasing it across twenty.
```mermaid
flowchart LR
subgraph WO [Without fail-fast checks]
direction LR
s1[source] --> st1[staging] --> m1[marts] --> d1[dashboards
wrong numbers]
end
subgraph WF [With fail-fast checks]
direction LR
s2[source] --> st2[staging] --> chk{CHECK} -->|fail| stop[STOP
bad data never reaches marts]
chk -->|pass| m2[marts]
end
```
You insert the checks as a *gate* between stages - after data lands in staging but before it's transformed
into the marts that dashboards read. If a check fails, the run aborts right there: the staging table might
be wrong, but the trusted downstream tables are never touched, so nobody reads a bad number while you fix
the source. Here's that gate as an annotated assertion the pipeline runs and reacts to:
```console
$ -- quality_gate.sql : the run aborts if this returns any rows
WITH violations AS (
SELECT 'stale' AS check_name
WHERE (SELECT NOW() - MAX(created_at) FROM staging_orders) > INTERVAL '26 hours'
UNION ALL
SELECT 'too_few_rows'
WHERE (SELECT COUNT(*) FROM staging_orders WHERE day = CURRENT_DATE) < 10000
UNION ALL
SELECT 'null_amounts'
WHERE EXISTS (SELECT 1 FROM staging_orders WHERE amount IS NULL AND day = CURRENT_DATE)
)
SELECT * FROM violations;
check_name
--------------
null_amounts
(1 row)
```
*What just happened:* This single query asks all three questions at once and returns **one row per failed
check** - here, `null_amounts` fired. The rule: *if this query returns any rows, fail the run.* The
orchestrator sees a non-empty result, aborts before the transform step, and turns the job red - the silent
failure from Phase 1 converted into exactly the loud crash you wanted: a named reason, and bad data
quarantined in staging.
💡 **Key point - the check's job is to fail the run, not just to log.** A quality check that writes
"warning: nulls found" to a log and lets the pipeline continue is barely better than no check at all. The
power comes from wiring the check to **halt the pipeline and go red**, so bad data physically cannot
proceed and a human is forced to look.
Most teams don't hand-write every assertion - testing frameworks built into transformation tools (dbt ships
`not_null`, `unique`, `accepted_values`, and relationship tests, plus freshness packages) let you declare
checks next to the models they guard. The *tool* varies; the four dimensions and fail-fast placement stay
constant.
## Recap
1. You don't test every number - you test a few **dimensions**: **freshness** (is it up to date?),
**volume** (is the row count sane?), **schema** (did columns change or vanish?), and **validity** (do the
values - nulls, uniqueness, ranges - make sense?).
2. Measure freshness from a timestamp **inside the data**, not from whether the job ran.
3. A volume or range check only works **against an expectation** - a floor or a historical comparison; the
raw number alone says nothing.
4. A schema check is a **contract** with your upstream; it turns a silent type change into a loud,
attributable failure.
5. **Run checks early and fail fast** - gate the transform step so a failure halts the run and bad data
never reaches the trusted downstream tables.
6. The check's job is to **fail the run and go red**, not merely to log - that's how you buy back the loud
crash.
Next: zooming out from individual checks to the whole system - seeing which downstream tables a broken
source poisons, alerting on these checks without drowning in noise, and setting SLAs so you catch the
silent failure before a human does.
## Try it yourself
A quick format check - which values are valid ISO dates (YYYY-MM-DD)?
```playground-regex
^\d{4}-\d{2}-\d{2}$
2026-06-20
2026-6-20
20260620
not-a-date
```
---
# Pipeline Observability
In Phase 2 you placed smoke detectors around individual tables. This phase wires them into a building-wide
alarm system - and gives you the map of the building. That's the difference between a *check* and
*observability*: a check tells you one table is wrong; observability tells you the *whole system's* health,
*which* downstream tables a broken source has poisoned, and gets a signal to a human the moment something
trips - ideally before anyone reads a bad number.
The mental model: a data quality check is a thermometer. Observability is the whole hospital monitor - many
readings, a history, a map of how the patient's systems connect, and an alarm that goes off at the right
bedside. You already have the thermometers. Now we connect them.
## Lineage - the map of what poisons what
*Lineage* is the dependency graph of your data: which tables are built from which other tables, all the way
from raw sources to the dashboards people read. It's the answer to two questions you'll be asked in every
data incident - *"if this source is broken, what's affected?"* (downstream) and *"this dashboard looks
wrong, where did the number come from?"* (upstream).
📝 **Terminology.** *Upstream* = the tables your table is built *from* (its inputs). *Downstream* = the
tables built *from* yours (what depends on you). *Lineage* = the full upstream-to-downstream map across the
whole platform.
When a freshness check fails on a raw source, lineage instantly tells you the blast radius - every mart and
dashboard that draws from it, so you know what to quarantine and who to warn.
```mermaid
flowchart TD
raw[raw] --> staging["staging_orders
freshness check FAILS here"]
staging --> rev[daily_rev]
staging --> cohorts[cohorts]
staging --> exec[exec_summary]
rev --> dash["'Revenue' dashboard
a human is about to act on this"]
```
*What just happened:* The diagram traces a single failure at `staging_orders` down to every table and
dashboard that depends on it. Without lineage, you'd find out about the blast radius the slow way - one
angry Slack message per affected dashboard, over hours. With it, you see in one glance that `daily_rev`,
`cohorts`, `exec_summary`, and the Revenue dashboard are all downstream of the break, so you can hold them
*before* anyone trusts them.
💡 **Key point.** Lineage is what makes a single check *systemic*. A check says "this table is bad."
Lineage says "...and therefore these twelve things built on it cannot be trusted until it's fixed." That
second sentence is what lets you protect decisions instead of just tables.
## Monitoring and alerting - getting the signal to a human
*Monitoring* is recording the results of your checks over time - not just pass/fail right now, but the
history (how fresh has this table been every day for a month? how has the row count trended?). *Alerting*
is the rule that decides when a result is bad enough to interrupt a human, and the mechanism that reaches
them. A check that fails into a void helps no one - monitoring turns each result into a durable signal, and
alerting routes the *important* ones to a person through a channel they actually watch.
Two failures, two very different responses - and the difference is the whole craft:
```text
FAILURE RESPONSE
───────────────────────── ───────────────────────────────────
raw revenue source stale, page on-call NOW - money number,
feeds the exec dashboard ─► decision-grade, every minute counts
───────────────────────── ───────────────────────────────────
optional 'notes' field has log it, fix in normal hours -
a few nulls, feeds nothing ─► nobody needs to wake up for this
```
*What just happened:* Both are check failures, but only one deserves to wake someone. The severity of an
alert should track the **severity of the data it guards** - measured by what decisions ride on it and how
many downstream tables (via lineage) it poisons - not by how easy the check was to write. Wiring every
check to the same loud channel is how you get to the next gotcha.
## ⚠️ Alert fatigue - the failure mode of doing this well
This is the trap that catches teams *after* they get good at quality checks, so it deserves its own
warning. Alert fatigue is what happens when you have so many alerts - many noisy, flaky, or unimportant -
that people stop reading them. The alarm becomes wallpaper, and the moment it does, the *one that matters*
scrolls past unread, landing you right back in Phase 1: a real silent failure, except now you technically
"alerted" on it and still nobody looked.
The instinct after Phase 2 is to check *everything* - every column, every table, every dimension - and
route it all to one channel. It feels thorough. It's actually counterproductive: a hundred low-value alerts
don't add up to vigilance, they manufacture the indifference that lets the high-value one slip through.
More checks is not more safety past a point; **more *trusted* checks is.**
**How to keep alerts meaningful:**
- **Test what matters, not what's easy.** Guard the columns and tables that feed real decisions. A null in
`amount` pages someone; a null in `notes` does not.
- **Match severity to blast radius.** Use lineage: a failure feeding twenty downstream tables and the exec
dashboard is a page; one feeding nothing is a log line.
- **Route by severity, not all to one firehose.** Critical → page/on-call. Warning → a channel people check
during the day. Info → a dashboard, no notification.
- **Kill flaky checks fast.** A check that cries wolf - fails on a normal day, needs a "just re-run it" -
trains everyone to ignore *all* checks. A check you've learned to ignore is worse than no check, because
it costs attention and gives nothing back. Tune it or delete it.
💡 **Key point.** The goal isn't the *most* alerts - it's that **every alert is worth reading.** A small set
of trusted, high-signal alerts beats a hundred that everyone has muted. You're protecting a scarce
resource: human attention. Spend it only where a wrong number would actually cost something.
## Data SLAs - promising trust, and measuring it
A *data SLA* (service-level agreement) is an explicit, written promise about the data - most commonly its
**freshness** ("the revenue table is updated by 7am every business day") and its **correctness** ("it
passes its quality checks before it's published"). It turns vague expectations into a number you can
monitor against and be held to.
📝 **Terminology.** *SLA* = service-level agreement, the promise you make to consumers ("fresh by 7am").
*SLO* = service-level objective, the internal target you actually engineer toward (often a bit tighter, e.g.
"fresh by 6:30am," to leave headroom). Borrowed straight from how production services are run.
An SLA is what makes "is the data trustworthy?" measurable rather than a feeling. It converts the freshness
and quality checks from Phase 2 into a *commitment*: not just "we noticed it was stale" but "we promised
7am, the check tripped at 6:15, on-call had 45 minutes to fix it before anyone opened the dashboard." That's
the silent failure caught and resolved *before* a human ever acted on a bad number, with time to spare
because you set a target with headroom.
## Tying it together - observability serves the pipeline you built
Everything here sits *on top of* the pipelines from [ETL & ELT Pipelines](/guides/etl-elt-pipelines). That
guide builds the machinery that moves data; this one watches the machinery and, more importantly, the
*truth of what comes out of it*. The same orchestration that runs your extract/transform/load steps runs
your quality gates, records their history, and fires your alerts - observability isn't a separate system
bolted on, it's the same pipeline, instrumented so it can tell you when it's lying.
The full picture, end to end:
```mermaid
flowchart LR
E[Extract] --> T[Transform]
T --> G{Quality gate
fail fast, go red}
G --> L[Load]
L --> M[Monitor history
lineage + alerts + SLA]
M --> H[A human - before the
bad number is trusted]
```
*What just happened:* This is the whole guide in one line of pipes. You build the pipeline (the ETL/ELT
guide), gate it with checks that fail fast and go red (Phase 2), and wrap it in lineage, monitoring,
severity-routed alerts, and an SLA (Phase 3). The payoff is the exact inverse of the Phase 1 nightmare:
instead of a green job quietly shipping a wrong number that surfaces in a meeting a week later, a tripped
wire turns the job red, names the broken table, shows you everything downstream it endangers, and reaches a
human while there's still time to fix it.
## Recap
1. **Observability** is the system-wide view - many checks, their history, a dependency map, and alerts -
not a single table's pass/fail.
2. **Lineage** is the map of what poisons what; it turns "a source broke" into "these exact downstream
tables and dashboards are now at risk," so you protect decisions, not just tables.
3. **Monitoring** records check results over time; **alerting** routes the important ones to a human, and
alert severity should track the **severity and blast radius of the data**, not the ease of the check.
4. ⚠️ **Alert fatigue** is the failure mode of doing this well - too many low-value alerts and people stop
reading the one that matters. **Test what matters, route by severity, and kill flaky checks.**
5. A **data SLA** turns trust into a measurable promise (freshness, correctness) you can monitor against
and catch *before* the deadline - with headroom (the SLO) to fix it in time.
6. Observability isn't bolted on; it's the **same pipeline** from the ETL/ELT guide, instrumented so it can
tell you when its output can't be trusted.
That's the whole arc: a green job never meant the data was true (Phase 1); checks make "is it true?" a
question the pipeline answers loudly (Phase 2); and observability makes sure the answer reaches a human
before a decision is built on a lie (Phase 3). The product was never the pipeline. It was the trust.
**Related:** [ETL & ELT Pipelines](/guides/etl-elt-pipelines) · [What Is Data Engineering?](/guides/what-is-data-engineering)