# Reading Dynatrace (What It's Showing You)

> What Dynatrace actually is, how to follow one request through a distributed trace, and how to read a 'Problem' and its proposed root cause without trusting it blindly.


---

# Reading Dynatrace (What It's Showing You)

Someone shares a Dynatrace link in the incident channel and says "the trace is right there." You open it
and find a wall of charts, a glowing map of boxes, a red banner, percentages, waterfalls, and a panel
confidently announcing a "root cause." It looks like it knows everything - and somehow that makes it
harder to read, not easier. Where do you even look first?

Here's the calm version. Dynatrace is not a magic oracle and it's not a hundred unrelated dashboards. It's
**one model of your system, kept continuously up to date**, with a few specific views layered on top. Once
you know what each view is *showing you* - and, just as important, what it's only *guessing* - the wall of
charts turns back into a story about one request, one service, one bad afternoon.

This guide assumes you already know the three pillars - logs, metrics, and traces - from
[Observability: Logs, Metrics & Traces](/guides/observability-logs-metrics-traces). Here we apply those
ideas to one specific tool, so you can read its screens instead of being read by them.

## How to read this

- **Handed a Dynatrace link mid-incident?** Jump to [Phase 3: Problems & Root Cause](03-problems-and-root-cause.md) -
  it walks you from the red alert to the actual cause, and tells you which parts to trust.
- **Want the tool to finally make sense?** Read in order. Phase 1 builds the mental model, Phase 2 teaches
  you to read a single trace, and Phase 3 puts it together when something is on fire.

## The phases

1. **[What Dynatrace Actually Is](01-what-dynatrace-actually-is.md)** - an always-on x-ray of your system:
   auto-instrumentation, the entity model, and Smartscape. The picture everything else sits on.
2. **[Reading a Service Flow & a Trace](02-reading-a-service-flow-and-a-trace.md)** - follow one request
   across services, read the response-time breakdown, and spot the slow tier or the failing dependency.
3. **[Problems & Root Cause](03-problems-and-root-cause.md)** - how Dynatrace folds many symptoms into one
   "Problem," proposes a cause, and why you verify that cause instead of trusting it blindly.

> Deep material - building custom dashboards, writing DQL queries, tuning alerting profiles and management
> zones, and the deployment/OneAgent setup itself - is deliberately left out. This guide is about *reading*
> what's already in front of you, not configuring the platform.


---

# What Dynatrace Actually Is

The first time you open Dynatrace it feels like ten products bolted together: infrastructure charts, a
service map, traces, user-session replays, a problems feed. The instinct is to treat each screen as its own
tool and try to memorize where everything lives. That instinct is exactly what makes it overwhelming.

Here's the secret that makes it calm: **all of those screens are windows onto one shared model.** Dynatrace
keeps a single, continuously-updated picture of your whole system - every host, every service, every
database, and crucially *how they talk to each other* - and each "view" is just a different lens on that one
picture. Learn the model and the screens stop being separate things to learn.

## Dynatrace is an APM / observability platform - an always-on x-ray

**What it actually is.** APM stands for *Application Performance Monitoring*. The mental model to hold:
Dynatrace is an **always-on x-ray of your running system**. An x-ray doesn't ask you to guess where the bone
is - it shows you the whole structure at once, and where something's wrong lights up. Dynatrace does the
same for your services: it's watching everything, all the time, and surfaces where the trouble is.

📝 **Terminology - observability vs. monitoring.** *Monitoring* answers questions you knew to ask ("is CPU
high?"). *Observability* is the property that lets you ask new questions after the fact ("why was *this one*
checkout slow at 2:14am?") without shipping new code. Dynatrace aims at the second - covered in depth in
[Observability: Logs, Metrics & Traces](/guides/observability-logs-metrics-traces).

**Why people get this wrong.** Newcomers think Dynatrace is "a dashboard tool" - a place where someone hand-
builds charts. You *can* build dashboards, but that's the smallest part. The heart of it is the live model
underneath, which is assembled for you whether or not anyone builds a single chart.

## Auto-instrumentation - why you didn't have to add tracing code

**What it actually is.** On each host runs an agent (Dynatrace calls it **OneAgent**). It hooks into your
application runtime and automatically detects your services, the requests flowing through them, and the calls
they make to other services and databases - *without you adding tracing libraries to your code*.

**What it does in real life.** This is the part that surprises people: you didn't write any trace-collection
code, yet full distributed traces appear. The agent instruments common frameworks and protocols at the
runtime level, so an incoming HTTP request and the outgoing database call it triggers are captured and
*stitched together* into one trace automatically.

⚠️ **Gotcha - "automatic" is not "total."** Auto-instrumentation covers the technologies Dynatrace knows how
to hook. A service written in an unsupported runtime, a call over an exotic protocol, or a third-party API
outside your hosts may show up as a vague "external" box or not at all. When a trace has a suspicious gap,
the first question is "is this tier actually instrumented?" - not "what broke inside it." An uninstrumented
hop looks a lot like a fast one.

## The entity model - everything is a thing with a type and an identity

**What it actually is.** Dynatrace doesn't store loose metrics floating in space. It models your system as
**entities**: a *host* is an entity, a *process* is an entity, a *service* is an entity, a *database* is an
entity. Each one has a stable identity, a type, and relationships to other entities ("this service *runs on*
that process," "this service *calls* that database").

**Why this matters for reading the UI.** Almost every screen is "an entity and the stuff attached to it."
When you click a service, you get *that service's* response time, failures, and the entities it depends on -
because the model already knows those links. You're never assembling the relationships yourself; you're
navigating ones the model already drew.

```mermaid
flowchart TD
  checkout["checkout-svc"] -->|calls| pricing["pricing-svc"]
  pricing -->|calls| ordersdb[("orders-db")]
  checkout -->|runs on| procA["process A"]
  pricing -->|runs on| procB["process B"]
  procA -->|on host| host1["host-01"]
  procB -->|on host| host2["host-02"]
```

*What just happened:* That diagram is the thing Dynatrace is really storing. Every chart, map, and trace you
see is rendered from these nodes and edges. This is why clicking around feels connected: you're walking a
graph, not flipping between unrelated reports.

## Smartscape & the service flow - the live map of who talks to whom

**What it actually is.** **Smartscape** is the visual, navigable form of that entity graph - a live map across
the tiers (applications → services → processes → hosts), showing the connections between them and updating as
your system changes. The closely-related **service flow** zooms in on one service and shows the chain of
services and databases that requests fan out to from there.

**What it does in real life.** When a service is slow, the map gives you the *neighborhood*: what calls this
service, what this service calls, and which of those neighbors is also unhealthy. That turns "the app is slow"
into "checkout calls pricing, pricing calls orders-db, and orders-db is the one lighting up." You get a
direction to look before you've read a single trace.

💡 **Key point.** The map shows you *topology and direction* - who depends on whom. It does **not** by itself
prove causation. A downstream service lighting up red might be the cause, or might just be the loudest victim.
The map narrows where to look; the trace (Phase 2) and the Problem analysis (Phase 3) are where you confirm.

**Why this saves you later.** When you understand that Smartscape, the service list, the traces, and the
problems feed are all the *same model* seen from different angles, you stop hunting for "the right dashboard."
You start from wherever you landed - a red service, an alert, a slow trace - and *navigate the relationships*
to the rest. The tool stops being a maze and becomes a map you already know how to walk.

## Recap

1. **Dynatrace is one live model of your system**, not a pile of dashboards - every view is a lens on it.
2. **OneAgent auto-instruments** your services, so distributed traces appear without you writing tracing
   code - but "automatic" only covers supported tech; gaps can be uninstrumented hops, not failures.
3. **The entity model** stores your system as typed nodes (host, process, service, database) with
   relationships - which is why clicking around feels connected.
4. **Smartscape / service flow** is that graph made visual: it shows topology and direction (who calls whom),
   narrowing *where* to look - but it shows correlation, not proven cause.

Now that you know what the model is, the next step is reading one request as it moves through it.


---

# Reading a Service Flow & a Trace

"It's slow" is the least useful sentence in an incident - and it's usually where you start. Someone points at
a trace and the obvious next question is the hard one: slow *where*? Slow in your code, slow waiting on a
database, slow because of one bad dependency three hops away? Staring at a chart of average response time
won't tell you. A single trace will.

A **distributed trace** is the record of *one request* as it travels through every service it touches. This
phase teaches you to read one: to follow the request across services, see where the milliseconds actually go,
and point at the tier that's eating the time. This is the difference between "checkout is slow" and "checkout
spends 90% of its time waiting on orders-db, which is fine until you look at the query."

> ⏭️ If "trace" and "span" are fuzzy, they're defined properly in
> [Observability: Logs, Metrics & Traces](/guides/observability-logs-metrics-traces). One line to carry in:
> a **trace** is the whole request's journey; a **span** is one timed segment of it (one service's piece, or
> one database call).

## What a trace is showing you - one request, broken into timed segments

**What it actually is.** Picture following a single customer's checkout click through your system. It hits
`checkout-svc`, which calls `pricing-svc`, which queries `orders-db`. The trace records each of those hops with
*when it started* and *how long it took*. Dynatrace draws this as a **waterfall**: nested bars, where bar
length is time and indentation is "who called whom."

**Why people get this wrong.** The common mistake is reading the waterfall like a to-do list - top to bottom,
as if each bar happens after the one above. It doesn't. A child bar sits *inside* its parent's time: the
parent is *waiting* while the child runs. The shape you're reading is "who is blocked on whom," not a sequence
of independent steps.

```text
   ONE TRACE AS A WATERFALL  (time flows left → right; illustrative ms)

   checkout-svc          |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■|   480 ms total
     ├ own work          |■■|                              20 ms (its own code)
     ├ pricing-svc       |  ■■■■■■■■■■■■■■■■■■■■■■■■■■■|    440 ms
     │   ├ own work      |  ■■|                            25 ms
     │   └ orders-db     |    ■■■■■■■■■■■■■■■■■■■■■■■■|     410 ms   ◄── the time sink
     └ render response   |                          ■■|    20 ms
```

*What just happened:* The total is ~480 ms, but `checkout-svc`'s *own* code only spent ~20 ms. Almost all the
time is inside `pricing-svc`, and almost all of *that* is one `orders-db` call at ~410 ms. The request isn't
slow because of checkout - checkout is mostly *waiting*. The waterfall just walked you three hops down to the
actual time sink. (Numbers here are illustrative, to show the shape - yours will differ.)

## The response-time breakdown - where the milliseconds actually go

**What it actually is.** For a service, Dynatrace summarizes *across many requests* where time is spent: how
much in the service's **own code** (CPU, in-process work) versus how much **waiting on calls to other
services and databases**. It's the waterfall idea aggregated - the same "self time vs. waiting time" split,
but for the service as a whole.

**What it does in real life.** This one breakdown settles the argument that wastes the most incident minutes:
*is it us or is it downstream?* If the bulk of a service's time is its own code, the problem is in this
service - a hot loop, a slow algorithm, garbage-collection pauses. If the bulk is waiting on a downstream
call, this service is healthy and you should follow the arrow to whatever it's waiting on.

💡 **Key point.** "Self time" (the service's own work) vs. "waiting time" (blocked on something downstream) is
the single most useful split on the screen. Read it first. It tells you whether to dig into *this* code or to
keep walking down the trace.

⚠️ **Gotcha - a fast bar can hide a slow truth.** A downstream span that looks quick might be quick because it
*failed fast* (errored in 5 ms instead of doing real work), or because that hop is uninstrumented and its real
time is invisible (see Phase 1). Don't read "short bar" as "healthy" on reflex. Cross-check with the failure
indicator on that span and with whether the tier is actually instrumented.

## Spotting the slow tier or the failing dependency

**What it actually is.** With the waterfall and the self-vs-waiting split, finding the culprit becomes a short
walk: start at the top, follow the longest child bar down, and stop where the time is being *spent* rather
than *passed along*. The deepest bar that's long *and* mostly its own work is your slow tier.

**A real example - reading a failing dependency.** Failures show up on the span itself. Suppose the same trace
comes back not slow but broken:

```console
Trace  checkout-svc  ·  POST /api/checkout  ·  Failed
  checkout-svc        18 ms   OK
    pricing-svc       12 ms   OK
      orders-db        6 ms   Failed
        error: connection refused (orders-db:5432)
  checkout-svc                Failed   HTTP 500 returned to caller
```

*What just happened:* The trace is fast - only ~18 ms - so this is *not* a performance problem. The failure
originates at the bottom: `orders-db` refused the connection in ~6 ms, `pricing-svc` couldn't get its data,
and the error bubbled up until `checkout-svc` returned a 500 to the user. Reading bottom-up, the *origin* of
the failure is the database connection, not the two services that faithfully reported it. The services aren't
broken; they're messengers.

📝 **Terminology - origin vs. impact.** The span where an error *first appears* is the **origin**. The spans
above it are the **impact** (they failed *because* the origin did). Incident time gets wasted when people debug
the impact. Trace the error to its origin span first.

**Why this saves you later.** A metric tells you a service got slower or started failing. A trace tells you
*which hop, and whether it's spending time or waiting, and where an error was born.* When you can read one
trace end to end, you can answer "is it us or them?" in under a minute - and stop debugging the service that's
merely downwind of the real problem.

## Recap

1. **A trace is one request's journey**; the **waterfall** shows nesting (who waits on whom), not a sequence -
   a child bar runs *inside* its parent's time.
2. **The response-time breakdown** splits a service's time into **own code** vs. **waiting on downstream** -
   read this first to settle "is it us or downstream?"
3. **Find the slow tier** by following the longest child bar to the deepest span that's long *and* mostly its
   own work.
4. **Find a failing dependency** by tracing the error to its **origin span** - the spans above it are impact,
   not cause.
5. **A short bar isn't automatically healthy** - it can mean a fast failure or an uninstrumented hop.

You can now read a single request. The last step is what Dynatrace does when *many* requests go wrong at once
- and how it tries to name the cause for you.


---

# Problems & Root Cause

It's 2:14am, a page goes off, you open the link, and Dynatrace shows you a single red **Problem** card that
says - confidently - what the root cause is. Part of you wants to believe it and go back to bed. Another part
remembers the time a tool blamed the wrong thing and a team spent an hour restarting an innocent service.

Both instincts are right. Dynatrace is genuinely good at the thing humans are bad at half-asleep: collapsing a
storm of correlated alerts into *one* incident and pointing at where it started. But its "root cause" is a
*proposal* built from correlation and topology, not a proof.

## The incident cheat-card

> **Paged right now? Find your situation, then read the section below.**

| You're looking at | The calm move |
|---|---|
| A red **Problem** card | Read what it's *grouping* - that tells you the blast radius (§1) |
| "Affected entities" / "affected users" | This is *impact*, not cause - it's who's hurting, not who's guilty (§1) |
| A proposed **root cause** | Treat as the #1 lead, not a verdict - open the evidence and confirm (§2) |
| The root-cause entity, but no *why* | Drop into its trace/logs for the code-level error (§3) |
| A cause that doesn't add up | Trust the trace and the timeline over the label - re-walk from impact (§2 ⚠️) |

---

## 1. What a "Problem" actually is - many symptoms, one incident

**What it actually is.** When something breaks, it rarely breaks quietly in one place. A slow database makes
the service above it slow, which makes the service above *that* slow, which trips the user-facing alert - five
red things, one underlying event. A **Problem** in Dynatrace is the platform's attempt to recognize that those
five red things are *one incident* and group them into a single card, instead of paging you five times.

**What it does in real life.** It uses the entity model from Phase 1 - the map of who depends on whom - plus
the *timing* of when each symptom started, to decide which alerts are part of the same story. The result is one
Problem with a timeline, a set of **affected entities**, and a proposed cause.

📝 **Terminology - Davis.** Dynatrace's built-in analysis engine is called **Davis**. When the UI says "Davis
detected" or shows an AI-flavored root cause, that's this engine correlating signals across the topology. Treat
"Davis says" as "the correlation engine's best inference," not as ground truth.

⚠️ **Gotcha - "affected entities" is impact, not cause.** This trips everyone. The list of affected entities
(and affected users) is *who is hurting* - the blast radius. It is **not** the culprit. The thing at the top of
the impact list is usually the most *visible* victim (the user-facing service), which is the furthest thing from
the cause. Read the affected list to understand *scope*; look elsewhere for *cause*.

```text
   ONE PROBLEM = MANY CORRELATED SYMPTOMS, ONE TIMELINE  (illustrative)

   time ─────────────────────────────────────────────►
   orders-db     █ slow queries start                    ◄── earliest symptom (likely origin)
   pricing-svc       ░ response time climbs
   checkout-svc          ░ errors climb
   user impact               ▲ ALERT fires, page sent     ◄── most visible, latest, NOT the cause
```

*What just happened:* Four red signals, but read left-to-right they're one cascade with a clear *order*:
`orders-db` degraded first, and the user-facing alert - the thing that actually paged you - came *last*. The
Problem card bundles all four so you triage one incident, and the timeline hints that the earliest symptom is
the place to look. (Order and timing illustrative.)

## 2. From alert to proposed root cause - and why you still verify

**What it actually is.** The proposed **root cause** is Dynatrace's inference about which entity and which
change started the cascade - derived from the dependency graph (what could affect what) and the timeline (what
degraded first). It's typically your single best starting lead.

**Why you don't trust it blindly.** It's an inference from *correlation plus topology*, and that has plain
failure modes:

- **Two things break at once.** If a deploy and an unrelated cloud-network blip happen in the same minute, the
  engine may fold them into one Problem and pick the wrong one as cause.
- **The real cause is uninstrumented.** If the true origin is a tier Dynatrace can't see (Phase 1), it can only
  blame the nearest thing it *can* see - the innocent neighbor of the real culprit.
- **Correlation isn't causation.** Two services that always move together can have the engine point at the more
  visible one.

**The calm move - confirm before you act.** Don't restart, roll back, or page a team on the label alone. Open
the Problem's evidence and check three things:

1. **Timeline order** - did the named cause really degrade *first*? (Phase 1's "earliest symptom" instinct.)
2. **A representative trace** - does an actual failed/slow request (Phase 2) pass *through* the named cause and
   spend its time or throw its error *there*?
3. **A correlated change** - was there a deploy, config change, or traffic spike on that entity at that minute?

If all three line up, you've upgraded a *proposal* into a *confirmed* cause and you can act with confidence. If
they don't, trust the trace and the timeline over the label, and re-walk from the impact down.

🪖 **War story - the day the label lied.** A team got a Problem blaming a payments service and spent twenty
minutes poking at it. The trace told a different story: every failed request died at a *cache* node the
analysis had folded in as a mere "affected entity." The cache had run out of connections; payments was just the
loudest victim. The lesson wasn't "the tool is wrong" - it was "the proposed cause is lead #1, and the trace is
the judge." (Details generalized.)

## 3. From the cause entity to the code-level *why*

**What it actually is.** Even a *correct* root-cause entity usually tells you *where*, not *why*. "Root cause:
`orders-db`" or "Root cause: `payments-svc`" names the tier - it doesn't tell you it was a null dereference, a
connection-pool exhaustion, or a bad query. For that, you drop from the topology down into the actual evidence:
the failing trace and the logs/exception attached to the offending span.

**What it does in real life.** Open a representative failed request on the named entity (Phase 2's trace view),
find the **origin span**, and read the exception it carries. That's where the abstract "this service is the
cause" becomes a concrete stack trace you can act on.

```console
Problem: Failure rate increase on payments-svc
  Root cause (proposed): payments-svc
  → open a failed trace → origin span: payments-svc · charge()
    Exception: NullPointerException
      at PaymentProcessor.applyDiscount(PaymentProcessor.java:88)
      at PaymentProcessor.charge(PaymentProcessor.java:51)
```

*What just happened:* The Problem named the *tier* (`payments-svc`). The trace named the *request*. The origin
span named the *line of code* - a null dereference in `applyDiscount`. Now you have something a human can fix,
not just a colored box. Reading that stack trace - top frame vs. the line that's actually yours, "caused by"
chains, and how to find the first frame you control - is its own skill, walked through in
[How to Read a Stack Trace](/guides/reading-a-stack-trace).

**Why this saves you later.** The full chain is *alert → Problem (scope) → proposed cause (lead) →
confirming trace (judge) → origin span (the code).* Each step narrows from "something's wrong" to "this exact
line." When you can walk it, an incident stops being a panic and becomes a short, repeatable descent from
symptom to cause.

## Recap

1. **A Problem groups many correlated symptoms into one incident** using the dependency graph plus the
   timeline - so you triage one story, not five pages.
2. **Affected entities are impact (the blast radius), not cause** - the most visible victim is usually the
   furthest thing from the culprit.
3. **The proposed root cause is your #1 lead, not a verdict** - confirm it with timeline order, a real trace,
   and a correlated change before you act.
4. **The cause *entity* is "where," not "why"** - drop into the origin span's exception, then read it with
   [How to Read a Stack Trace](/guides/reading-a-stack-trace).

That's the whole tool. The thing to keep, across all three phases: **Dynatrace is excellent at surfacing
*where* - the slow tier, the failing hop, the correlated incident. You still bring the *why* - the trace you
confirm, the timeline you check, the line of code you read.** The x-ray shows the break; the doctor still
reads the film.