Metrics That Inform vs Vanity Metrics

Picture two numbers on a dashboard. One says Total signups: 1,284,000 - big, green, always climbing, great in a board deck. The other says Active users this week: 8,200, down 4% from last week. Only one will ever make anyone do anything: the first can literally never go down (it's a cumulative count), so it can't tell you whether you're winning or losing. The second just told you something's slipping.

That difference is the whole game of this phase. A vanity metric feels good and informs nothing; a metric that informs changes a decision. Here's how to tell them apart, and how to build numbers that carry meaning.

The dividing line: does it change a decision?

A vanity metric mostly goes up and to the right, looks good, and wouldn't change anyone's plans no matter what it did. A useful metric is one where a different value leads to a different action.

📝 Terminology. Vanity metric - a measurement chosen because it flatters (big, always rising, easy to grow) rather than because it informs a decision. The classic tell: you can't imagine an action you'd take if it dropped.

Run any metric through the question from Phase 1: "If this changed, what would someone do?"

Vanity metrics tend to be cumulative all-time totals; useful ones are rates, recent windows, and comparisons - the next three sections show why.

⚠️ Vanity metrics aren't lies - they're just unactionable. "Total signups" is a real, correct number. The problem isn't accuracy; it's that it can't lose. Anything that only ever goes up can't tell you when you're in trouble, and a metric that can't deliver bad news can't drive a decision.

Choosing the right aggregation

An aggregation is how you squash many rows into one number - count, sum, average, median, percentile. The one you pick decides what story the number tells, and the wrong one quietly lies.

"Average" is the reflex, and it's often the wrong one - averages get dragged around by a few extreme values, medians don't.

   Page load times (seconds) for 9 sessions:
      0.4  0.5  0.5  0.6  0.6  0.7  0.8  0.9  14.0
                                              └── one stuck session

   Average (mean): 2.1 s   ← dragged up by the single 14 s outlier
   Median:         0.6 s   ← the typical experience, unmoved

What just happened: Eight of nine users had a snappy sub-second load; one session hung at 14 seconds. The average says "2.1 seconds - kind of slow," painting almost everyone as having a bad time when they didn't. The median says "0.6 seconds - the middle user is fine," true, but it hides that one user had an awful time. Neither is "right" alone - they answer different questions. Typical experience? Median. Anyone having a terrible time? A percentile (like the 95th) that surfaces the tail.

💡 Key point. The aggregation is a decision, not a default. Typical → median; total volume → sum; worst-case → a high percentile. Reaching for "average" out of habit is how dashboards mislead while being technically correct.

The gotcha. Many BI tools default new tiles to sum or average with one click. Always ask whether summing or averaging this column actually means anything - summing a column of percentages is nonsense, and the tool will happily do it anyway.

Choosing the right denominator

A raw count rarely means anything until you divide it by something. "47 errors" - out of how many requests? The denominator turns a count into a rate, and a rate is what you can compare and act on.

Counts grow as your business grows, which makes them sneakily misleading - more users means more errors, more tickets, more everything, even while things get better per user.

                     January        June
   Support tickets      300          900     ← "tickets tripled! we're drowning!"
   Active users      10,000       60,000
   ─────────────────────────────────────
   Tickets per user    0.030       0.015     ← actually HALVED per user

What just happened: The raw ticket count tripled, which looks alarming and would push you to panic-hire support staff. But you grew 6x over the same period - per active user, tickets halved. Support improved, not worsened. The raw count pointed at exactly the wrong decision; the denominator (active users) rescued it.

⚠️ Pick a denominator that matches the question. "Errors per request" answers "how reliable is the service?" "Errors per user" answers "how many people got hurt?" Same numerator, different denominators, different decisions.

Add context: a number alone means nothing

"Revenue: $84,000." Good? Bad? No idea - a single number has no meaning without something to compare it against. Context converts a number into a judgment. Three kinds are worth adding, and the best tiles have all three.

1. Comparison - versus what? Show the number next to a reference point: last week, last month, same period last year. The comparison is where the meaning lives.

   ┌─────────────────────────────┐
   │  Revenue (this week)        │
   │                             │
   │     $84,000                 │
   │     ▲ 12% vs last week      │   ← the comparison turns a number into news
   └─────────────────────────────┘

What just happened: "$84,000" became "$84,000, up 12% from last week" - suddenly a story you can act on. Without the comparison, a viewer has to remember last week's number to know if this is good. The dashboard's job is to carry that memory so they don't have to.

2. Target - versus where we wanted to be. A goal line tells the viewer whether the number is good enough, not just whether it moved.

3. Trend - which way and how fast. One number is a dot; a small line chart behind it shows direction. "Up 12% this week" is fine, but a sparkline showing four straight weeks of climbing tells a more decision-ready story than one that's bouncing around.

💡 Key point. A bare number is a trivia question. Number + comparison + target + trend is a decision. When a tile feels useless, it's almost always missing one of these three.

Recap

1. The line between vanity and useful is "would it change a decision?" Cumulative all-time totals almost always fail this; rates and recent windows pass it.
2. Aggregation is a choice. Average gets dragged by outliers - reach for median for "typical," a percentile for "worst case." Don't ship the tool's default blindly.
3. Counts need denominators. A rate (per user, per request) survives growth; a raw count gets misleading as you scale.
4. A number alone means nothing. Give it a comparison, a target, and a trend.

Next: arranging these numbers so the most important answer hits the eye first - and the visual traps that mislead even with the right metrics.

← Phase 1: What BI Actually Is · Phase 3: Designing One People Actually Use →