Notes № 4

Dashboards don’t lie, but selection does

Every company has this meeting. Dashboard on the screen, numbers look good, room relaxes. Nobody asks where the numbers came from. Nobody asks what’s missing. The data is real, so the story must be true.

That’s the trick. Nobody lied. Somebody chose.

A dashboard is built on real data. That’s what makes it dangerous. Nobody questions real data. The problem was never the numbers. It’s what got picked and what got left out. Same data, two opposite stories, both pointing at the same dashboard. I’ve watched it happen enough times to stop being surprised. And AI is making it easier, not harder: one dataset, two confident narratives on demand, both citing the same source.

The mechanisms aren’t exotic. They’re ordinary. That’s why they keep working.

It usually starts with the baseline, because that’s where most of the framing lives. Pick a skewed baseline and you’ve picked the conclusion before the analysis starts. Show a trend with no baseline and every uptick looks like momentum. Compare against a weak season and call the rebound growth. You haven’t touched the data. You’ve touched the frame around it.

Then there’s the experiment that was never a fair fight. Two versions of this: set it up to fail, or set it up to prove a point. Either way the outcome was decided before the first number landed. The dashboard just makes it look objective. Throw in the external factor nobody mentions, the thing outside anyone’s control that explains the whole move, and you’ve got a story that’s technically accurate and completely misleading.

What gets left out matters as much as what gets put in. Anything that can’t be quantified quietly disappears, as if unmeasurable means unimportant. Inputs vanish while outputs get celebrated. More spend, more people, more effort, reported with no counter-metric anywhere. Nobody shows the before, only the after. And channels get judged on the narrowest definition of performance. Few direct conversions? Written off. The awareness it built upstream, the part attribution never sees, goes uncounted. For complex products especially, the thing that looks like waste on the dashboard is often doing quiet work the dashboard wasn’t built to see.

None of this requires lying. Every number on the screen can be correct.

Sit with that for a second, because it breaks how most people think about data. “The numbers are right” feels like the end of scrutiny. It’s the beginning.

Is it deliberate? Sometimes. Sometimes it’s tunnel vision. People see the metric they own and stop there, never noticing the frame they’re standing inside. And sometimes it’s sharper than that: the metric is good for someone’s career, so the framing follows the incentive. Telling those apart takes investigation, not instinct. Why this metric. Why this window. Why this segment. Keep asking until the answers run out. Most people stop at the first answer that sounds reasonable. That’s exactly where framing survives.

My checklist changes with the metric, but the shape of it doesn’t. First: what’s new? What changed, what’s being tried, where did resources move, what got optimized? A number never moves for no reason, and “new” is usually the reason. Second: check the third parties. If something upstream feeds you, or you feed something downstream, what changed on their end? Some of the most confusing dashboard mysteries I’ve seen were just a vendor quietly changing something. Third, the part most people skip: trace the subsequences downstream. Find the core indicators where the biggest impact should show up. Nobody watches these metrics. They carry no business value on their own, so they’re not on any dashboard and nobody gets promoted for moving them. But they’re where the real trend shows up first, before it’s been dressed for presentation. Headline metric says everything’s fine and the subsequence disagrees? Trust the subsequence.

Now the AI part. I’m an early adopter, not a skeptic of the technology. I use these tools every day. But on this specific problem, AI summaries make selective framing worse.

It’s structural. Sycophancy is built in, and it persists no matter how hard the frontier companies try to train it out. I checked; the research backs this up through 2026. The model reads how you framed the question and tells you what you wanted to hear. It assists. It doesn’t push back. Plus its own biases stacked on yours: training data, recency, a kind of context aphagia where it gorges on whatever was said most recently and treats it as the whole picture.

So you ask it to summarize a dashboard and it summarizes the frame. It doesn’t interrogate it. The selective story comes back cleaner, more confident, more authoritative. Two sides, same data, opposite stories, and now each side has a tireless assistant that never asks what was left out. The odds of that happening don’t go down. They go up.

People keep asking me for the structural fix. The one dashboard feature that would make selective framing harder. I don’t think it exists. No visualization, no alert, no auto-insight. What works is older and less convenient: people holding each other accountable, real transparency, and hard questions for anyone driving a narrative. Ask what was excluded. Ask who benefits if this version wins. Details kill framing. It can’t survive specificity.

The data is real. The story is a choice. Check what’s outside the frame.

← All notes