The Gauges Were All Green and the Line Was Down: Five Systems Lied to Me in a Day, and That's the Whole Security Story
- Patrick Duggan
- 44 minutes ago
- 6 min read
*There is a scene in The Phoenix Project where Erik drags Bill out of the war room, away from the dashboards, and makes him stand on the actual plant floor and watch the actual work. Because the dashboards were green and the product was still coming out broken, and no amount of staring at green was going to fix that. I lived that scene this week — as the guy whose dashboards were lying. I caught five of my own systems reporting "success" while doing nothing at all, in a single afternoon. And the reason that story matters to a CISO, a VP, or anyone who has ever read a status report and believed it, is that the lie is never in the breach. The lie is in the gauge that stayed green while the line went down.*
Five green lights, zero working machines
Let me show you the floor before I give you the sermon, because the sermon is worthless without the floor.
A threat-hunting system we run went dark for seventeen days and nobody noticed — because a single malformed identifier silently failed every write it attempted, and the job dutifully reported SUCCESS every single day. Green. Doing nothing.
A vendor-risk view that ranks the government's known-exploited-vulnerability catalog was confidently serving the word "Unknown" for all 1,630 entries — because it was reading one empty field instead of the one next to it that had the data. Green. Answering every question wrong.
An enrichment job ran on its schedule, reported success, and tagged exactly zero records — because the thing it was supposed to join against was empty and nobody had ever checked. Green. A ritual with no result.
An AI-perception audit was quietly flattering us, scoring us higher than reality, because it was asking a model a question the model couldn't actually answer. Green. Comforting and wrong.
And the honest one, the one that should scare you most because it was mine and it was confident: an analysis I built to prove we detect threats before the government proudly announced we were 1,316 days ahead of CISA. It was garbage — it was measuring whether our database mentioned a vulnerability, not whether we found it first, and since we mirror the entire catalog, the answer was a meaningless "always." I nearly shipped it to a schedule. It looked green the whole way.
Five systems. Five green lights. Five lies. One afternoon.
Green is a claim, not a fact
Here is the part that isn't about security at all, which is exactly why a VP should read it. A green dashboard is a claim your system makes about itself. It is not a fact about the world. The threat hunt claimed success; the fact was seventeen days blind. The enricher claimed success; the fact was a hollow ritual. The gap between the claim and the fact is where every quiet catastrophe lives, and it is invisible precisely because nobody is hurt by it in the moment. Silent decay does not page you. That is its defining, lethal feature.
Substitute your own dashboards. The project status that says "on track" while the technical debt compounds. The KPI that is green because the data behind it stopped updating in March. The team that reports velocity — activity — while the outcome the velocity was supposed to move sits exactly where it was. If you manage the metric instead of the mission, you will be confidently, invisibly wrong, and the first time you find out will be the incident review. The discipline is brutal and simple: distrust the green light, and go verify the outcome. Did the write land. Did the number move. Did the customer get protected. Not "did the job run."
The war moved. Your budget stayed put.
Now Erik points at the actual threat, and it is not where your org chart is aiming. Verizon's own breach data now says the quiet part out loud: software vulnerabilities start more breaches than stolen passwords. For twenty years we built an industry on the premise that the human is the weakest link — the phishing tests, the awareness posters, the click-training. And attackers, pragmatists that they are, simply stopped needing the human. Why trick someone into opening a door when there is an unpatched one standing wide open around the back?
That is a painful sentence for a security leader, because it means the shape of your spend may be fighting the last war. There is a mature line item for teaching people not to click, and a perennial afterthought called patch management — and the threat inverted underneath both. This week alone, China walked into the Department of Homeland Security through a Microsoft SharePoint flaw that had been on the government's own must-patch list for a year. No one clicked anything. The door was simply never locked.
The good news: the risk is a short, legible list
Here is where the data stops being grim and starts being useful, and it is the most actionable thing I can hand a CISO. Exploitation is not random. It is legible. Rank the known-exploited catalog by product and you get a short, knowable roster — Microsoft, Cisco, Adobe, Ivanti — the same names, again and again.
And we tested something specific: does that risk stick to proven-soft products, or does it spread to new ones? The honest answer, measured, was roughly half and half. A real correlation (0.55, for the analytically inclined) that products with a history of being exploited keep getting exploited — but also that half of new danger arrives on products entering the exploited set for the very first time.
That gives you a two-sentence prioritization policy you can hand down tomorrow: patch the chronic offenders first, because they will be hit again — and watch the newcomers, because half your risk is products you have never had to worry about before. Not "defend everything equally," which is how security budgets drown. Defend the legible short list, and keep one eye on the door that just appeared.
The newest doors, the oldest keys
And the punchline the whole week kept repeating, which is the most expensive thing to tell a board because you cannot procure your way out of it: the newest maximum-severity vulnerabilities are the oldest attacks on the oldest technology. Adobe ColdFusion — thirty-year-old software with fifteen prior known-exploited flaws — earned a fresh "perfect ten" this week, and the exploit was upload a file, get a shell, a trick older than the commercial web. The AI coding agents everyone is racing to adopt are being cracked with shell tricks from 1989.
The novelty is almost never the attack. It is the door the new technology left standing open, and attackers are far too experienced to invent something new when something ancient still works. Which means the defense is never a new product on the show floor. It is an old discipline — patch, least privilege, do not expose the admin panel to the internet — finally applied to the surface everyone got excited about and forgot to lock.
The one habit under all of it
Erik's real gift to Bill was never a tool. It was a habit: make the work visible, and look — on a cadence, not when you are worried, but because it is Tuesday and Tuesday is when you look. You cannot feel a model going stale, a metric jamming, a hunt going dark. Nothing hurts until you look. So the only defense against the silent lie is scheduled, deliberate, suspicious inspection of your own systems — including, and especially, the green ones.
That is the whole thing, and it collapses into five instructions for anyone with direct reports, in security or anywhere else:
Measure outcomes, not activity. Distrust the green light and verify the fact underneath it. Inspect on a cadence, before you have a reason to. Move your investment with the threat, not with your comfort or your org chart. And never lose the plot of what the metric is for — it serves the person it is supposed to protect, or it is theater.
Why I am telling you this against my own dashboards
We will cap it at ninety-five percent, as we always do, because certainty is the first false-green. But I am telling this story with my own five lies at the center of it, on purpose, because the credible version of "your dashboards are betraying you" is the one told by someone who caught his own doing it and killed them the same day — rather than the one selling you a new dashboard to replace the old. The systems that protect people are only as honest as the discipline that keeps checking them. The gauges were all green. The line was down. The only thing that ever fixes that is walking out onto the floor and looking — at the machine, not the meter.
The threat feed this post is built on
1.14M+ IOCs, STIX 2.1, precursor signals, supply-chain detection. Free API key in 30 seconds.