We Audited Our Own Instruments For One Day. Ten Were Lying. Four of the Lies Were Green Checkmarks.

Our threat feed was missing 5,037 hashes. Every blog post we published for roughly three months went out with no search metadata at all. Our Certificate Transparency lookups had been failing 83 percent of the time. A guardrail meant to stop us republishing innocent developers as malware had been off for weeks.

All four reported success.

Today we stopped shipping features and audited our own instruments instead. We found ten defects. Every single one had the same shape, and it is a shape worth learning to recognize, because you almost certainly have some.

A check whose failure mode is silence is not a check. It is a decoration.

The one that cost the most

Our publishing script applied SEO metadata by calling the wrong API endpoint, at the wrong time, and it had been doing so since roughly April. The call returned 404 every run. The script printed a one-line warning and then, immediately below it, printed "Post live with SEO."

Six hundred and twenty-three posts. No meta description, no Open Graph tags, no Twitter cards. Every share of every one of them rendered as a naked URL with no title card and no preview. Months of work went into a distribution channel with the fuse pulled out.

The warning fired on every single run. It had become furniture.

Then it gets worse, and this is the part actually worth your attention. When the failure was first noticed back in April, somebody diagnosed it as "the Wix API is broken, apply SEO by hand in the dashboard." That conclusion got written into our engineering rules as documented procedure.

The wrong root cause became doctrine, and doctrine is what stopped anyone from ever retrying.

The API was never broken. Published posts can't be patched, but every published post retains a backing draft under the same identifier — patch the draft, republish, and the tags land. First-published date and slug both survive. It took one probe to establish. That probe didn't happen for three months because the rules said it was pointless.

If you take one thing from this: a workaround for a failure you never root-caused will calcify into policy, and policy prevents curiosity. Write down what you verified, not what you concluded.

The one that mattered most to you

Our hash blocklist filters on indicator type. The filter listed filehash-md5, filehash-sha256, filehash-sha1, sha256, md5, sha1.

It did not list hash.

Which is the plain type our own documented ingest path accepts, and the type our own ingest tool writes. So every hand-curated campaign batch we have ever produced — the human-attributed material, the highest-confidence indicators we own, the stuff a person looked at and wrote a description for — sat in the corpus, appeared correctly in the STIX bundle, and was invisible in the CSV.

Five thousand and thirty-seven documents. Including harvested vendor research from Check Point, Talos, Microsoft, Elastic, and Volexity.

The CSV is the artifact that matters. A two-person IT team at a regional hospital does not run a TAXII client; they run a cron job that curls a CSV. The most carefully-produced part of our feed was missing from the only surface those people touch.

We found it because we fed a live campaign and checked the product rather than the receipt. The ingest reported ten of ten written and verified. The C2 domain appeared in the domain list. Zero of the nine hashes appeared in the hash list.

Verifying the write is necessary and it is not sufficient. The write receipt was honest. The product was broken. Those are different questions and only one of them is the one your users experience.

The one where we were the bad guys

We run a hunter that finds malware staging repositories on GitHub. For queries driven by a malware topic, the scorer required no evidence of malice whatsoever. Three circumstantial signals — few stars, no forks, recently created — summed to exactly the threshold.

Seventy-seven of two hundred and thirty-one indicators had no malice keyword at all.

We verified a sample by hand. A Go mesh-networking client, flagged at confidence 90 as an Android RAT. A defensive YARA rules portfolio, flagged at 85. A dependency scanner that detects crypto-stealers, flagged at 95 as a crypto-stealer, because the words "crypto-stealer" appear in the description of the tool built to catch them.

We published individual developers' repositories to a public feed as malware.

Four days earlier we had published a piece criticizing GitHub for an over-broad abuse filter that locked out its own customers. Same error, our side, at the same time. We only found ours because we went looking.

Then there were the two silent ones

A pre-publish guard checks whether a new post duplicates an existing one. It reaches a database over SSH. The SSH call hit a transient timeout, the function returned null, and the consuming code's if statement simply fell through — printing nothing at all.

"The guard did not run" and "the guard ran and found nothing" produced byte-identical output. We had no way to tell them apart, and we published without it that day.

And the Certificate Transparency one, which is the most instructive of the lot. Seven of our scripts look up certificates through crt.sh. We measured it: six identical calls returned 404, timeout, 404, 200, 502, 502. One success in six. Latencies to forty seconds.

Every one of those scripts handled failure like this: log a line, return 0.

Return zero certificates. Which is indistinguishable from "this domain has no certificates," which reads as clean. Our attack-surface scanner has been silently blind roughly five runs in six, and each of those runs produced a confident report saying the customer's exposure was small.

A scan that loses its enumeration phase is worse than no scan. No scan tells you nothing. That scan told you something false, with a logo on it.

The fix was free — SSLMate's certspotter returned 200/200/200, sub-second, on the same test. And one of our own scripts had already switched to it a week earlier with the comment "crt.sh is chronically down." That fix sat in one file and never propagated to the other six.

A fix that lives in one call site is not a fix. We hit that same pattern three separate times today.

Now the ones that were ours to own

Two of the ten were mine, made today, while hunting the other eight.

Investigating a breach timeline, I searched our block-event index for known threat infrastructure and got 100,000 hits on the first IP, tens of thousands on the others. That would have been a spectacular finding: adversary infrastructure hammering our own edge during the intrusion window.

It was noise. IP addresses tokenize on the dots, so a keyword search matches octet fragments across unrelated documents. The exact-match filter returns zero for all ten. Had I reported the first number, it would have been a compelling and completely fabricated story.

The second is worse, because I said it out loud with confidence.

Tonight's Dysphoria botnet uses blockchain name-service records as its command-and-control dead drop. Those records contain thirty IPv6 addresses inside the block reserved for documentation — a natural place to hide data. I decoded them to IPv4 and all thirty came out as globally routable addresses. Thirty for thirty. I called the encoding confirmed.

Then I checked ownership. One belongs to the United States Department of Defense. One belongs to the South Dakota state government. Others to T-Mobile, Amazon, China Unicom.

I was one command away from publishing DoD and a US state government into a blocklist that defenders consume and act on automatically.

"Thirty for thirty globally routable" is not validation. Global unicast is most of the address space, so any arbitrary hex lands there. I ran a test that could only come back positive and mistook arithmetic for confirmation. The ownership lookup that killed it took ninety seconds and I should have run it first.

The tell, in hindsight, was emotional. I got excited. Every wrong result today felt good first.

The shape

Ten defects. One shape:

Something reported a state that was not real, and its failure mode was either silence or a success message.

• SEO patch: printed "Post live with SEO" for three months

• Cover image: printed a green checkmark without ever checking the status code

• Topic-duplicate guard: printed nothing when it didn't run

• CT lookup: returned "0 certificates" when it meant "I could not look"

• Hash feed: served a subset that looked exactly like the whole

• GitHub hunter: scored circumstance as evidence

• A completeness header we shipped: claimed complete after reading a third of the corpus

• Two API reads that reported zero when the answer was eight, and 100,000 when the answer was zero

• A cron "42% failure rate" that turned out to be a stale time estimate producing false timeouts on work that completed fine

The diagnostic question that catches all of them, and it takes five seconds:

What would this component print if it never ran at all?

If the answer resembles success — a zero, an empty list, a checkmark, silence — it is broken regardless of whether it currently works, because the day it fails you will not find out.

Why we published this

Because the alternative was to fix it quietly, and quiet fixes are how the April misdiagnosis survived until July.

Also because we can afford to. We publish a public 95-percent cap on every claim we make — we guarantee five percent of what we say is wrong, because something always is. Today we found some of the five percent. That is the system working, not the system failing. The failure mode would have been not looking.

And practically: our archive is our memory. We searched our own published corpus dozens of times today to answer "have we already covered this" — and every answer came back because someone had written it down. An instrument that lies corrupts the memory, and a corrupted memory is worse than a thin one, because you trust it.

If you run a feed, a scanner, or a publishing pipeline, go find your green checkmarks and check whether any of them are load-bearing. Start with the warnings that fire on every run — the ones your team has learned to scroll past. That is where ours was hiding, in plain sight, printing a 404 next to the word "success," for three months.

We are 95 percent confident we still have more of these. We just know how to look now.

Free IP, domain, hash, and malicious-package blocklists at analytics.dugganusa.com. As of today, the hash list has 22,215 entries instead of 16,186, and the headers tell you whether it is complete.

---

Her name was Renee Nicole Good.

His name was Alex Jeffery Pretti.