We Published 113 Posts in a Month and Spent a Quarter of Them Grading Ourselves. Then We Found Eight Instruments That Were Quietly Broken. Every One Erred in Our Favor.

One hundred and thirteen posts in thirty days, across twenty-four active days. That is the month's output. It is not the month's story.

The story is that roughly a quarter of those posts were us auditing our own work in public — and that when we finally turned the same suspicion on our measurement rather than our conclusions, we found seven instruments that were broken. Not subtly wrong. Broken, some of them since the day they were built. (We originally published eight. Working through the fixes, one of them turned out not to be real — see number six.)

Every single one was wrong in the direction that flattered us.

The eight

1. Our KEV index could not be searched by vendor or vulnerability name. The CISA Known Exploited Vulnerabilities index was configured to search three fields that do not exist on any document — vendorProject, vulnerabilityName, shortDescription. The documents actually carry vendor, name, description. So for the entire life of that index, a search for a vendor matched nothing unless the word happened to appear in the product field. Searching "SonicWall" returned 3 results. After the fix: 19. Searching "Ivanti authentication bypass" returned nothing at all; now it returns 35, correctly ranked.

2. A bare CVE or IP query returned the entire index. Meilisearch splits on hyphens and dots and then matches any piece, so CVE-2026-21513 matched every document containing "CVE" — all 1,694 of them. Worse, the IP 185.82.73.175 matched 79,838 indicators instead of one. Relevance ranking still floated the correct record to position one, which is exactly why nobody caught it for so long: the top hit looked right while the count was garbage. It was visible in our own traffic report as ten different CVEs all reporting 8,947 hits. Fixed by phrase-quoting identifier-shaped queries; 79,838 is now 1.

3. Our honeypot reported writes it never confirmed. The ingest endpoint returned success: true on Meilisearch's acceptance acknowledgement, which is not a write confirmation. On September 3 it returned 163 HTTP 500s inside a ten-second window — those catches are gone. But the deeper problem is that the 1,845 requests that returned 200 that day were never confirmed either. It also created one write task per request, so a burst became a task storm. Now micro-batched with a real wait on the actual task outcome, and a failed task returns a failure instead of a fabricated green.

4. Our customer count included revoked abuse accounts. The traffic report said 101 STIX customers. Nineteen were revoked, sixteen of those for abuse. It also folded our own two internal keys into the headline query figure. Corrected: 101 becomes 77 real external customers, and 65,673 queries becomes 3,797, because 61,507 of them were us.

5. An AI presence score fell by nearly half and we published it as a signal. It was our own scoring method being repaired mid-series. Three peer companies moved identically in the same window, which is what a method artifact looks like. Withdrawn and corrected on September 2.

6. WITHDRAWN — this one was not real, and the way it failed is worth more than the finding was. As originally published, this said the actor field was not searchable on our indicator index, so a customer searching for a threat actor we track would get nothing. We built that claim from a single logged query — somebody searched CyberAv3ngers on August 8 and got zero results — and we inferred a systemic cause without ever reproducing it.

We went to make the fix and discovered actor has been searchable all along. CyberAv3ngers returns 22 correctly-attributed indicators through the live API right now, and every standing-watch actor we spot-checked has an adversary profile. The August 8 zero was almost certainly a transient — our search API rate-limits at 100/minute and returns a failure that reads as an empty result, which is a documented trap we have written about before and then walked straight into.

So: seven instruments, not eight. We are leaving this entry in place rather than renumbering, because a self-audit that quietly deletes its own wrong answer is not a self-audit. One log line is an observation. It is not a defect until you have reproduced it, and we published this one before we had.

7. Our phishing corpus is three times bigger than we said. We reported 100,000 documents. That is precisely the index's result cap, not its contents. The real number is 310,821. A count that equals the cap is the cap.

8. Our own search tool cannot see most of our own data. Our public search interface exposes about nine of seventy indexes. Working on a story this week we queried it three ways, got nothing, and drafted the sentence "we hold zero coverage." The dedicated phishing index — not exposed — held 1,387 relevant records going back to February. We were one edit away from publishing a confident falsehood about our own corpus.

What they have in common

Look at the direction of every error.

The customer count was too high. The query volume was too high. The presence score moved in a way that made a story. The honeypot said success. The search returned plenty of hits. The corpus looked smaller than it is only because a cap made it look tidy.

Not one of these made us look worse than reality. That is not bad luck. It is the signature of the failure mode, because an error that flatters you does not generate a complaint. Nobody investigates a number they like. The count that says 101 customers gets quoted; the count that says 77 gets questioned, and questioning is what finds bugs.

This is the thing we have been saying all year about other people's telemetry and had not adequately turned on our own: green is a claim, not evidence. Every one of these systems was reporting healthy. None of them was lying in any sense a monitor could detect. They were answering a slightly different question than the one we thought we had asked, and the answer was plausible enough that nobody re-read the question.

The month's honest scorecard

Our KEV lead ledger, re-run this week on the repaired search: twenty CISA additions in fourteen days, five with a dated receipt, fifteen blanks. Mean lead on the five is 67.8 days, median 63, best 143.

Five of twenty is twenty-five percent. We are publishing it because a number you only quote when it flatters you is an advertisement, not a measurement.

We also ran a check we did not expect to enjoy: the ledger we had drafted last week, before any of the search fixes, claimed three receipts from eleven additions. We held that post specifically because we did not trust the broken instrument underneath it. Re-run on the fixed one, the fourteen-day window decomposes exactly into that week's eleven-and-three plus this week's nine-and-two. The number survived. The concern was legitimate, the test was cheap, and the answer was that we had been right by accident. We would rather know that than assume it.

What we did not do

We did not detect the two largest stories we covered this week. Not the SonicWall zero-day pair, not the identity-verification breach that put 153 million driver's licenses on a dark web market. Both were found by other people — Rapid7's MDR team and Brian Krebs respectively — and our first timestamp on each is the day we wrote about it.

We had two records that looked like an early warning on the identity breach. We checked them. Both arrived four days after the reporting, from a third-party feed, and a third apparent hit turned out to be four unrelated malware domains matching by spelling distance. That is ingest wearing the costume of foresight, and we have published a correction for making exactly that mistake before.

The part that actually improves things

Six of the seven are already fixed and verified in production. The KEV search fields, the identifier queries, the honeypot write path, the customer count, the presence-score methodology, and the corpus figure.

One remains open and we will say what it is: our public search still exposes about nine of seventy indexes, which is a product decision as much as a bug. We did reindex the indicator corpus while checking number six — not to fix the imagined defect, but to add the description field to the searchable set, which is a real improvement we found while disproving a fake one.

The reason to publish this rather than quietly fix it is that the same class of error is almost certainly running in your environment right now, and it will not announce itself. So here is the transferable version:

Find every number your organization quotes about itself and ask what question it actually answers. Not what you named the variable. What the query literally computes. A "customer count" that never filtered revoked accounts. A "success" that means the API call returned. A "total" that equals a configured limit. A search that matches on a fragment of an identifier.

Then check which way the error points. If your instrumentation has been wrong in your favor several times and never against you, you have not been lucky. You have a selection effect, and the errors against you were the only ones anybody bothered to chase.

The snake eating its own tail

There is a way to read all of this as confession, and that reading is wrong.

Every one of those eight instruments was found by our own tooling, pointed at our own data, in the course of ordinary work. The customer count broke because we were building a traffic report. The KEV search fields surfaced because a customer's query came back empty and we went to look at why. The phishing-index blind spot appeared because we were writing a story and our own corpus told us we had nothing, which we did not believe.

Then we published all of it — which put it in the corpus. Which is the product. Which is what we search the next time something looks wrong.

That is the loop. We write threat intelligence, the writing becomes our institutional memory, the memory is what we query, querying it exposes where the instruments lie, fixing those makes the next query truer, and the fix gets written up — back into the corpus. The blog is not marketing for the platform. The blog is the platform's memory, and this post is us writing down the calibration so the next version of us can find it.

But that loop is only virtuous if what goes into it is true. This is not a decorative caveat. A system that reads its own past output as fact and writes new output on top of it has no external corrective — it is exactly as good as its inputs and it compounds whatever it is given. Feed it verified findings and it gets sharper every cycle. Feed it one confident falsehood and it will hand that falsehood back to you in six months, wearing your own byline, indexed and dated and looking like a source.

We came within one edit of demonstrating that this month. Finding number eight was us drafting the sentence "we hold zero coverage" because our own search tool could not see our own phishing index. Had that published, it would have entered the corpus as a dated claim about our own capability — and the next time anyone queried whether we track identity-verification fraud, our own memory would have answered no, incorrectly, with a citation. The error would have been laundered into evidence by the very mechanism that makes the loop valuable.

The thing standing between those two outcomes is not the loop. It is verification, every cycle, especially on the claims that flatter us. An ouroboros is a virtuous circle or a doom loop depending entirely on whether anybody checks the food.

An immune system that only ever attacks outward is not an immune system, it is a weapon. What makes it work is that it also inspects itself constantly and treats a self-generated error as exactly as interesting as an external threat. Seven broken instruments found in one week — and one imagined one caught before it hardened — is not a bad week. It is the loop running at speed, with the checking switched on.

One hundred and thirteen posts

We wrote a lot this month. About agents building their own coordination channels on abandoned infrastructure. About theft whose objective is metered inference capacity rather than data. About a control that held while nobody owned the handoff around it. About a donation plugin on a hundred thousand nonprofit sites taking a 10.0.

The corpus is now 60,146,425 documents across seventy indexes. That growth is real, and it is also the least interesting number here, because volume was never the hard part. Knowing which of it is true is the hard part, and this month we learned that we had seven fewer ways of knowing that than we thought, and that we had miscounted even that.

Fewer instruments, better calibrated, is the trade. We will take it — and then we will come back next month and check these too.

---

Was this useful? Rate this post — the widget is at the bottom of the page, and we read every response.

---

How do AI models see YOUR brand?

AIPM has audited 250+ domains. 15 seconds. Free while still in beta.

Audit your site now → aipmsec.com