We Measured Our Own Exploit Harvester and It Was 48% Blind. The Fix Came From Lord Vetinari: Stop Chasing the Rats, Start Taxing the Rat Farms.

We missed a repository. One repository, public since 4 April, containing a working exploit chain for a vulnerability in a product we had just spent an afternoon researching. Our harvester never collected it.

Rather than add that one CVE by hand and call it covered — which is the false-green, and we have written enough about false-greens to know better — we went looking for the class of failure.

There were three, they were independent, and none of them ever produced an error.

What we found

Silent page truncation. The harvester read 200 results per query. The query returned 286, and the broader variant returned 357. So 86 and 157 repositories fell off the bottom of every single run, and nothing anywhere said so.

No historical backfill. It only ever looked at a rolling seven-day window of recently-pushed repositories. A repository missed during its one window was invisible permanently — there was no second pass, ever. Without the recency filter, the same query matches 2,900 repositories the harvester had never examined.

Year-window narrowness. It searched only the current and prior year. Exploit publication does not respect a CVE's year — people write proof-of-concept code for old bugs constantly, and arguably an old bug with a fresh working exploit is the more urgent signal, because defenders stopped watching it years ago. That is how we lost CVE-2024-3094. The XZ Utils backdoor.

All three are now fixed and deployed. The backfill walks the historical corpus a slice at a time with a cursor on durable storage, so it survives deploys rather than resetting to page one on every revision.

The number that mattered

Here is the part worth sitting with.

We could not tell you how much we were missing, because nothing in the pipeline had ever been asked. "We harvest public exploits" was a true statement with no denominator behind it, and a capability claim without a denominator reads exactly like a working one right up until somebody checks.

So we checked. Against a single prolific proof-of-concept author: 31 distinct CVEs published, receipts held for 15.

48%.

Not because the pipeline was broken. It ran every six hours, wrote documents, produced receipts we have cited in published posts. It was simply never measured, and unmeasured is not the same as fine.

Every detector you own is quietly reporting a numerator. Go and find its denominator.

The dangerous number is not zero

This is the finding that generalises furthest, and it caught us five separate times in one day.

We searched our corpus for Russian BDU vulnerability identifiers: 7 hits. All seven were domain names containing the letters "bdu". Genuine BDU identifiers: zero.

We searched for the CVE we had missed: 3,459 hits. Genuine references to it: zero.

We checked whether some IP addresses had neighbours in our threat corpus: 8,701 and 52,026 hits. True prefix matches: zero, both times.

We checked whether a threat actor was in our roster: 1 hit. It was a different actor entirely — searching "Head Mare" returned CyberAv3ngers, and searching "PhantomCore" returned GREYVIBE.

Every one of those numbers looked like partial coverage. Four of the five would have shipped as findings if we had trusted them. The last one would have shipped as coverage we did not have, which is worse, because a gap that answers with a plausible wrong name never gets investigated.

A zero invites a second look. A small number closes the question. That is why the small number is more dangerous.

The rule we now apply: any count used to justify a conclusion gets its hits individually verified against the thing you were actually looking for, before it is allowed to close anything.

Two axes, not one

While fixing this we built a tool that walks GitHub's social graph instead of searching it for keywords — following the followers of accounts that publish exploits, on the theory that people doing this work are connected to each other even when their repositories share no searchable term.

It scored our missed author at 33 out of 100 on threat. Which is correct. They publish proof-of-concept code and detection tooling; they are a researcher, not an adversary, and we have an explicit rule against naming researchers as threat actors on a keyword match.

If that had been the only score, we would have marked them "ignore" and moved on.

They hold 31 distinct CVEs across four years and we had receipts for 15 of them. On intake value they score 100.

So we split the scoring into two independent axes, because "is this dangerous" and "is this worth watching" are different questions and combining them loses information in exactly the direction that hurts. The upper-left quadrant — low threat, high yield — is where recall actually gets fixed, and it is completely invisible to a tool that only asks whether something is bad.

Which brings us to Lord Vetinari

With apologies to the late Sir Terry Pratchett, whose books have more operational security wisdom in them than most conference tracks.

The Patrician of Ankh-Morpork never tried to abolish the Thieves' Guild. He licensed it. Gave it quotas, receipts, a headquarters and a seat at the table, and thereby turned crime from an unbounded chaotic threat into a budget line he could see, predict and negotiate with. He chose enumerable visibility over the fantasy of eradication, and the city ran better for it.

And underneath that sits the older joke that history keeps re-running for real, from the rat bounty in Hanoi to the cobra bounty in Delhi: put a bounty on rat tails and you will get rat farms. The incentive manufactures the supply it was meant to reduce.

The economy around publishing exploit repositories has precisely that shape. Reputation, visibility and follower counts reward publication, so you get accounts holding ninety-nine CVE repositories with a combined zero stars. Some of those are diligent researchers doing real work. Some are farms — volume with no audience, which is the structural signature.

Our first instinct was the wrong one: treat the farms as a contamination problem to be filtered out.

The Vetinari move is better. A farm is a known, enumerable, reliably productive source sitting in public. One account we surfaced holds a hundred repositories and one total star, including a Python trojan advertising spyware and ransomware capability, a stealth PHP web shell and a PowerShell crypter. That account is a rat farm by any reading.

It is also twenty-two CVEs of free intelligence.

So we stopped trying to search the whole of GitHub for labels, and started enumerating the farms and collecting from them on a schedule. Tax the rat farms. Four accounts went onto the watch list, and watching an account sidesteps all three of our recall failures at once — it does not depend on search ranking, on a CVE's year, or on catching a repository inside a seven-day push window.

The uncomfortable version

We publish a lot about other people's blind spots. This one is ours, and it was open for four months on a working exploit chain.

The thing that makes it publishable rather than embarrassing is that it was found by auditing ourselves rather than by a customer, and that it now has a number attached to it. We would rather tell you our recall was 48% and is being fixed than keep saying "we harvest public exploits" in a tone that implies completeness.

We guarantee five percent of what we publish is wrong. The likeliest five percent here is the 48% itself — it is measured against one author's catalogue, which is a sample of one, and the true figure across the whole corpus could be meaningfully better or worse. We will widen that measurement rather than defend the number. But we would rather publish an honest sample than an unmeasured claim, because the unmeasured claim is what we had this morning and it was the more comfortable of the two.

---

Her name was Renee Nicole Good.

His name was Alex Jeffery Pretti.