That Ain't Lake Minnetonka: A One-Man SOC, 462,545 Requests, and a Bug That Had Been There All Week

Friday evening in Minnesota. It is gorgeous out — the kind of August light that makes you forgive the state for February. There is a MacBook Air on a patio table on the banks of Lake Minnetonka, and at 22:45 UTC it started making noise.

Four hundred and sixty-two thousand requests in seventy-five minutes. A hundred and seventy-four times normal. Gateway timeouts by the tens of thousands. Frankfurt.

What follows is the honest reconstruction, including the part where the conspiracy was better than the truth, right up until it wasn't.

First, the theory, because the theory was magnificent

Here is what it looked like from the patio.

We had spent the week publishing research about AI-generated attack tooling and adversary behaviour. We had named things. We had annoyed, by our own estimate, at least three different categories of people who own server capacity. And then, on a Friday night, four machines in Frankfurt — Google Cloud, europe-west3 — began walking our entire archive with headless Chrome while we were mid-sprint.

Retaliation. Obviously. Somebody read what we wrote about them and decided to take the corpus.

And if you want the full-fat version: Prince was from here. Prince was, by any reasonable measure, too cool to be allowed to continue, and the Illuminati handled it in 2016, and now they have come for a threat-intelligence blog nine miles from the lake he made famous. The waters of Lake Minnetonka are right there. The symmetry writes itself.

I want to be clear that we enjoyed this theory for approximately forty minutes and that it was, in every particular, wrong.

The data killed it in four moves

Move one: the timing. We published three posts that evening — 20:02, 22:52, and 23:01 UTC. The traffic step happens in the 22:45 bucket. That is seven minutes before the second post existed and sixteen before the third. Whatever this was, it was already running when we published into it. The sprint did not summon it.

And the sprint was never a plausible cause anyway. We publish eight at a clip on an ordinary Tuesday. If three posts in seventy minutes could knock over a blog, we would have discovered that in March.

Move two: the shape. 1,579 distinct posts were fetched, out of a corpus of about 1,641. That is 96.2% — all but sixty-two. Nothing produces that except somebody deliberately walking the sitemap. But look closer at the distribution and the second mechanism falls out: the median post was fetched fifteen times, and not one single post in the entire corpus was fetched exactly once. A harvester takes a page once. Fifteen times each is retry.

So it was two things stacked: a full-corpus harvest, and a retry loop riding on top of it.

Move three: who was actually failing. Here is the fact that unravelled the whole thing. Every single one of the 504s landed on a cache miss. Zero on a hit. Zero on bypass, dynamic, or revalidated. Eighty-two thousand cached requests sailed through the event without a scratch — which is precisely why, the entire time this was happening, I could sit on the patio and read my own blog perfectly well. I was reading warm pages. The harvester was requesting cold ones.

Move four, the one that ended the conspiracy and started a much worse conversation. If cold reads were failing at 78% under the flood, what do cold reads do on a normal day?

They fail between 28% and 82.5% of the time. Every day. At any volume. The mean across the week is 52.7%. The event day, at 77.7%, sits inside the ordinary range. The single worst day of the week was 18 August — a nothing Tuesday, no flood, no sprint, no Germans — which quietly produced 14,109 gateway timeouts that not one person observed.

There was no outage. There has been a chronic, unmeasured, half-the-time failure to serve our own archive, and a stranger's badly-written scraper is the only reason we know.

So what actually happened

The blog cannot reliably serve cold pages. That is the root cause, it is not new, and it is not load-dependent.

A full-corpus harvest is, by construction, almost pure cold traffic. Nobody has recently read post number nine hundred. Walking the sitemap means requesting 1,579 pages that are all guaranteed cache misses, which means sampling that chronic defect 163,953 times in one evening.

A 504 never populates a cache, and the harvester had no backoff. Miss, timeout, retry, miss, timeout, retry. The retry could never be served warm, because failure is not cacheable. That is a closed loop with no exit — and it is the exact same shape as the bug we published about earlier that same week, where our own feed required an API key to discover that it required an API key. We wrote the essay on closed loops and then stood inside one.

The flood was not the disease. The flood was a very expensive, very rude diagnostic.

The part where the one-man SOC earns it

I am not going to be modest about this next bit, because the machine did exactly what it was built to do and it did it while I was outside.

At 23:13:09 — twenty-eight minutes after onset, with no human in the loop — our behavioural burst detector identified all four source addresses and listed every one of them in the same second, at confidence 90, tagged as scraper. Not a signature. Not a threat feed lookup. Those four addresses were on nobody's blocklist; the ASN is Google's, the user agents were real Chrome. It caught them on shape — the burst geometry of a single network doing something no visitor does. Confidence at or above 80 flows automatically to our published blocklist, so within the hour those four addresses were in the free feed that anyone on earth can pull.

Then the human part. Somewhere around 23:30 I went into the edge shield and hand-wrote the rules to back it off. The traffic went 134,761, then 44,017, then 482. Firewall blocks fell along with it, which is how you know they stopped arriving rather than stopped succeeding. By midnight we were below the evening's baseline. Rev 338 deployed at 23:47:49 with the replica ceiling raised from two to six — which removed the timeouts, and therefore removed the amplifier, and was insurance rather than the kill.

Onset to baseline: about seventy-five minutes. Detection: automated, twenty-eight minutes, nobody watching. Mitigation: one person, on a patio, on a laptop, with a lake behind him.

A conventional shop books this event at roughly fifty-five to sixty person-hours across seven or eight roles — tier one triage, tier two log pull, an IR lead on root cause, threat intel on attribution, a network engineer on the WAF, an SRE on the scale ceiling, detection engineering, and a manager writing it up. Call it a person-week and a half of skilled labour, four or five thousand dollars, and several days of calendar time while the ticket queues between people who each need the context explained to them again.

That is the honest comparison and it comes with an honest asterisk: three hours is the marginal cost of responding, not the cost of being able to respond. The detector and the pipeline were months of prior work. This incident spent that capital; it did not create it.

Five things we got wrong, in order

This is the part I actually want on the record, because everybody publishes the incident where they were right.

One. We announced "four nines, nothing but net" while it was happening, on the strength of curling a single healthy endpoint and getting a 200. The real number on our API host was 88.373%. A green probe on a warm path is not a measurement of anything.

Two. We nearly wrote up our own IP addresses as adversary infrastructure. Three addresses from a rotating AT&T prefix, 297 events, sitting in the firewall log looking exactly like a participant — because they were the investigation itself. Caught only because a human recognised his own address. Any forensic process that does not exclude the investigator's own prefix will eventually indict the investigator.

Three. The first query we ran was capped at the top 250 result groups and we read the truncation as a finding. It said three posts and twenty-three thousand requests, so we concluded nobody was harvesting a corpus. Uncapped, it was 1,579 posts and forty-five thousand requests, and somebody was very much harvesting a corpus. We had drawn the opposite conclusion from the same query, confidently, in public.

Four. We then published a piece titled, in effect, we DDoS'd ourselves — and that was wrong too. The onset predates two of the three publishes. We did not do this to ourselves. Somebody did it to us, and separately we had a chronic defect that made it forty times worse than it needed to be.

Five. We conflated two unrelated clusters into one villain. The Frankfurt four are one thing. A Hong Kong host rotating twenty user agents in the same window is a different thing, and merging them made a tidier story than the evidence supported.

The one part that genuinely was our fault

The blog is hosted by a platform. Our analytics API is ours. Those are two separate failure domains and there is no reason on earth for one to bleed into the other — except that we hang a tracking beacon off every blog page, so every scraped post fired two uncached requests at our own API host. Fifteen thousand of them, waved straight through by a firewall rule we had scoped to an entire hostname instead of to the two paths that needed it.

We welded somebody else's cold-read problem onto our own front door and then acted surprised when it rang.

Fixed now: the beacon script is served from the edge cache, the tracking endpoint is rate-limited per address per minute, and the firewall exception has been narrowed to the paths that actually require it.

The cluster keeps its stupid name

We designated this group BOATYMCFUCKFACE on the night, and after reviewing everything I am keeping it, deliberately.

The temptation once it turned out to be real is to re-christen it into something with fangs. That would be exactly the error the naming rule exists to prevent — the evidence did not change, only our reading of it. And the evidence is unglamorous: rented cloud VMs, off-the-shelf headless Chrome, a sitemap walk, and zero backoff through a 47% error rate. No probe at an admin path. No credential surface touched. No attempt at the API. Nothing was attacked. They wanted the words.

That is a badly-engineered bulk corpus harvester — training data collection or blog aggregation — and a crew that hammers straight through half its own requests failing is not disciplined. Discipline looks like exponential backoff.

A frightening name gets you frightened analysis, and everything downstream inherits the fear before anybody checks. Keeping this one ridiculous forces every future claim about them to stand on evidence rather than on menace the name loaned it for free.

That ain't Lake Minnetonka

In Purple Rain, the Kid tells Apollonia she has to purify herself in the waters of Lake Minnetonka. She strips, she jumps in, she comes up freezing, and he tells her: that ain't Lake Minnetonka.

The ritual was invented. The plunge was real. She got in the water either way.

That is the whole shape of this thing. The Germans were not retaliating. The Illuminati did not come for the blog. Nobody was punishing us for our opinions about AI-assisted attackers, and Prince, so far as the firewall logs are concerned, remains uninvolved. The conspiracy was ours, we made it up on a patio, and we enjoyed it enormously.

But we went into the water. And at the bottom of it, where the good story was supposed to be, we found something considerably more useful than a nemesis: an archive we have been telling the world is our institutional memory, which fails to serve its own cold pages about half the time, and had been doing so for at least a week without a single alarm.

Every post we have written about instruments that measure their owner instead of the world — the feedback widget counting our own test entries, the badge scoring our own commits, the feed statistics counting blocked scrapers as consumers — was a rehearsal for this one. This time the instrument was the absence of one. Nobody was watching the cold path, so the cold path was free to be broken, and it took a stranger's rude scraper to file the bug report.

The heroic version is true, incidentally. Four hostile-shaped hosts identified unattended in the same second by behaviour rather than reputation. One person, one laptop, seventy-five minutes, on a patio, in the best weather Minnesota gets. Five separate corrections published rather than buried.

We just did not get to be heroes for the reason we wanted. We got to be heroes for reading our own numbers until they stopped flattering us.

Which, honestly, is the better badge. It is just a much worse Friday night.

---

Was this useful, or did we miss something? Rate this post below — we read every one, and the low scores are the ones that change what we build.

---

How do AI models see YOUR brand?

AIPM has audited 250+ domains. 15 seconds. Free while still in beta.

Audit your site now → aipmsec.com

---

Was this useful? Thirty seconds, no cookies, no tracking, no third parties, your address hashed and never stored. If the box below does not load, the same question lives at https://analytics.dugganusa.com/nps.html?post=that-ain-t-lake-minnetonka-a-one-man-soc-462-545-requests-and-a-bug-that-had-been-there-all-week