Somebody Crawled Our Entire Blog in One Evening. Our Own Tracking Beacon Turned It Into 209,066 Requests. Here Is Everything We Got Wrong Working It Out.

CORRECTION, 22 August 2026 — the central claim of this post is wrong. Re-queried with uncapped result sets and fifteen-minute buckets, the traffic surge begins at 22:45 UTC — seven minutes before the second post published and sixteen before the third. We did not do this to ourselves. A full-corpus harvest was already running when we published into it, and the real root cause is chronic: cache-miss requests to this blog have been returning HTTP 504 between 28% and 82.5% of the time every day, at any volume, with the event day sitting inside that ordinary band. The corrected account is published in full as That Ain't Lake Minnetonka. The original text is left standing below rather than rewritten, because the sequence of wrong readings is the useful part.

On the evening of 21 August our traffic went to 209,066 requests against a baseline of about eighteen thousand a day. Cloudflare logged 195,799 in a single hour. Our origin scaled to its ceiling, a deployment failed to find a slot, and four Google Cloud addresses in Frankfurt sat at the top of the source list.

It looked exactly like being attacked. Then it looked exactly like we had done it to ourselves. The truth turned out to be both, and it took five corrections to get there — every one of them because a number we trusted was measuring something other than what we thought.

Somebody fetched 1,579 distinct posts that evening. We have about 1,641. So yes, we were crawled, comprehensively. But the crawl alone does not explain 209,066 requests, and the thing that turned a corpus crawl into a ten-times-normal traffic event was our own analytics beacon, waved through a hole in our own firewall, uncached, straight to our own origin.

This is the whole incident: the mechanism, the three unrelated defects we found while looking, and the five separate times during the investigation that we read our own equipment as an adversary. That last part is the useful part, and it is why this post exists at all.

The loop

A new post goes live. Something re-renders it with a real headless browser. Because the blog runs on a platform with a chatty client, each render fires roughly eighteen platform API calls — access tokens, tag manager, blog settings, permissions, laboratory flags, comment translations. The backend runs out of headroom. It returns 504. The renderer retries. The retry costs another eighteen calls.

The timeouts cause the volume. That is the whole mechanism, and it is self-feeding rather than malicious.

Correction, and it is a big one. The first version of this post said only 214 distinct posts were fetched, and concluded "nobody harvesting a corpus behaves like that." That was wrong, and wrong for an embarrassing reason: the query behind it stopped at the top 250 result groups, so I read a truncated top-N as the whole population. Re-run without the ceiling, the real figures are 1,579 distinct posts and 45,001 post requests — against a blog of roughly 1,641 posts. They fetched essentially the entire corpus, and exactly one post was fetched two times or fewer.

So it was both. A full-corpus crawl with a retry storm riding on top, concentrated where the failures were worst. The three newest posts took 35% of all post traffic despite being 0.2% of the posts, and the mean across everything was 28.5 fetches per URL — far too many for a clean harvest. The loop is real. The harvest I dismissed was real too, and it was the larger half.

Better still, we accidentally ran the experiment. We published the same article twice that evening, in English and in Italian, two hours apart. The Italian one — newest at the time the storm peaked — took 5,539 requests at a 33% failure rate. The English one, published earlier and already settled, took 514 requests at 7%.

Same content. Same server. Ten times the load and five times the failure rate, purely as a function of recency.

What the forensics turned up on the way

We captured Cloudflare's firewall events while the event was still running, because that data has short retention and we did not want to be reconstructing it on Monday. Three findings fell out that had nothing to do with the storm.

Our API host had no managed firewall. Cloudflare's managed ruleset blocked 11,291 of 11,329 events on the blog. On our API host it blocked ten out of 16,097. Everything else was skipped by a custom rule of ours whose match condition was a single wildcard: /api/*.

That rule was written for a good reason. A managed ruleset will happily maul a real API — SIEM clients with odd user agents, TAXII pulls, JSON-RPC. But /api/* is every path under the API, on every host, which meant our admin routes and our key-issuance endpoints had been running with the WAF switched off. Not because anyone decided that; because a rule that was correct when written stayed in place while the thing behind it grew.

Our own analytics beacon was the amplifier. 93% of what that rule waved through was two paths: blog-tracking/track at 7,589 requests and blog-tracking/embed.js at 7,559. The chain is short and unlovely — a scraper renders a blog page, the page loads our beacon from the API host, the beacon fires a hit back at the API host, the custom rule ensures neither is ever blocked, and neither is cached. Every scraped page cost us two uncached origin requests and one inflated page-view number.

The edge was absorbing 28% of it. Cache misses, bypasses and dynamic responses came to roughly 259,000 requests against 102,000 served from cache. During a two-hundred-times-normal event, three quarters of it was our servers' problem rather than Cloudflare's — which is why we hit the replica ceiling, and why a deployment sat in a queue for forty minutes behind a fight our own blocklist was already winning.

Now the lumps

Four times during this, we looked at our own equipment and saw an attacker. This is the part worth writing down.

One. We claimed four nines while serving 504s. Throughout the incident we reported that production was unaffected. That was based on curling one health endpoint several times and getting 200 back. The actual number for our API host across the window: 50,272 requests, 5,841 of them HTTP 504, an 88.373% success rate. Not four nines. Not two. A handful of green probes on a healthy path is not a measurement of system health, and we said otherwise out loud, repeatedly, while the data to contradict us sat in an API we were already querying.

Two. We nearly published our own traffic as an audience. Earlier the same day we had looked at 81,590 server-side page views against 612 in Google Analytics and described the gap as a hidden, JavaScript-blind readership roughly a hundred times larger than the measured one. It is not an audience. It is the difference between requests and people, and the bot classification separating them works on user-agent strings — which the headless Chrome in this very incident would sail straight past into the "human" column. The honest label was never "audience." It was "unknown."

Three. Our own addresses were in the incident data. Three IPv6 addresses from the investigator's own network appeared in the firewall events — 297 of them, from API work done during the session that produced the investigation. The prefix rotates, so excluding a single address would not have caught it; the /64 is the unit. It was noticed because a human recognised his own IP, not because anything flagged it. Any forensic process that does not exclude the investigator's own prefix will eventually indict the investigator.

Four. The scary-looking German host was a health check. A Hetzner address with ninety events, sitting in the middle of an apparent attack, turned out to be mcpbeat/0.1 — a liveness monitor that identifies itself in its user agent and publishes an opt-out URL. It was checking whether our MCP server was up. We had it queued for enrichment as adversary infrastructure.

And a fifth, which is a different flavour: the genuinely suspicious host in that dataset was one nobody noticed. A single address presenting twenty different mobile user-agent strings — iPhones, Redmis, Samsungs, okhttp — patiently rotating its fingerprint against our API. 432 events. Our burst detector missed it entirely, because it was quiet, and quiet is not the shape that detector looks for. The loud thing was innocent and the quiet thing was not.

What our own kit got right

It is not all lumps. The behavioural detector saw a single-ASN burst shape and listed all four Google Cloud addresses in the same second, at confidence 90, without anyone asking. Anything above our confidence threshold flows automatically into the published blocklist, and all four were confirmed there within the hour. Production never went down. The autoscaler did its job.

The detection worked. The scoping did not. Those are different failures and it is worth being precise about which one you had.

What it would have cost a real SOC

We are two people. The interesting question is what this same incident consumes in an organisation that staffs it properly.

Roughly 55 to 60 person-hours — about 1.5 FTE-weeks of skilled security labour, spread across seven or eight people who each need context handed to them. At a loaded blended rate of $85/hour that is $4,700 to $5,100 for one event, before you count the calendar days lost while a ticket queues between the WAF engineer and the SRE.

Ours took about three hours, and the honest asterisk matters: that is the marginal cost of responding, not the cost of being able to respond. The detector, the confidence pipeline and the blocklist plumbing were months of prior work. This incident spent that capital rather than creating it.

What automation genuinely removed is the part that repeats — triage, enrichment, listing, and the context-transfer tax between seven people. What it did not remove was the thinking. Four times that evening the numbers looked conclusive and were wrong, and each time it took a person and a machine arguing to notice.

Naming, and why we called it something stupid

The four Google Cloud hosts were designated BOATYMCFUCKFACE while unattributed, and that designation is now retired — the cluster resolved to a platform retry loop, not an actor. Keeping a threat name on a solved non-event is how phantom adversaries get into corpora and never leave. Somebody quotes it in a report eighteen months later and it has a life of its own.

The name moved to the thing that actually deserves scrutiny. COSTUMETRUNK: a single host at a Hong Kong cloud provider presenting twenty different mobile user agents — iPhones, Redmis, Samsungs, an okhttp client — patiently rotating its fingerprint against our API. 432 events, no burst, no noise.

Third-party correlation on it: AbuseIPDB reports zero. VirusTotal, zero malicious, reputation zero. Our own corpus, nothing.

That is not a clean bill of health. Unreported is not clean; it is first observation. And it is precisely the case our loud burst detector cannot see, because it was built to notice volume and this thing is quiet on purpose.

We are tracking it and deliberately not publishing it as an indicator. One address, 432 events, zero corroboration is a hunch, and shipping a hunch into a blocklist that other people consume is the laundering we spend most of our time complaining about.

The operating guide, as far as we have one

If this happens to you:

Freeze the evidence first. Firewall-event retention is short and you will want it after the adrenaline. We had raw captures on disk within eight minutes and every number in this post came from those files rather than from memory.

Check whether it is you. Correlate the spike against your own publish log, deploy log, and cron schedule before reaching for attribution. Ours correlated perfectly with three publish timestamps.

Look at the three-column signature. Requests up 10×, page views up 5×, unique visitors flat — 3,207 against 3,236 the day before. A real audience surge moves uniques. Volume without uniques is few hosts doing a lot, every time.

Count what they actually took. 214 distinct posts across 23,710 requests is a loop. Thousands of distinct URLs fetched once each is a harvest. Those need different responses.

Exclude yourself from your own data, by network prefix rather than by address.

Do not call it an attack until the evidence says attack. We named this cluster boatymcfuckface precisely so nobody could accidentally take it seriously before attribution was earned. A frightening name propagates into every downstream conversation and quietly does your thinking for you.

And measure your uptime from the log, not from a curl.

Watch what nobody reports. The four loud hosts were innocent. The one host worth a second look had zero reports anywhere on earth, and was invisible to a detector tuned for bursts. If your entire detection posture keys on volume, you have built something that finds the noisy and misses the patient.

What we changed

The beacon is now cached at the edge and rate-limited per address, so it can never again amplify somebody else's traffic into our origin. The firewall rule now carves out admin, key issuance and the beacon paths, so those get their managed ruleset back — for the first time in months. The four source addresses are in the published blocklist. The replica ceiling was raised so a deployment cannot be starved by a load spike again.

The one honest caveat: the firewall change was verified against nineteen events, because by then the storm had ended. That is not a verification, and saying so is the entire point of this post.

The shape

Every instrument eventually measures its owner. The analytics counted our own beacon. The bot filter classified headless Chrome as a person. The firewall logged the investigator. The health check reported green while one request in nine timed out.

None of those are exotic failures. They are all the same failure, which is that a measurement taken from inside the system includes the system, and the flattering reading is always available and never announces itself as the flattering one.

We spent the day building detectors that refuse to score an unmeasurable thing as zero. Then we read 191,000 requests from a machine having a bad time and very nearly called it growth.

---

Was this useful, or did we miss something? Rate this post below — we read every one, and the low scores are the ones that change what we build.

---

How do AI models see YOUR brand?

AIPM has audited 250+ domains. 15 seconds. Free while still in beta.

Audit your site now → aipmsec.com