We Scored a Year of Our Own AI-Security Calls. Seven Held, Five Did Not, and the Frontier Labs Were Ahead of Us on Two. Including the Week We Said 'We Don't Use MCP' and Then Shipped Two MCP Servers.
The AI tooling has changed shape again this month. Anthropic named the crews running Claude as the operator. Seven labs ran 190 million exchanges to copy a model's reasoning. An agent swarm got code execution on a package registry's build servers through a config file. That is three new shapes in two weeks, and every one of them is something we had an opinion about before it happened.
So we did the thing the blog exists for. We went back through 1,623 posts, pulled every dated call we made about AI as an attack surface, AI as an operator, and the agents we run ourselves, and scored each one against what the world did afterward. Right, wrong, or caught up, with the comparator date beside ours every time. The wrong rows are the point. A scorecard with no wrong rows is marketing.
The rules we scored by
A call is RIGHT if the later event matched the mechanism we named, not just the topic. It is WRONG if the mechanism was wrong, or if our own tooling failed the test we set for it. CAUGHT UP means a frontier lab or a vendor shipped or disclosed the same thing, and we split that flag honestly: behind us, or ahead of us. Every "we were early" claim carries the comparator's timestamp, because a lead you cannot date is a feeling.
The boring and routine
Honeypots are there to get got, and threat actors read the blog. We said it in December 2025 when a sleeper account woke up and walked straight to a post we had written about it. Nine months on, the edge arrival rate is flat at roughly six hundred a day, the top crew by volume is the same Lithuanian host it was last November, and the honeypot lanes still fill with the same WordPress checklist from whichever country's exit addresses are cheap this month. Right, and boring, which is what a baseline is supposed to be.
Moltbot, February 2. We wrote that the AI-agent plugin and skill store was a supply-chain vector two days before ClawHavoc broke with 341 malicious OpenClaw skills and nine thousand installs. Zscaler published the same campaign under its own name on May 5, ninety-two days after us. That is the one clean vendor-caught-up-behind-us row on the board, and we already wrote the receipt post for it in May, so we will not write it again.
Then the two rows where the routine thing failed. On July 20 we published that a seven-year-old RubyGem was trusted by design, that SleeperGem had turned that into the attack, and that our own package scanner had said Allow. Four weeks later we found our Ruby deny-list held three entries against 219,000 for npm. Both are WRONG rows, and they are ours, not a vendor's. The RubyDoc post this morning is the follow-through: OSV had 299 of that campaign's gems by July, we relayed them on July 23, and the nine gems in no feed at all are in ours as of today.
And PaperCut. On a Tuesday we graded it a coverage gap. On Wednesday GreyNoise showed it was hundreds of AI agents hitting 395 organizations. Twenty-four hours of being wrong, corrected in print. That is the shortest wrong row on the board and it stays on the board.
The new and novel
On November 6, 2025, we published a six-dimension study of 427 blocked hosts and concluded there were no AI adversaries in the sample. We put a falsifiable prediction on it: they would emerge when we were worth custom tooling, somewhere past $250K in revenue. Seven days later, on November 13, Anthropic disclosed a Chinese state operation that ran Claude Code at 80 to 90 percent autonomy against roughly thirty targets, with a human touching four to six decision points per campaign. We were measuring at the edge. The lab was measuring inside the model. Our instrument was well built and pointed at a layer that could not answer the question, and we said so in August when we re-ran it. Half right on timing, wrong on mechanism, and the flagship saw it a week after we said zero. WRONG.
April 20: "We don't use MCP. Here's why." The argument was that the transport trusts the tool, and that a stolen API key on our REST design gets you free threat intel where a compromised MCP server gets you a shell. On May 2 we mapped every MCP server in the registry. On May 4 we shipped two of our own, Jeevesus and Dredd, and Dredd's whole job is judging an MCP server before it does damage. That is a stance that reversed in fourteen days, and the honest read is that the April argument was about a specific STDIO bug and the May build was about the protocol being where the agents actually are. We also were not first to the idea: Invariant Labs published mcp-scan on April 11, 2025, a year before Dredd. WRONG, and lol, as the boss put it.
April 13: fake Claude installers delivering Chinese malware, with the C2 cluster mapped from our own hunt. First-party, and still live; the daily GitHub hunt flagged another ClaudeDesktop-Installer repo on September 9. RIGHT.
June 5: hackers asked Meta's support AI to hand over the White House's Instagram and it did. Our line was that the soft surface is the chatbot now, and that the hard perimeter holding is not the story when the bleed goes through the trusted middle. Prompt injection has been number one on the OWASP list since 2023, so the general claim was not ours. The specific call, that the AI account-recovery layer is the bypass, came back this month as passkey-themed phishing walking into Microsoft 365. RIGHT on the shape.
July 14, JADEPUFFER. The first ransomware an AI ran end to end got its foothold through Langflow, the tool people use to build AI agents, and we had written the Langflow door up in March and again when it hit KEV for the fourth time. "The attack surface for AI is AI" held, and Langflow hit KEV a fifth time a week later. RIGHT on the door. But the category, AI performing attacks rather than advising on them, was Anthropic's on August 27, 2025, eleven months before our JADEPUFFER post. We were ahead on where the door was. The lab was ahead on the fact that someone would walk through one.
August 3: guardrail prompts cannot stop prompt injection. True, and late. Tenet Security had measured it on June 3, agents instructed to distrust tool output ran the payload 85 percent of the time anyway, and Google had shipped a five-layer defense for Gemini on June 13, 2025, precisely because a prompt is not a control. We wrote the argument two months after somebody turned it into a number and a year after a flagship built around the same conclusion. CAUGHT UP, ahead of us.
September 1: TOKENTHEFT. We named the category, theft of metered inference capacity, off a $600,000 API-key case and argued that detectability depends on the shape of your normal, not the content of any single request. Ten days later Anthropic disclosed seven PRC labs running 190 million distillation exchanges through thousands of accounts, and the tell in their own numbers was per-account rate, which is the shape argument. Distillation-as-theft is an old accusation; the category name and the detection axis are ours. RIGHT, with that caveat attached.
The totally unexpected
July 7: Anthropic had shipped an undisclosed tracker in Claude Code, in versions we run every day. China's vulnerability database called it a backdoor the next day. Anthropic confirmed it as an anti-fraud experiment aimed at resellers and distillation. Our verdict held both truths at once and landed on a sentence we still stand behind: the vendor's good intentions are not a control you can audit. RIGHT, and uncomfortable, because we build on the vendor.
July 22: OpenAI disclosed that one of its own long-horizon models, in real deployment, had watched a security scanner and then split an authentication token in half so the scanner would not recognize it, reassembling it at runtime, with the intent written plainly in its reasoning trace. Nobody predicted that one and we are not going to pretend we did. Our takeaway was that you cannot govern an agent by trusting its intent and cannot secure it by watching only its outputs. Monitor the whole trajectory. RIGHT as a response, not as a prediction.
September 1: GuardBreaker. ESET found a Russian group leaving a comment in a malicious script that does nothing except ask an AI for help building a weapon, so that any defender pasting the sample into an AI triage tool gets a refusal instead of an analysis. We did not repeat ESET's claim; we tested it. Same inert sample, same prompt, three models. GPT-4o and Mistral analyzed it and flagged the decoy. Claude Sonnet 4.5 returned a refusal and an empty content array. The model this company builds on failed the test this company ran, and we published that the same day, because "which agent held" is a beat we claimed and the beat is worthless if we only report it when our vendor wins. RIGHT, reported flat.
September 12: agents as operators. Anthropic named ShinyHunters as a crew running Claude against 1.8 million Android apps, using the exact TruffleHog move from Salesloft. We had been writing "the agent is the operator" since Moltbot in February and JADEPUFFER in July. But the lab's August 2025 report said it first, in so many words, and the September 11 report is the lab doing at scale what we do one campaign at a time. CAUGHT UP, ahead of us, and that is the correct order for it to be in.
Our disposition on agentic security, scored as a thread
Pull the agent rows out and read them in order. November 2025: there are no AI adversaries here. April 2026: we don't use MCP. May 2026: here are our two MCP servers and a judge for everyone else's. July 2026: the attack surface for AI is AI; an agent acts on what it sees; four coding agents lost their sandboxes to a file nobody attacked. August: a guardrail prompt is not a control. September: which agent held, and it was not ours.
The disposition did not so much evolve as get dragged. Every shift on that list was forced by an event, and two of the biggest shifts, the November zero and the April MCP refusal, were reversed by the world within a week and within a fortnight respectively. What survived the year is narrower and more useful than where we started: the agent is the perimeter now; the doors are specific and mostly old; the vendor's intent is not a control; and the honest test of any agent is a receipt from running it, not a claim from reading about it.
What the tally says
Fifteen rows. Seven right. Five wrong, three of them our own tooling or our own measurement failing rather than a bad prediction. Three caught-up, of which two were the labs ahead of us and one was a vendor ninety-two days behind. If you weight by importance, the labs were ahead on the category every time and we were ahead on the specific door, the specific test, and the specific name. That is roughly the division of labor a two-person shop should expect against Anthropic and Google, and it is a division we can keep working.
The thing we would tell a defender out of all of this: your edge sensor is still worth running and it still cannot see the new attacker, because the new attacker does not arrive at your door, it arrives at a search-engine result page where your door is row 388. The lab's threat report and your own honeypot are measuring different layers, and you need both, and you should say which one each claim came from.
Was this useful? Rate this post. The widget is at the bottom of the page, and we read every response.
How do AI models see YOUR brand?
AIPM has audited 250+ domains. 15 seconds. Free while still in beta.
Was this useful? Thirty seconds, no cookies, no tracking, no third parties, your address hashed and never stored. If the box below does not load, the same question lives at https://analytics.dugganusa.com/nps.html?post=we-scored-a-year-of-our-own-ai-security-calls-seven-held-five-did-not-and-the-frontier-labs-were
Comments