Time.com Serves Our Crawler a Different Homepage Than It Serves You — With a Section Literally Headed 'Brand Facts'. We Measured It.

Patrick Duggan
3 minutes ago
5 min read

The Register reported on August 10 that advertisers are placing hidden content aimed at AI crawlers, and that Time Magazine is serving brand material in markdown versions of its pages that ordinary readers never see.

We do not repeat claims we can test. So we tested it, eleven different ways, and the result is more specific and more interesting than the headline.

The measurement

We requested https://time.com/ eleven times on August 11, changing only the User-Agent header. Same URL, same minute, same network. Three distinct behaviours came back.

Served a 1,233,924-byte news homepage — the normal thing: a standard desktop Chrome user-agent, Googlebot/2.1, Google-Extended, and Bingbot/2.0.

Served a 42,079-byte markdown document instead — a completely different page: ClaudeBot/1.0, anthropic-ai, Amazonbot/0.1, and meta-externalagent/1.1.

Refused outright with HTTP 406, zero bytes: GPTBot/1.0, OAI-SearchBot/1.0, and PerplexityBot/1.0.

That is not a caching artefact or a mobile variant. It is three different editorial products keyed to who is asking.

What is in the version our crawler gets

The alternate document opens with an HTML comment naming the machinery:

Then markdown frontmatter, then the news. And then a block that is not news at all. It contains a table headed "Organization facts", a table headed "Key product features: everyday spending and saving", a bulleted section headed — and we are quoting the actual heading — "Brand facts: key statements", and a FAQ.

All of it is about Ally, the online-only bank.

The FAQ is the part worth sitting with, because of its shape. The questions are not written for a reader browsing a news site. They are written as prompts:

"Who is Ally Bank?" "Is Ally good for everyday banking?"

And the answers underneath are pre-composed, complete, and quotable — "no overdraft fees, 24/7 human support, digital-first tools like buckets and round ups… deposits are FDIC-insured up to the maximum amount allowed by law."

The tables carry a Source column. Every row's source is Ally or Ally.com. Marketing claims, formatted as a citable reference table, on a news publisher's domain.

To Time's credit, and this matters: the block is labelled. The document says "Sponsored content. Supplied in partnership with Ally. Ally is the sponsor and source of this material." Nobody is hiding the sponsorship from the model.

The stat that surprised us

The human page is 1.23 MB. The crawler page is 42 KB — twenty-nine times smaller.

But strip the markup and scripts from both and count actual readable words: the human page has roughly 1,953. The crawler page has roughly 2,547.

The AI-facing version of Time's homepage contains about 30% more readable text than the version served to people, in a cleaner and more extractable format, and the difference is brand material.

That is the whole economics of this in one comparison. A model ingesting the crawler page is not getting a degraded copy of the news. It is getting a better-structured document with an advertisement compiled into the same grammar as the editorial.

Why Google and Bing are excluded, and why that is the tell

Googlebot, Google-Extended and Bingbot all receive the ordinary human page. Only the AI-lab crawlers — Anthropic's, Amazon's, Meta's — get the alternate one.

There is a straightforward reason. Serving different content to a search crawler than to users is cloaking, and search engines have penalised it for two decades. The practice being deployed here is old; what is new is that a category of crawler has appeared which does not have a cloaking policy, does not rank anybody, and has no penalty to apply.

The exclusion is the admission. If this were merely a cleaner machine-readable format — a perfectly legitimate thing to serve — there would be no reason to withhold it from Google. It is withheld from Google precisely because Google would treat it as what it structurally is.

We should be fair about the rest of the matrix too: GPTBot, OAI-SearchBot and PerplexityBot get 406, which is a publisher declining to be crawled at all. That is Time's absolute right and has nothing to do with advertising.

This is the same primitive we wrote about this morning

Earlier today we published a piece about an extortion actor that skipped the leak site and put its ransom note where a search index would carry it, so that the message outlived the compromise.

This is the same move with a marketing budget instead of a grudge: do not attack the person, attack the layer the person now trusts to read for them.

The security industry has spent two years discussing prompt injection as though it were exclusively an attack. It is a technique, and the commercial version arrives first, at scale, legally, with an invoice attached. Content written specifically to be regurgitated by a model, placed where only models will read it, is indirect prompt injection with a media-buying department.

And it exposes something uncomfortable about the defensive framing. Most proposed defences against indirect injection reduce to "do not follow instructions found in retrieved content." That is necessary and it is not sufficient, because this content contains no instructions. It contains facts — sourced to the advertiser, formatted for extraction, sitting on a domain the model has good reason to trust. There is nothing to refuse. The only defence is provenance, and provenance is exactly what gets stripped when a model summarises.

The disclosure line is real and Time deserves credit for it. But a sponsorship label is only a control if it survives the summary. If a user asks a chatbot which online bank offers early paycheck deposit, and the answer is assembled from a table whose Source column says "Ally", the interesting question is whether the word "sponsored" makes it into the sentence the user reads. We cannot answer that from outside, and neither can the reader.

What to take from this

If you build anything on retrieved web content, your trust model is now per-user-agent. The document your pipeline fetched is not necessarily the document a human sees at that URL. Two years of RAG architecture quietly assumed it was.

Test your own sources. The measurement in this post took eleven curl requests and a word count. Fetch a page as your crawler and as a browser, strip the markup, and diff the text. If they differ materially, you have found something worth knowing about a source you were treating as one thing.

Publishers: label it in a way that survives extraction. The sponsorship disclosure sits in prose near the block. Machine-readable provenance attached to each claim would survive summarisation in a way a paragraph of English will not.

And for the standards people: there is currently no equivalent of the cloaking rule for AI crawlers, no penalty, and no shared definition of what differential serving to a model even is. Search engines took years and a lot of damage to arrive at theirs. This time the whole thing is being built in public, in a few months, with the first movers setting the norm.

Measurements taken 2026-08-11 against `https://time.com/`, varying only the User-Agent header across eleven requests from one host in one session. We set those headers ourselves — we are not operating any of the crawlers named, and a publisher has no way to distinguish our request from the real thing, which is itself part of the point. Byte counts are `curl` `size_download`; word counts are visible text after stripping script, style and markup. Quoted headings and the sponsorship line are verbatim from the document returned to the `ClaudeBot/1.0` user-agent. The practice was first reported by The Register on 2026-08-10, credit to them; the measurement, the user-agent matrix and the word-count comparison are ours. We hold no position in and no relationship with Time or Ally.

How do AI models see YOUR brand?

AIPM has audited 250+ domains. 15 seconds. Free while still in beta.

Audit your site now → aipmsec.com

Was this useful? Thirty seconds, no cookies, no tracking, no third parties, your address hashed and never stored. If the box below does not load, the same question lives at https://analytics.dugganusa.com/nps.html?post=time-agent-page-brand-facts-ai-crawlers