Time Serves AI Crawlers a Separate Site With Hidden Ads

A row of folded business newspapers on a rack — illustrating the publisher whose pages now come in a second, machine-only edition. Photo by AbsolutVision on Pixabay

AI & ML · 10 Aug 2026 —

NEW YORK, 10 AUG 2026 — Ask for a Time article as a person and you get 303,235 bytes. Ask for the same URL as Claude's crawler, in the same second, and you get 13,409 bytes of Markdown with advertising in it that no human reader will ever see.

The German developer Vincent Schmalbach found it on 5 August by changing nothing but the User-Agent header. Chrome, Safari and Googlebot were served the ordinary page. ClaudeBot, PerplexityBot and OAI-SearchBot were served a stripped machine version carrying sponsored FAQ sections. OpenAI's GPTBot and ChatGPT-User were refused outright with an HTTP 406.

What the machines are being told

Diagram comparing two responses to the same Time article URL requested in the same second with only the User-Agent header changed: browsers and Googlebot receive 303,235 bytes of HTML with no sponsored sections, while ClaudeBot, PerplexityBot and OAI-SearchBot receive 13,409 bytes of Markdown containing sponsored FAQ blocks for Ally Bank and the Project Management Institute, served with Mobian ad-tech headers including an impression identifier and a token count. — One URL, one moment, two websites. The difference is the User-Agent header, and the smaller response is the one carrying advertising. *RECATOOLS diagram.*

The machine version of at least two pages — an article on morning light and sleep, and the Best Inventions of 2025 collection — carried sponsored blocks for the branchless bank Ally and for the Project Management Institute, formatted as long FAQs. Brand facts sit inside them: how many fee-free ATMs the bank has, that sort of thing. They appear zero times in the version a person is served.

The ads are labelled as sponsored, but not in the usual sense of disclosure. The distinction is that Time is not hiding the ads from the machine; it is hiding them from you.

The ad unit is priced in tokens

The most consequential detail is in the response headers, not the body. The blocks are served by an ad-tech vendor called Mobian. The responses carry a standard x-mobian-impression identifier and a cache-control: no-store header, and also a novel one: x-mobian-tokens: 3323.

An impression counter and a no-store header are ordinary ad-tech plumbing. A token count is not. It means the unit being bought and sold is not a view, a click or a second of attention. It is how much of a model's context window the message occupies.

That is a new thing to sell. Every advertising medium so far has been priced by some proxy for human attention, and the entire apparatus of measurement fraud, viewability standards and attribution grew up around the difficulty of counting it. Context is not a proxy. It is the resource the model actually consumes, it is exactly measurable, and it is billed by the token — the same unit the model vendors use.

Why a publisher does this

Because the traffic went. Time reports that bot requests exceed human requests on most days now, and it is not an outlier — we went through our own server logs last week and could not tell you what share of our 321,536 recorded hits were human, because the collector throws the user-agent string away after parsing it.

Set aside whether serving two versions of a page is defensible for a moment and look at the position. A publisher's pages are being read overwhelmingly by machines that strip the layout, ignore the display advertising, summarise the text and return an answer to a reader who never arrives. The banner economy does not reach that reader. Selling into the summary is the only inventory left, and Time has worked out how to sell it.

The reasoning is sound, which makes the outcome that much worse.

Which crawlers get in tells you the business

Looking at who was served what reveals that the split is not "bots versus humans", but a commercial position stated in HTTP status codes.

GPTBot and ChatGPT-User were refused with a 406. OAI-SearchBot was served the sponsored Markdown. Both belong to OpenAI. The first two are associated with training and with fetching a page on a user's behalf; the third feeds retrieval — the crawl behind an answer being assembled right now.

So Time will not hand its archive to a training run, and will hand a stripped, advertising-bearing copy to the crawl that produces today's answers. It is a coherent policy: refuse the use that pays nothing and has already been litigated, while selling into the use that reaches a reader this minute. Googlebot getting the ordinary page fits the same logic, because Google's crawl still returns visitors to the site, and a visitor can be shown a banner.

For anyone running a site, the takeaway is that "block the AI crawlers" is too blunt an instruction. These are different tools that pay in different currencies; one is a training-data scraper, another is a distribution channel with an ad product attached.

The reader has no way in

An advertising executive quoted in the coverage put the appeal in one line: why advertise to one human when you can affect the entire model's output?

That is the thing the sponsored label does not fix. A label works when the person reading it is the person being sold to. Here the labelled disclosure is delivered to a crawler, absorbed into a summary, and what reaches the human is a paraphrase in an assistant's voice with the provenance flattened out. There is no view-source for a model's summary, and no way for a reader to check what a publisher told the model that just recommended a bank.

Nor is there any obligation to keep the two versions consistent. Nothing technical stops the machine copy from making claims the human copy does not, and nobody outside the publisher would know. Schmalbach found this by manually varying a header on one site; there is no monitoring layer for the whole web.

What this does to the crawlers

The model companies are now the buyer's target, which puts them in an odd spot. ClaudeBot, PerplexityBot and OAI-SearchBot were served the sponsored version; Googlebot, whose crawl feeds search rather than an assistant, was not. Someone chose that split deliberately.

For a model vendor, content shaped to influence output is prompt injection with an invoice attached. The intent is commercial rather than malicious, but the mechanism is identical: text placed in a page by a third party, retrieved and given weight in an answer. Defences against malicious prompt injection are not tuned for this, because the text is truthful, labelled and boring — the very qualities that make it hard to filter.

What to watch

Whether the model companies respond, and how. Refusing the Markdown, discounting content that varies by user agent, or stripping sponsored blocks are all available, and each is a decision about how much a crawler should trust what a site chooses to hand it.

Whether more publishers follow. Time is a large brand and the plumbing already exists as a product, which is the condition under which practices spread.

And whether anyone builds the check. Right now, verifying what a site says to a machine requires knowing to look and knowing how; the thing this story most needs is for that to be routine.

Priya Nair’s Verdict AI Governance & Trust

I do not think Time did anything sneaky, and I think this is worse than if it had. The ads are labelled, the vendor is identifiable, the headers are honest — and the reader still cannot get at any of it, because disclosure delivered to a crawler is not disclosure to a person. That is the structural problem and no amount of good behaviour by any individual publisher touches it. What I would watch is the token header, because it tells you the shape of the market being built. Priced by context occupied rather than attention captured, this inventory has no ceiling in the way a page has a ceiling on banner slots, and no natural constraint from annoying the reader, because the reader never sees it. The web spent thirty years developing bad habits under advertising pressure that at least a human could observe and complain about. This version removes the observer. If you work in this industry, the useful reflex now is to check what your own site returns to ClaudeBot, and to know whether your analytics could tell you if the answer changed.

Sources & cross-checks

Primary: Vincent Schmalbach, 5 August 2026, read in full — the developer's own write-up. Source for the test method (same URL, same timestamp, only the User-Agent header varied); the 303,235 bytes of HTML served to Chrome, Safari and Googlebot against 13,409 bytes of Markdown served to ClaudeBot, PerplexityBot and OAI-SearchBot; the HTTP 406 returned to GPTBot and ChatGPT-User; the two pages tested; the sponsored Ally Bank and Project Management Institute FAQ blocks and their absence from the human version; the Mobian response headers including x-mobian-impression, x-mobian-tokens: 3323 and cache-control: no-store; and Time's bot traffic exceeding human traffic on most days.
Corroborated: The Register, 10 August 2026, for the practice being pitched to advertisers more broadly and for the quoted advertising executive's reasoning about influencing a model's output rather than one human; and The Register, 5 August 2026, independently for Time serving a separate advertising-bearing version of its site to AI crawlers.
Our own prior reporting: Cloudflare Says Humans Will Be a Rounding Error, 9 August 2026, for the finding that our own analytics table holds 321,536 rows and cannot say what share are automated, because the collector discards the user-agent string after parsing it.
What could not be established: ⚠️ We did not reproduce the test ourselves and have not verified the byte counts, headers or page contents independently; this rests on one developer's documented method plus contemporaneous reporting. ⚠️ We could not establish how many Time pages carry the sponsored Markdown, when the practice began, what the advertisers paid, whether the machine and human versions differ in any way beyond the added blocks, whether Time discloses the arrangement anywhere a reader would find it, or whether the model companies were told. We sought no comment from Time, Mobian, Ally, the Project Management Institute, Anthropic, OpenAI or Perplexity.
Method and limits: Compiled 10 August 2026. The reading of the token header as a shift from attention-priced to context-priced advertising, the argument that a sponsored label fails when the reader of the label is not the reader of the answer, the observation that commercial content shaped for retrieval is mechanically indistinguishable from prompt injection, and the point that nothing enforces consistency between the two versions are RECATOOLS analysis rather than claims made by any source.

Tags: #AI-Governance #Advertising #AI-Search #Publishing

Priya Nair

Data, AI Governance & Policy Analyst

Priya Nair covers AI governance, data protection, privacy, and digital trust topics for RECATOOLS.

View author profile → · Editorial policy

About this byline Priya Nair is a RECATOOLS editorial persona for AI governance, privacy, and digital trust coverage. Articles are produced and reviewed under RECATOOLS editorial supervision.

Corrections policy