Who Actually Blocks the AI Crawlers

Every publisher has an opinion about AI companies training on their journalism. Most of those opinions live in editorials and legal filings. There is one place where the opinion has to be written down in a form a machine can act on, and it is public: robots.txt.

We read that file for nineteen news publishers — eleven across ASEAN, eight global outlets for comparison — and checked it against twelve named AI crawlers. Publishers fall into two camps, with almost nothing in between.

The decision is binary

Of the nineteen publishers, six turn away eleven or twelve of the twelve agents. Nine turn away none or one. Only four sit anywhere in the middle, and all four of those block exactly eight.

No publisher in the sample blocks three agents, or five. Once a publisher decides to act, they act broadly — which suggests this is a policy decision taken once at an institutional level, not a per-vendor negotiation.

The regional split is the finding

ASEAN publishers block a mean of 3.9 of the twelve agents. The global outlets block 7.4 — very nearly double.

Underneath that average, six of the eleven ASEAN publishers block nothing at all, against two of the eight global ones. And the three largest English-language outlets in Singapore — The Straits Times, CNA and The Business Times — are all in the zero column.

We are not going to tell you that is a mistake. A publisher that wants its journalism quoted by an assistant, or that has a licensing deal, or that has decided the traffic is worth more than the training data, is making a defensible call. The narrower point is that on the one public record where the position has to be explicit, most of the region has not taken one.

Some of the zeroes are deliberate, and you can tell which

There is a difference between a robots.txt that has never heard of GPTBot and one that names it and lets it through. We track this distinction because it gets to the heart of the matter—whether a zero is a deliberate choice or a sign of inaction.

businesstimes.com.sg names seven AI agents and blocks none of them. Someone sat down, listed the crawlers, and chose to allow every one. Reuters names two and blocks neither; the Wall Street Journal names three and blocks none.

Those are positions. The other zeroes — straitstimes.com, channelnewsasia.com, thestar.com.my, thejakartapost.com, bangkokpost.com — name no AI agent at all. Their files predate the question.

Training and search are being separated

The most interesting pattern in the per-agent numbers is that publishers do not treat one company as one thing.

OpenAI's training crawler is blocked by eight publishers. Its search crawler and its user-initiated fetcher are blocked by six. Four publishers explicitly name the search agent and allow it while turning something else away.

That is a coherent position, and it is the one most publishers seem to be feeling their way toward: do not train on us, but do send us readers. Whether the distinction survives contact with how these systems actually work is a separate argument.

One other number runs against expectation. Anthropic's crawler is blocked more often than OpenAI's, by ten publishers against eight. We would not read much into a two-publisher gap in a sample of nineteen, and we mention it mainly because the assumption usually runs the other way.

What robots.txt is, and what it is not

robots.txt is a published request. It has no enforcement behind it. A crawler that ignores it is being impolite, not breaking in — the file is a sign on the lawn, not a lock on the door. Nothing we measured tells you whether any crawler obeyed any of these instructions, and we make no claim that they did.

What the file is good for is exactly what we used it for: it is the publisher's stated position, in public, in a form nobody can later claim to have misunderstood.

If you run a site, read your own file properly

The obvious way to check whether you block GPTBot is to search your robots.txt for the string. That test is wrong, and it fails in both directions.

robots.txt is a grouped format. A User-agent line opens a group, several consecutive User-agent lines share one group, and the Disallow and Allow lines that follow apply to all of them. So the string "GPTBot" appears in your file whether you are blocking it, explicitly allowing it, or mentioning it in a comment. Our harness parses the groups and asks whether this agent's own group disallows the site root, because a substring is not a policy.

The second trap is the wildcard. A User-agent: * group with Disallow: / does not reliably stop AI crawlers, because most of them look for their own name first and follow the most specific group that matches. If you want a named crawler out, name it.

Our AI crawler robots.txt builder generates the groups for the current set of agents, and our llms.txt generator covers the newer convention for stating terms to AI systems directly.

If you are weighing the other side of this — what an AI system can and cannot be trusted to have learned honestly — what you are trusting when you download a model measures how often that provenance is stated at all.

The limits of nineteen publishers

This is a sample. It is a deliberately chosen one — the largest English-language general news outlets we could identify per market, plus a global comparison set — but nineteen is nineteen, and the regional means rest on eleven sites and eight.

We would defend the two-camps pattern hardest, because it depends on the sample size less than the means do. Six publishers at eleven-or-twelve and nine at zero-or-one is a shape, and one that stark rarely comes from noise.

We also read only the root robots.txt at one moment in time. Publishers change these files, sometimes in response to a negotiation that concluded that week. Figures are from 19 August 2026, and the script is in our repository if you want to see what has moved since.