Every publisher has an opinion about AI companies training on their journalism. Most of those opinions live in editorials and legal filings. There is one place where the opinion has to be written down in a form a machine can act on, and it is public: robots.txt.
We read that file for sixteen news publishers — eight across ASEAN, eight global outlets for comparison — and checked it against twelve named AI crawlers. Publishers fall into two camps, with almost nothing in between.
The decision is binary
Of the sixteen publishers, six turn away eleven or twelve of the twelve agents. Six turn away none or one. Only four sit anywhere in the middle, and all four of those block exactly eight.
| Position | Publishers |
|---|---|
| Blocks 11–12 of 12 | nst.com.my, kompas.com, rappler.com, nytimes.com, bbc.com, bloomberg.com |
| Blocks 8 | vnexpress.net, theguardian.com, washingtonpost.com, ft.com |
| Blocks 0–1 | thestar.com.my, thejakartapost.com, bangkokpost.com, inquirer.net, reuters.com, wsj.com |
No publisher in the sample blocks three agents, or five. Once a publisher decides to act, they act broadly — which suggests this is a policy decision taken once at an institutional level, not a per-vendor negotiation.
The regional gap is real, and smaller than you might expect
ASEAN publishers block a mean of 5.4 of the twelve agents. The global outlets block 7.4 — two more agents per publisher, on average.
Underneath that average, three of the eight ASEAN publishers block nothing at all, against two of the eight global ones.
We are not going to tell you that is a mistake. A publisher that wants its journalism quoted by an assistant, or that has a licensing deal, or that has decided the traffic is worth more than the training data, is making a defensible call. The narrower point is that on the one public record where the position has to be explicit, three of the eight ASEAN outlets we read have not taken one.
Some of the zeroes are deliberate, and you can tell which
There is a difference between a robots.txt that has never heard of GPTBot and one that names it and lets it through. We track this distinction because it gets to the heart of the matter—whether a zero is a deliberate choice or a sign of inaction.
Reuters names two AI agents and blocks neither; the Wall Street Journal names three and blocks none. Someone sat down, listed the crawlers, and chose to let them through.
Those are positions. The other zeroes — thestar.com.my, thejakartapost.com, bangkokpost.com — name no AI agent at all. Their files predate the question.
Training and search are being separated
The most interesting pattern in the per-agent numbers is that publishers do not treat one company as one thing.
| Agent | What it does | Blocked by |
|---|---|---|
| ClaudeBot | Anthropic, training | 62.5% |
| CCBot | Common Crawl | 62.5% |
| Bytespider | ByteDance | 62.5% |
| Applebot-Extended | Apple, training opt-out | 62.5% |
| GPTBot | OpenAI, training | 50.0% |
| Google-Extended | Google, training opt-out | 43.8% |
| OAI-SearchBot | OpenAI, search index | 37.5% |
| ChatGPT-User | OpenAI, fetches when a user asks | 37.5% |
OpenAI's training crawler is blocked by eight publishers. Its search crawler and its user-initiated fetcher are blocked by six. Three publishers name the search agent and let it through. Only one of them, vnexpress.net, does so while turning other agents away; Reuters and the Wall Street Journal name it and block nothing.
That is a coherent position, and it is the one some publishers seem to be feeling their way toward: do not train on us, but do send us readers. Whether the distinction survives contact with how these systems actually work is a separate argument.
One other number runs against expectation. Anthropic's crawler is blocked more often than OpenAI's, by ten publishers against eight. …in a sample of sixteen, and we mention it mainly because the assumption usually runs the other way.
What robots.txt is, and what it is not
robots.txt is a published request. It has no enforcement behind it. A crawler that ignores it is being impolite, not breaking in — the file is a sign on the lawn, not a lock on the door. Nothing we measured tells you whether any crawler obeyed any of these instructions, and we make no claim that they did.
What the file is good for is exactly what we used it for: it is the publisher's stated position, in public, in a form nobody can later claim to have misunderstood.
If you run a site, read your own file properly
The obvious way to check whether you block GPTBot is to search your robots.txt for the string. That test is wrong, and it fails in both directions.
robots.txt is a grouped format. A User-agent line opens a group, several consecutive User-agent lines share one group, and the Disallow and Allow lines that follow apply to all of them. So the string "GPTBot" appears in your file whether you are blocking it, explicitly allowing it, or mentioning it in a comment. Our harness parses the groups and asks whether this agent's own group disallows the site root, because a substring is not a policy.
The second trap is the wildcard. A User-agent: * group with Disallow: / does not reliably stop AI crawlers, because most of them look for their own name first and follow the most specific group that matches. If you want a named crawler out, name it.
Our AI crawler robots.txt builder generates the groups for the current set of agents, and our llms.txt generator covers the newer convention for stating terms to AI systems directly.
If you are weighing the other side of this — what an AI system can and cannot be trusted to have learned honestly — what you are trusting when you download a model measures how often that provenance is stated at all.
The limits of sixteen publishers
This is a sample. It is a deliberately chosen one — large general news outlets in Malaysia, Indonesia, Thailand, the Philippines and Vietnam, plus a global comparison set — but sixteen is sixteen, and the regional means rest on eight sites and eight.
We would defend the two-camps pattern hardest, because it depends on the sample size less than the means do. Six publishers at eleven-or-twelve and six at zero-or-one is a shape, and one that stark rarely comes from noise.
We also read only the root robots.txt at one moment in time. Publishers change these files, sometimes in response to a negotiation that concluded that week. Figures are from 19 August 2026, and the script is in our repository if you want to see what has moved since.
View author profile → · Editorial policy
