Almost everything written about optimising for AI search is written by people who do not publish their own numbers. So here are ours.

Between 5 June and 30 July 2026 this site recorded 214,712 page hits, from a counter that drops declared bot user-agents before it writes anything down. Of those, 3,167 arrived with an external referrer. Every AI assistant in the world, combined, accounted for 59 of them.

External referrals to recatools.com between 5 June and 30 July 2026: Google about 2,106, Bing 605, DuckDuckGo 282, Yandex 69, Baidu 35, for a search total of 3,108. Every AI assistant combined sent 59: Perplexity 23, Microsoft Copilot 17, ChatGPT 16, and Claude and DeepSeek 3 between them.
Same scale, both blocks — the AI bars are not a rendering error

That is 1.9% of referred traffic and 0.027% of everything. From a site with 89 guides and 877 tools, live for eight weeks.

We aren't saying AI search is irrelevant. We're showing you the arithmetic from inside one small site, because the alternative is another article describing a channel nobody has put a number to.

Three things in that log we did not expect

Perplexity beat ChatGPT. Twenty-three referrals against sixteen. Microsoft Copilot, at seventeen, also beat ChatGPT. If you were going to pick one assistant to court on the strength of its name recognition, you would have picked the wrong one.

Bing is 19% of our search referrals (605 against Google's roughly 2,106), over-performing its usual single-digit market share by about four times. The reason is structural, not flattering: Bing's index is what Copilot reads, and it has fed ChatGPT's search too. A large part of "being visible to AI" shows up in a referrer log as ordinary Bing traffic, not as an AI referral at all.

Only 1.5% of hits carried any external referrer. The rest is internal navigation and requests with no referrer at all. Which brings us to the measurement problem underneath all of this.

The number you cannot have

When an assistant reads your page and answers the question without sending anyone to you, it leaves no referrer. Nothing appears in your logs. There is no impression count, no citation count, no "your page was used" event.

So our 59 is not "how often an AI used us". It is "how often an AI used us and the reader clicked through afterwards". The gap between those two numbers is invisible from our side of the wire, and it is almost certainly the larger of the two.

This has two consequences. First, anyone quoting a confident figure for AI-search visibility is estimating from panel data or vendor telemetry, not your logs; treat it like any other survey. Second, our own counter has a softer edge than it looks: it filters user-agents that declare themselves bots, and undeclared scrapers walk straight through. "Did not say it was a bot" is not the same as "was a person", and every traffic figure on every small site carries that caveat whether or not anyone says so.

The one change worth making today

If you do one technical thing in response to all this, make it your robots.txt — and specifically, stop treating "AI crawlers" as a single category.

Training crawlers, which learn from you: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Bytespider and CCBot. Answering crawlers, which cite you: OAI-SearchBot, ChatGPT-User, PerplexityBot and Amazonbot. Blocking the training bot does not remove you from the answers; blocking the answering bot does.
Two different jobs, and the robots.txt line is different for each

OpenAI's own documentation is unambiguous about this. GPTBot "is used to crawl content that may be used in training our generative AI foundation models." OAI-SearchBot "is used to surface websites in search results in ChatGPT's search features." A third, ChatGPT-User, visits a page when a specific user asks a specific question.

Those are three different bots doing three different jobs, and the consequence is the whole ballgame:

  • Blocking GPTBot doesn't remove you from ChatGPT's search results — it just stops your pages feeding the next model.
  • Blocking OAI-SearchBot, however, does remove you from those results.

Most publishers, asked plainly, want the first and not the second: don't train on me, but do cite me. That position is expressible in four lines of robots.txt, and a great many robots.txt generators — including ours, until this week — could not express it, because they offered a single "block AI bots" button that swept both into the same bucket.

Google is the awkward exception. Google-Extended opts you out of Gemini training, but there is no separate token for AI Overviews: the crawler that feeds them is Googlebot, so opting out of Overviews means opting out of Google Search. That is not an oversight you can route around, and any guide implying otherwise is selling something.

What still works, which is most of it

The useful part of spending a week on this question was finding how little of the old SEO advice actually broke.

An assistant answering a question is still reading pages, and it still has to find them, parse them and decide they are worth quoting. Clean titles, honest meta descriptions, a real sitemap, fast pages, structured data, headings that match user questions — all of it helps a language model for the same reason it helps a ranking algorithm.

What has changed is the shape of the reward. A ranking gets you a click. A citation gets you a mention, sometimes a click, and sometimes nothing you can see. The things that were always weakly rewarded — being unambiguously correct, specific, the single page that answers the question — are now rewarded in a channel you can't measure. It's an uncomfortable truth.

Still worksDescriptive titles, accurate meta descriptions, a current sitemap, fast pages, headings that match real questions.
Still works, differentlyStructured data. It rarely wins a rich result any more, but it is a machine-readable statement of what your page contains.
Newly load-bearingBeing citable — specific, correct, and self-contained enough that a paragraph of you can stand alone in someone else's answer.
Newly load-bearingYour robots.txt, at the granularity of individual crawlers rather than "AI: yes or no".
Never workedKeyword density as a target. It has been a diagnostic and not a lever since Panda, in February 2011.
Quietly narrowedFAQ rich results. Google restricted them in August 2023 to authoritative government and health sites; the markup still describes your page honestly.
What to actually do

Split your robots.txt by crawler job, not just by "AI". Block the training bots if you want, but leave the answering ones alone unless you want to vanish from results. Check your own logs before you believe anyone's numbers, ours included — the ways your traffic differs from this report are the useful part. Don't chase the AI channel at the expense of search: on our data it is two orders of magnitude smaller, and the work serves both. Watch Bing. For a small site it is the most visible, measurable edge of the AI era. And accept that some of your best work will now be read without being visited. That's the trade.

What this audit found in our own tools

Every guide here starts by reading the source of the tools it will link to. The roadmap's prediction for this batch was blunt: our twelve SEO tools were "the most likely on the site to encode obsolete assumptions."

That prediction was mostly wrong. Our FAQ schema generator already documents Google's August 2023 narrowing in its own words, and argues the nuanced position rather than the "FAQ schema is dead" headline. Our keyword density checker never claimed density was a ranking factor — it walks through the history and treats the number as a diagnostic. The title and meta-description limits are the conventional published figures.

One tool was wrong: our robots.txt generator. Our robots.txt generator knew four AI crawlers — GPTBot, ClaudeBot, Google-Extended, CCBot — while our own AI crawler robots.txt builder, sitting in a different category, knew ten. Two tools on the same site answering the same question differently, with nothing comparing them.

It was worse than a short list. The page's own explanatory text named PerplexityBot, while the generator behind it had no entry for PerplexityBot and could not emit a rule for it. Its single preset, "Block AI training bots", blocked GPTBot — the one action that doesn't do what the label promises.

The crawler list now has eleven AI entries, each with its job specified. The single preset has been replaced by two: "Block AI training (stay citable)", which leaves OAI-SearchBot, ChatGPT-User, PerplexityBot and Amazonbot unblocked, and "Block AI entirely".

Sources
  • All traffic figures are from this site's own analytics_daily table on production, queried 1 August 2026 and covering 5 June to 30 July 2026 — 56 days and 214,712 page hits. Recording is done by middleware that skips declared bot user-agents before writing; undeclared scrapers are not excluded, which is why the guide treats the totals as an upper bound on human traffic.
  • The referrer breakdown was aggregated from the same table's stored referrer counts across all recorded days: search 3,108 (Google ~2,106, Bing 605, DuckDuckGo 282, Yandex 69, Baidu 35, Ecosia 11) and AI assistants 59 (Perplexity 23, Microsoft Copilot 17, ChatGPT 16, Claude 2, DeepSeek 1), out of 3,167 externally referred hits in total.
  • The descriptions of GPTBot, OAI-SearchBot and ChatGPT-User are quoted verbatim from OpenAI's published crawler documentation, read 1 August 2026. A fourth crawler, OAI-AdsBot, validates advertisement landing pages and is not discussed here.
  • Google's August 2023 restriction of FAQ rich results to authoritative government and health sites, and the February 2011 Panda update, are as described in our own FAQ schema generator and keyword density checker respectively — both of which predate this guide and were verified as part of its audit.
  • The crawler counts for our two robots.txt tools, and the PerplexityBot discrepancy, were read directly from the tools' source on 1 August 2026 and fixed in the same session.

General observations about search and crawler behaviour, drawn from one site's data over eight weeks. The sample is small and deliberately published anyway, because a real small number is more useful than a confident estimate. Crawler names and platform behaviour in this area change quickly; every figure carries the date it was read.