Robots.txt Generator

The robots.txt file is the oldest still-standardised contract on the web. It was drafted in 1994 by Martijn Koster after a runaway crawler accidentally took down his employer's server, and the format has barely changed in three decades. The fundamentals are simple: a plain-text file at the root of your domain, listing which user-agents are allowed (or not allowed) into which folders. Every well-behaved crawler — Googlebot, Bingbot, GPTBot, ClaudeBot — fetches /robots.txt before it touches anything else and obeys what it finds. Misbehaved scrapers ignore it, but those are a different problem solved at the firewall layer.

What robots.txt is not

Robots.txt is a request, not a wall. It does not block access — it asks crawlers to skip certain paths. A page disallowed in robots.txt can still appear in Google search results (as a URL-only entry without snippet) if other sites link to it. To genuinely hide content, you need authentication, a noindex meta tag, or HTTP-level blocking. Use robots.txt to manage crawl budget (don't waste Googlebot's time on /admin) and to express preferences (please don't train your model on my content), not as a security boundary.

The most common robots.txt mistake is blocking /wp-admin/ on a WordPress site without also allowing /wp-admin/admin-ajax.php — which breaks half the dynamic features that depend on AJAX calls.

AI crawlers — the new arrivals

In the past two years, the robots.txt landscape has gained a new class of visitor: AI training crawlers. GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google's Bard/Gemini opt-out token), and CCBot (Common Crawl, which feeds many open-source models) all respect robots.txt. If you want your content excluded from future model training, the Block AI bots preset adds the right Disallow lines. This is now a meaningful editorial decision — major publishers (Reuters, BBC, NYT) have made the call, and so should you.

The APAC search-engine landscape

Across APAC, Google dominates organic search in most countries — Singapore, Malaysia, Indonesia, Philippines, Thailand, Vietnam, India, Australia, Taiwan, Hong Kong — but several markets have heavyweight local players you should explicitly handle. China is Baidu's home turf (60%+ market share), so Baiduspider is non-negotiable for Chinese-language SEO; Sogou and 360Spider are smaller but worth allowing. South Korea sees Naver hold around 55% of search, served by NaverBot and the older Yeti agent. Japan is split: Yahoo Japan (powered by Google) and Google direct, plus Slurp for Yahoo's legacy index. Russia's Yandex (YandexBot) matters if you have Russian-speaking visitors anywhere in APAC, and Yandex is the only major engine that still respects the Host: directive — useful when you have www and non-www variants and want to declare a canonical host inside robots.txt itself.

Crawl-delay — sparingly

The Crawl-delay directive asks crawlers to wait N seconds between requests. Bingbot, YandexBot, and most secondary crawlers respect it. Googlebot does not — Google's position is that crawl rate should be controlled via Search Console settings, not robots.txt. Set a Crawl-delay only if a specific bot is overloading your server; the default behaviour of major search engines is usually fine for shared hosting and definitely fine for a VPS or anything bigger.

Validating after you deploy

After uploading your robots.txt, paste it through Google Search Console's robots.txt Tester and Bing Webmaster Tools' equivalent. Both will tell you whether specific URLs are allowed or blocked under your current rules. Re-test every time you change a Disallow line — the cost of accidentally blocking / for Googlebot is a complete deindex within 48 hours, and recovery takes weeks. Always keep the previous robots.txt in version control so you can roll back in seconds.