Robots.txt Generator

CRAWLERS SEO WEBMASTER
Share:
By Checked against sources by , Founder & Architect, RECATOOLS Published Updated Last reviewed

Build a valid robots.txt with allow/disallow rules per bot, sitemap pointers, and ready-made presets for common stacks.

RT-SEO-008 · SEO & Marketing
Your browser is too old for this tool’s interactive features. Please update to Safari 13.1, Chrome or Edge 80, Firefox 74 or newer.

Robots.txt Generator

Sitemap URLs

    Generated robots.txt 
    
        
    
        
        
    
            How crawlers will read it (parsed the way RFC 9309 defines)
            
      
        
      

      



      
    
      
        Advertisement
        
      
            After results · AD-W1
            Responsive · Post-tool — peak engagement
        
      
    
      

      


      
    
      How to use the robots.txt generator
      
    
      
        
      Start with a preset (optional)
      If your site is WordPress, Laravel, or you want to block all AI training bots, click the matching preset. The rule groups, sitemap field, and allow/disallow lists fill in for you. From there, tweak by hand.
      
        
      Add rules per user-agent
      Each user-agent group starts with one bot (or * for all). Type a path under Allow or Disallow and press Enter to add it. Paths must start with / and are case-sensitive — write /Admin/ if that is how your URLs are spelled. The tool adds a missing leading slash and tells you.
      
        
      Add your sitemap URL
      Paste the full sitemap URL (https included) in the Sitemap field. You can add multiple sitemaps; each gets its own Sitemap: line. A sitemap line is not tied to any user-agent group, so where it sits in the file does not matter.
      
        
      Copy or download
      Click Copy to grab the text, or Download to save as robots.txt. Upload to the root of your domain — it must live at example.com/robots.txt, never in a subfolder.
      
    
      

      



      
    
      
        Advertisement
        
      
            After how-to · AD-W2
            Responsive
        
      
    
      

      


      
    
      Robots.txt — the polite handshake between you and the crawlers
      
    
      
    
      The robots.txt file is one of the oldest conventions on the web: Martijn Koster defined the method in 1994, and it became a formal standard only in September 2022, as RFC 9309. It is a plain-text file at the root of a host that tells crawlers, group by group, which paths they may fetch. Well-behaved crawlers read it before crawling; others ignore it, and that is a problem for the firewall, not for this file.
      

    
      The mistake this tool used to make
      

    
      Until 30 September 2026 this generator wrote an empty group as a bare User-agent: * line. In RFC 9309 a group is one or more User-agent lines followed by rules, and blank lines do not end it — so a bare User-agent: * followed by User-agent: GPTBot and Disallow: / is a single group, and its Disallow: / applies to every crawler. Three presets, "Block AI training", "Block AI entirely" and "Block SEO scrapers", produced exactly that file: it told Googlebot to crawl nothing. Every group now ends with a rule (an empty Disallow: allows everything and closes the group), and the page reads the finished file back the way RFC 9309 does and shows you whether Googlebot, Bingbot and the main AI crawlers may fetch your home page.
      

    
      
        A blank line does not end a robots.txt group. Write two user-agent lines with nothing between them but white space, and they share one set of rules.
    
      

    
      What robots.txt is not
      

    
      It is a request, not a lock. Google's own documentation warns against using it to hide pages: if other pages link to a URL with descriptive text, Google "could still index the URL without visiting the page". To keep a page out of results, use password protection or a noindex tag — and do not also block that page in robots.txt, or crawlers never see the tag.
      

    
      AI crawlers: training, search and user fetches are different bots
      

    
      OpenAI, Anthropic and Perplexity each document separate tokens. OpenAI: GPTBot crawls content "that may be used in training", OAI-SearchBot surfaces sites in ChatGPT search, and ChatGPT-User fetches pages when a user asks, for which OpenAI says "robots.txt rules may not apply". Anthropic: ClaudeBot for training, Claude-SearchBot for search, Claude-User for user requests. Perplexity: PerplexityBot is for search and "not used to crawl content for AI foundation models", while Perplexity-User "generally ignores robots.txt rules". Google's Google-Extended covers both training Gemini and grounding Gemini's answers, and "does not impact a site's inclusion in Google Search". Apple's Applebot-Extended opts content out of training Apple's models while Applebot keeps crawling for search. So "Block AI training (stay in AI search)" blocks the training tokens only; "Block AI entirely" blocks the search and user-request ones as well.
      

    
      Crawl-delay
      

    
      Crawl-delay is not part of RFC 9309, and Google lists it among the fields it does not support. Some other crawlers honour it. Google also removed Search Console's crawl-rate limiter on 8 January 2024: Googlebot now slows down by itself when a server responds with errors or slowly. If you set a delay here, the tool writes it into each group, because a line after the last group would belong to that group alone.
      

    
      Checking after you deploy
      

    
      The file must be at the root of each host and protocol — https://example.com/robots.txt covers neither http://example.com nor https://blog.example.com. Google ignores content past 500 KiB. Search Console's robots.txt report shows which robots.txt files Google found for your site and when it last fetched them. Keep the previous version, so a bad change can be rolled back at once.
      

      

      



      
    
      Koster 1994, RFC 9309 in 2022, and why a blank line does not end a group
      
    
      
        
      01
      Martijn Koster defined the robots exclusion method in 1994. It became an IETF standard, RFC 9309, in September 2022 — written by Koster with Gary Illyes, Henner Zeller and Lizzi Sassman.
      
        
      02
      RFC 9309 defines a group as user-agent lines followed by rules, with empty lines allowed in between. So an empty line never separates two user-agent lines into two groups.
      
        
      03
      Google enforces a 500 KiB limit on robots.txt; content after that point is ignored.
      
        
      04
      Google supports four fields — user-agent, allow, disallow and sitemap — and says other fields "such as crawl-delay aren't supported".
      
        
      05
      When rules conflict, the most specific one wins: RFC 9309 defines it as the match with the most octets. If an allow and a disallow are equally long, the allow should be used.
      
        
      06
      A crawler follows only the group that names it most specifically. A bot with its own group ignores the * group completely — its rules are not added together.
      
        
      07
      A rule's path must start with / and is case-sensitive: Disallow: /admin does not cover /Admin.
      
        
      08
      A sitemap line must be a fully qualified URL, need not be on the same host, and is not tied to any user-agent group.
      
        
      09
      Naver, Korea's largest search engine, documents its crawler as Yeti — so a rule for "NaverBot" matches nothing.
      
        
      10
      Google removed Search Console's crawl-rate limiter on 8 January 2024; Googlebot now adjusts its crawl speed from how the server responds.
      
    
      

      



      
    
      FAQ
      

    
        

      • 
        
        
        
            
        At the root of your domain — https://example.com/robots.txt. Crawlers will not check any other location. Subdomains need their own file (e.g. blog.example.com/robots.txt).
        • 

    
      • 
        
        
        
            
        No — it is a request. Well-behaved crawlers such as Googlebot, Bingbot, GPTBot and ClaudeBot obey it; scrapers ignore it. Some AI companies also say their user-triggered fetchers may not follow it (OpenAI for ChatGPT-User, Perplexity for Perplexity-User). To actually block access you need authentication, IP blocking or a web application firewall.
        • 

    
      • 
        
        
        
            
        Yes. Google says that if other pages link to a URL with descriptive text, it could still index the URL without visiting the page. To keep a page out of results, use noindex or password protection — and leave the page crawlable, or Google never sees the noindex.
        • 

    
      • 
        
        
        
            
        Use the "Block AI training (stay in AI search)" preset. It disallows GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Gemini — which also stops Gemini's answers from grounding on your pages, though not Google Search), Applebot-Extended (Apple), Amazonbot and CCBot (Common Crawl), and leaves OAI-SearchBot, Claude-SearchBot and PerplexityBot allowed, so you can still be cited in AI search. PerplexityBot is a search crawler, not a training one, by Perplexity's own description.
        • 

    
      • 
        
        
        
            
        The longest matching rule wins, measured in octets; on a tie, the allow wins. So Allow: /wp-admin/admin-ajax.php beats Disallow: /wp-admin/ for that one file — which is exactly the default robots.txt WordPress itself generates.
        • 

    
      • 
        
        
        
            
        Yes. Rule paths are case-sensitive whatever your server does: Disallow: /admin does not block /Admin. Match the spelling of your real URLs.
        • 

    
      • 
        
        
        
            
        Only if images should be treated differently from pages. A crawler uses the most specific group that names it and ignores the rest, so a Googlebot-Image group replaces — not adds to — whatever would otherwise apply to it.
        • 

    
      • 
        
        
        
            
        No. Google does not support the field, and it removed Search Console's crawl-rate limiter on 8 January 2024; Googlebot now slows down on its own when your server returns errors or responds slowly. Some other crawlers do honour Crawl-delay.
        • 

    
      • 
        
        
        
            
        Yes if your site has multiple sitemap files — common for large sites split by content type (sitemap-posts.xml, sitemap-products.xml). Or use a sitemap index file and point to that single URL. Either approach is valid.
        • 

    
      • 
        
        
        
            
        This page reads the file back the way RFC 9309 does and shows whether Googlebot, Bingbot and the main AI crawlers may fetch your home page. After deploying, Search Console's robots.txt report shows which robots.txt files Google found and when it last fetched them.
        • 

      

      





                        
                        
      
        
      
            
      Method & sources
      
            
        
      

        
                    
      
                
      What this tool implements
      
                
        
                                            
      • Until this review an empty group was written as a bare "User-agent: *". RFC 9309 §2.2: group = startgroupline *(startgroupline / emptyline) *(rule / emptyline), so the next group's user-agent joined it — the three bot-blocking presets told Googlebot to crawl nothing. The kernel's test rebuilds the old output and proves it.
        • 
                                            
      • Bot tokens per their operators' pages: GPTBot/OAI-SearchBot/ChatGPT-User, ClaudeBot/Claude-SearchBot/Claude-User, PerplexityBot (search, not training)/Perplexity-User, Google-Extended (Gemini training AND grounding, not Search), Applebot-Extended, Naver's Yeti (not "NaverBot").
        • 
                                            
      • Google supports only user-agent, allow, disallow and sitemap; ignores content past 500 KiB; removed the crawl-rate limiter on 8 Jan 2024. The WordPress preset now matches core's do_robots() default.
        • 
                                    
      
            
      
        
        
      
            
      Sources
      
            
        
      

        
                    
      Abridged — the full review record for this tool runs longer than the list above.
      
            
      
                        
                                                
                        
                        
                        


                    
      
                    
                
      
            
      
        
      
    
      

    
    
      
        
      
            
      
                Advertisement
                
      
                    Pre-footer · AD-W3
                    728 × 90