Robots.txt for AI Crawlers: Allow or Block GPTBot and More

To allow GPTBot in robots.txt, add a group that names it with `User-agent: GPTBot` and `Allow: /`. To block it, use `Disallow: /` instead. But GPTBot is a training crawler. ChatGPT search uses a different one, OAI-SearchBot. The same split between search and training exists at Google, Anthropic and Perplexity. So the real job is to decide, crawler by crawler, what you want, then write rules that say exactly that. You'll need access to robots.txt and your CDN or firewall. This guide is part of our AI search optimization guide.

The AI crawlers you can name in robots.txt

Each company documents its own tokens. The ones that matter most fall into three jobs: search, training and fetches a person triggers.

Two details are easy to miss. Google-Extended has no user agent of its own. It's a control token only, and Google says it doesn't affect Search inclusion or ranking. And blocking Googlebot to stay out of AI Overviews would also remove you from regular search results.

How crawlers pick the rules that apply

A crawler reads robots.txt and looks for a group that names it. If one exists, it follows that group only. The catch-all `User-agent: ` group applies only when no group names the crawler. The Robots Exclusion Protocol standard, RFC 9309, sets this order, and Google's documentation adds that specific groups and the `` group are never combined.

That has a sharp consequence. If your `*` group blocks /admin/ and you add a GPTBot group with only `Allow: /`, GPTBot may now crawl /admin/. Every named group must repeat the private-path rules you care about.

Inside a group, the longest matching path wins. So `Allow: /blog/` beats `Disallow: /` for blog URLs.

Steps to set up robots.txt for AI crawlers

1. 01
   Decide what each type of crawler should get.

   Write down one decision per job. Search crawlers decide whether AI answers can cite you, so most businesses allow them. Training crawlers are a values and licensing choice. User-triggered fetchers act for a person asking about your site. Get sign-off from whoever owns the content, and record the date. You'll want that record when someone asks later why a bot is blocked.

2. 02
   Read your live robots.txt.

   Fetch what crawlers actually receive, not what your CMS settings page claims:

   bash

   curl -s https://example.com/robots.txt

   Note every group, the paths in your catch-all group and any rules a plugin or platform added. Check that the file returns a 200 status. Google treats most 4xx responses as having no restrictions, and a 5xx can pause crawling of the whole site.

3. 03
   Add groups for the AI search crawlers.

   If you want to be citable, name the search bots and allow them. Being explicit protects you from a future catch-all rule blocking them by accident:

   text

   User-agent: OAI-SearchBot
   User-agent: PerplexityBot
   User-agent: Claude-SearchBot
   Allow: /
   Disallow: /admin/

   Several user-agent lines can share one group.

4. 04
   Set your training-crawler choice.

   Add a group for the training tokens with the rule you decided in step 1. To opt out of training while staying in search:

   text

   User-agent: GPTBot
   User-agent: ClaudeBot
   User-agent: Google-Extended
   Disallow: /

   OpenAI and Google both say these settings are independent of search. Blocking GPTBot doesn't affect ChatGPT search, and blocking Google-Extended doesn't affect Google Search.

5. 05
   Repeat private-path rules inside every named group.

   Go back through each group you added and copy in the disallow rules from your `*` group that still matter, such as admin, cart or staging paths. Then remember that robots.txt is public and is not a security control. RFC 9309 says so plainly. Anything truly private needs a login, not a robots.txt line.

6. 06
   Plan for user-triggered fetchers.

   OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt, because a person asked for the page. Don't rely on robots.txt to stop them. If a page must not be fetched, put it behind authentication. If you want people to get accurate answers about you, make sure these fetchers aren't blocked by your firewall either.

7. 07
   Match your firewall to your robots.txt.

   A CDN or firewall can block a bot that robots.txt allows. OpenAI and Perplexity publish IP lists for their crawlers, at openai.com/searchbot.json and perplexity.com/perplexitybot.json, and recommend allowing them. Perplexity advises rules that match the user agent and the IP range together, because user agents are easy to fake. Google publishes a way to verify Googlebot as well.

8. 08
   Wait, then check your logs.

   OpenAI says changes take about 24 hours to reach its search systems, Perplexity says up to 24 hours, and Google generally caches robots.txt for up to a day. After that, check your access logs for each bot and the response codes it gets. 200s on pages you allowed, and no hits on paths you blocked, mean the rules work.

A complete example robots.txt

Here is how the steps fit together for a site that wants AI search visibility, opts out of model training and keeps an admin area off limits. Adjust the paths to your own site.

User-agent: Googlebot
User-agent: OAI-SearchBot
User-agent: PerplexityBot
User-agent: Claude-SearchBot
Allow: /
Disallow: /admin/

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

User-agent: *
Allow: /
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml

Each named group repeats the admin rule, because named groups ignore the `*` group. Your own choice on training crawlers may differ. The structure is what matters.

If you'd like a second pair of eyes on your rules, our in-house SEO team handles crawler access as part of our SEO work, end to end.

Troubleshooting

AI crawlers ignore your new rules

Cause: usually timing or caching. Each company takes up to about a day to pick up a change, and a CDN may cache robots.txt itself, so crawlers can still get the old file. A second cause is a typo in the token. Bots match their own name, so a misspelled token creates a group that applies to nobody.

Fix: fetch robots.txt from outside your network and confirm the new rules are live. Purge the CDN cache for that one file if needed. Compare each token, letter by letter, with the company's crawler page linked in our sources. Then wait a full day before judging. If a user-triggered fetcher is the one you're seeing, remember robots.txt may not apply to it at all.

A named group opened pages you meant to keep private

Cause: the group-matching rule from earlier. Once a crawler finds a group with its name, it ignores the `*` group completely. A short group with only `Allow: /` therefore cancels every disallow line you wrote for everyone else.

Fix: copy the needed disallow lines into each named group, then retest. Google's robots.txt report in Search Console shows how Google fetched your file, not how other bots read it. For AI bots, read the file carefully group by group, or test with a robots.txt parser that follows RFC 9309. For pages that must stay private, move them behind a login, since robots.txt is publicly readable and only a request.

Your robots.txt returns an error

Cause: a misconfigured route, a redirect loop or a server error. Google treats most 4xx responses as if no robots.txt exists, meaning no restrictions. A 5xx is more serious. Google says it stops crawling the site for the first 12 hours while it retries.

Fix: request the file with curl and check the status code and any redirects. It should return 200 and plain text from the root of the exact host, such as https://example.com/robots.txt. Fix the route or server rule, then confirm with a fresh fetch. Once the file loads, review your crawl rules for AI bots as above. For the next layer of AI readiness, see llms.txt explained and structured data for AI search. For ChatGPT specifically, see how to get cited by ChatGPT.