Guide

Should you block GPTBot? AI crawlers and robots.txt, explained

By Mandel Sherrod · Updated October 19, 2026

Blocking "AI bots" can quietly remove you from AI answers, or do nothing you intended. Here's what each crawler does, according to the companies that run them.

Three kinds of AI crawler

"Block AI bots" sounds like one switch. It's really three, because AI companies run different crawlers for different jobs:

• Training crawlers collect public pages that may be used to train future models. Blocking them is a statement about how your content may be used. It doesn't directly take you out of search answers.
• Search crawlers build the index an assistant searches when it answers with links. Blocking these takes you out of that assistant's cited answers.
• User-initiated fetchers load a specific page because a person asked the assistant to, for example by pasting your URL. OpenAI and Perplexity say robots.txt may not apply to these, because a human triggered the fetch.

The crawlers that matter, in one table

Based on each company's own documentation as of October 2026. Vendors add and rename crawlers, so recheck their pages before making big changes.

Sources: OpenAI crawlers, Anthropic crawlers, Perplexity crawlers, Google AI features.

How to decide what to allow

For a business that wants customers to find it through AI, here's a sensible default:

1. Allow every search crawler: Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot and PerplexityBot. These decide whether you can be cited at all.
2. Leave user-initiated fetchers alone. When a prospect pastes your URL into ChatGPT and asks "is this any good?", you want the assistant to be able to read the page.
3. Make a deliberate call on training crawlers. Allowing them gives future models more chances to learn who you are, which can help you get named from memory. Blocking them protects content you consider proprietary. Publishers and marketing-led businesses often decide differently, and both are reasonable.

Whatever you choose, write it down in robots.txt with explicit groups so the intent is clear and doesn't break when someone edits the wildcard rule.

A robots.txt you can copy

This version allows search and training crawlers explicitly, and keeps private areas out for everyone:

User-agent: *
Disallow: /admin/
Disallow: /cart/

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Googlebot
User-agent: Bingbot
Allow: /
Disallow: /admin/
Disallow: /cart/

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Allow: /
Disallow: /admin/
Disallow: /cart/

Sitemap: https://www.example.com/sitemap.xml

To opt out of training but stay in AI search, change the second group to Disallow: /. Note that a crawler follows only the most specific group that names it. That's why each named group repeats the private-area rules instead of inheriting them from *.

The trap: your CDN may be blocking them anyway

robots.txt is a request, read by well-behaved crawlers. Your CDN, firewall or hosting platform can block a crawler before it ever reaches robots.txt. Several providers now offer one-click "block AI bots" or "AI crawl control" settings. Some apply broad bot-fighting rules that challenge anything that isn't a browser. A site can have a perfect robots.txt and still return a 403 or a challenge page to OAI-SearchBot.

Check your CDN's bot settings, and allow the verified search crawlers you want. Where vendors publish IP ranges, as OpenAI does for its crawlers, use them to verify traffic instead of trusting the user-agent string alone, because anyone can fake that string.

What about llms.txt?

llms.txt is a proposed convention, introduced in 2024 at llmstxt.org. It's a Markdown file at your site root that summarizes the site and links its most useful pages for language models. It's easy to make and does no harm.

Keep expectations realistic. No major AI search provider has said it uses llms.txt to decide what to cite. Google states that you don't need new machine-readable files or AI text files to appear in its AI features. Add one if you like. Don't put it ahead of crawl access, clear pages and consistent facts.

How to check what crawlers actually see

1. Run our free checker. The AI visibility checker reads your robots.txt rules for each crawler and fetches your homepage using each search crawler's user agent, so a CDN block shows up as a non-200 response.
2. Check your server or CDN logs for the tokens above. Steady visits from OAI-SearchBot, PerplexityBot and Bingbot are a good sign. Seeing none at all may point to a block.
3. Test from the command line: curl -A "OAI-SearchBot/1.0" -I https://yoursite.com/ shows the status code your server returns to that user agent. Some CDNs verify crawlers by IP, so a 403 here can be a false alarm. A normal 200 is reassuring.

Questions