AI crawlers in robots.txt: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended

Three kinds of AI bot

1. Training. They download content that may be used to train future models. Blocking them does not remove you from today's search-based answers: it only keeps your content out of the next training runs.
2. Search. They index the web so the assistant can find you and cite you with a link when it searches in real time. Blocking them means disappearing from those answers.
3. On a user's request. They read a specific page when someone asks the assistant to, for example by pasting a link. Because they act on behalf of a person, several of them do not apply robots.txt.

Who is who

Three nuances that change things:

• OpenAI: blocking OAI-SearchBot does not make you fully invisible. If OpenAI learns your URL from a third-party search provider or from other pages' links, it may show the link and the title. To prevent that, use noindex and let the bot crawl the page so it can see it.
• Anthropic: unlike OpenAI, Perplexity or Google, it says all its bots respect robots.txt, including the one acting on a user's request.
• Microsoft has no AI token of its own: Bing and Copilot crawl with bingbot. Control comes through meta tags: with NOARCHIVE the page is not used in Copilot answers or to train models, and with NOCACHE only the URL, title and snippet are used. Either way, the page still appears in Bing.

Apple and Meta also run other agents: Applebot is the one that crawls for Apple's search, and Meta uses Meta-WebIndexer for Meta AI's searches and Meta-ExternalFetcher for fetches a user asks for.

Google is different: its AI is not controlled from robots.txt

Google's AI Overviews and AI Mode use Googlebot, the same crawler as regular search. You cannot block them in robots.txt without leaving Google, and Google-Extended was never that switch: it controls Gemini training, not Search.

What does exist since 31 August 2026 is a Search Console control (Settings, "Search generative AI") to exclude your site from AI Overviews, AI Mode and Discover's AI features. You are included by default. Excluding yourself does not affect your rankings in the rest of Search, or model training. The nosnippet, data-nosnippet and max-snippet tags also limit what Google shows from your pages, inside and outside its AI features.

What we recommend

For a business that wants to be found, always allow the search bots and the user bots. Block them and you disappear from assistants' answers, which is exactly where more and more customers ask.

The training bots are a business decision. If your content is your product (a publication, courses, paid reports), blocking them makes sense. If your site exists so people get to know you and hire you, being in the memory of the next models works in your favour.

If you block nothing, you do not need to name any bot:

User-agent: *
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

If you want to be in AI search but not in training:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: CCBot
Disallow: /

User-agent: *
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

Two technical details. A bot obeys only the most specific group that names it, so if you create a group for GPTBot, that bot ignores whatever you put under User-agent: *. And changes are not instant: OpenAI and Perplexity talk about roughly 24 hours to apply them.

Check your CDN and your firewall too

robots.txt is a request, not a barrier, and it is not the only door: your CDN or firewall may be turning bots away before they even read it. The most common case is Cloudflare:

• Since July 2025, Cloudflare blocks AI crawlers by default on new domains, and asks when they are added.
• In September 2025 it added a content signals line to its managed robots.txt that, by default, allows search and refuses training.
• Since 15 September 2026, its AI controls are split into search, training and agents, and the "Block" option also blocks Googlebot, Bingbot and Applebot. In other words, it takes you out of search engines too. To refuse training without losing search, there is the new "Disallow AI Training" option.

If your site is on Cloudflare, open AI Crawl Control and check that search is allowed.

How to see who actually gets in

• Server or CDN logs. Filter by user agent (GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot…) and look at the status codes they get. Repeated 403s mean something is stopping them.
• Beware of impostors. Anyone can claim to be GPTBot. OpenAI, Perplexity and Common Crawl publish their crawlers' IP ranges so you can check.
• An audit. Our free GEO audit reads your robots.txt, tells you which AI crawlers you let through and which you do not, and checks what they see without running JavaScript.

Sources

• OpenAI: Overview of OpenAI crawlers and Publishers and Developers FAQ
• Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?
• Perplexity: Perplexity crawlers
• Google: common crawlers, including Google-Extended and AI features and your website
• Search Console Help: Search generative AI control
• Bing: robots meta tags Bing supports
• Apple: About Applebot
• Meta: Meta web crawlers
• Common Crawl: CCBot
• Cloudflare: control content use for AI training (July 2025), Content Signals Policy (September 2025) and accountable mixed-use crawlers (September 2026)