Should you block AI crawlers? A plain guide to robots.txt and AI bots
By Maquinable · Published September 24, 2026
Should you block AI crawlers? Some people say you should, to protect your content. Others say you should let AI in so it recommends you. Both can be right at once, because each AI assistant runs several bots that do different things. This guide explains what each one does, what each company says in its own documentation, and which setup fits a small business website.
What robots.txt is, and what it doesn't do
robots.txt is a plain-text file at yoursite.com/robots.txt. Before crawling a site, bots read it to see what they're allowed to visit. Each block starts with User-agent, the bot's name, followed by Disallow (stay out of here) or Allow (this is fine) rules.
Three things to keep in mind:
- It's a request, not a lock. The big AI companies say their crawlers honor it. A malicious scraping bot just ignores it.
- Blocking doesn't erase. If a bot already collected your pages, blocking it today doesn't remove what it took.
- It doesn't deindex pages. To keep a page out of Google you use a
noindextag, not robots.txt.
If you've heard of llms.txt, it's a different file: robots.txt grants or denies access, while llms.txt only summarizes your site for AI tools. If robots.txt blocks a bot, your llms.txt doesn't matter to it. More in what is llms.txt.
The three kinds of AI crawlers
Each AI company runs several bots, and what each one does decides whether you want to let it in.
Training crawlers. They collect content to train future models: GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl, an open archive many models are trained on) and Meta-ExternalAgent (Meta). Google-Extended and Applebot-Extended work differently. They don't crawl on their own; they're names you use in robots.txt to tell Google and Apple not to use your content to train their AI.
AI search crawlers. They index your site so an assistant can cite and link to it when someone asks a question: OAI-SearchBot (ChatGPT search), Claude-SearchBot, PerplexityBot and Meta-WebIndexer. Block them and those assistants have a harder time mentioning you with a link.
User-triggered fetchers. They visit when a person asks an assistant to read a specific page: ChatGPT-User, Claude-User, Perplexity-User, Meta-ExternalFetcher and Google-Agent. OpenAI, Perplexity, Google and Meta say in their docs that these fetchers may not follow robots.txt, because they act on a person's behalf. Anthropic is the exception: it says all its bots, Claude-User included, honor robots.txt.
Which bot belongs to whom
| Company | Bot | Job | Follows robots.txt? |
|---|---|---|---|
| OpenAI | GPTBot | Training | Yes |
| OpenAI | OAI-SearchBot | ChatGPT search | Yes |
| OpenAI | ChatGPT-User | User requests | Not guaranteed |
| Anthropic | ClaudeBot | Training | Yes |
| Anthropic | Claude-SearchBot | Search | Yes |
| Anthropic | Claude-User | User requests | Yes |
| Perplexity | PerplexityBot | Search | Yes |
| Perplexity | Perplexity-User | User requests | Generally no |
| Google-Extended | Permission to train Gemini | Yes (it's a robots.txt rule) | |
| Apple | Applebot-Extended | Permission to train Apple's AI | Yes (it's a robots.txt rule) |
| Common Crawl | CCBot | Open archive used for training | Yes |
| Meta | Meta-ExternalAgent | Training | Yes |
| Meta | Meta-ExternalFetcher | User requests | May bypass it |
| ByteDance | Bytespider | No official documentation | Not documented |
Sources, checked on September 24, 2026: OpenAI, Anthropic, Perplexity, Google, Apple, Common Crawl and Meta.
Don't block facebookexternalhit. It's the bot that builds the link preview (title, image, description) when someone shares your site on Facebook, Instagram or WhatsApp.
Should you block AI crawlers? Three setups for a small business
There's no single answer, but for a small business one of these three almost always fits.
Setup 1: allow everything
For businesses that live on being found: local services, B2B firms, online stores. You block nothing, and any assistant can read you, cite you and train on your content. It's what we do on maquinable.com, because our business depends on people finding us.
Setup 2: opt out of training, allow search
The balanced choice for most sites. You ask AI companies not to train on your content, but you let in the bots that can cite you with a link. Add this to your robots.txt without deleting what's already there:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: meta-externalagent
Disallow: /OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot keep access because they aren't named. If you also want to block Bytespider, add the same block with its name, knowing ByteDance doesn't document whether it follows it.
This is also how you block GPTBot and keep appearing in ChatGPT search: GPTBot only trains, and ChatGPT search uses OAI-SearchBot.
Setup 3: block everything
For paid or highly original content you don't want summarized by AI. You also block the search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot).
You lose the chance of ChatGPT, Claude or Perplexity citing you with a link. And you don't fully close the door: several user-triggered fetchers may still come in, and content already used for training stays used.
In every case, allowing search crawlers doesn't guarantee an assistant will recommend you.
How to edit your robots.txt without breaking anything
Where you edit it depends on your site:
- WordPress: Yoast SEO and Rank Math both include a robots.txt editor in their tools.
- Shopify: edit the theme's
robots.txt.liquidtemplate. - Wix and Squarespace: they have their own, more limited settings; if you can't find them, ask their support.
- Custom-built site: send the block for your chosen setup to whoever maintains it.
The most common mistakes:
- Writing
User-agent: ChatGPT. No bot has that name, so the rule does nothing. Use the names in the table. - **Putting
Disallow: /underUser-agent: *.** It blocks every bot, Googlebot included, and your site can drop out of Google. - Blocking Googlebot to block Google's AI. Googlebot is what puts you in search results. Gemini training is controlled with Google-Extended.
To check the result, open yoursite.com/robots.txt in your browser. If you use Google Search Console, its robots.txt report flags errors.
Check Cloudflare: it may block AI bots whatever robots.txt says
Many small business sites run through Cloudflare without the owner knowing, because an agency or host set it up. Cloudflare can block AI bots with a firewall rule on its network, before they ever read your robots.txt.
What changed, by date:
- July 2024: a one-click toggle to block AI bots, on all plans. Opt-in.
- July 1, 2025: every new domain is asked at signup whether to allow AI crawlers.
- September 15, 2026: bots are split into three categories (search, agent and training) with new defaults. On pages that display ads, training and agent bots are blocked, and search bots are allowed. According to Cloudflare's July 1, 2026 press release, this applies to new customers, new sites, and existing free-plan sites that hadn't changed their settings by that date.
If your site is on Cloudflare and has pages with ads, ChatGPT-User or Claude-User may be blocked there even if your robots.txt allows them. And if someone turned on the general block, they may all be out.
Where to check: in your Cloudflare dashboard, open your domain and look for AI Crawl Control. It shows which AI bots try to get in and whether they're blocked, and lets you change it. It's available on every plan, including Free. Cloudflare also offers a managed robots.txt that adds AI rules to your file; it's off by default.
If you'd rather have us look, our free scan checks whether AI bots can read your site. More guides are in Guides.
Google-Extended and AI Overviews: the most common mix-up
Many site owners block Google-Extended thinking it keeps them out of Google's AI answers. It doesn't. According to Google's documentation, updated July 14, 2026, Google-Extended controls whether your content is used to train future Gemini models, and "does not impact a site's inclusion in Google Search nor is it used as a ranking signal."
AI Overviews and AI Mode are part of Google Search, controlled by Googlebot and the nosnippet and noindex tags. Blocking Google-Extended doesn't remove you from them. Blocking Googlebot does, but it also removes you from Google entirely.
FAQ
Will blocking AI crawlers hurt my Google rankings?
Not as long as you don't block Googlebot. Google says Google-Extended doesn't affect inclusion in Google Search or rankings. Bots from OpenAI, Anthropic or Perplexity have no effect on Google either.
If I block GPTBot, will my business still show up in ChatGPT?
It can. GPTBot only collects content for training. ChatGPT search uses OAI-SearchBot, and ChatGPT-User visits when a user asks to read your page. Block GPTBot, allow OAI-SearchBot, and ChatGPT can still cite you with a link.
Do AI bots actually respect robots.txt?
The training and search crawlers from the big companies say they do. User-triggered fetchers generally don't: OpenAI, Perplexity, Google and Meta warn they may bypass it. Anthropic says all its bots honor it. Bytespider has no official documentation.
Is Cloudflare blocking ChatGPT on my site?
It might be. Since September 15, 2026, Cloudflare blocks training and agent bots by default on pages with ads for new sites and untouched free-plan sites. Someone may also have turned on its AI block. Check AI Crawl Control in your dashboard.
Can blocking AI bots keep my site out of Google's AI Overviews?
Not with Google-Extended. AI Overviews are part of Google Search and use Googlebot. You can limit how a page appears with the nosnippet or noindex tags, but blocking Googlebot removes you from Google Search altogether.
If I block AI crawlers now, does that remove what they already took?
No. robots.txt only affects future visits. Content already collected or used as training data stays where it is.