Short answer
There are two kinds of AI crawler and they need opposite decisions. Training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended) collect content to train models — blocking them is a legitimate choice with no traffic cost. Answering crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Perplexity-User) fetch pages to cite in AI answers and send referral traffic — blocking them removes you from those answers. For most businesses the right setting is: allow everything on public pages, block only private areas.
The split that decides everything
Every major provider now runs at least two separate user-agents, and they are governed by separate robots.txt rules. A single line written in 2023 to “keep AI out” usually names only the training bot, or names both and silently removes the site from AI answers.
| Provider | Training crawler | Answering crawler | Sends referral traffic |
|---|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot, ChatGPT-User | Yes, via ChatGPT search |
| Anthropic | ClaudeBot, anthropic-ai | Claude-SearchBot, Claude-User | Yes, via Claude citations |
| Perplexity | — | PerplexityBot, Perplexity-User | Yes, sources are listed |
| Google-Extended | Googlebot (feeds AI Overviews) | Yes | |
| Apple | Applebot-Extended | Applebot | Yes, via Siri and Spotlight |
| Common Crawl | CCBot | — | No |
| ByteDance | Bytespider | — | No — documented ignoring robots.txt |
Note the Google row: Google-Extended controls training only. AI Overviews are fed by ordinary Googlebot crawling, so blocking Google-Extended does not remove you from them — and blocking Googlebot removes you from Google entirely.
What sites actually do
The 2026 data shows the market converging on a middle path rather than a blanket block:
- GPTBot is the most blocked AI crawler, named in roughly 5.5% of disallow rules in Q1 2026, ahead of CCBot (~5.1%), ClaudeBot (~4.9%), Google-Extended (~4.4%) and Bytespider (~4.2%).
- Among top sites, GPTBot blocking plateaued at about 25% — up from ~5% in early 2023 and ~25% through 2024, flat since.
- About 30% of top sites now block training bots while allowing answering bots — the deliberate middle path.
The plateau is the interesting part. The first wave of blocking was reflexive; the second wave was informed, and it stopped at the line between training and answering.
Decide by what your content is
| Your situation | Training crawlers | Answering crawlers | Why |
|---|---|---|---|
| Small or mid-sized business, marketing site | Allow | Allow | You have far more to gain from being described and cited than to lose from being in a training set. |
| Local services (trades, clinics, agencies) | Allow | Allow | Local recommendation questions are answered from pages the answering bots can reach. |
| Publisher or media, content is the product | Block | Allow | The middle path: keep the archive out of training, stay in the answers that drive readers. |
| Paid courses, research, proprietary data | Block | Allow marketing pages only | Put the product behind authentication; let the crawlers see what sells it. |
| Anything with personal or client data | Block | Block | Not a robots.txt question: it needs authentication. |
A robots.txt you can copy
This is the allow-everything-public, block-everything-private pattern — the right default for most businesses. Repeat the block for each user-agent you care about; wildcards in User-agent are not reliably supported.
| Directive | Meaning |
|---|---|
| User-agent: GPTBot … Allow: / | Public pages readable; repeat the block per user-agent. |
| Disallow: /app/, /admin/, /api/, /account/ | Private areas closed to every crawler. |
| Allow: /llms.txt | Needed if you use Disallow: / with an allow-list. |
| Sitemap: https://example.com/sitemap.xml | Discovery, once at the end of the file. |
Three mistakes we see constantly: a CDN or security plugin blocking AI user-agents without anyone knowing; a Disallow: / allow-list that forgets /llms.txt and the CSS the page needs to render; and a login wall that returns HTTP 200 with a sign-in page for every unknown path, so a crawler records your whole site as one login form.
robots.txt is a request, not a lock
The major providers document their user-agents and honour robots.txt. Bytespider and stealth crawling attributed to Perplexity have both been documented ignoring it. Anything that genuinely must not be read needs authentication or an IP rule — a disallow line is a polite notice, and it is published for anyone to read.
Frequently asked questions
Should I block AI crawlers?
It depends which. Training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended) collect content to train models. Answering crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Perplexity-User) fetch pages to cite in AI answers and send traffic back. If you want to appear in AI answers, allow the answering crawlers.
What is the difference between GPTBot and OAI-SearchBot?
GPTBot collects content for training. OAI-SearchBot crawls for ChatGPT's search feature, which cites sources and links back. Separate user-agents, separate rules: blocking GPTBot does not remove you from ChatGPT search; blocking OAI-SearchBot does.
How many sites block AI crawlers?
GPTBot is the most blocked, named in about 5.5% of disallow rules in Q1 2026. Among top sites, GPTBot blocking rose from ~5% in early 2023 to ~25% and has been flat since; roughly 30% now block training bots while allowing answering bots.
Do all AI crawlers respect robots.txt?
No. The major providers do; Bytespider and stealth crawling attributed to Perplexity have been documented ignoring it. Use authentication for anything that must not be read.
Does blocking AI crawlers hurt my SEO?
Not your Google rankings — Googlebot and Google-Extended are separate. But blocking answering crawlers removes you from AI answers, where a growing share of qualified traffic starts.
What should a small business allow?
Everything on public pages; block only accounts, APIs, dashboards and checkout.
Free diagnostic
Not sure what your robots.txt is actually doing?
We check which crawlers reach your pages, what the AI engines say about your business today, and where the two disagree. Remote, worldwide, in English or French.
Request the diagnostic →Read the llms.txt guide