# AI Crawler Blocking: Which Bots to Allow

> OpenAI, Anthropic, Perplexity and Google run separate bots for training, search and user requests. How robots.txt and firewalls block them, and how to check.

Page: https://trylectern.com/blog/ai-crawler-blocking

By Edgar Li, October 24, 2026.

AI companies run different bots for different jobs, and blocking the wrong one removes you from AI answers. Training bots collect content for models. Search bots index pages for an assistant's search feature. User-triggered fetchers load a page because a person asked a question. You can block the first without touching the others, but only if you know which is which. Two things can block any of them: a rule in robots.txt, or a firewall or bot challenge that refuses the request.

## Key Takeaways

- Each major vendor documents separate bots for training, search and user requests, and says blocking one does not block the others.
- robots.txt is a request. The firewall is enforcement. A site can allow a bot in robots.txt and still refuse it at the edge.
- Vendors differ on user-triggered fetchers. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt. Anthropic lists Claude-User as something you can disable.
- In our October 2026 probe of 435 publisher sites, 125 blocked or refused at least one AI assistant.
- Blocking training is a legitimate choice. Blocking search or user fetchers costs citations.

## The Bots

Names below come from each vendor's public documentation, read on 24 October 2026.

| Vendor | Bot | Purpose |
|---|---|---|
| OpenAI | GPTBot | Crawls content that may be used to train OpenAI's generative AI foundation models |
| OpenAI | OAI-SearchBot | Surfaces sites in ChatGPT's search features |
| OpenAI | ChatGPT-User | Visits a page when a user's ChatGPT or Custom GPT request needs it |
| Anthropic | ClaudeBot | Collects web content that could contribute to model training |
| Anthropic | Claude-SearchBot | Analyses content to improve search result relevance and accuracy |
| Anthropic | Claude-User | Accesses sites when a Claude user asks a question that requires it |
| Perplexity | PerplexityBot | Surfaces and links websites in Perplexity search results; not used to crawl for foundation models |
| Perplexity | Perplexity-User | Visits a page to support a user's action within Perplexity |
| Google | Google-Extended | A robots.txt token, not a crawler, controlling use of content for Gemini training and grounding |

Sources: [OpenAI](https://developers.openai.com/api/docs/bots), [Anthropic](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler), [Perplexity](https://docs.perplexity.ai/guides/bots), [Google](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers).

Google-Extended has no separate user agent string. Google crawls with its existing agents, and Google says Google-Extended does not affect inclusion or ranking in Google Search. Google's page names Gemini Apps and Grounding with Google Search on Vertex AI as the grounding uses; it does not say how AI Overviews are affected, so we make no claim about them here.

## How Blocking Works

**robots.txt.** A rule like `User-agent: GPTBot` followed by `Disallow: /` asks that bot to stay out. Well-behaved crawlers read the file before fetching. Rules are per bot, so you can disallow GPTBot and allow OAI-SearchBot. OpenAI says changes can take about a day to take effect.

**Bot challenges and firewalls.** A CDN or web application firewall can return a 403 or a challenge page to requests it does not trust, whatever robots.txt says. This is how sites block AI assistants by accident: a bot-protection setting is turned on for scrapers and catches the assistants too. Perplexity advises that if you use a WAF you may need to allow its bots explicitly, matching both the user agent and its published IP ranges.

**What robots.txt does not control.** OpenAI says that because users initiate ChatGPT-User's visits, robots.txt rules may not apply. Perplexity says Perplexity-User generally ignores robots.txt. If you want to stop those fetchers, the firewall is the lever, and the cost is that a person who asked an assistant about you gets nothing back.

## What the Probe Found

In October 2026 we fetched the placement pages of publisher sites that list themselves as citable in AI search. We requested each page once as a browser and once each as ChatGPT-User, Claude-User and Perplexity-User, and judged robots.txt with Lectern's matcher. Of 435 readable sites, 125 (29 percent) blocked or refused at least one AI assistant. 83 blocked an AI search crawler outright in robots.txt, which is the strongest evidence. 36 only refused our requests, which is weaker, because one request from one address is one reading, and a CDN that verifies bots may treat the real agent differently.

Blocks were per assistant. A site can be open to ChatGPT and closed to Claude. This is why one check per assistant beats one check for "AI".

If you buy placements, treat them as labelled commercial content that complements earned coverage, and look at whether the outlet lets assistants read it. Payment does not change what an outlet's robots.txt or firewall says.

## How to Check Your Own Site

1. Open yourdomain.com/robots.txt and read each AI-related rule. Sort them into training, search and user-triggered.
2. Check Google-Extended separately, because Google documents it as covering grounding as well as training.
3. Test the edge. A robots.txt that allows everything says nothing about what the firewall does. Request a page with each agent's user-agent string and compare with a browser.
4. Or run the free [agent access check](https://trylectern.com/agent-access), which does steps 1 to 3 for ChatGPT-User, Claude-User and Perplexity-User and the tracked crawlers, and names the rule or edge behind each block. Its requests come from Lectern's servers, so it reports what those requests got.
5. Look at the history. [Our robots.txt blocked AI crawlers for five months](https://trylectern.com/blog/we-blocked-ai-crawlers-for-five-months) inside unrelated commits.

## Fixing a Block

- For a robots.txt rule, delete or narrow the Disallow line for the bot, then re-fetch the file to confirm.
- For a firewall refusal, look in the CDN's bot settings for the rule that caught the agent and allow the vendor's published user agent and IP ranges. Anthropic and Perplexity publish IP lists. Anthropic warns that blocking by IP can stop bots reading robots.txt.
- Wait for caches. Vendors say changes can take up to a day.
- Re-check after the next deploy.

## The Tradeoffs

- **Block training, allow search.** You keep out of model training and stay citable in live answers. Models may still know about you from other sources.
- **Block everything.** Most control, no presence in assistants that rely on fetching.
- **Allow everything.** Maximum reach, no say over how your content is used.

None of these guarantees a citation, and none is permanent. Decide on purpose, write it down, and re-check.

This is the access half of AI visibility. [AX vs AEO](https://trylectern.com/blog/ax-vs-aeo) covers how it relates to being cited, and the [agent readiness score](https://trylectern.com/blog/agent-readiness-score) explains how to weigh the checks. What assistants then say about you is measured in [AI share of voice](https://trylectern.com/blog/ai-share-of-voice). See [pricing](https://trylectern.com/pricing) for the plans.