robots.txt Generator
Build safe robots.txt rules for search engines and AI crawlers.
AI Crawler Directory
Compare search, AI answers, model training and user-triggered fetching in one directory.
26 tokensrobots.txtPurpose split
| Crawler | Owner | Purpose |
|---|---|---|
Googlebot | Google Search, including AI surfaces that use Google Search infrastructure | |
Bingbot | Microsoft | Bing Search and Bing-grounded experiences |
Applebot | Apple | Apple search and discovery |
DuckDuckBot | DuckDuckGo | DuckDuckGo Search |
YandexBot | Yandex | Yandex Search |
| Crawler | Owner | Purpose |
|---|---|---|
OAI-SearchBot | OpenAI | OpenAI search discovery and answer retrieval |
Claude-SearchBot | Anthropic | Claude search discovery |
PerplexityBot | Perplexity | Perplexity search and answers |
DuckAssistBot | DuckDuckGo | DuckDuckGo AI answer features |
Amzn-SearchBot | Amazon | Amazon search and AI answer discovery |
MistralAI-Index | Mistral AI | Mistral AI index and search discovery |
| Crawler | Owner | Purpose |
|---|---|---|
GPTBot | OpenAI | Model improvement and training crawler |
ClaudeBot | Anthropic | Model improvement and training crawler |
Google-Extended | Control token for Gemini and Vertex training or grounding. Not a separate search crawler. | |
Applebot-Extended | Apple | Control token for generative model use |
CCBot | Common Crawl | Common Crawl archive, frequently used in model datasets |
meta-externalagent | Meta | Meta AI model data collection |
Bytespider | ByteDance | ByteDance crawler |
Amazonbot | Amazon | Amazon crawler used across search and AI products |
MistralAI-Training | Mistral AI | Mistral AI training crawler |
| Crawler | Owner | Purpose |
|---|---|---|
ChatGPT-User | OpenAI | Fetch started by a person using ChatGPT |
Claude-User | Anthropic | Fetch started by a person using Claude |
Perplexity-User | Perplexity | Fetch started by a person using Perplexity |
meta-externalfetcher | Meta | Fetch started by a person using Meta AI |
Amzn-User | Amazon | Fetch started by a person using Amazon AI |
MistralAI-User | Mistral AI | Fetch started by a person using Mistral |
How to use it
Separate the purpose, then choose the token.
- Use the directory as a starting point for a robots.txt decision. The provider’s current documentation stays authoritative.
- Google-Extended and Applebot-Extended are control tokens, not separate search crawlers.
- Blocking a training crawler does not require blocking search visibility.
Limitations
This page is not a live block test.
- Use the robots.txt tester to see whether a token is actually blocked on your file.
- Names and policies change. Do not treat the list as a legal opt-out by itself.
Sources
The checks follow public specifications and official documentation.
Related work
Pages that answer the same decision
robots.txt Tester
Check whether Googlebot and AI crawlers can access a URL and see the deciding rule.
robots.txt and AI Crawlers: What Should You Block?
Learn how to separate AI search, training and user-triggered bots when making robots.txt decisions.