Behlül Göktepe
Free technical SEO tool

robots.txt Generator

Separate search crawlers from AI training crawlers in one readable file.

RFC 9309Purpose splitFree

robots.txt

Choose settings and generate the file.
How this works

Presets follow what the token is for.

  • The training preset disallows GPTBot, ClaudeBot, Google-Extended, CCBot and the other training or control tokens.
  • That same preset leaves search and answer tokens such as Googlebot, Bingbot and OAI-SearchBot allowed.
  • Blocking every non-search AI token also blocks user-triggered fetchers, which can break a person asking a model to open your page.
Limitations

This file is not an access-control firewall.

  • robots.txt asks compliant crawlers to behave. It does not protect logged-in, paid or private URLs.
  • Block everything ignores extra paths and closes the wildcard group.
  • Test the exact tokens you care about before publishing. Provider names change.
Data approach

Submitted addresses are not stored to build a persistent user profile.

A URL check sends a server-side request to the target site. Do not submit local-network, metadata, authenticated or private addresses.

Sources

The checks follow public specifications and official documentation.

Related work

Pages that answer the same decision

Tool

robots.txt Tester

Check whether Googlebot and AI crawlers can access a URL and see the deciding rule.

Tool

AI Crawler Directory

Compare search, answer, training and user-triggered AI crawlers in one directory.