How does Google read robots.txt?

Each crawler looks for its own group. A crawler uses the group whose User-agent line names it; multiple groups for the same name are merged. If it has no group of its own, it follows the * group. Googlebot-Image and Googlebot-News use Googlebot's rules when they have no group of their own.
The longest rule wins. Of the rules that match an address, the longest (most specific) one applies: with Disallow: /blog/ and Allow: /blog/rehber/ together, /blog/rehber/ can be crawled. If the lengths are equal, Allow wins.
The * and $ characters. * means any sequence of characters and $ means the end of the address: Disallow: /*.pdf$ blocks all PDFs, Disallow: /*? blocks addresses with parameters.
Blocked ≠ not indexed. robots.txt blocks crawling, not indexing: a blocked page that gets links from other sites can still appear without a description, even though its content is never read. To remove it from the index, allow crawling and use noindex.
The status code matters. If robots.txt returns a 4xx (except 429), Google treats everything as crawlable; if it returns a 5xx or 429, Google temporarily treats the whole site as blocked. Only the first 500 KiB of the file is read.

What does each bot do?

The tool tests all of the crawlers below; you can add one that isn't listed from the results screen.

Bot (User-agent)	Type	What does it crawl for?
`Googlebot`	Search engine	Google Search
`Googlebot-Image`	Search engine	Google Images (Googlebot rules if it has no group)
`Googlebot-News`	Search engine	Google News (Googlebot rules if it has no group)
`Bingbot`	Search engine	Bing; Microsoft Copilot answers use this index too
`YandexBot`	Search engine	Yandex
`Applebot`	Search engine	Apple (Siri, Spotlight, Safari suggestions)
`OAI-SearchBot`	AI search (GEO)	Citing sources in ChatGPT search results
`Claude-SearchBot`	AI search (GEO)	Citing sources in Claude search results
`PerplexityBot`	AI search (GEO)	Citing sources in Perplexity search results
`ChatGPT-User`	User request	Opening a page a user asks for in ChatGPT
`Claude-User`	User request	Opening a page a user asks for in Claude
`Perplexity-User`	User request	Opening a page a user asks for in Perplexity
`GPTBot`	Model training	OpenAI model training
`ClaudeBot`	Model training	Anthropic model training
`Google-Extended`	Model training	Gemini training and grounding (doesn't affect search rankings)
`Applebot-Extended`	Model training	Apple Intelligence training
`CCBot`	Model training	Common Crawl open dataset (many models are trained on it)
`meta-externalagent`	Model training	Meta AI models
`Bytespider`	Model training	ByteDance (TikTok) AI models
`Amazonbot`	Model training	Amazon (Alexa and AI services)

Frequently asked questions

Should I block AI crawlers?

Search and answer crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) crawl your site to cite it as a source in AI answers; blocking them costs you GEO visibility. Blocking model-training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended), on the other hand, is a choice about whether your content goes into training data and doesn't affect search rankings.

Will blocking Google-Extended hurt my Google rankings?

No. Google-Extended isn't a separate crawler; it's a token that decides whether content collected by Googlebot can be used for Gemini training and grounding. It doesn't affect visibility in Google Search.

How do I test a change before publishing it?

Use the “Edit and test” button to change the file on the page; the matrix updates as you type. The file on your site isn't changed. If you're happy with the result, save it with “Download” and upload it to your server.

Why can I test robots.txt content that isn't on my site?

With the “Paste robots.txt” option you can test a file that isn't live yet, or text a client sent you, directly.

To see the technical health of your whole site, use the Technical SEO Audit tool.

Robots.txt Tester