Article

robots.txt vs AI crawlers: training bots, search bots, and the ones that just fetch

Published October 26, 2026. Topics: robots txt, ai crawlers, ai search, technical seo.

robots.txt is a small text file with a big job: telling crawlers where they’re welcome. It’s also the file most likely to have been edited once, in 2014, by someone who has since moved to Montana.

AI has made it more interesting. There are now several kinds of bots with different purposes, and blocking the wrong one can quietly remove you from places you’d like to be. Here’s how to tell them apart and read your own file.

Three kinds of AI-related bots

The big AI companies now split their bots by purpose. Roughly:

1. Training crawlers

These collect public web content that may be used to train AI models.

• GPTBot (OpenAI): OpenAI says disallowing it “indicates a site’s content should not be used in training generative AI foundation models.” (OpenAI)
• ClaudeBot (Anthropic): Anthropic says restricting it signals that future materials should be excluded from training. (Anthropic)
• Google-Extended (Google): not a separate crawler but a robots.txt token. Google says it controls whether content it crawls may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI, and that it doesn’t affect inclusion or ranking in Google Search. (Google)

2. Search crawlers

These index pages so an AI search product can find and cite them.

• OAI-SearchBot (OpenAI): OpenAI says sites opted out of it “will not be shown in ChatGPT search answers, though can still appear as navigational links.”
• Claude-SearchBot (Anthropic): Anthropic says disabling it may reduce your visibility in its search results.
• Googlebot: Google says its AI Overviews and AI Mode have no additional requirements beyond normal Search, so Googlebot access is what matters there. (Google)

3. User-triggered fetchers

These visit a page because a person asked an assistant to look at something right now.

• ChatGPT-User (OpenAI): OpenAI says that because these actions are user-initiated, “robots.txt rules may not apply,” and that it isn’t used to decide search inclusion.
• Claude-User (Anthropic): Anthropic says disabling it prevents retrieval in response to a user query.
• Google’s user-triggered fetchers: Google says they “generally ignore robots.txt rules” because a user requested the fetch. (Google)

The key idea: these are separate choices. You can opt out of training and still allow search. Whether you should is your call; there’s no single right answer.

How to read your robots.txt

Open https://yoursite.com/robots.txt in a browser. The rules are standardized in RFC 9309. A few things the standard says that trip people up:

• Groups. A file is made of groups. Each starts with one or more User-agent: lines, followed by Allow: and Disallow: rules.
• A crawler follows the group that names it. It only falls back to the User-agent: * group if no group matches it. So if you have a GPTBot group, GPTBot ignores your * rules entirely.
• Most specific match wins. When Allow and Disallow both match a URL, the rule with the longer path wins.
• No file is usually fine. If robots.txt returns a 4xx error (like 404), crawlers may access anything. If it returns a 5xx server error, crawlers must assume everything is disallowed. A broken server can hide you more effectively than any rule.
• It’s not security. RFC 9309 says these rules “are not a form of access authorization.” Listing a secret path in robots.txt is like hanging a sign that says “definitely nothing valuable in this drawer.”

Example snippets

These are examples, not recommendations. Adapt them to your site and your goals.

Example A: allow everything.

User-agent: *
Allow: /

Example B: allow search crawlers, opt out of some training.

# Example only
User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /

User-agent: *
Allow: /
Disallow: /wp-admin/

Sitemap: https://www.example.com/sitemap.xml

Because OAI-SearchBot, Claude-SearchBot and Googlebot don’t match the first group, they fall back to the * group and are allowed.

Example C: the accidental vanishing act.

# Don't copy this unless you mean it
User-agent: *
Disallow: /

This asks every well-behaved crawler, Googlebot included, to stay out of the whole site. It’s often left over from a staging site that went live.

Common pitfalls

1. The blanket Disallow. Example C, still sitting there months after launch.
2. Blocking search along with training. A list like User-agent: GPTBot plus User-agent: OAI-SearchBot plus Disallow: / goes further than “don’t train on my content.”
3. Forgetting the named group overrides *. You add a Googlebot group with one rule and accidentally drop all your * rules for Googlebot.
4. The firewall disagrees. Your host, CDN or security plugin can block bots outright, whatever robots.txt says. Anthropic also notes that blocking its IPs can stop it from reading your robots.txt at all.
5. Expecting instant results. OpenAI says changes can take about 24 hours to be reflected for search. Others may take longer.
6. Treating robots.txt as a guarantee. It’s a request that well-behaved crawlers honor. And allowing a crawler doesn’t mean any assistant will mention you.

This article is general information, not legal or professional advice. Bot names and behaviors change; check each company’s documentation before you edit your file.

Check yours in a minute

Reading robots.txt by hand is fine; having a tool do it is faster. Our free AI Crawler Checker fetches your robots.txt and shows, bot by bot, which AI search and training crawlers it allows or blocks. No sign-up. If the file looks fine but bots still can’t reach you, see our guide on unblocking AI crawlers at the firewall or CDN.

All articles