The short answer
For most SMEs it pays to allow the AI providers' search crawlers. These include OAI-SearchBot, Claude-SearchBot and PerplexityBot. Only then can your site appear as a source in ChatGPT, Claude or Perplexity. Whether you also allow the training crawlers such as GPTBot, ClaudeBot or Google-Extended is a question of principle. According to Google and Apple, it has no influence on ranking in their search [4][5].
Important: the names and purposes of the crawlers change. Everything here is as of October 2026 and comes from the providers' documentation. Check the sources before you change your robots.txt.
Which kinds of AI crawlers are there?
Almost every major provider now runs several bots. They can be divided into three groups:
- Training crawlers. They collect content that future AI models can be trained on. If you block them, your new content no longer ends up in this training data. Data already collected isn't affected.
- Search crawlers. They build an index for web search in the AI assistant. If you block them, you're linked as a source there less often or not at all.
- User fetches. They fetch a page at the moment someone asks about it in the chat or pastes a link. That isn't crawling in the classic sense. Some providers therefore say robots.txt doesn't apply here, or only to a limited extent [1][3].
This distinction is the core of the whole decision. Many websites block "all AI bots" across the board and then wonder why ChatGPT never names them as a source.
Which crawler belongs to which provider?
The following table summarises the official information. The name in the first column is the entry you use in the robots.txt under User-agent.
| Crawler | Purpose according to the provider | Blocking it means |
|---|---|---|
| GPTBot (OpenAI) | Training the models | Content isn't used for training [1] |
| OAI-SearchBot (OpenAI) | Search in ChatGPT | Not shown in ChatGPT search answers [1] |
| ChatGPT-User (OpenAI) | Fetch at user request | robots.txt may not apply [1] |
| ClaudeBot (Anthropic) | Training the models | Future content not in training [2] |
| Claude-SearchBot (Anthropic) | Search index for Claude | Not indexed for search [2] |
| Claude-User (Anthropic) | Fetch at user request | Claude can't fetch the page [2] |
| PerplexityBot (Perplexity) | Search index, according to the provider not training | Linked as a source less often [3] |
| Perplexity-User (Perplexity) | Fetch at user request | Generally ignores robots.txt [3] |
| Google-Extended (Google) | Training and grounding for Gemini | No effect on Google Search [4] |
| Applebot-Extended (Apple) | Training Apple's models | No effect on Apple search [5] |
Two details from the documentation are practically relevant. OpenAI writes that changes to the robots.txt take about 24 hours to take effect for search [1]. Anthropic additionally supports the non-standard Crawl-delay directive, with which you can throttle the fetch rate [2].
Google-Extended and Applebot-Extended aren't crawlers of their own. They're only tokens in the robots.txt. Crawling continues to be done by Googlebot and Applebot. The entry only governs what the data may be used for [4][5]. For Google-Extended that also applies to grounding in the Gemini apps. So anyone who blocks it may appear there as a source less often [4].
What about Google and Bing?
This is where the biggest misunderstanding lies. The AI Overviews and AI Mode in Google Search use the normal Google index. For control, Google names Googlebot and the familiar snippet rules such as nosnippet or max-snippet [6]. So Google-Extended doesn't switch off AI Overviews. Anyone who doesn't want to appear there would also have to restrict the normal snippets.
It's similar with Microsoft. Copilot relies on the Bing index, built by Bingbot. There's no separate AI crawler to block. Instead Bing offers two meta values: with NOCACHE the AI chat only uses URL, title and snippet. With NOARCHIVE the page doesn't appear in chat answers at all. In normal Bing search it stays in both cases [7]. Bing announced this in 2023 for Bing Chat, Copilot's predecessor.
In short: blocking Googlebot and Bingbot almost never makes sense for an SME. You'd disappear from classic search along with it.
Which robots.txt suits you?
The robots.txt always sits in the root directory, i.e. at yourdomain.com/robots.txt. A crawler follows exactly one group of rules – the one with the most specific matching user-agent [10]. If GPTBot has its own group, it ignores the rules under User-agent: *. Here are three typical strategies, each as an example.
Strategy 1: allow everything
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xmlResult: all bots that respect robots.txt may read everything. Your content can appear in search answers and flow into training. For service providers who want to be found, it's the simplest choice. That's how tacobite.de does it too.
Strategy 2: block training, allow search
# As of October 2026 – check names regularly
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xmlResult: search crawlers such as OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and Bingbot may keep reading. Your site remains possible as a source in AI answers. New content no longer flows into training at these four providers. That fits if your content itself is your product, for example for specialist publishers or course providers. Downside: grounding in Gemini may suffer, see above.
Strategy 3: block all AI crawlers
# As of October 2026 – check names regularly
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /
User-agent: *
Allow: /Result: you largely disappear from ChatGPT, Claude and Perplexity answers. According to the provider, user fetches like Perplexity-User still don't always respect it [3]. AI Overviews on Google and Copilot answers remain possible, because Googlebot and Bingbot are still allowed. For an SME that wants to win customers, this is rarely the right choice in our view.
How do I decide in five minutes?
Answer these four questions honestly. In most cases they lead straight to one of the three strategies:
- Do you earn money with the content itself? If your texts, courses or data are the product, a lot speaks for strategy 2. If the website is mainly meant to bring enquiries, rather for strategy 1.
- Do your customers ask AI assistants for providers like you? Then you need the search crawlers. If you block OAI-SearchBot, according to OpenAI ChatGPT won't show your site in search answers [1].
- Are there areas that shouldn't be read at all? Block only those directories, such as a customer area, instead of the whole domain. Real confidentiality still needs a login.
- Who maintains the file? Decide who compares the list with the providers' pages once a quarter. New bots appear, old ones get renamed.
From our project experience, most service providers and trades businesses end up with strategy 1. Visibility is worth more to them than protecting individual texts.
What robots.txt can't do
- It isn't protection. robots.txt is a request, not a lock. Reputable crawlers respect it, others don't [9]. Anything confidential belongs behind a password.
- It doesn't remove anything from the index. A blocked URL can still appear on Google if other pages link to it. For that you need
noindex[9]. - It doesn't work retroactively. What has already been crawled stays in existing datasets.
- IP blocks aren't a better solution. Anthropic points out that an IP block prevents the bot from reading your robots.txt at all [2].
Do I need an llms.txt?
The llms.txt is a Markdown file in the root directory. It summarises what's on a website and links the most important pages. The proposal comes from Jeremy Howard in September 2024 and is explicitly an open proposal, not an adopted standard [11].
Google is clear about it: you don't need AI text files for Google Search. Search ignores llms.txt, but it doesn't do any harm either [6][8]. None of the providers named here documents on its crawler pages whether it evaluates the file.
Our honest assessment: an llms.txt costs little and can't do any harm. We maintain one ourselves. But it replaces neither good content nor a clean robots.txt. If you only have time for one, it's better invested in clear answers to real customer questions. What such content looks like is in the article Blog & content.
How do I check whether AI crawlers visit my site?
The most reliable source is your hosting's server logs. Every fetch is recorded there with the visitor's user agent. Analytics tools don't help here, because bots usually don't execute JavaScript.
- Get the logs. With most hosts you'll find the access logs in the customer menu. Sometimes logging has to be activated first.
- Filter by name. Search for the names from the table above. On a Linux server, for example,
grep -ci "gptbot" access.logis enough to count the fetches. - Look at the paths. Which pages does the bot fetch? Does it also reach your most important service pages, or only the homepage?
- Check authenticity. Anyone can fake the user agent. Google explains how to verify Googlebot via a reverse DNS lookup or via published IP lists [12]. For other bots you'll find corresponding information, where available, in their documentation.
- Test the robots.txt. After every change, open
yourdomain.com/robots.txtin the browser. One typo in a name and the rule doesn't apply.
By the way, whether Google shows your pages in AI Overviews isn't visible in the logs but in Search Console. How to set it up and read it is in the article Google Search Console for SMEs.