AI Crawler Bots: What Should You Allow, and What Should You Block?
Summary
AI crawlers fall into three types: training, AI search, and user-initiated. Even a single company, such as OpenAI or Anthropic, operates different bots for different purposes.
Blocking GPTBot (training) doesn’t stop you from managing ChatGPT search visibility separately through OAI-SearchBot. Conversely, if you block OAI-SearchBot, your site won’t appear in ChatGPT search answers.
Google-Extended and Applebot-Extended are not bots that actually visit your site. They are robots.txt tokens, so they must be configured in robots.txt rather than in server blocking rules.
robots.txt only works on bots that follow its rules. Bots that impersonate crawler names, user-initiated bots, and AI agents that don’t identify themselves must be managed through official IP verification and behavior-based policies.
Even if you allow search bots in robots.txt, bot-blocking rules in your firewall or CDN may still block those requests. Checking actual access results in your logs is the final step.
If you open your server logs, you’ll likely see a growing number of unfamiliar bot names, such as GPTBot, ClaudeBot, and PerplexityBot. Most of them are crawlers sent by AI services. The problem is that the name alone doesn’t tell you whether a bot wants to use your content for training, cite it in AI search answers, or visit on behalf of a customer. As AI becomes more accessible and AI search citations matter more for AEO and GEO, deciding which AI bots to allow is becoming increasingly important.
If you apply a blanket “block all AI bots” rule without understanding their purposes, you could lose AI search visibility. If you leave everything open for bots to explore, you also end up allowing data collection you never wanted. In this article, we organize the major AI crawlers by operator and look at the criteria for deciding whether to allow or block each one.
What Is an AI Crawler?
An AI crawler is a program that AI services use to automatically read web pages. AI crawler bots fall into three broad types depending on why they read your pages.
Type | What It Does | Impact on Your Site | Example Bots |
|---|---|---|---|
Training crawler | Collects large volumes of content for AI model training | Little to no direct traffic | GPTBot, ClaudeBot, CCBot |
AI search crawler | Indexes pages to cite in AI search answers | Source visibility in AI answers and opportunities for traffic | OAI-SearchBot, Claude-SearchBot, PerplexityBot |
User-initiated bot | Reads a page in real time when an AI user asks a question or shares a link | A channel through which real customers view your site via AI | ChatGPT-User, Claude-User, Perplexity-User |
On top of these, there are traditional search engine bots such as Googlebot and Naver’s Yeti, as well as social bots that generate link-sharing previews. The key point is that even AI bots from the same company should be handled differently if they serve different purposes.
Major AI Crawlers by Operator
Each operator publishes the purpose and IP ranges of its bots in its official documentation. The tables below are organized by the names that appear in the User-Agent field of your server logs.
OpenAI
Bot | Type | What It Does |
|---|---|---|
GPTBot | Training | Collects content that may be used for purposes such as model training |
OAI-SearchBot | AI search | Collects pages to show in ChatGPT search results |
ChatGPT-User | User-initiated | Accesses specific pages at the request of ChatGPT users |
OAI-AdsBot | Ad review | Reviews landing pages submitted as ChatGPT ads |
One thing to note here: OpenAI states that sites that block OAI-SearchBot will not appear in ChatGPT search answers. If you want to be cited in ChatGPT, check OAI-SearchBot, not GPTBot.
Anthropic (Claude)
Bot | Type | What It Does |
|---|---|---|
ClaudeBot | Training | Collects content for model training |
Claude-SearchBot | AI search | Indexes pages for Claude search answers |
Claude-User | User-initiated | Accesses pages at the request of Claude users |
Bot | Type | What It Does |
|---|---|---|
Googlebot | Search | Indexes pages for Google Search |
Google-Extended | robots.txt token | A name used to indicate that content should not be used for AI training such as Gemini. No bot visits under this name. |
GoogleOther | General-purpose | Collects content for Google’s internal products and research |
Perplexity
Bot | Type | What It Does |
|---|---|---|
PerplexityBot | AI search | Indexes pages for Perplexity search |
Perplexity-User | User-initiated | Reads pages when answering Perplexity users’ questions |
Meta
Bot | Type | What It Does |
|---|---|---|
Meta-ExternalAgent | Training | Collects content for AI training and product improvement |
Meta-ExternalFetcher | User-initiated | Fetches links at the request of Meta AI users |
facebookexternalhit | Social | Generates link previews for Facebook and Instagram |
Apple
Bot | Type | What It Does |
|---|---|---|
Applebot | Search | Powers Apple search features such as Siri and Spotlight |
Applebot-Extended | robots.txt token | A name used to indicate that content should not be used for Apple AI training. No bot visits under this name. |
Other AI Bots You May See
Bot | Operator | Type | What It Does |
|---|---|---|---|
Yeti | Naver | Search | Indexes pages for Naver Search |
bingbot | Microsoft | Search | Indexes pages for Bing Search |
CCBot | Common Crawl | Training data source | A public web archive used in many AI training datasets |
Bytespider | ByteDance | Training | Collects content for AI services |
Amazonbot | Amazon | Training and services | Collects content to improve Amazon services and AI |
DuckAssistBot | DuckDuckGo | AI search | Collects content for DuckDuckGo AI answers |
MistralAI-User | Mistral AI | User-initiated | Accesses pages at the request of Le Chat users |
Adding Google-Extended or Applebot-Extended to a blocklist won’t catch any traffic. Actual collection happens under the Googlebot and Applebot names, so these two names only take effect when configured in robots.txt.
How to Allow or Block AI Crawlers with robots.txt
robots.txt is a file placed at the root of your site (e.g., example.com/robots.txt) that tells crawlers how much of your site they may collect. Because you can write separate rules for each bot name (User-agent), you can block training while keeping search visibility open.
However, there are a few things to keep in mind when setting it up.
Changes take time to apply. According to OpenAI, it can take about 24 hours for robots.txt changes to be reflected in search results.
Allowing a crawler doesn’t guarantee visibility. Allowing a search bot is only a prerequisite for visibility. Whether your content is actually cited is up to each AI service.
It’s a good idea to block paths such as login and checkout separately. You can separate the pages you want to make public from the paths you want to protect and specify the latter with
Disallow.
Why robots.txt Alone Isn’t Enough
The important thing to remember is that robots.txt is a “request,” not “enforcement.” It only works on bots that follow its rules, and what actually blocks or allows a request is your server and security policies. That is why robots.txt alone can’t manage the following three cases.
1. It May Not Apply to User-Initiated Bots
OpenAI notes that because ChatGPT-User accesses pages at a user’s direct request, robots.txt rules may not apply to it. User-initiated bots from other AI services are often in a similar situation. Since these bots are also a channel through which real customers reach your site via AI, it’s better to monitor which pages they visit and what they do there rather than blocking them outright.
2. Some Bots Simply Borrow AI Crawler Names
Anyone can change the name in a User-Agent string. If a malicious scraping bot arrives using the GPTBot or Googlebot name, robots.txt and name-based rules can’t tell the difference. That is why major operators publish the official IP ranges of their bots. Whether a bot is a genuine AI crawler should be verified by its source IP, not by its name.
3. More AI Agents Don’t Identify Themselves
Some AI browsers and agents access sites like ordinary browsers, without a bot name. Neither robots.txt nor bot name lists apply to this kind of traffic, so it must be evaluated based on behavior, such as request speed and navigation paths.
There is also a problem in the opposite direction: you may allow search bots in robots.txt while bot-blocking rules in your firewall or CDN are still blocking those requests. This is why OpenAI recommends allowing requests from its published IP ranges in addition to allowing its bots in robots.txt.
Which Bots Should Your Site Allow, and Which Should It Block?
The answer depends on what your site values most. The table below provides examples to help you decide.
Type | AI search visibility matters most (brands, B2B, media) | Content and data protection matters most (news, pricing, databases) | Transaction fairness matters most (ticketing, commerce, finance) |
|---|---|---|---|
General search bots | Allow | Allow | Allow |
AI search crawlers | Allow | Monitor, then decide | Allow (public pages) |
Training crawlers | Decide based on your content policy | Block | Monitor or block |
User-initiated bots | Monitor | Monitor | Monitor, with separate controls for login and checkout |
Social preview bots | Allow | Allow | Allow |
If you’re not sure, it’s safer not to block right away, but to spend two to four weeks checking which bots visit, how often, and which pages they access. Once you see the actual numbers, you can clearly distinguish between bots you need for visibility and bots that only add load.
Be especially careful with user-initiated bots. Because there is a real person behind them, blocking them entirely also blocks customers who want to view your site through AI. In this case, one approach is to keep access open while separately filtering out abnormal repeated requests in sensitive sections such as ticketing, checkout, and sign-up.
AI Bots: Not ‘Should We Block Them?’ but ‘What Should We Open, and to Whom?’
As AI continues to evolve, AI crawler traffic is likely to grow in volume and become more varied. Once you know the names and purposes of these bots, you can configure settings that protect both your search visibility and your content.
However, as the number of bots grows, it becomes difficult to keep making these decisions manually. Ongoing security attacks are also adding to these concerns.
That is why you need a solution that can verify major AI crawlers against their operators’ official IP ranges and help you decide whether to allow, detect, or block each bot. Do you know which AI bots are visiting your site, and how often?
FAQ
If I block GPTBot, will my site also disappear from ChatGPT search?
No. ChatGPT search visibility is handled by OAI-SearchBot. Even if you block GPTBot, your site can still appear in search answers as long as you allow OAI-SearchBot. However, whether it appears is up to OpenAI.
What is the difference between ClaudeBot and Claude-User?
ClaudeBot collects content for model training, while Claude-User reads pages when a Claude user requests it. Claude search indexing is handled separately by Claude-SearchBot.
If I block Google-Extended in robots.txt, will my site disappear from Google Search?
No. Google-Extended is a token that only indicates whether your content may be used for AI training. It has no effect on Googlebot, which handles Google Search indexing.
Is configuring robots.txt enough?
It’s enough for bots that follow its rules, but bots that impersonate crawler names, user-initiated bots, and agents that don’t identify themselves need to be managed separately. Also check that your firewall or bot management settings aren’t blocking the bots you’ve allowed.
How can I tell whether a bot in my server logs is the real GPTBot?
Check whether the source IP falls within the official IP ranges published by the operator. Major operators such as OpenAI and Google publish IP lists for each of their bots.