---
title: "Free AI Bot Checker: Is GPTBot Blocked on Your Site?"
description: "Check which AI crawlers your robots.txt allows or blocks: GPTBot, ClaudeBot, PerplexityBot, Google-Extended and more, with the rule that decides each."
canonical: https://luckpanda.app/chatbot/tools/ai-crawler-checker
updated: 2026-10-14
---

# AI bot checker: which AI crawlers can read your website?

Canonical URL: https://luckpanda.app/chatbot/tools/ai-crawler-checker
Language: en
Updated October 14, 2026

Enter your website address. The checker reads your robots.txt and your page’s robots tags, and shows for 16 AI crawlers and agents whether each is allowed, partly blocked or blocked, and by which rule.

Free tool: https://luckpanda.app/chatbot/tools/ai-crawler-checker (analyzer, website)

Agents: run it over MCP (https://luckpanda.app/mcp) with chatbot_public_tool_run { "tool": "ai-crawler-checker", "lang": "en", "inputs": { … } }; chatbot_public_tool_list gives the inputs.

## What the AI bot checker does

> An AI bot checker reads your site’s robots.txt and tells you which AI crawlers may read your pages. This one checks 16 crawlers and agents from OpenAI, Anthropic, Perplexity, Google, Apple, Amazon, DuckDuckGo, Meta, Common Crawl, ByteDance and LuckPanda. Each shows as Allowed, Partly or Blocked, with the robots.txt rule that decided it. It also flags noai tags on your page.

1. **Enter your address.** Your home page, or a page you care about, such as your blog or your pricing page.
2. **The checker reads two files.** Your /robots.txt and the page itself, for its robots meta tag and X-Robots-Tag header.
3. **Read each bot’s seal.** Bots are grouped by what they do: AI search, model training, fetches for a user’s question, and website chatbots.
4. **Check the matched rule.** Under each bot you see the user-agent group that applies to it, so you know which lines to edit.
5. **Edit robots.txt and check again.** Use the examples below, publish the file, and run the check again to confirm.

## Search, training and user fetches: three kinds of AI bot

Most AI companies now run more than one bot, each with its own robots.txt token. Blocking one does not block the others, which is why the difference matters.

**AI search**
- Reads pages so an AI search product can show and link them in answers.
- Examples: OAI-SearchBot, Claude-SearchBot, PerplexityBot.
- Block it and your pages stop appearing in that product’s answers.

**Model training**
- Collects pages that may be used to train future AI models.
- Examples: GPTBot, ClaudeBot, Google-Extended, CCBot.
- Block it to opt out from now on. It does not undo past collection.

**Fetches for a user’s question**
- Visits a page when a person asks an assistant about it or pastes its link.
- Examples: ChatGPT-User, Claude-User, Perplexity-User.
- Some companies say robots.txt may not apply to these, because a person asked.

## Every bot the checker tests

The tokens below are the names each company documents for robots.txt. Write them exactly as shown in a User-agent line. Matching is not case sensitive.

| Company | robots.txt token | What it does | Group |
| --- | --- | --- | --- |
| OpenAI | OAI-SearchBot | Finds pages to show and link in ChatGPT search results. | AI search |
| OpenAI | ChatGPT-User | Visits a page when a ChatGPT user asks for it. | User fetch |
| OpenAI | GPTBot | Collects pages that may be used to train OpenAI’s models. | Training |
| Anthropic | Claude-SearchBot | Reads pages to improve the search results in Claude. | AI search |
| Anthropic | Claude-User | Visits a page when a Claude user asks for it. | User fetch |
| Anthropic | ClaudeBot | Collects pages that may be used to train Anthropic’s models. | Training |
| Perplexity | PerplexityBot | Indexes pages for Perplexity’s search answers. | AI search |
| Perplexity | Perplexity-User | Visits a page when a Perplexity user asks about it. | User fetch |
| Google | Google-Extended | A control token, not a separate crawler: says whether pages Googlebot collects may train Gemini models. Google Search is not affected. | Training |
| Apple | Applebot-Extended | A control token, not a separate crawler: says whether pages Applebot collects may train Apple’s AI models. | Training |
| Amazon | Amazonbot | Amazon’s crawler, used for Amazon services such as Alexa’s answers. | AI search |
| DuckDuckGo | DuckAssistBot | Fetches pages for DuckAssist, DuckDuckGo’s AI answers. | AI search |
| Meta | meta-externalagent | Meta’s crawler for training AI models and improving its products. | Training |
| Common Crawl | CCBot | Builds a free, open archive of the web that many AI models are trained on. | Training |
| ByteDance | Bytespider | ByteDance’s web crawler, used to collect data for its AI models. | Training |
| LuckPanda | LuckPandaBot | Reads a site when someone builds a LuckPanda chatbot from it or runs one of these free tools on it. | Website chatbot |

Tokens as each company documents them, checked October 2026. Companies add and rename bots, so check their documentation before relying on a list.

## How to read the result

- Allowed: the bot may read your home page and the sample paths the checker tries (/blog/, /products/, /pricing, /docs/, /help/ and the page you entered).
- Partly: the home page is allowed, but at least one of those paths is disallowed. That is often on purpose, for example a private /docs/ area.
- Blocked: the bot may not read your home page. In practice it is kept off the whole site.
- Matched rule: the user-agent line of the group that applies. A group that names the bot wins over “User-agent: *”. A bot follows only one group, so the rules under * no longer apply to it once it has its own.
- No robots.txt: every bot shows Allowed, because nothing asks them to stay out, and the file check asks you to review it.
- noai tags: shown for review when your page’s robots meta tag or X-Robots-Tag header contains noai or noimageai.

The checker reads rules, not behavior. It cannot see a firewall, CDN or hosting setting that blocks bots before they reach robots.txt, and it cannot tell whether a bot actually obeys your file. If a bot shows Allowed here but never visits, look at your host’s bot or security settings too.

## Should you block AI crawlers? The trade-offs

There is no single right answer. It depends on whether you want to be found in AI answers, and how you feel about your work training models.

| Choice | What you keep | What you give up |
| --- | --- | --- |
| Allow everything | The widest chance of appearing in AI answers and assistants. | Your pages may be used to train models. |
| Block training, allow search and user fetches | You can still appear and be linked in AI search, and assistants can open your pages when asked. | Little for most businesses. Some companies use one bot for both jobs, so the split is not always clean. |
| Block search bots too | Your pages stay out of those AI search products. | Visitors who ask those assistants are less likely to find you. |
| Block everything AI | The strongest signal you can send with robots.txt. | Visibility in AI search and assistants, and the option of a website chatbot built from your pages. |

> Two facts to keep in mind. robots.txt is a request: well-behaved crawlers follow it, but nothing forces a bot to. And a block only works from the day you add it. It does not remove content a company has already collected, or take it out of models already trained.

To stay out of Google’s AI features in Search, Google-Extended is not the switch: those use Googlebot, which also runs normal Search. Google’s snippet controls, such as nosnippet, limit what it can show, in AI features and in regular results alike.

## robots.txt examples

Copy one, replace the sitemap address with yours and publish it at the root of your site, for example yourbusiness.com/robots.txt. Each subdomain needs its own file.

**Block AI training, allow AI search and user fetches**

```text
# AI model training: no
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: CCBot
User-agent: Bytespider
Disallow: /

# AI search and fetches for a user's question: yes
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: DuckAssistBot
Disallow: /checkout/
Disallow: /account/

# Everyone else
User-agent: *
Disallow: /checkout/
Disallow: /account/

Sitemap: https://www.yourbusiness.com/sitemap.xml
```

The private paths are repeated in the AI search group, because a bot that has its own group ignores the rules under *.

**Allow every AI bot, keep private areas out**

```text
User-agent: *
Disallow: /checkout/
Disallow: /account/
Disallow: /admin/

Sitemap: https://www.yourbusiness.com/sitemap.xml
```

**Block every AI bot on this page’s list**

```text
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Amazonbot
User-agent: DuckAssistBot
User-agent: meta-externalagent
User-agent: CCBot
User-agent: Bytespider
Disallow: /

User-agent: *
Disallow: /admin/
```

This leaves out LuckPandaBot, so you can still build a website chatbot from your pages. Add it to the list if you want it blocked too.

To check by hand whether GPTBot is blocked, open yourbusiness.com/robots.txt and look for a group that starts with “User-agent: GPTBot”. If there is one, its rules apply. If not, the rules under “User-agent: *” do. A line “Disallow: /” in the group that applies blocks the whole site.

## The noai tag and its limits

Some sites add “noai” or “noimageai” to their robots meta tag or X-Robots-Tag header to say their content should not be used for AI.

**noai as a meta tag and as a header**

```text
<meta name="robots" content="noai, noimageai">

X-Robots-Tag: noai, noimageai
```

These values are not part of any robots standard, and few crawlers are documented to honor them. The AI companies in the table above publish robots.txt tokens as their opt-out, so treat noai as a statement of your wishes, not a block. If you add it, make sure you did not also write noindex: that removes the page from search engines entirely.

## Common mistakes

- Blocking OAI-SearchBot when you meant GPTBot. The first keeps you out of ChatGPT search, the second only out of training.
- Shipping a staging robots.txt with “Disallow: /” for every user agent. It blocks search engines too.
- A robots.txt address that returns your home page’s HTML, which happens on some single-page apps. Crawlers find no rules, and the checker treats it as no file.
- Giving a bot its own group and forgetting that it then ignores the rules under *.
- Editing robots.txt on www.yourbusiness.com and forgetting shop.yourbusiness.com. Every host has its own file.
- Blocking Googlebot to stay out of AI answers. It takes you out of Google Search as well.
- Expecting a new block to remove your pages from answers that already use them. It only affects what is collected next.
- Turning on a host’s or CDN’s one-click “block AI bots” setting without checking what it covers. Some block search bots and user fetches too.

## What sites usually allow, by type

Common choices, not rules. Decide with whoever owns your content and your website.

| Site | Common choice | Why |
| --- | --- | --- |
| Online store | Allow AI search and user fetches | Product pages work like ads: being named in an answer can bring a buyer. |
| Local service business | Allow AI search and user fetches | People ask assistants for a nearby plumber, dentist or tutor, and want hours and prices. |
| Publisher or blog | Often block training, allow search | The writing is the product, but links from AI search still bring readers. |
| Software company | Allow search and user fetches for docs and pricing | Buyers ask assistants to compare tools and explain setup. |
| Membership or course site | Block paid areas for all bots | Paid content should sit behind a login anyway; robots.txt does not protect it. |

## Your own website chatbot and robots.txt

LuckPandaBot is the crawler behind LuckPanda’s website agents. It reads your site when someone builds an agent from it or runs one of these free tools on it. When it builds an agent, it follows your robots.txt.

If you block it, a support agent cannot learn your pages, but you can still upload them as documents. If you allow it, the agent reads up to 50 pages, skips anything robots.txt or a noindex tag keeps out, and answers your visitors from what it read, day and night.

**Build a support agent from your pages** Paste your address, watch the agent learn your site, and ask it what your customers ask. [See the AI customer service agent](https://luckpanda.app/chatbot/ai-customer-service-agent)

## Questions people ask

### How do I check if GPTBot is blocked?

Enter your address in the checker above and look at the GPTBot row: Allowed, Partly or Blocked, with the rule that decides it. By hand, open yoursite.com/robots.txt and find the group for GPTBot, or the * group if there is none. “Disallow: /” there blocks it.

### Does blocking GPTBot remove my site from ChatGPT?

No. GPTBot collects pages for training. ChatGPT search uses OAI-SearchBot, and pages a user asks ChatGPT to open are fetched by ChatGPT-User. Block only GPTBot and you opt out of future training while staying visible in ChatGPT search.

### Does robots.txt stop AI companies from using my content?

Only partly. robots.txt is a request that well-behaved crawlers follow. It works from the day you add the rule, and it does not remove content already collected or used in trained models. For content that must stay private, put it behind a login.

### Will blocking Google-Extended hurt my Google rankings?

Google says Google-Extended does not affect inclusion or ranking in Google Search. It only controls whether pages Googlebot collects may be used for Gemini models. It also does not keep pages out of AI features in Google Search, which use Googlebot.

### What is the difference between ClaudeBot and Claude-User?

ClaudeBot collects pages that may be used to train Anthropic’s models. Claude-User visits a page when a person using Claude asks about it. Claude-SearchBot reads pages for search results in Claude. Each has its own robots.txt token.

### What does the noai meta tag do?

It states that a page should not be used for AI. It is not part of any standard and few crawlers are documented to honor it, so use robots.txt rules for the specific bots you want to stop.

Try it on your website: https://luckpanda.app/chatbot/try