English

Free tool

AI bot checker: which AI crawlers can read your website?

Enter your website address. The checker reads your robots.txt and your page’s robots tags, and shows for 16 AI crawlers and agents whether each is allowed, partly blocked or blocked, and by which rule.

What the AI bot checker does

An AI bot checker reads your site’s robots.txt and tells you which AI crawlers may read your pages. This one checks 16 crawlers and agents from OpenAI, Anthropic, Perplexity, Google, Apple, Amazon, DuckDuckGo, Meta, Common Crawl, ByteDance and LuckPanda. Each shows as Allowed, Partly or Blocked, with the robots.txt rule that decided it. It also flags noai tags on your page.

Enter your addressYour home page, or a page you care about, such as your blog or your pricing page.
The checker reads two filesYour /robots.txt and the page itself, for its robots meta tag and X-Robots-Tag header.
Read each bot’s sealBots are grouped by what they do: AI search, model training, fetches for a user’s question, and website chatbots.
Check the matched ruleUnder each bot you see the user-agent group that applies to it, so you know which lines to edit.
Edit robots.txt and check againUse the examples below, publish the file, and run the check again to confirm.

Search, training and user fetches: three kinds of AI bot

Most AI companies now run more than one bot, each with its own robots.txt token. Blocking one does not block the others, which is why the difference matters.

AI search

Reads pages so an AI search product can show and link them in answers.
Examples: OAI-SearchBot, Claude-SearchBot, PerplexityBot.
Block it and your pages stop appearing in that product’s answers.

Model training

Collects pages that may be used to train future AI models.
Examples: GPTBot, ClaudeBot, Google-Extended, CCBot.
Block it to opt out from now on. It does not undo past collection.

Fetches for a user’s question

Visits a page when a person asks an assistant about it or pastes its link.
Examples: ChatGPT-User, Claude-User, Perplexity-User.
Some companies say robots.txt may not apply to these, because a person asked.

Every bot the checker tests

The tokens below are the names each company documents for robots.txt. Write them exactly as shown in a User-agent line. Matching is not case sensitive.

Company	robots.txt token	What it does	Group
OpenAI	OAI-SearchBot	Finds pages to show and link in ChatGPT search results.	AI search
OpenAI	ChatGPT-User	Visits a page when a ChatGPT user asks for it.	User fetch
OpenAI	GPTBot	Collects pages that may be used to train OpenAI’s models.	Training
Anthropic	Claude-SearchBot	Reads pages to improve the search results in Claude.	AI search
Anthropic	Claude-User	Visits a page when a Claude user asks for it.	User fetch
Anthropic	ClaudeBot	Collects pages that may be used to train Anthropic’s models.	Training
Perplexity	PerplexityBot	Indexes pages for Perplexity’s search answers.	AI search
Perplexity	Perplexity-User	Visits a page when a Perplexity user asks about it.	User fetch
Google	Google-Extended	A control token, not a separate crawler: says whether pages Googlebot collects may train Gemini models. Google Search is not affected.	Training
Apple	Applebot-Extended	A control token, not a separate crawler: says whether pages Applebot collects may train Apple’s AI models.	Training
Amazon	Amazonbot	Amazon’s crawler, used for Amazon services such as Alexa’s answers.	AI search
DuckDuckGo	DuckAssistBot	Fetches pages for DuckAssist, DuckDuckGo’s AI answers.	AI search
Meta	meta-externalagent	Meta’s crawler for training AI models and improving its products.	Training
Common Crawl	CCBot	Builds a free, open archive of the web that many AI models are trained on.	Training
ByteDance	Bytespider	ByteDance’s web crawler, used to collect data for its AI models.	Training
LuckPanda	LuckPandaBot	Reads a site when someone builds a LuckPanda chatbot from it or runs one of these free tools on it.	Website chatbot

Tokens as each company documents them, checked October 2026. Companies add and rename bots, so check their documentation before relying on a list.

How to read the result

Allowed: the bot may read your home page and the sample paths the checker tries (/blog/, /products/, /pricing, /docs/, /help/ and the page you entered).
Partly: the home page is allowed, but at least one of those paths is disallowed. That is often on purpose, for example a private /docs/ area.
Blocked: the bot may not read your home page. In practice it is kept off the whole site.
Matched rule: the user-agent line of the group that applies. A group that names the bot wins over “User-agent: *”. A bot follows only one group, so the rules under * no longer apply to it once it has its own.
No robots.txt: every bot shows Allowed, because nothing asks them to stay out, and the file check asks you to review it.
noai tags: shown for review when your page’s robots meta tag or X-Robots-Tag header contains noai or noimageai.

The checker reads rules, not behavior. It cannot see a firewall, CDN or hosting setting that blocks bots before they reach robots.txt, and it cannot tell whether a bot actually obeys your file. If a bot shows Allowed here but never visits, look at your host’s bot or security settings too.

Should you block AI crawlers? The trade-offs

There is no single right answer. It depends on whether you want to be found in AI answers, and how you feel about your work training models.

Choice	What you keep	What you give up
Allow everything	The widest chance of appearing in AI answers and assistants.	Your pages may be used to train models.
Block training, allow search and user fetches	You can still appear and be linked in AI search, and assistants can open your pages when asked.	Little for most businesses. Some companies use one bot for both jobs, so the split is not always clean.
Block search bots too	Your pages stay out of those AI search products.	Visitors who ask those assistants are less likely to find you.
Block everything AI	The strongest signal you can send with robots.txt.	Visibility in AI search and assistants, and the option of a website chatbot built from your pages.

Two facts to keep in mind. robots.txt is a request: well-behaved crawlers follow it, but nothing forces a bot to. And a block only works from the day you add it. It does not remove content a company has already collected, or take it out of models already trained.

To stay out of Google’s AI features in Search, Google-Extended is not the switch: those use Googlebot, which also runs normal Search. Google’s snippet controls, such as nosnippet, limit what it can show, in AI features and in regular results alike.

robots.txt examples

Copy one, replace the sitemap address with yours and publish it at the root of your site, for example yourbusiness.com/robots.txt. Each subdomain needs its own file.

Block AI training, allow AI search and user fetches

# AI model training: no
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: CCBot
User-agent: Bytespider
Disallow: /

# AI search and fetches for a user's question: yes
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: DuckAssistBot
Disallow: /checkout/
Disallow: /account/

# Everyone else
User-agent: *
Disallow: /checkout/
Disallow: /account/

Sitemap: https://www.yourbusiness.com/sitemap.xml

The private paths are repeated in the AI search group, because a bot that has its own group ignores the rules under *.

Allow every AI bot, keep private areas out

User-agent: *
Disallow: /checkout/
Disallow: /account/
Disallow: /admin/

Sitemap: https://www.yourbusiness.com/sitemap.xml

Block every AI bot on this page’s list

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Amazonbot
User-agent: DuckAssistBot
User-agent: meta-externalagent
User-agent: CCBot
User-agent: Bytespider
Disallow: /

User-agent: *
Disallow: /admin/

This leaves out LuckPandaBot, so you can still build a website chatbot from your pages. Add it to the list if you want it blocked too.

To check by hand whether GPTBot is blocked, open yourbusiness.com/robots.txt and look for a group that starts with “User-agent: GPTBot”. If there is one, its rules apply. If not, the rules under “User-agent: *” do. A line “Disallow: /” in the group that applies blocks the whole site.

The noai tag and its limits

Some sites add “noai” or “noimageai” to their robots meta tag or X-Robots-Tag header to say their content should not be used for AI.

noai as a meta tag and as a header

<meta name="robots" content="noai, noimageai">

X-Robots-Tag: noai, noimageai

These values are not part of any robots standard, and few crawlers are documented to honor them. The AI companies in the table above publish robots.txt tokens as their opt-out, so treat noai as a statement of your wishes, not a block. If you add it, make sure you did not also write noindex: that removes the page from search engines entirely.

Common mistakes

Blocking OAI-SearchBot when you meant GPTBot. The first keeps you out of ChatGPT search, the second only out of training.
Shipping a staging robots.txt with “Disallow: /” for every user agent. It blocks search engines too.
A robots.txt address that returns your home page’s HTML, which happens on some single-page apps. Crawlers find no rules, and the checker treats it as no file.
Giving a bot its own group and forgetting that it then ignores the rules under *.
Editing robots.txt on www.yourbusiness.com and forgetting shop.yourbusiness.com. Every host has its own file.
Blocking Googlebot to stay out of AI answers. It takes you out of Google Search as well.
Expecting a new block to remove your pages from answers that already use them. It only affects what is collected next.
Turning on a host’s or CDN’s one-click “block AI bots” setting without checking what it covers. Some block search bots and user fetches too.

What sites usually allow, by type

Common choices, not rules. Decide with whoever owns your content and your website.

Site	Common choice	Why
Online store	Allow AI search and user fetches	Product pages work like ads: being named in an answer can bring a buyer.
Local service business	Allow AI search and user fetches	People ask assistants for a nearby plumber, dentist or tutor, and want hours and prices.
Publisher or blog	Often block training, allow search	The writing is the product, but links from AI search still bring readers.
Software company	Allow search and user fetches for docs and pricing	Buyers ask assistants to compare tools and explain setup.
Membership or course site	Block paid areas for all bots	Paid content should sit behind a login anyway; robots.txt does not protect it.

Your own website chatbot and robots.txt

LuckPandaBot is the crawler behind LuckPanda’s website agents. It reads your site when someone builds an agent from it or runs one of these free tools on it. When it builds an agent, it follows your robots.txt.

If you block it, a support agent cannot learn your pages, but you can still upload them as documents. If you allow it, the agent reads up to 50 pages, skips anything robots.txt or a noindex tag keeps out, and answers your visitors from what it read, day and night.

Build a support agent from your pages

Paste your address, watch the agent learn your site, and ask it what your customers ask.

See the AI customer service agent

Questions people ask

How do I check if GPTBot is blocked?

Enter your address in the checker above and look at the GPTBot row: Allowed, Partly or Blocked, with the rule that decides it. By hand, open yoursite.com/robots.txt and find the group for GPTBot, or the * group if there is none. “Disallow: /” there blocks it.

Does blocking GPTBot remove my site from ChatGPT?

No. GPTBot collects pages for training. ChatGPT search uses OAI-SearchBot, and pages a user asks ChatGPT to open are fetched by ChatGPT-User. Block only GPTBot and you opt out of future training while staying visible in ChatGPT search.

Does robots.txt stop AI companies from using my content?

Only partly. robots.txt is a request that well-behaved crawlers follow. It works from the day you add the rule, and it does not remove content already collected or used in trained models. For content that must stay private, put it behind a login.

Will blocking Google-Extended hurt my Google rankings?

Google says Google-Extended does not affect inclusion or ranking in Google Search. It only controls whether pages Googlebot collects may be used for Gemini models. It also does not keep pages out of AI features in Google Search, which use Googlebot.

What is the difference between ClaudeBot and Claude-User?

ClaudeBot collects pages that may be used to train Anthropic’s models. Claude-User visits a page when a person using Claude asks about it. Claude-SearchBot reads pages for search results in Claude. Each has its own robots.txt token.

What does the noai meta tag do?

It states that a page should not be used for AI. It is not part of any standard and few crawlers are documented to honor it, so use robots.txt rules for the specific bots you want to stop.

Keep reading

For AI agents

Build it with your AI agent

Give an agent your site and it can create the chatbot, read your pages and test it through LuckPanda's API and MCP. It prepares the work; going live stays with you.

Use LuckPanda's MCP server (https://luckpanda.app/mcp) to build a website chatbot for me. Ask for my website address, call chatbot_create_from_website with it, wait until the pages are learned (chatbot_source_list), then run a few test questions with chatbot_test_message and show me the answers. Do not put it live: I will do that myself.

Instructions for AI agents

Needs a LuckPanda agent key (Settings › Developers) with the chatbot.manage grant. MCP endpoint: https://luckpanda.app/mcp (Streamable HTTP, bearer key).

chatbot_create_from_website { url }: creates a draft chatbot named after the site and starts reading it.
chatbot_source_list: pages learned, failed or skipped, with a reason for each.
chatbot_test_message: asks the bot a question as the owner. Test chats never email anyone.
Going live (chatbot_update with status live) is a person's action; an agent gets APPROVAL_REQUIRED.

Try it on your website

Paste your address and watch the chatbot learn your pages. No account and no card.

This page in other languages

AI bot checker: which AI crawlers can read your website?

Build a support agent from your site

What the AI bot checker does

Search, training and user fetches: three kinds of AI bot

AI search

Model training

Fetches for a user’s question

Every bot the checker tests

How to read the result

Should you block AI crawlers? The trade-offs

robots.txt examples

The noai tag and its limits

Common mistakes

What sites usually allow, by type

Your own website chatbot and robots.txt

Build a support agent from your pages

Questions people ask

Related pages

Build it with your AI agent

Try it on your website