Free AI crawler checker with live request test
This checker reads your robots.txt and sends one live request per crawler from our server, so you see both rules and reality in one table, with no account needed.
Want to know if Google is finding and indexing your pages? Run a free SEO audit from your Search Console data to see which pages lost clicks and what to fix first. If your site is built with code and AI assistants are missing pages, RankBrain AI finds what they cannot read and applies the fix through your AI coding tool after approval.
How do you use this AI crawler checker?
Paste any page address and click Check crawlers. The tool reads your robots.txt and sends one request per crawler. The result table shows each crawler's robots.txt rule alongside the HTTP status our server received.
- Type or paste any public web address into the box, for example https://example.com or https://example.com/blog.
- Click Check crawlers. The tool fetches your robots.txt, parses the rule for each crawler, then sends one live request using that crawler's published user-agent string.
- Read the table: the robots.txt column shows whether that path is allowed or blocked by your rules, and the live column shows what your server actually returned to that crawler name.
What do the robots.txt column and the live column each mean?
The robots.txt column reads your file and applies the longest-match rule for each crawler, the same way Google documents it. The live column shows what your server returned when our server sent a request using each crawler's published user-agent string.
The four live results:
- Reached: the server returned a status below 400, the same as a normal page load.
- Refused: the server returned an error status (such as 403 or 429) that the browser request did not get, so something is treating that crawler name differently.
- Same as browser: the browser request got the same error, so the block applies to our server's address for everyone, and the bot rows alone show nothing specific to AI crawlers.
- No answer: the request timed out or the connection failed.
One limit to keep in mind: the live requests come from our server's address, carrying each bot's name. A firewall that checks both the address and the user-agent may treat these as impersonation attempts. A refused result means your firewall is looking at that crawler name, but the real bot, arriving from the bot company's own addresses, may get a different answer.
Which AI crawlers does this checker test, and what does each one do?
The checker tests 11 crawlers across five companies. Each crawler serves a different purpose: training, search answers, or fetching pages when a user asks. Your robots.txt can allow or block each one separately.
| Crawler | Company | What it does | Follows robots.txt |
|---|---|---|---|
| GPTBot | OpenAI | Crawls content that may be used to train OpenAI models | Yes |
| OAI-SearchBot | OpenAI | Shows websites in ChatGPT search features | Yes |
| ChatGPT-User | OpenAI | Fetches pages when a user's ChatGPT question leads there | Rules may skip user visits |
| ClaudeBot | Anthropic | Collects web content for training | Yes |
| Claude-SearchBot | Anthropic | Crawls to improve Claude search answers | Yes |
| Claude-User | Anthropic | Fetches pages when a Claude user's question needs them | Yes |
| PerplexityBot | Perplexity | Surfaces and links websites in Perplexity search results | Perplexity asks sites to allow it |
| Perplexity-User | Perplexity | Fetches pages when a user asks a question | Generally ignores it |
| Google-Extended | robots.txt token controlling use of crawled content for Gemini training and grounding | Yes, as a token | |
| Googlebot | Google Search crawler | Yes | |
| Bingbot | Microsoft | Bing Search crawler | Yes |
Checked on 20 November 2026 on each company's own crawler documentation. Sources: OpenAI "Overview of OpenAI Crawlers", Anthropic Help Center article on blocking crawlers, Perplexity crawler documentation, Google "Google's common crawlers" page, Google robots.txt documentation, Microsoft Bing Webmaster documentation.
Which robots.txt lines allow or block AI crawlers?
Paste one of these blocks into your robots.txt above your User-agent: * group. RFC 9309 says a named group applies only to that crawler, so copy any Disallow lines from your * group that should still apply, such as /admin or /cart, into the named block too.
1. Let AI search and AI answer bots in, keep training bots out (GPTBot, ClaudeBot and Google-Extended disallowed).
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /
2. Keep every AI crawler listed here out.
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
Disallow: /
3. Let every AI crawler listed here in, but keep one private folder closed (replace /admin/ with your own path).
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
Allow: /
Disallow: /admin/
Perplexity says Perplexity-User generally ignores robots.txt rules when a user's question triggers the visit. OpenAI says robots.txt rules "may not apply" to ChatGPT-User because a user starts those visits. OpenAI's documentation says search-related robots.txt changes for OAI-SearchBot can take about 24 hours to take effect.
What happens when a firewall blocks a crawler your robots.txt allows?
robots.txt is a signal crawlers choose to follow. A firewall or CDN decides independently which requests get an answer, regardless of what robots.txt says.
When the tool shows a "refused" result for a crawler row but your robots.txt allows it, your server returned an error status to that bot's user-agent while the browser request from the same address got through. This can mean your firewall is blocking that specific crawler, or it is blocking requests that carry a bot name from an address it does not recognise as that bot's own. Check your firewall or CDN's bot settings to confirm which crawlers you are allowing or blocking there. Cloudflare's AI Crawl Control feature, for example, shows which AI services visit your site and lets you allow or block each crawler (checked on 20 November 2026). Anthropic advises site owners to avoid blocking its crawlers by IP address, since doing so may stop its bots from reading robots.txt at all.
What does a real check look like?
Here are three cases run with this tool on 20 November 2026, showing what each result type looks like in practice.
- Example robots.txt (an example, to show how the tool reads rules):
User-agent: GPTBot
Disallow: /
User-agent: *
Allow: /
The tool reads the named group for GPTBot and returns: blocked by "Disallow: /" in the User-agent: GPTBot group. The other 10 crawlers have no named group, so the User-agent: * group applies: each is allowed by "Allow: /".
- maitreyalabs.com, checked on 20 November 2026: the robots.txt answered 200 and has 53 non-empty lines in one
User-agent: *group. All 11 crawlers are allowed at the home page. The browser request got 200. All 10 live crawler requests got 200. Google-Extended has no live request.
- A large news site checked the same day blocks every AI crawler in this list by name in its robots.txt. Its server also answered 403 to every request from our server, including the browser request. The tool marks those bot rows "same as the browser": the block is coming from that server address for all visitors, so the bot rows alone say nothing about how AI bots are treated. Check your CDN or firewall settings to separate the two causes.
What does this checker cover, and what does it leave out?
This checker reads your robots.txt and sends one live request for each crawler that has its own user-agent, plus one as a normal browser. It covers one address per check, one path, 10 checks per hour from one connection, and public websites only.
Ports 80 and 443 only. Private, local and internal network addresses are refused. Each request waits up to 5 seconds. The robots.txt verdict on this tool covers only the path you entered; meta robots tags and X-Robots-Tag headers are outside what it reads. The live requests carry each bot's user-agent string and come from our server's address. A "refused" result can mean your firewall blocks that bot, or that it blocks requests using that bot's name from an address it does not recognise as the real bot. The tool stores no addresses and no results after the answer is shown. No sign-up and no email are needed. The result can be copied as plain text.
Related guides and tools
These guides cover Google indexing problems and traffic drops; the last link gives you more free tools.
- fix Discovered - currently not indexed: step-by-step guide to pages Google found but chose to skip indexing.
- why your website traffic dropped: what Search Console signals say about a traffic drop and where to start.
- Crawled - currently not indexed check: pages Google crawled but held out of its index, with causes and fixes.
- free SEO audit tools compared: a side-by-side look at free audit tools and what each one checks.
- more free tools from Maitreya Labs: other free tools for site owners and marketers.
Frequently asked questions
Here you'll find answers to common questions about AI crawlers, robots.txt and this checker.
Does blocking GPTBot or Google-Extended hurt my Google Search rankings?
Google says Google-Extended controls only whether your content may be used to train future Gemini models. Google also states it has no effect on your site's inclusion in Google Search and is no ranking signal. Blocking GPTBot is an OpenAI signal with no connection to Google's crawlers. Your Google Search rankings depend on Googlebot, which is separate from both.
Which AI crawlers should I allow to appear in ChatGPT, Claude and Perplexity answers?
For ChatGPT search results, allow OAI-SearchBot. For visits triggered by user questions, allowing ChatGPT-User and Claude-User gives each assistant access when a user asks for your page. For Perplexity search results, allow PerplexityBot, and for Claude search answers, allow Claude-SearchBot. Allowing these bots gives each service access to your content; whether and how often your pages appear in answers depends on each company's own systems.
Why does the checker show "refused" when my robots.txt allows the bot?
robots.txt is a signal crawlers choose to follow, and your server or CDN can still refuse a request regardless of what it says. The "refused" result means the server returned an error status to that bot's user-agent while the browser request from the same address got through. This can mean your firewall is blocking the bot's name, or blocking requests with that bot name from an address it does not recognise as the real bot. Check your firewall or CDN bot settings to confirm.
Is the check free, and does the tool store anything?
The check is free and needs no account or email. The tool stores no addresses and no results: everything is cleared after the answer is shown. You can run up to 10 checks per hour from one connection.
How long until a robots.txt change takes effect?
OpenAI's documentation says search-related robots.txt changes can take about 24 hours to take effect for OAI-SearchBot. For other crawlers, the timing varies: check each company's own crawler documentation for the latest guidance.