Block or Allow AI Crawlers: GPTBot to PerplexityBot

An AI company can send a crawler at your site for three different reasons, and each vendor runs a separate user agent for each reason.

Retrieval. The bot is building or refreshing an index that gets searched when a user asks a question. When your page wins a slot in the answer, the assistant usually shows a link to it. This is the door that produces referral traffic.

Live user fetch. Somebody pasted your URL into an assistant, or asked about a page by name, and the agent goes and gets it on their behalf. There is a human waiting on the other end.

Training. The bot is collecting text to train or improve a model. Nothing links back, nothing gets cited, and the fetch produces no visitor now or later. Any return arrives much later and cannot be measured.

Those three doors carry completely different consequences, and the user agent string is the only thing that tells them apart. GPTBot and OAI-SearchBot are both OpenAI. One is training, one is search. Block the wrong one and you have made a decision you never intended, with no error message and nothing in any analytics product to warn you.

Vendors document their own agents, and those pages are the only source worth trusting. The list below is current as of publication and it will age.

Primary sources for the four that matter most: OpenAI's bots documentation, Anthropic's crawler article, Perplexity's bots guide and Google's common crawlers list. Common Crawl documents CCBot separately.

Two entries in that table trip people constantly.

Google-Extended is not an agent that appears in your logs. It is a token you write into robots.txt, and Google's own documentation states it does not affect inclusion in Google Search. Since AI Overviews and AI Mode are served from the Search index Googlebot builds, blocking Google-Extended does not remove you from them. Plenty of sites added it in 2024 and 2025 believing it would.

Applebot-Extended works the same way: a training opt-out token that does not change what Applebot crawls for Siri and Spotlight.

Microsoft is the odd one out. There is no Copilot crawler to allow or block, because Copilot answers draw on the Bing index that bingbot builds. Microsoft's Bing Webmaster Tools help documents nocache and noarchive robots meta tags as the content usage controls there, so that lever is a meta tag on the page, not a user agent line in robots.txt.

For the fuller picture of how each engine assembles an answer once it has your page, see how AI assistants pick sources.

Strip the ideology out and the two decisions have different ledgers.

That table is the whole argument. Blocking a training crawler is a low cost decision with a defensible rationale. Blocking a retrieval crawler is a traffic decision, and it is the one people make by accident.

The invisibility is what makes it dangerous. Fall out of ordinary search results and Search Console shows it: impressions drop, positions vanish. There is no equivalent report for a citation that never happened. This matters more than your position data suggests, because AI answers frequently cite pages that do not rank in the top ten, so a site with mediocre rankings can have plenty to lose here.

One caveat on the upside. Retrieval traffic is not a substitute for search traffic and the volumes are not comparable. An assistant that answers in full is a zero click outcome for most readers, and only a minority follow the citation. Expect a smaller stream of unusually well qualified visitors, not a replacement channel.

Here is the mechanic that turns a careful robots.txt into an accident.

RFC 9309, the 2022 specification for the Robots Exclusion Protocol, says a crawler uses the group with the most specific matching user agent, and only that group. Rules are not inherited. Rules are not merged. The group headed User-agent: * applies only to crawlers that match no other group in the file.

So this file, which looks like a reasonable way to welcome Perplexity, does something else entirely:

User-agent: *
Disallow: /checkout/
Disallow: /wp-admin/
Disallow: /internal-search/

User-agent: PerplexityBot
Allow: /

PerplexityBot now has an explicit group of its own, so it ignores the three Disallow lines above completely. Your checkout flow, your admin path and your internal search results are open to it and closed to everybody else. The file reads as "allow Perplexity" and behaves as "exempt Perplexity from every rule on the site."

The fix is to restate the shared rules inside every group you create:

User-agent: *
Disallow: /checkout/
Disallow: /wp-admin/
Disallow: /internal-search/

User-agent: PerplexityBot
Disallow: /checkout/
Disallow: /wp-admin/
Disallow: /internal-search/

Tedious, and there is no way around it. A per agent group is a full replacement, not an amendment. Four more rules follow from the same spec:

• One group can head several agents. Stack User-agent: lines with no blank line between them and the rules below apply to all of them. That is how you keep the file short without losing the inheritance battle.
• Token matching is case insensitive but not fuzzy. Do not rely on a group headed User-agent: Claude to catch ClaudeBot, Claude-User and Claude-SearchBot. Write the exact tokens the vendor publishes, one per line.
• The most specific path rule wins, and Google documents that a tie between Allow and Disallow goes to the least restrictive rule. So Allow: /blog/ beats Disallow: / under /blog/.
• robots.txt is per host and per scheme. https://www.sparkcliks.com/robots.txt does not govern https://docs.sparkcliks.com/. Subdomains need their own file, and that is where a large share of real world misconfigurations live.

Three coherent positions. Pick one, and do not end up mixing them by accident.

There is no universal answer, because the two models are not selling the same thing. A publisher sells attention by the page view, so an answer that satisfies a reader without a click is lost inventory. A lead generation site sells an action that happens after a visit, so being named as the answer is free distribution it would otherwise pay for.

The publisher row is the one with genuine disagreement inside it, and anybody who tells you it is settled is selling something. Large publishers have gone both ways inside the same year, some blocking and then reversing, some signing licensing deals that make the robots.txt question moot. In that category, treat your decision as a hypothesis with a review date, not a policy.

Run these in order. They resolve most cases in about ten minutes.

1. Does a reader who sees a good summary of this page still need to visit it? If yes, retrieval is free distribution and you should allow it. If no, you are a page view business and the retrieval question is a real trade rather than an obvious win.
2. Do you get paid per page view, or per action after the visit? Per view pushes you toward caution on retrieval. Per action pushes you hard the other way, because one qualified visitor is worth thousands of impressions.
3. Is your content the product, or the marketing for the product? Content that is the product is the strongest case for blocking training. Content that is marketing has no case at all.
4. Six months from now, will you be able to tell what the decision did? With no baseline and no way to check, the honest move is to allow retrieval, block training, and revisit once measurement exists. Reversing a block is easy. Recovering the answers you were never in is not.

One check before you write anything: see whether you already have a rule you forgot about. Plenty of sites blocked GPTBot in 2023, when it was the only AI user agent anyone had heard of, and never reopened the file after OAI-SearchBot appeared. That file is now expressing a 2023 opinion about a 2026 landscape.

Writing the rule is the easy half. Confirming that it does what you meant, at every layer, takes about twenty minutes.

Be clear about what you are buying, because several popular reasons for blocking are not served by this file at all.

• It is a request, not a fence. RFC 9309 defines a format and a parsing model. It defines no enforcement. Compliance is voluntary, and the agents that ignore it are the ones that worried you in the first place.
• It is not retroactive. Blocking GPTBot today does nothing about content collected last year, or about copies of your pages in Common Crawl archives stretching back more than a decade. If removal is the goal, this is not the instrument.
• It is not a copyright notice and not a licence. It expresses a crawl preference. It creates no rights and waives none, and it is no substitute for terms of use.
• It cannot stop an agent that does not announce itself. In August 2025 Cloudflare published findings that it had observed crawling attributed to Perplexity from undeclared user agents and rotating addresses, and Perplexity publicly disputed the analysis. The dispute is unresolved, and the lesson holds either way: a text file governs only the crawlers that read it.
• It cannot express "index but do not train." The protocol grants or denies a fetch and stops there. Anything finer depends on a vendor honoring a separate token, which is why Google-Extended and Applebot-Extended exist as bolt-ons.
• Blocking a page also blocks the noindex on it. A crawler barred from fetching the page cannot read the meta tag telling it to stay out of an index. That failure mode is covered in full in robots.txt vs noindex, and it applies to AI crawlers exactly as it does to search engines.

Anything written on this topic has a short shelf life. Four areas are genuinely unsettled, and pretending otherwise would be dishonest.

The agent lists keep growing. OAI-SearchBot did not exist when GPTBot launched. Claude-SearchBot came after ClaudeBot. Every vendor has revised its published list at least once, so any blocklist pasted from an article, this one included, starts aging the day it is written. Diary a quarterly re-read of the four vendor pages above.

There is no standard vocabulary yet. The IETF has a working group drafting a way to express content usage preferences, carried in an HTTP header or a robots.txt extension, precisely because "allow crawling, refuse training" cannot be said in the current protocol. Those drafts are in progress, not a published standard, and nothing you write today can depend on them. llms.txt is not the answer either: it is a content index rather than a crawler directive, and no major vendor documents obeying it.

Paid access is being tested. Cloudflare launched a pay per crawl beta in 2025, letting site owners charge AI crawlers for access instead of choosing between open and closed. Whether that becomes normal infrastructure or a footnote is not knowable yet.

Agentic browsing does not fit the taxonomy. An agent booking a table or completing a purchase for somebody is not training a model and is not building a search index. It is a customer's proxy. The categories above predate that use case, and the vendors have not settled how to label it.

One thing has not changed. The fetch matters far less than what the crawler finds when it arrives. A page that is allowed but unreadable, buried in client rendered JavaScript or written so that no passage stands on its own, loses the citation anyway. That is a content problem, and it is the subject of how AI answer engines retrieve chunks rather than pages.