Allow AI Crawlers in robots.txt Without Opening Everything

Most AI crawler guides open with a table of bots. That's the wrong end of the problem once the decision is made, because robots.txt does not control agents at the page level. It controls them at the path pattern level, and path patterns come from your templates.

So the first artifact you need isn't a bot list. It's a template inventory: every distinct URL shape your CMS can emit, with one representative URL for each.

A typical small content site emits more shapes than its owner expects:

Here's the part that gets skipped. Your exposure surface is the URL set, not the page set. One guide is one page in your head and often a dozen URLs in reality: the canonical, the tracked variant from your newsletter, the sorted view, the paginated view, the uppercase link somebody pasted, the print route. A rule that opens /guides/ opens all of them. A retrieval agent that samples the sorted-and-tracked variant chunks a page whose main content sits below a filter bar, and that becomes the version of you it quotes. A canonical tag doesn't rescue this, because the crawler has to fetch the page to read the canonical, and by then it has the content.

Write the inventory down before you write a single rule. Everything after this section gets applied to that table.

Run each template through two questions. Both have to pass.

Test 1: does a single URL under this path answer a question completely, without the rest of the site? Retrieval systems split pages into chunks and rank the chunks, so a URL that only makes sense inside a browsing session has nothing to contribute. A glossary term passes. A category archive doesn't. There's more on how that splitting works in how AI answer engines break a page into retrieval chunks.

Test 2: would a quote from this path still be true and safe ninety days from now? Retrieval indexes lag. Anything you expose can be repeated back with your brand attached long after you changed it. Price pages and spec tables pass this test only if you accept that a stale figure will surface at some point.

Paths that clear both, on a typical content site:

The glossary row isn't filler. Definitional pages are the cheapest citations available: one URL, one question, one answer, no navigation required to make sense of it. If your template inventory has no glossary and no docs, that's a content gap robots.txt cannot fix. What earns the citation once the door is open is covered in how AI assistants pick their sources.

The instinct is to sort paths into public and private. Wrong axis. Almost everything in the table below is technically public. The reason to close it is that a quote from it would be wrong, thin, or somebody else's words wearing your domain name.

Internal search deserves its own paragraph, because it's the path people leave open by accident. A search results page publishes user input under your domain. Anyone can request /search?q= followed by any phrase they like, get it rendered inside your header and footer with your logo above it, then link to that URL from somewhere a crawler will find. If a retrieval agent is allowed on that path, the phrase becomes a chunk attributed to your site. No exploit is involved and there's nothing to patch. The fix is one Disallow line, and it belongs in every group in the file.

Deep pagination and thin tag archives fail for a duller reason. They're real pages with real content, but the content is a list of headlines. A retrieval index doesn't know a listing page from an article. It knows this URL on your domain mentions the query term nine times, which turns the listing page into a direct competitor of the article you wanted cited.

This is where most circulating AI crawler templates fall apart, and it's mechanical enough to check by hand.

Two rules, both documented in Google's robots.txt specification and consistent with RFC 9309, the Robots Exclusion Protocol standard published in September 2022:

1. The rule that applies is the most specific match, measured by the number of characters in the rule's path, wildcards included.
2. When two rules are the same length, the least restrictive one wins, which means Allow beats Disallow.

Google's own worked cases, worth memorizing because every argument about robots.txt collapses into one of them:

Source: Google Search Central, robots.txt specifications. Two wildcards are supported, for zero or more of any character and $ for end of URL. A trailing is ignored, so /blog/* and /blog/ are the same rule.

A small content site with a blog, guides, a glossary, a members area holding one free sample, an internal search and a store. The stance on training agents was settled in the decision post, so it sits in its own group at the bottom and doesn't get re-argued here.

# https://www.sparkcliks.com/robots.txt
# These rules apply to this host and this scheme only. An apex domain
# or an http variant that also serves the site needs its own copy.

# Group 1: every crawler not named in another group.
User-agent: *
Disallow: /account/
Disallow: /checkout/
Disallow: /cart/
Disallow: /search?
Disallow: /search/
Disallow: /preview/
Disallow: /go/
Disallow: /*?

# Group 2: AI retrieval and user-triggered fetch agents.
# Repeated in full, because a named group inherits nothing from *.
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Disallow: /account/
Disallow: /checkout/
Disallow: /cart/
Disallow: /search?
Disallow: /search/
Disallow: /preview/
Disallow: /go/
Disallow: /tag/
Disallow: /members/
Allow: /members/sample/
Disallow: /members/sample/*?
Disallow: /*?

# Group 3: training-only agents. Your stance from the decision post lives here.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
Disallow: /

Sitemap: https://www.sparkcliks.com/sitemap.xml

The reasoning, for every line that isn't self-explanatory:

Two limits worth knowing before the file grows. Google enforces a 500 kibibyte cap on robots.txt and ignores everything past it, and it generally caches the file for up to 24 hours. A change you push at 9am is not a change that's in effect at 9:05.

robots.txt governs the next fetch. It's not a delete request, and the format has no directive that means "forget what you already have."

Split by where the copy actually lives:

The counterintuitive part, and the one that costs people the most time: to get a wrong answer corrected, you have to leave the door open. If a retrieval index holds a page saying something you no longer want quoted, and you respond by adding Disallow: for that path, you've frozen the last version the crawler saw. The agent can't re-fetch, so it can't discover the page changed. The sequence that works is: fix the page, keep it crawlable, wait for a re-crawl, confirm in your logs that the new version was fetched, and only then decide whether the path still needs closing.

Nobody publishes a service level agreement on re-crawl timing for these agents, and neither will this post. Watch your access log for the next hit on that URL and treat that as your clock.

These three get used interchangeably in conversation and they do genuinely different jobs. For AI search, the distinction that matters most is the last column.

Google's robots meta tag documentation was updated to state that nosnippet and max-snippet apply to AI Overviews and AI Mode as well as ordinary results, which makes them the only per-page instrument that speaks to an AI surface. The trade is blunt: nosnippet also removes your ordinary search snippet.

The asymmetry is the point of this section. Per-page AI controls are a Google-only lever. OpenAI, Anthropic and Perplexity document robots.txt user-agent tokens and nothing per-page. For those agents, path-level control in robots.txt is the entire toolbox, which is why the inventory you built in section one carries so much weight.

There's also a trap that mirrors the classic robots.txt versus noindex problem, one layer up: a Disallow on a path stops the crawler from reading the nosnippet on that path. If your goal is "indexed, ranking, not used as AI input," the page has to stay crawlable and carry nosnippet. Blocking it in robots.txt gets you neither result cleanly. The full mechanic, including why blocked pages still show up in results, is in robots.txt vs noindex.

Two things are moving and neither is a control you can ship today. Cloudflare's Content Signals Policy, launched in September 2025, adds search, ai-input and ai-train signals as a comment block inside robots.txt and is deployed across a large number of domains, but no crawler operator has committed to honoring it. The IETF AIPREF working group is drafting a shared vocabulary for the same idea, and as of 2026 the attachment specification that would carry it in robots.txt still hasn't shipped as a standard. Treat both as directional. A related file that does exist and is worth understanding sits in llms.txt explained.

Writing the file is the easy half. Here's the check, worked against the file above.

Ranked by how often they turn up in a real file:

1. Decorative Allow: lines. They grant nothing and out-specify the Disallow rules you needed.
2. Prefix collisions. Check every short Disallow against your real slug list.
3. Blocking a path, then expecting the tags on it to work. A blocked page's noindex, nosnippet and canonical are all unreadable.
4. Only one host covered. The apex serves its own robots.txt, or the http variant does, and neither one got the rules.
5. Blocking /assets/ or /wp-content/. Retrieval agents that render before chunking need the stylesheet and the script. Block those and your page gets evaluated as an unstyled column of text.
6. A Disallow line that publishes the path. robots.txt is public and people read it. Writing Disallow: /internal-pricing-2027/ tells the world that directory exists. Anything that genuinely must not be reached needs authentication, not a line in a public text file.
7. Trailing wildcards mistaken for precision. /blog/* is the same rule as /blog/. If you wrote the star to mean something, it didn't.