How to Check If AI Crawlers Can Read Your Page

Three separate layers stand between an AI answer engine and your words, and each one fails in a way the other two cannot see.

Permission. robots.txt says whether a named agent is allowed to request the URL. This is the layer everyone writes about, and it is the layer least likely to be your actual problem.

Delivery. Your CDN, WAF or bot management rules decide whether the request survives long enough to reach your application. They never consult robots.txt, they answer first, and they frequently return a 200 status with a challenge page in the body.

Payload. Assuming the request got through, the crawler now has some bytes. The question nobody checks is whether your article text is actually in those bytes, or whether it arrives later when a browser runs the JavaScript bundle that fetches and renders it.

That third layer is where the damage usually is. Vercel published an analysis in December 2024 based on traffic across its own network, reporting that OpenAI's and Anthropic's crawlers requested JavaScript files but did not execute them, while Google's Gemini rides on Googlebot infrastructure that does render. Treat any such list as a snapshot. Vendors change their stacks, and none of them publish a changelog for rendering behavior.

There is a consequence worth sitting with. Googlebot does render JavaScript, in a deferred second pass, and AI Overviews and AI Mode are served from the same index Googlebot fills. So a client rendered page can be visible inside one search engine's AI features and completely blank to a retrieval crawler like OAI-SearchBot or PerplexityBot. Your exposure is not one number. It differs per engine, and the only way to know is to look at what your own server sends.

That is the point of everything below. You never have to trust a vendor list, or this article. You can measure it.

One note before the commands. Every request here goes to your own server, for your own page, using a user agent string you type yourself. That is diagnostics, not scraping and not evasion. Do not run these against somebody else's site.

Run them in this order, because each one narrows what the next can mean. A 403 at check 5 makes check 1 pointless, and a passing check 1 makes a robots.txt argument moot.

Nothing in that table promises a citation. That is check 8, which is not a check at all, and it gets its own section near the end.

A crawler sends an HTTP request, reads the response, and stops. No cookie jar carried over from a previous visit, no JavaScript engine, no waiting for a network idle event. curl behaves almost exactly the same way, which makes it the right instrument.

curl -sL \
  -A "Mozilla/5.0 (compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)" \
  https://www.sparkcliks.com/blog/ -o raw.html

curl -sIL \
  -A "Mozilla/5.0 (compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)" \
  https://www.sparkcliks.com/blog/

The first command saves the body. The second shows the header chain, including every redirect hop. Read three things in those headers: the final status code, content-type (it should be text/html, and a page served as text/plain is a broken configuration), and whether -L had to follow a redirect to a login, a locale picker or a consent domain.

Then look at the size:

wc -c raw.html

Under about 5 KB for what should be a full article is a strong hint you received a shell rather than a page. That is a heuristic, not a rule, so confirm it with check 2 instead of concluding from the number alone.

This is the check that catches the invisible failure, and it is a comparison rather than a single reading. You need two word counts for the same URL: one from the bytes your server sent, one from the DOM after the browser finished working.

The raw count. Strip scripts, styles and tags, then count what is left:

curl -sL -A "Mozilla/5.0 (compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)" \
  https://www.sparkcliks.com/blog/ \
  | perl -0777 -pe 's/<(script|style|noscript)\b.*?<\/\1>//gsi; s/<[^>]+>/ /g; s/&[a-z#0-9]+;/ /gi; s/\s+/ /g' \
  | wc -w

The rendered count. Open the same URL in a normal browser, let it settle, open the developer console and run:

document.body.innerText.trim().split(/\s+/).length

Now compare. Here is a worked example. The numbers are illustrative rather than measurements from any real site:

A raw share above roughly 80 percent means the substance is in the source. Below about 50 percent, something material only exists after hydration. The middle band needs judgment, because a large navigation menu or a cookie notice can move the number without touching the article.

Then find out what kind of page you are dealing with:

curl -sL -A "OAI-SearchBot" https://www.sparkcliks.com/blog/ \
  | grep -o '__NEXT_DATA__\|__NUXT__\|__INITIAL_STATE__\|data-reactroot\|ng-version' \
  | sort | uniq -c

Finding a hydration payload is not itself a failure. Frameworks that server render still ship one, and that is fine, because the text is in the HTML too. An empty root element plus a large JSON blob and no prose is the failing pattern. The tabbed docs case above is the sneakiest version of all, because it survives a casual look: the source has words in it, just not the words in tabs two through five.

While you are here, confirm the machine readable layer survived:

curl -sL -A "OAI-SearchBot" https://www.sparkcliks.com/blog/some-post/ | grep -c 'application/ld+json'

Schema injected by a tag manager after page load is invisible to a crawler that does not run scripts, which quietly undoes the work described in structured data versus prose for AI search.

Retrieval systems do not fetch your page and read it top to bottom. They pull passages, which is the mechanism covered in how AI answer engines retrieve chunks rather than pages. So the useful question is narrower than "is the page readable". It is: is the specific sentence I want an engine to quote present in the bytes?

Pick the definition, the number or the direct answer you actually want attributed to you, take a distinctive fragment of four to six words from it, and grep for it:

curl -sL -A "Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" \
  https://www.sparkcliks.com/blog/some-post/ \
  | grep -c "clicks that arrive from an answer"

Two traps make this test return a false negative:

• Entity encoding. An apostrophe in the source is often ' and an ampersand is &, so a fragment containing either will not match even when the text is there. Choose a fragment with no punctuation.
• Inline tags. If any word in your fragment is bolded or linked, the raw HTML has a or an sitting in the middle of your string, and a literal grep fails. Pick a run of plain words.

A count of zero after avoiding both traps means the passage does not exist for that crawler. Run this against the three or four pages that carry your best answers, not across the whole site.

curl -sI https://www.sparkcliks.com/robots.txt
curl -s  https://www.sparkcliks.com/robots.txt

You want a 200 and content-type: text/plain. A single page app that answers every unknown path with an HTML shell is serving an HTML document as robots.txt, which parses to no rules at all.

The parsing rule that trips people up: a crawler obeys exactly one group, the one whose User-agent line names it, and it ignores every other group in the file including . So a site that allows everything under and then adds a narrow group for GPTBot has just handed GPTBot a completely different rulebook, and any Disallow in the * group no longer applies to it. Which agents belong on which side of that decision, and the exact syntax for each position, is the subject of block or allow AI crawlers. This post is only asking whether the file says what you think it says.

Two things are worth remembering while you read it. A disallowed page cannot be fetched, so a crawler can never see a noindex tag sitting on it, which means blocking a URL and de-indexing it are opposite instructions. And an llms.txt file is not a crawler directive and changes nothing about permission, for the reasons set out in llms.txt explained.

Your edge answers before your application does, and it has never read your robots.txt. Cloudflare announced in July 2025 that new domains joining its network default to blocking AI crawlers unless the owner opts in, so a site that configured nothing at all can still be closed. Run a status and size matrix across the agents you care about:

for ua in "OAI-SearchBot/1.0" "ChatGPT-User/1.0" "PerplexityBot/1.0" \
          "Claude-SearchBot/1.0" "ClaudeBot/1.0" "Googlebot/2.1" "bingbot/2.0"; do
  out=$(curl -s -o body.html -w "%{http_code} %{size_download}" \
        -A "Mozilla/5.0 (compatible; $ua)" https://www.sparkcliks.com/blog/)
  printf "%-24s %s\n" "$ua" "$out"
done

The size column is the part most people leave off, and it is the part that catches soft blocks.

Confirm a suspicious 200 rather than guessing:

grep -il "just a moment\|checking your browser\|enable javascript\|verify you are human" body.html

Then repeat the whole loop for a second path, ideally one deep in the site rather than a section index. Edge rules are frequently written per path and per method, so a clean home page proves very little about /blog/.

A crawler arrives with no cookies, no session and no consent record, every single time. Whatever your site does for a first time anonymous visitor is what it does for every AI crawler, forever.

The distinction that matters: a gate that hides text with CSS still delivers the text, and a gate that withholds it until an event fires does not. Test with the raw fetch, not with your eyes.

Interstitials cost you on the human side of the visit too, but that is a conversion problem. Here the only question is whether the words shipped.

Everything so far was a simulation you ran yourself. Logs are evidence of what really happened. Count hits, but do not stop at hits, because the response you gave them is the more interesting column:

awk '/OAI-SearchBot/ {n++; s[$9]++; b+=$10}
     END {printf "hits=%d avg_bytes=%d\n", n, (n ? b/n : 0);
          for (c in s) printf "  status %s  %d\n", c, s[c]}' access.log

In the common combined log format, field 9 is the status code and field 10 is the response size. That size field is the cheapest hydration detector anyone has, and almost nobody uses it. If a browser session for the same URL pulls 90 KB and every crawler request pulls 4 KB, you have proof from your own server that you are serving two different documents. No developer console needed, and it keeps working long after you stop paying attention.

One caveat applies to every count here. A user agent string is a claim, not an identity, and anyone can send a request labeled ClaudeBot. OpenAI, Anthropic, Perplexity and Google each publish IP ranges for their crawlers and several support reverse DNS verification, so check the addresses in your log against the vendor's current published list before concluding anything about who visited.

The last step is qualitative, and it is the only one that touches the outcome you care about. Take five to ten questions you would genuinely want to be the source for, ask them in the assistants your audience uses, and record what gets cited.

Three honesty requirements for reading that sheet. Answers are not deterministic, so the same question asked twice can return different sources, and a single run is an anecdote rather than a measurement. Results vary by account, region and model version, so a colleague's screenshot is not your result. And a citation appearing once does not mean it is stable.

A measurement design that does not fool you. Fix the query list, run it on a set day each month, and keep every result in one sheet. Take a 28 day baseline before you fix a rendering problem and a 28 day window after it, hold back a control set of comparable pages you deliberately leave untouched, and read two things: crawler hits and average response size in your logs, plus assistant referral sessions in Google Analytics 4 under Reports, then Acquisition, then Traffic acquisition, with Session source / medium as the primary dimension filtered to the assistant hostnames. Expect small, noisy numbers, and expect a real share of assistant referrals to land in Direct because no referrer is sent. A handful of sessions moving on a base of a few dozen is noise. Signal is a change large enough to see without squinting, holding across two consecutive windows, and absent from the pages you did not touch.

Every check in this runbook establishes a precondition. None of them produce a citation, and nothing here should be read as a promise of visibility, referral traffic or inclusion in any answer.

Being readable puts you in the pool of candidates. What happens next is selection, and it turns on whether your passage answers the question better than the other candidates, whether the engine treats your site as a source worth naming, and what it has been built to prefer. That selection process is the subject of how AI assistants pick their sources, and it is why AI answers routinely cite pages that do not rank in ordinary search results.

The asymmetry is what makes the runbook worth an afternoon. A readable page might get cited or might not. An unreadable page cannot, no matter how good the writing is. You are removing a disqualification, not buying a result.

Run this checklist against your three most important pages before you spend another hour on content:

• [ ] Raw fetch returns 200 with text/html and a plausible byte count
• [ ] Raw HTML word count is at least 80 percent of the rendered DOM count
• [ ] The exact sentence you want quoted appears in the raw source
• [ ] Tabbed, accordion and lazy loaded sections are in the HTML, not fetched on click
• [ ] JSON-LD is server rendered, not injected by a tag manager
• [ ] The robots.txt group that applies to each named agent says what you intended
• [ ] Status and response size are consistent across every agent you allow
• [ ] The page is readable with no cookies, no session and no consent record
• [ ] Logs show real agents arriving and receiving full size responses
• [ ] Crawler identity verified by IP range or reverse DNS, not by user agent string