How AI Assistants Pick Sources: ChatGPT to Gemini

Every assistant that shows links behind an answer runs the same skeleton. The branding differs, the vocabulary differs, the shape does not.

Two of these fail silently. If Access fails you were never a candidate, and if Selection fails you were fetched, read and dropped. No dashboard anywhere reports "retrieved but not cited," which is why most people optimizing for AI search are guessing about which half of the funnel is broken.

OpenAI publishes three user agents and they do different jobs, documented at platform.openai.com/docs/bots. GPTBot collects data for model training. OAI-SearchBot builds the index behind ChatGPT's search results. ChatGPT-User fetches a page live when a prompt points at a specific URL.

That split matters more than anything else in this section. Blocking GPTBot does not remove you from ChatGPT's search results. Blocking OAI-SearchBot does.

ChatGPT also searches conditionally rather than on every prompt, and when it does cite, it cites sparingly: a handful of links for an answer where Perplexity would show twenty. Your pressure point here is Selection. Being retrievable is the entry fee. Being the clearest available statement of the specific claim is what wins the slot.

One caution on tactics you may read elsewhere. Ignore advice about optimizing for whichever third party index is currently rumored to sit behind ChatGPT. OpenAI documents its own crawler, that crawler shows up in your access logs, and a rumor does neither.

Anthropic documents ClaudeBot for training data, Claude-SearchBot for search indexing and Claude-User for user triggered fetches, described in Anthropic's crawler documentation.

The structural difference is upstream of all of that. Claude's web search is a tool the model chooses to call, so the first gate is not your content at all: it's whether the model decided the question needed the web. A question it believes it can answer from what it already knows never triggers retrieval, and no citation is possible no matter how good your page is.

The queries that reliably trigger a search are the ones about recent, changing, local or narrowly specific facts. That makes your realistic surface area with Claude narrower than with Perplexity, and it makes being current worth more than being comprehensive. When search does run, results come back with source metadata attached and Claude cites the pages it drew on.

Perplexity is the retrieval led one. It searches on essentially every query rather than conditionally, runs PerplexityBot for indexing and Perplexity-User for live fetches (documentation here), and attaches numbered citations inline as it writes, often several in a single paragraph.

Two things follow. The retrieved set is wider, so mid authority pages get cited here far more often than anywhere else: this is the friendliest of the four for a site that is good but not dominant. And because attribution happens per sentence, a page can be retrieved, read, and still never cited, because no sentence in the finished answer traces cleanly back to it.

That second point is the practical lesson. Perplexity rewards pages that make one checkable claim per sentence. Long narrative paragraphs where the useful fact is distributed across four clauses get read and discarded.

The naming here causes more confusion than the technology. Gemini the assistant app, AI Overviews in search results, and AI Mode are different surfaces sitting over overlapping infrastructure.

For the search surfaces, one fact matters above the rest: there is no separate AI index. AI Overviews and AI Mode are served from the same Search index Googlebot builds, which Google states directly in its AI features and your website documentation. If a page is not indexed, it cannot be cited there. Indexing is a hard gate, not a ranking factor you can compensate for elsewhere.

Google describes AI Mode as using a query fan out technique: your single question is decomposed into multiple related searches issued at once, and the answer is assembled from all of them. You are not competing for the query the user typed. You are competing for a set of synthetic sub-queries you will never see in a keyword tool or in Search Console.

Then there is the control most people get backwards. Google-Extended is not a crawler. It's a robots.txt token that governs whether your content helps improve Gemini apps and Vertex AI generative APIs, and Google's crawler documentation states it does not affect inclusion in Google Search. Blocking it does not take you out of AI Overviews. Many sites added it in 2024 and 2025 believing it would.

These lists change: OpenAI added OAI-SearchBot alongside the older GPTBot, and that split is exactly the trap described in the next section. Every cell above traces to a vendor documentation page linked in the sections before it. Check the live page before you write a rule against it.

The most expensive mistake in this area is treating "AI crawlers" as one category. There are three doors, and they have completely different consequences.

The common failure runs like this. Somebody reads that AI companies are scraping the web, finds a list of AI user agents, and adds a blanket Disallow for all of them. The training door closes, which may well have been the intent. The search door closes too, which was not. Months later the site is never cited and nobody connects the two, because nothing in any analytics product or webmaster tool reports a citation that did not happen.

Decide the doors separately. They are separate decisions with separate trade-offs.

Different architectures, converging incentives. The overlap is short and unglamorous.

• Content that exists in the raw HTML. Googlebot renders JavaScript. Several of the other crawlers perform a plainer fetch, so text that only appears after client side hydration may not exist for them at all.
• A visible, accurate date. Recency is a retrieval and selection signal across all four, and it is the one thing Claude's conditional search actively looks for.
• Claims that are checkable. A sentence carrying a number, a named entity or a date can be attributed to you. "Best practices vary depending on your goals" cannot be attributed to anyone, which is precisely why it never gets cited.
• Passages that stand alone. Selection operates on chunks rather than whole pages, so a paragraph that only makes sense after reading the two above it is a weak candidate even on a strong page.
• Corroboration. A claim that appears on several independent sites survives selection more often than a claim only you make. This is uncomfortable for original research and it is how these systems behave.

About thirty minutes of work, and it tells you which stage is actually failing instead of leaving you to guess.

None of this is a route to a guaranteed citation. Every engine here can change its retrieval stack, its crawler names and its citation behavior without notice, which is why every vendor claim in this post links to a page you should re-check rather than take on trust.

It also has nothing to do with traffic volume, which is worth stating plainly given what we sell. SparkCliks provides search engine clicks and website visits through SERP Clicks, Sparky Traffic Bot and the Website Traffic and Realistic Traffic products. Those change what your analytics records about visits to your site. None of them put a page into a retrieval index, and no assistant consults your traffic numbers when deciding what to cite. Anyone selling AI citations as a traffic product is selling something they cannot deliver.

There is a real overlap, and it is worth describing precisely because it is usually overstated. The work that makes a passage citable, namely clear claims, honest dates and structure that survives extraction, is the same work that makes a search snippet worth clicking. That is a shared cause, not a causal link running from one to the other. Doing the first does not buy you the second.