How AI search engines choose sources
How an AI answer is assembled in six steps, what decides whether a page is quoted, an answer-first page and robots.txt, how to measure presence in AI answers and why AI does not quote a site.
In short
AI search engines — Google’s AI Overviews and AI Mode, ChatGPT search, Perplexity, Claude — do not invent an answer from memory: they run ordinary searches, often several at once, take the pages found, cut out the passages that answer the question directly and assemble a reply with links. So a page is chosen in three steps: first it must be found and readable by the robot, then it must contain a passage that answers on its own, and finally that passage must look trustworthy — facts, dates, a clear author, consistency with other sources. Pages with the answer in the first paragraph, clean HTML without scripts, tables and fresh dates are quoted more often than long texts where the answer is hidden in the middle.
How an AI answer is assembled: 6 steps
-
1. The question is split
“Which sofa for a small flat” becomes several searches: sizes, folding mechanisms, materials, prices.
-
2. Ordinary search
Each query goes to a search index — pages that are not indexed do not take part.
-
3. Pages are read
A robot loads the page; content that appears only after scripts is often not seen.
-
4. Passages are cut out
From each page — a few paragraphs, rows of a table or a list that answer the question.
-
5. The answer is assembled
The model chooses the clearest and most consistent passages and writes a reply from them.
-
6. Links are attached
The pages whose passages were used get a link — the new “position” in search.
What decides whether a page is quoted
Each step has its own filter. A page that fails one of them never reaches the answer.
| Step | What matters | What to do |
|---|---|---|
| Search | indexing and ordinary SEO | a sitemap, canonical addresses, speed |
| Access | the robot is allowed in | robots.txt and protection settings for search assistants |
| Reading | text in the HTML | content rendered on the server |
| Passage | an answer that stands alone | the answer first, a question in the heading |
| Facts | concrete numbers and terms | tables, units, dates |
| Trust | author, date, consistency | an author, an update date, no contradictions |
| Freshness | current information | update pages and the date on them |
| Mentions | the brand appears in other sources | reviews, directories, articles about you |
A page and access: 2 examples
A page built so that its first paragraph can be quoted, and robots.txt that separates search from training.
An answer-first page
The question in the heading, the answer in the first paragraph, details in a table, a visible update date.
<!-- The answer comes first: one paragraph is enough to quote the page -->
<article>
<h1>How long does delivery of furniture take?</h1>
<p>We deliver within the city in 3 working days; outside the city — in 5–7.
Delivery is free for orders over 30 000.</p>
<p class="updated">Updated: <time datetime="2026-09-15">15 September 2026</time></p>
<h2>Delivery times by zone</h2>
<table>
<tr><th>Zone</th><th>Time</th><th>Price</th></tr>
<tr><td>City</td><td>3 working days</td><td>1 500</td></tr>
<tr><td>Region</td><td>5–7 working days</td><td>3 000</td></tr>
</table>
</article>Search yes, training no
Search robots and training robots of the same companies have different names, so the two decisions are separate.
# Search assistants may read the site — they quote it and link to it
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
# Training of models is a separate decision, independent of search
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /How to measure presence in AI answers
-
01
A list of target questions
20–50 questions your customers ask — checked in each assistant every month.
-
02
Mentions and links
Whether the brand is named, whether the link leads to you, which page is quoted.
-
03
Visits from assistants
Analytics shows traffic from chatgpt.com, perplexity.ai and other assistants as separate sources.
-
04
Visits of robots
Server logs show which AI robots read which pages and how often.
-
05
Accuracy about you
Wrong prices or terms in AI answers are a signal to fix the source pages.
-
06
Comparison with competitors
Who is quoted instead of you, and what their pages do better.
Why AI does not quote a site
-
Robots are blocked
Protection against bots on the hosting or the CDN blocks search assistants too.
-
Content after scripts
The robot gets an empty page and a loading spinner.
-
The answer in the middle
Three paragraphs of introduction before the point — the passage is taken from someone else.
-
Facts in pictures
Prices and terms in an image or a PDF are often not read at all.
-
Contradictions
Different terms on different pages make the model choose another source.
-
Old dates
A page last updated years ago loses to a fresh one.
Questions about AI search
How does ChatGPT choose sources?
It searches the web, reads the pages found and quotes passages that answer the question clearly and consistently.
Do I need SEO to appear in AI answers?
Yes: AI search starts with ordinary search, and a page outside the index does not take part.
Should I block AI robots?
Search robots — no, they bring mentions and links; training robots — your own decision.
Does llms.txt help?
It is an extra map of the site for models; it does not replace clean pages and indexing.
Why did traffic fall while positions stayed?
The answer is given right in the results; the goal now is to be the source quoted in it.
How fast do changes show in AI answers?
As fast as the page is re-read and re-indexed; fresh pages are visited more often.
Online form
Get into
AI answers
I check where your site appears in AI answers today and rebuild the pages so they are found, read and quoted. Tell me about your site — I answer within one working day.