AI SEO: what changes, and what still matters most?

Google says AI features need no special files or new markup. Crawler rules, clear answers and separate evidence still decide what you can prove.

A smartphone screen showing an AI assistant interface.

Do you need a separate AI search plan, or should you keep improving the pages people already reach?

Direct answer

AI search does not replace the basics. Google says its AI features need no additional technical requirements, special files or new markup. What changes is what you monitor. Check crawler access for each named bot, keep answers clear on the page, and record Search Console data and dated assistant checks as separate evidence.

Methodology, checked 7 November 2026: I read Google's guide to AI features, its people-first content guidance, its list of common crawlers and its robots.txt introduction. I also read OpenAI's crawler page and Anthropic's crawler help article. I did not test any live AI answer. For Microsoft, I read one Bing Webmaster Blog post on its AI Performance report, used only in the common questions below. No client data, rankings or citation results were available, and none are claimed.

What changes in AI search, and what stays the same?

Google says its AI features need no extra technical work beyond normal search eligibility. The change is in the outputs you watch. A page can be crawled, indexed, shown in a result, used as a supporting link or named in an assistant's answer. Each of these needs its own evidence, and none proves the next.

Decision matrix: what changed, what to do, what to keep doing and what proof you need
Change in AI searchWhat to doWhat to keep doingEvidence needed
AI Overviews and AI Mode show supporting links for some queries. Google says they "often don't trigger."Do not write a separate page for each AI query. Check that the pages you want found are indexed and eligible to show with a snippet.Improve the pages people already reach through Search.Dated query record, Search Console data for the Web search type, and the link shown for each query.
Google's AI features need no special files or markup.Stop treating AI text files or AI-only rewrites as Google requirements.Keep normal crawl access, clear titles and internal links.Google's published text and a dated check of the live site.
OpenAI's search crawler decides whether a site appears in ChatGPT search answers.Decide on OAI-SearchBot separately from training. Blocking it removes the site from ChatGPT search answers.Keep the robots.txt file readable and consistent across the site.The live robots.txt file, dated, and OpenAI's description of each crawler.
Training crawlers are separate from search crawlers. These include GPTBot, ClaudeBot and Google-Extended.Decide training permission on its own. Do not let it decide search access by accident.Keep search crawlers allowed if you want search visibility.The rule as written and the vendor's description of what the token controls.
Structured data is read alongside the page.Remove markup that describes things the visible page does not show.Keep accurate business and page details.Validation output and a field-by-field check against visible text.

Google's guide says AI Overviews and AI Mode have no additional technical requirements, and that Google's systems decide when an AI Overview adds value. [1]

What to do: before adding any AI-specific work, confirm the main service pages can be reached, read and listed.

Which crawlers should your robots.txt treat separately?

Robots.txt controls crawling, not indexing. Googlebot rules affect Google Search. Google-Extended controls whether Google may use content to train Gemini models or for grounding, and Google says it does not affect inclusion in Search. OpenAI and Anthropic publish separate crawlers for training and for search or user requests, so each needs its own decision.

OpenAI says GPTBot is used for training generative models and that its setting is independent of OAI-SearchBot. It says OAI-SearchBot surfaces websites in ChatGPT search features, and that ChatGPT-User is not used to decide whether content appears in Search. [5] Anthropic says ClaudeBot collects content that could contribute to training, and that its bots honour robots.txt. [6] Anthropic's page also lists Claude-User and Claude-SearchBot. This guide does not add them to the checklist, so read that page before writing rules for them.

Google also says robots.txt is not a way to keep a page out of Google. For that, it points to a noindex directive or password protection. [4]

What should you check on the site?

Work through these groups in order. Access comes first, because a page that cannot be crawled cannot be indexed or used by a search feature. Record each check with a date, the page address and the result, so later changes can be compared with a known starting point.

Crawl access and robots

  • Open the live robots.txt file and read the rules for Googlebot, Google-Extended, OAI-SearchBot, GPTBot and ClaudeBot.
  • Allow Googlebot and OAI-SearchBot if you want those systems to read pages for search.
  • Decide GPTBot, ClaudeBot and Google-Extended separately, because they relate to training.
  • Check that a CDN or hosting firewall does not block these crawlers.

Indexability

  • Check that important pages return a normal page response.
  • Check that no noindex tag or header is present unless you intend it.
  • Check Search Console for each priority page, and note any reason given for it not being indexed.
  • Do not use robots.txt to hide a page from Google.

Clear answers on the page

  • Put the buyer question in a heading.
  • Answer it in the first lines below the heading, in words that still make sense when read alone.
  • Use a list where the content is a set of steps.
  • Name the service, place or price basis only where the page already shows it.

Entity and about information

  • Make the business name, owner, location and service area match across the about page, contact page and footer.
  • Show who wrote each guide and why they know the subject. Google asks for clear authorship and sourcing. [2]
  • Use the same business description on every platform you control.

Structured data that matches visible content

  • Mark up only what the visible page shows.
  • Google says no special schema is required for its AI features, and that structured data should match the visible text. [1]
  • Validate the markup, then compare each field with the page.

Measurement in Search Console

  • Read the Performance report with the Web search type.
  • Google says sites shown in AI features are included in overall Search traffic in Search Console. [1]
  • Note the date of each check and any site change made near it.

Google says recrawling after a change can take days to months, and that you can request a recrawl. [1] A pass on any check shows what was tested on that day. It does not promise a result in AI search.

Need the access and markup fixes made? See technical improvements.

How does a page move from crawl to an AI answer?

A page first has to be reached by an allowed crawler, then kept in the index, then shown in search. An AI feature can only use pages that have passed these steps. Passing them does not guarantee a mention or a link. The diagram shows the search path and the separate training branch.

How a page moves from crawl to an AI answer

  1. 1. CrawlAllowed crawler reads the page
  2. 2. IndexPage is indexed and not noindex
  3. 3. ResultEligible to show with a snippet
  4. 4. AI answerMay use it as a supporting link
  5. 5. MeasureSearch Console, dated AI checks
Training branch, separate GPTBot, ClaudeBot, Google-Extended Controls training use. Search crawlers are set separately.
Read left to right. Each step needs its own evidence, and passing one step does not prove the next.

[1] [5] [6]

How often should you review AI search readiness?

Review access every month, and review answers and structure every quarter. A monthly check catches robots.txt or hosting changes early. A quarterly review gives Search Console data and dated assistant checks enough time to show a pattern. Record what you looked at even when nothing changed.

  1. Monthly, access: open the live robots.txt file, confirm the crawler rules have not changed, and check five priority pages return a normal response with the intended main address.
  2. Monthly, Search Console: read the Performance report with the Web search type for those pages. Note changes, but do not draw conclusions from one month.
  3. Monthly, assistants: run the same saved questions in each assistant you track. Record the date, answer, whether the business is named and whether a source link appears.
  4. Quarterly, answers: review the question heading and the first answer on each priority page. Check that it still matches the visible content.
  5. Quarterly, facts: check that the about, contact and service details agree across pages, and that structured data still matches the visible text.
  6. After any release: repeat the access and indexability checks. Google says recrawling can take days to months, so record the date and check again later.

What does each check prove, and what does it not prove?

Each check answers one narrow question on one date. A robots.txt rule proves what the file said when it was read. It does not prove the page was crawled, indexed or quoted. Writing the limit beside each result stops a team from reading a pass as a promise.

Evidence limits for AI search checks
CheckWhat it provesWhat it does not prove
Robots.txt ruleThe live file allows or blocks the named crawler on the date read.That the crawler visited, or that the page is indexed or used in an answer.
Noindex tag or headerThe directive is present in the HTML or response that Google receives, when checked with URL Inspection.When Google will recrawl the page.
Search Console Performance, Web search typeImpressions and clicks for pages in Search, including pages shown in AI features.Which feature produced a click, or that AI features led to an enquiry.
AI Overview or AI Mode linkA dated record shows the page as a supporting link for one query.That the page will appear again, or that the result is typical.
Business named in an assistant answerThe assistant named the business in one saved answer.That the page was used, or that the mention will repeat.
Source link in an assistant answerThe saved answer linked to the page.That the wording came from the page, or that anyone visited it.
Structured data validationThe markup parses and matches the fields that were checked.That the markup is required, or that it caused a citation. Google says no special schema is required. [1]
AI text file such as llms.txtThe file is served at its address.That Google reads or uses it. Google says no such file is required. [1]

How do you triage common AI search problems?

Start with the symptom you can see, then test the narrowest cause first. Do not change content until you know whether the problem is access, answer quality, facts or measurement. This tree sets the first investigation. It does not prove a cause.

  1. Not cited by AI tools?
    Check the page first. Is it public, indexed and reachable by the crawler? If not, fix access before changing content. If yes, check whether the page answers the saved question directly under a matching heading, then record the prompt, platform and date.
  2. AI bot blocked?
    Open the live robots.txt file and read the rule for that bot. Then check any CDN or hosting firewall. Write down the crawler, its purpose and what you expect to change before editing a rule. OpenAI says robots.txt rules may not apply to ChatGPT-User, because user-initiated actions are involved. [5]
  3. Wrong facts about the business?
    Compare the answer with the about page, the service page and the structured data. Correct the source page first, then record the wrong answer, the platform and the date. Do not rely on an AI text file to correct it.
  4. Traffic down but impressions up?
    Check the query and page in Search Console with the Web search type, and compare clicks with a dated baseline. Look at the query mix and the page before blaming crawling. An impression is not a visit.

What can you use to record each priority page?

Use one record for each priority page. Start with the access card, then the answer card, then the evidence log. Keep a mention, a source link, a click and an enquiry in separate fields, so one good result does not hide the others.

Three records per page. Kept apart.

Copy these cards into a spreadsheet or project ticket. Fill them with your own pages and dated results. The example values below are illustrative only.

01

Page access and listing

One row for each priority page.

Page address
The clean address you want found
Response
Normal page response, checked on a date
Crawler rules
Rule read for each named crawler
Indexed
Search Console status, with any reason given
Owner
Person who checks it each month
Example (illustrative)

Page: /services/your-service
Response: normal, checked on the date recorded
Rules: Googlebot allowed; GPTBot decided separately
Indexed: yes, per Search Console

02

Answer and facts

Check what the page says before checking what an assistant says.

Buyer question
The question the page should answer
Heading matches
Yes or no, and the heading text
Direct answer
First lines under the heading, read alone
Facts checked
Name, location, service area and dates against sources
Markup matches
Structured data fields against visible text
Example (illustrative)

Question: What does the service include?
Heading matches: yes
Facts checked: service area matches the contact page
Markup matches: yes, on the date recorded

03

Evidence log

One line for each dated check, kept apart from the others.

Date
When you checked
Platform and prompt
Where you checked, and the exact question
Business named
Yes or no, with the answer saved
Source link shown
Yes or no, and the link
Next step and owner
The agreed action and who takes it
Example (illustrative)

Date: check date
Platform and prompt: one assistant, one saved question
Business named: yes
Source link shown: no, so no link is recorded as shown

Keep mentions, source links, clicks and enquiries as separate events. A named business is not the same as a linked source, and neither is a sale.

What else do people ask about AI search?

These answers use only what Google and Microsoft say in their own documentation, checked on 7 November 2026. Where a question has no published answer, the reply says so instead of guessing at how the systems work. Each one is written for a site owner, not a developer.

Do I need a special file or markup to appear in AI Overviews?

No. Google says you do not need to create new machine-readable files, AI text files or markup to appear in AI features, and it says no special schema.org structured data is required. Google also says structured data should match the visible text on the page. The page must be indexed and eligible to show in Google Search with a snippet.

Does blocking AI crawlers affect my place in Google Search?

Blocking Googlebot stops Google from crawling the page, which does affect Search. Google-Extended is a separate robots.txt token. Google says it does not impact a site's inclusion in Google Search and is not a ranking signal. Blocking it is a choice about training and grounding use. To keep a page out of Google, Google points to noindex or password protection, not robots.txt.

Can I track how often my pages appear in AI answers?

Google's Search Console counts AI feature traffic in the Performance report, under the Web search type. Google's page does not describe a separate AI report. Bing Webmaster Tools has an AI Performance report, announced in February 2026 as a public preview. It counts citations in AI answers. The Bing article does not say it measures clicks, and it calls the data a sample.

Does my existing SEO work still count?

Google says standard SEO practices still apply to AI Overviews and AI Mode. Pages that feed these features must be indexed and eligible to show in Google Search with a snippet. Google's people-first guidance still asks for original information, clear authorship and sourcing. It warns against pages primarily made to attract visits from search engines.

Primary sources

  1. AI features and your website, Google Search Central. Checked 7 November 2026.
  2. Creating helpful, reliable, people-first content, Google Search Central. Checked 7 November 2026.
  3. Google's common crawlers, Google Search Central. Checked 7 November 2026.
  4. Introduction to robots.txt, Google Search Central. Checked 7 November 2026.
  5. Overview of OpenAI crawlers, OpenAI. This address replaced platform.openai.com/docs/bots. Checked 7 November 2026.
  6. Does Anthropic crawl data from the web, and how can site owners prevent crawling?, Anthropic Help Center. Checked 7 November 2026.
  7. Introducing AI Performance in Bing Webmaster Tools Public Preview, Bing Webmaster Blog, Microsoft, 10 February 2026. Checked 7 November 2026.
ClaroSi logo
ClaroSi

At ClaroSi, we run evidence-led website and search reviews for Irish startups, solo founders and small businesses. Read about the approach.