Key takeaways
- Access comes first. Allow the AI search and user crawlers in robots.txt and in your firewall.
- Put the content in the server HTML. Most AI crawlers do not run JavaScript.
- Structured data and clear headings help machines understand and attribute a page.
- Measure readiness here, and visibility in the assistants themselves. No tool can predict citations.
On this page
How assistants get their information
AI assistants use three sources for what they say about the web:
- Training data, collected months or years ago by training crawlers.
- A search index, built by their own search crawlers or bought from a search engine, queried when a question needs fresh information.
- Live fetches, when a person pastes a link or asks about a specific page and the assistant opens it.
The second and third are what put your current pages, with links, into answers. Both depend on a crawler reaching your page and reading it.
Step 1: let the right crawlers in
Read the guide to robots.txt for AI crawlers and decide on a policy. For visibility in AI answers, the search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and the user agents (ChatGPT-User, Claude-User, Perplexity-User) must be allowed.
Then check the layer most people forget: bot protection. Run the AI crawler checker. If AI user agents get a 403 or a challenge page while a browser gets the page, your CDN or firewall is blocking them regardless of robots.txt.
Step 2: put the content in the HTML
Google renders JavaScript. GPTBot, ClaudeBot, PerplexityBot and most other AI crawlers generally read only the HTML the server sends. A client rendered application shows them a near empty shell.
- Use server side rendering or static generation for any page you want read.
- Keep the main text, headings, links, titles and structured data in the server response.
- Avoid putting key information behind tabs, accordions or “load more” buttons that fetch it on demand.
The JavaScript SEO checker counts the words in the HTML before scripts run.
Step 3: make it easy to understand
Assistants pick passages to answer specific questions. Pages that are easy to parse are easier to quote accurately.
- One topic per page, named in the title and the H1.
- Descriptive headings that read like the questions people ask.
- Direct answers near the top of each section, then the detail.
- Structured data for the organisation, the articles, the products. It tells machines who publishes what. See the structured data guide.
- Facts that can be checked: dates, numbers with units, named sources.
Step 4: optional extras for machines
- llms.txt gives models a curated map of the site. Useful for docs and developer sites. See llms.txt explained.
- Markdown versions of pages, linked with
<link rel="alternate" type="text/markdown">, give agents clean text. - Agent discovery files, such as an MCP server card or an OpenAPI document, let agents find and call your services.
These are emerging conventions. They cost little and help the tools that use them. They are not ranking levers.
Step 5: measure what you can
The AI search readiness checker measures access, delivery and readability. It cannot measure whether you are cited, because that depends on the assistant, the question and the day.
For visibility:
- Ask the assistants about your brand, your products and your main topics, and note which pages they cite.
- Watch referral traffic from chatgpt.com, perplexity.ai, claude.ai and similar in your analytics.
- Keep a monitor on your key pages so a CDN change that starts blocking AI crawlers is caught the week it happens.
What not to do
- Do not serve AI crawlers different content from what people see. It is cloaking, and providers treat it as manipulation.
- Do not stuff pages with phrases aimed at language models. Clear writing for people is what assistants quote.
- Do not trust any tool that claims to predict AI citations. No one outside the providers can measure that reliably.
Check your own site
The AI Search Readiness Checker runs the checks from this guide on any public page, free and without an account.
Open the AI Search Readiness CheckerFrequently asked questions
What is the difference between SEO and GEO?
They share the technical base of crawlable, readable, well structured pages. GEO adds attention to the AI crawlers specifically, to content that works without JavaScript, and to how assistants pick and quote passages.
Do I need llms.txt to appear in ChatGPT?
No. There is no public commitment from the major assistants to read llms.txt. It is a cheap, useful extra, not a requirement.
How do I know whether assistants cite my site?
Ask them. Search your brand and your key topics in ChatGPT, Claude, Perplexity and Google, and check your analytics for referrals from those products.
Does Google AI Overviews use a separate crawler?
No. AI Overviews draw on Google's search index, built by Googlebot. Google-Extended controls training and grounding for Gemini, not AI Overviews in Search.