Why Your Site Needs to Be Readable by AI Too, Not Just Google

01

AI Is Where People Ask Now, Not Just Google

How AI picks its answers is different from how Google ranks its top ten.

People now ask ChatGPT or Perplexity directly for all sorts of things, from "recommend a service in this city" to "what's the difference between product A and B". If you sell a product or a service, this is a new channel that's starting to carry its own weight, separate from regular Google search results.

The problem is that how AI picks its sources is different from how Google ranks its top ten results. The Ahrefs study mentioned above found AI Overview often pulls from outside the top ten — sometimes from pages that don't even make the top hundred [ahrefs]. So ranking well on Google still matters, but it's no longer the only thing that decides whether you get mentioned.

02

What AI Crawlers Actually Read

These three things matter, not the same old SEO tricks repackaged.

AI systems like ChatGPT and Perplexity don't read your site live when answering a question. They rely on crawlers — GPTBot, ClaudeBot, PerplexityBot, and similar — that visit ahead of time to gather content. If your robots.txt blocks these crawlers, sometimes without you even realizing it because of an overly strict default template, your business will never show up in an AI answer. No amount of other SEO work fixes that; the door was already closed.

Two other things matter just as much. First, your main content needs to exist in plain HTML, not only appear after JavaScript runs — not every AI crawler executes JavaScript, so content that depends entirely on it risks never being read at all. Second, headings structured cleanly from H1 through H3. AI crawlers read that heading hierarchy as a map of your argument's structure, not just as different font sizes.

• robots.txt allows GPTBot, ClaudeBot, PerplexityBot, and Google-Extended
• Main content exists in HTML, not only after JavaScript runs
• H1 through H3 follow the argument's flow, not just visual styling

03

llms.txt Is Not a Magic Key — Google Says So Itself

Fine to add, but don't make it your only effort.

You may have read advice to add an llms.txt file at your site's root so "AI immediately understands every important page." The idea sounds reasonable, but there's no proof behind it yet. Google's John Mueller addressed llms.txt's benefit directly: "nobody knows — for now this is purely speculative," adding that files like it have existed for years without a single AI system actually using them [google].

We still include an llms.txt check in the Free Website Check, because it costs almost nothing to add and there's no downside. But we won't call it "essential" or "the key fix," because it simply isn't proven yet. If anyone sells you an "llms.txt optimization" package as the main solution, ask what evidence backs it up.

John Mueller's statement on llms.txt

04

Structure That Makes Your Answers Easy to Pull

This part you can act on directly in your own content.

Beyond crawler permissions, there's a part you can act on directly in your own content: making it easy to break apart and quote. AI answer engines tend to pull short definitions placed at the start of a section, numbered lists, or comparison tables — not long paragraphs where the answer only shows up in the fifth sentence.

• A one-sentence definition at the start of each section, not buried mid-paragraph
• Numbered lists or tables for comparisons, not just long narrative text
• Short Q&A blocks whose answers stand alone without needing prior context

05

Check Your Own Site, Free

Two of the Free Website Check's 18 checks cover exactly this.

Of the 18 checks in the Free Website Check, two cover exactly this: whether your robots.txt allows AI crawlers, and whether you have an llms.txt file. Both sit in their own category we call GEO, worth 12 of the 100 total score points — as much as the domain security category. Results come back in seconds, and you'll know right away which part to fix first.

Free Website Check