Want to optimize for AI? First check if bots can actually read your site

Most guides on “AI optimization” start with content. Write clearly, add structured data, answer questions, build authority. All correct. But they skip a step that comes one level earlier: whether bots from ChatGPT, Perplexity, or Google can actually reach your site and read it.

When ChatGPT, Perplexity, or Google in AI mode answers a question, it usually cites someone. It pulls content from pages its bots have crawled and processed. If bots can’t reach you, you’re not in the answers. No amount of content tweaking will change that — because you’re polishing content that AI never read.

Why address this now

Traffic from AI assistants is growing fast. The share of visits sites receive from links in ChatGPT and similar tools grew roughly sevenfold over 2025, from 0.02% to 0.15% of total traffic. Still a minority, but growing sharply. And this kind of visit can be valuable: someone who clicks a source from an AI answer usually arrives with a specific intent, not just starting to look around. In e-commerce, such visits convert more readily than average search traffic — in an analysis of 94 e-shops, traffic from ChatGPT converted 31% better than non-branded organic search.

At the same time, new barriers have appeared. Cloudflare, which runs a large portion of the web, has been blocking AI bots by default on new domains since July 2025. Many sites are now restricting AI. Some owners decided this consciously; many have no idea — the infrastructure decided for them.

Where AI access most often breaks down

In practice, it tends to break at four places. None of them are visible when you open your site in a browser, because you see it as a human from a regular IP address — not as a language model’s crawler.

The robots.txt file. Rules where you tell your site which bots may go where. The problem is that every AI bot has its own name (token). OpenAI’s GPTBot, ClaudeBot, PerplexityBot, bingbot (which also feeds Copilot), Google-Extended for Gemini — each accesses the site differently. It’s easy to let Googlebot through while accidentally blocking the rest. On top of that, Cloudflare broadly adds robots.txt lines to many sites declaring “yes for search, no for AI training” — often without the owner knowing.

Why a robots.txt ban doesn't always mean blocked

Some bots don’t follow robots.txt at all — intentionally. When a user in ChatGPT asks specifically about your site, the bot that goes to fetch that page doesn’t follow robots.txt rules, according to OpenAI’s documentation. So “banned in robots.txt” doesn’t mean “blocked” for certain bots. Mixing up these categories makes any audit meaningless.

A firewall in front of your site. A common block you won’t notice. Your robots.txt is fine, you allow bots — but the protective layer in front of your site (a CDN or firewall, typically Cloudflare) rejects any request that identifies itself as an AI bot before it ever reaches your content. The site works normally for humans. For a bot like Perplexity’s, it returns a “verify you’re not a robot” page or an outright block. There’s no trace of this in robots.txt, so a routine check finds nothing.

Content rendered only by JavaScript. Many sites send an almost empty page to the browser and only then render the text via a script in the browser. Googlebot, bingbot, and Applebot can “compute” such a page, so Google in AI mode, Gemini, and usually Copilot and Apple will see the content. But bots from ChatGPT, Perplexity, and Claude don’t run JavaScript. They may download a nearly empty page and see nothing more. If your product description or article body is loaded this way, it’s as if it weren’t there at all.

Indexing directives. Small instructions in the page header like noindex, nosnippet, or noarchive can silently remove a page from AI answers even if it’s otherwise fully accessible. A developer often leaves them from a staging environment and no one notices.

Why you can’t check this reliably by hand

It’s tempting to take the shortcut — open robots.txt and call it done. It’s not that simple.

There are dozens of bots and they fall into three categories evaluated differently. A classic crawler respects robots.txt. A user-triggered bot usually ignores it. A third category, like Google-Extended, isn’t a bot at all — it downloads nothing from your pages. It’s just a robots.txt switch that lets you allow or block Google from using your content for AI training. Mix up those categories and you’re either solving a block that doesn’t exist, or missing the one that’s actually costing you a spot in AI answers.

Going through each token individually, testing HTTP access against a control visit from a regular browser, identifying whether a firewall is blocking, comparing raw and rendered pages for JavaScript issues — and throughout all of that, staying clear on what’s evidence and what’s just a hint. Done by hand, it takes hours and is easy to get wrong.

What we can do about it

re:LLMO has a tool for this — a Claude Code extension. Give it a website address and it crawls the way AI bots actually see it.

It takes a sample of different page types from the sitemap — homepage, category, product, article — and checks access for each. For every important bot it reads the robots.txt rules against the current standard. It tests HTTP access under each bot’s identity against a control visit and detects when a firewall or CDN is blocking. It compares the page before and after JavaScript runs, showing exactly what ChatGPT, Perplexity, and Claude’s non-rendering bots miss. And it carefully distinguishes between a proven block and an indication that needs verification in server logs.

The output is a clear report: who reads you, who doesn’t, where the problem is, and what to do about it. No paid API, no sign-up, a few minutes of autonomous agent work.

A critical first step that’s worth taking before you invest more time and money in content creation. First make sure AI can read you. Then worry about how it cites you.