GEO and AEO: what the evidence actually supports

Training, citation and live fetch are three different things

Each assistant runs more than one crawler, and they do different jobs. Blocking the wrong one has consequences nobody intended.

OpenAI also notes a propagation delay of roughly a day between a robots.txt change and its systems reflecting it, so a check made minutes after an edit proves nothing.

Whether ChatGPT runs on Bing's index

This is repeated everywhere and it was never quite true. What existed was a privacy disclosure: OpenAI's help documentation said ChatGPT might share disassociated search queries with third-party search providers such as Bing. Bing was named there as a recipient of queries, and the index was not described at all.

That wording has since changed twice. It became "sometimes partners with other search providers", and Shopify was added as a second named provider, so Bing is no longer even the only one listed. The restructured Microsoft and OpenAI agreement of October 2025 covers compute, intellectual property and Azure exclusivity, and says nothing about supplying a search index.

What OpenAI does say plainly is that OAI-SearchBot must be allowed to crawl a site, and that the site's host or CDN must allow traffic from OpenAI's published address ranges. That crawler is what gates citation. One detail trips people up: OAI-SearchBot crawls from Azure address space, because OpenAI runs on Azure, and that gets misread as evidence of a Bing dependency.

Bing does matter in one real place. Microsoft Copilot genuinely runs on Bing, with no separate crawler of its own, and Bing Webmaster Tools is currently the only publisher-facing report that shows AI citations at all. Google's AI surfaces run on Google's index. Perplexity runs its own crawler and claims its own index. Anthropic has not documented what backs Claude's web search.

So being in Bing is worth having, and it is not a proxy for anything else. The Bing Search APIs were switched off in August 2025, which also removed the mechanism by which third-party products used to piggyback on that index.

Check nothing is already blocking the crawlers

Before any of the content advice matters, the requests have to arrive. Some sites block AI crawlers without anyone having decided to, because the control sits at the CDN and was applied there rather than chosen.

Cloudflare separates its controls into training, search and live user fetches. That separation is the part worth understanding: an owner who blocks AI crawlers to keep content out of model training will, unless careful, also switch off eligibility to be cited. Those are opposite intentions served by adjacent switches.

Checking it takes one command per crawler, and it works against any domain from any machine.

curl -A "OAI-SearchBot" -o /dev/null -w "%{http_code}" https://example.com/

Requesting a real page as OAI-SearchBot and getting anything other than a 200 with the actual HTML means the content advice below is irrelevant until that is fixed. The same test applies to Claude-SearchBot, PerplexityBot, Googlebot and bingbot. A CDN that returns a challenge page to a crawler has removed the site from that surface as effectively as a robots.txt disallow, and it does so silently.

Also worth checking whether robots.txt as served matches robots.txt as written, since some CDNs now manage or prepend to that file.

One dated change makes this urgent rather than theoretical. Cloudflare's documentation for its legacy Block AI bots option now carries the heading "Deprecating on September 15, 2026", and states that from that date "Mixed-purpose crawlers that combine Search and Training will also be blocked by all configurations to block AI training, including the legacy 'Block AI bots' option." Cloudflare's announcement of 1 July 2026 names them: "multi-purpose crawlers such as Googlebot, Applebot, and BingBot will be blocked by customers who have selected to block Training".

Today that toggle excludes crawlers that do both jobs, so Googlebot passes. After that date the same untouched setting is evaluated against every behaviour a crawler has, most restrictive winning. Nobody has to change anything for the outcome to change. Anyone who switched on AI blocking once, on principle, and moved on is the person this affects, and the remedy is to set Search to Allow and record the opt-out for mixed-purpose crawlers before the date.

Structured data is being read as text, not as structure

Adding schema markup for AI assistants is close to universal advice. A test by Mark Williams-Cook in May 2026 was a simple one: publish deliberately invalid JSON-LD containing a fabricated address, on a page where that address appeared nowhere in the visible text, then ask the assistants about the business.

ChatGPT and Perplexity both returned the fabricated address, and Perplexity said it had found it in the page's embedded structured data. On that evidence the markup was reaching them as text on the page rather than as parsed structure.

That result is one test, not a literature, and it should be held that lightly. What can be said is that no published study demonstrates a citation benefit from schema, and that this test offers a mechanism for why there might not be one.

None of which is an argument against schema. It earns its place for Google's rich results, where the benefit is documented and mechanical. The narrower claim, that assistants parse it as structure, is the one without support.

The file this site serves that nobody has confirmed reading

llms.txt is a proposed convention: a plain text index at the root of a site, pointing at markdown versions of the important pages, on the theory that an assistant would rather read clean prose than parse a page full of navigation.

This site serves one, along with a markdown mirror of every content page. No AI vendor has confirmed that they read it. Not one has documented support for the convention.

It is served here anyway, and the reasoning should be stated precisely because it is not evidence. The cost is close to zero, the mirrors are useful to anyone reading in plain text, and if the convention is adopted the work is already done. That is a bet on a cheap option. It is presented as established practice in a lot of published advice, and it is not.

Reddit is read constantly and cited rarely

Reddit's prominence in AI answers has produced a tactic: get the brand mentioned in threads, and the assistants will repeat it. The measurement does not support the tactic, and the numbers are more interesting than either side of the argument suggests.

Ahrefs analysed 1.4 million ChatGPT prompts and found Reddit cited in 1.93 per cent of retrievals, while making up 67.8 per cent of the URLs that were retrieved and then not cited. Assistants read Reddit constantly and quote it seldom. Yext, across 6.8 million citations from 1.6 million responses, put forums at around 2 per cent. SE Ranking, testing 100,000 keywords in June 2024, found Reddit had dropped out of the top ten linked domains in AI Overviews entirely.

Citation does not track quality either. Semrush, looking at 248,000 Reddit URLs, found 80 per cent of cited posts had fewer than twenty upvotes. The rate is also unstable in a way content cannot explain: ChatGPT's Reddit citation rate fell roughly sixfold over six weeks in late 2025, with no change to Reddit's content over the period.

The licensing arrangements are also less settled than the retelling suggests. The figure everyone quotes for the Google deal was never confirmed by either party, and Reddit's filings name no counterparty at all, referring only to two partners. Google's own announcement contains a sentence that rarely survives into the summaries: the arrangement "does not change Google's use of publicly available, crawlable content for indexing, training, or display". Reddit's robots.txt is a blanket disallow with no carve-outs, Googlebot included. The access runs through a contractual arrangement instead.

There is a second reason not to do it, separate from the absence of evidence. Reddit's rules prohibit paid promotion outright, and the stated sanctions extend to bans at the level of a domain rather than an account. When a university research team ran language model personas in a subreddit, with ethics approval, Reddit's chief legal officer called it "deeply wrong on both a moral and legal level" and issued formal demands. A consultant doing the commercial version has no such cover. That is enough on its own to advise a client against it.

The question nobody has measured

Here is the gap at the centre of the subject. Every study showing that a content change improves AI visibility starts from pages that are already being retrieved. The GEO paper measured how a generator treats sources it has already found. The citation studies measure which retrieved pages get quoted.

Whether any on-page change causes a page that was never cited to start being cited has not been measured by anyone. That is the step every site owner actually cares about, and the confident advice about it is guesswork.

One thing is not guesswork, because it is mechanical: a page that the citation crawlers cannot fetch cannot be cited. That is the crawler and CDN question at the top of this page, and it is the only part of the subject where cause and effect are documented by the vendors themselves. Everything downstream of it is currently belief, including the parts of this page that describe what I do on my own site.

What this site does, and what I cannot prove about it

This site serves llms.txt, a markdown mirror of every content page, JSON-LD on every page including a shared entity node, and a sitemap. Nothing at the CDN blocks any AI crawler, which I verified by requesting pages as each of them rather than by reading the settings page.

I cannot demonstrate that any of it causes a citation. The site is new, the traffic is negligible, and a handful of clicks is not a dataset. What I can say is what it costs, which is almost nothing, and what it would take to know, which is a controlled test nobody has run.

The reason to write this down is that the subject is being sold hard by people who are not distinguishing between the things they have measured and the things they assume. The gap between those two is most of the published advice.

Questions this raises