The Fault Lines: Where AI Search Platforms Disagree, and What llms.txt Actually Does

At the end of the last post I left you with a tension. The five layers of entity existence are real, and the fintech founder's numbers proved it. But the same work gives you different results on different platforms, because each platform is reading a different slice of the web. I promised to map those differences and to settle the llms.txt argument as far as it can be settled. That is this post.

A warning before we start. This is the part of the field where people are most certain and least correct. I am going to tell you what I have measured across client accounts, what the platforms have said in public, and where I am guessing. I will label the guesses.

Four readers, four libraries

Think of each AI platform as a reader who has been given a library card to a different library. The reader is smart. The library decides what they can know.

ChatGPT reads two libraries. The first is its training corpus, which leans heavily on Wikipedia, licensed news, books and the better-written half of the open web. The second is a live index it queries when the question needs current information. OpenAI's own crawler builds part of that, and OpenAI has said in public that Bing's index underpins search in ChatGPT. The practical effect is that ChatGPT rewards the things Wikipedia rewards: notability, independent coverage, clean facts repeated in reputable places. If you exist in the knowledge graph layer, ChatGPT finds you easily. If you exist only on your own site, it finds you when it searches and forgets you when it does not.

Perplexity reads the open web at the moment you ask. It runs its own crawler and its own index, and it retrieves fresh pages for nearly every answer. That makes it the platform most like classic search, and the one where page-level work still pays directly. It also leans on Reddit and forum content more than ChatGPT does, because that is where fresh, specific, opinionated answers live. If your category is discussed in a subreddit and you are absent from the discussion, Perplexity will not mention you, whatever your schema says.

Google AI Overviews and AI Mode read Google's index, which is the largest library of the four, but with a bias that surprised a lot of SEOs. Since Google's 2023 and 2024 updates that elevated forum and first-hand content, and since the 2024 licensing agreement that gave Google structured access to Reddit, Overviews cite Reddit, YouTube and Quora far more often than their share of the web would predict. Google is also the only platform with its own Knowledge Graph, so a Knowledge Panel for your brand feeds straight into how AI Mode describes you.

Claude reads its training corpus, which is curated more conservatively than the others, and a live web index when search is turned on. In my testing it is the most cautious about naming brands it is not sure of. It would rather say "several providers offer this" than guess. That makes it hard to win and hard to lose. The brands Claude names are the ones with the cleanest, most consistent public record.

Put those four side by side and the tension from the last post becomes concrete. A Wikidata entry moves ChatGPT and Google. It does almost nothing for Perplexity next week. A strong Reddit presence moves Perplexity and Google Overviews. It does little for ChatGPT's training-time memory of you. Consistency moves Claude. Nothing moves everything.

Where the platforms disagree, and why it matters for you

I keep a scorecard for every client. For each of the five layers from the last post, I note which platforms respond to it, and how quickly. It is not scientific. It is twenty months of running the same prompts weekly across thirty-odd brands and watching what changes. This is the shape of it.

Entity home and structured data (Layers One and Two) move every platform, but slowly and quietly. They do not make you appear. They stop you from being described wrongly once you do appear. The founder's hallucinated "headquartered in Singapore" problem disappeared within six weeks of fixing the About page and the schema. His citation rate did not move in that window. Those are different jobs.

Knowledge graph presence (Layer Three) moves ChatGPT and Google the most, Claude somewhat, Perplexity least. The Wikidata entry showed up in ChatGPT's answers about his company within roughly two months. I cannot tell you the mechanism, and anyone who says they can is guessing. I can tell you the correlation has held across every client where we did it.

Co-mention density (Layer Four) is the one that moves Perplexity and Google Overviews fastest, and here the fight over where the mentions happen matters more than how many. A comparison article in a trade publication helps ChatGPT at the next training cycle. A thread on the right subreddit helps Perplexity on Thursday.

Consistency (Layer Five) is the floor for all four. The one platform that punishes inconsistency most visibly is Claude, which will simply decline to name you.

So the honest answer to "which platform should I build for" is "which platform are your buyers using". That is a question you can answer with data rather than opinion, and most companies have not asked it. Your sales team knows which AI tool prospects quote back at them. Your support inbox knows. Your referral traffic in GA4, filtered to the AI domains, knows. Pick the one or two that matter and weight the layers to suit.

For the fintech founder, the answer was ChatGPT first, Perplexity second. His buyers were Series A founders in the UK and Nigeria, and in both markets those two were what people reached for. We put Google third, which would have been the wrong order for an e-commerce client and the right order for him.

Reddit, and the uncomfortable part

I have to address Reddit directly, because it sits underneath two of the four platforms and most B2B marketers would rather it did not.

Reddit matters to AI search for a dull reason. It is the largest public archive of real people answering specific questions in their own words, and both Google and OpenAI have paid for access to it. When a model needs to know "which business bank do founders actually like", a thread with forty replies is better evidence than any landing page, and the models have been built to treat it that way.

That creates a temptation, and the temptation is to fake it. Seeded threads, employees posing as customers, agencies selling "Reddit presence" packages. Do not. Reddit's moderation is better than people assume, and the models are trained on the downvotes as well as the upvotes. A thread where your brand is recommended and then torn apart is worse than no thread.

What works is slower. Founders and senior staff answering questions in their category under their own names. Sharing something useful without a link. Being present in the two or three subreddits where your buyers already are, for months, before anyone mentions your product. When the brand does come up, it comes up because someone else raised it. That is the only version of this that compounds.

Two more sources deserve a line each. YouTube matters to Google Overviews and increasingly to ChatGPT, and the transcript is what gets read, so say the brand name and the category out loud. Quora is cited more than it deserves to be, and a clear, honest answer there under a real name still gets picked up.

The llms.txt argument, settled as far as it can be

Now the file.

The proposal came from Jeremy Howard at Answer.AI in September 2024. The idea is simple. You put a Markdown file at /llms.txt on your domain that describes what the site is, links to the pages that matter most, and tells a language model where to look. A companion convention, llms-full.txt, carries the full text of your key documentation in one file so a model can read it without crawling.

Supporters point to adoption. Thousands of sites have published one, including documentation sites from Anthropic, Stripe, Cloudflare and a long list of developer tools. Several AI coding assistants and documentation tools read the file. The argument is that it costs almost nothing and gives a model a clean map.

Sceptics point to the absence of evidence that it affects search answers. John Mueller at Google compared it to the old keywords meta tag in 2025, a self-declaration with no outside check, and said Google does not use it. No major chat platform has confirmed reading it for search. Log analyses published by several SEO teams through 2025 found the big crawlers rarely requested the file at all. Nobody I know of has shown a controlled lift in citations from adding one.

Both sides are right about different things, and the argument has been confused because people are using one name for two different jobs.

Job one is documentation retrieval. A developer's AI assistant needs to read your API docs. Here llms.txt and llms-full.txt do exactly what they claim. The tools that read them are real, the benefit is immediate, and if you have documentation you should publish both. This is not search. It is a reading aid.

Job two is search visibility. Getting ChatGPT or Perplexity to recommend you. Here the file does nothing that can be measured, and the mechanism it proposes runs against how the platforms actually work. They do not ask your site what it is about. They ask the rest of the web, and they verify against the knowledge layer. A file you wrote about yourself is the one source they trust least. That is the whole lesson of the last post.

So my position, after a year of watching this. Publish llms.txt if you have documentation, a product people integrate with, or a large content library that models should be able to navigate. Spend half an hour on it. Then stop thinking about it, and do not let anyone sell you "llms.txt optimisation" as if it were a ranking lever. If your site is a brochure, skip the file entirely and put the half hour into Wikidata.

For what it is worth, this site has no llms.txt as I write this. It has a Wikidata entry, a complete sameAs array, and four press mentions. I know which of those moved the needle, because I can see it in the answers.

The things everyone is fighting about that do not matter

While we are here, three other arguments you will meet, and how much weight I give each.

Blocking AI crawlers. A growing number of publishers block GPTBot, ClaudeBot and the rest in robots.txt, on the principle that training should be paid for. For a publisher whose product is the content, that is a legitimate commercial decision. For a company whose product is something else and whose content exists to be found, it is self-harm. You cannot be recommended by a model you have told to go away. Decide which kind of site you are, and do not copy a newspaper's robots.txt because it looked principled.

"GEO" as a separate discipline. Generative engine optimisation is a useful label and a poor strategy. Nearly everything that moves AI answers is a thing good SEO and good PR already did: be a clearly defined entity, be talked about by others, be consistent, be fast and crawlable. The parts that are new are the platform weightings I described above, and the knowledge-layer work most SEOs never learned. Buy those two skills. Do not buy a rebrand of the rest.

AI-generated content at scale. There is a camp that says you should publish thousands of machine-written pages to seed the corpus with your brand. It fails on both platforms that matter. Google's spam policies on scaled content abuse were written for exactly this. And the models are trained to discount text that looks like text they would have written. Co-mention is earned by being mentioned by someone else. You cannot mention yourself ten thousand times and call it authority.

What I would do on Monday

If you have read both posts and want the short list, this is it. In order.

1. Find out which platforms your buyers use. Ask sales. Check referral sources. Pick two.
2. Fix your entity home and schema. One week. The About page reads like an encyclopedia entry; the Organization or Person schema carries a full sameAs array and consistent facts. This is the floor, and it is still missing on most sites.
3. Get into the knowledge layer. Wikidata first, because you can do it yourself in an evening if you have independent coverage to cite. Crunchbase second. Wikipedia only when the coverage is there; do not force it.
4. Weight co-mention to your platforms. ChatGPT means trade press and comparison content. Perplexity and Google Overviews mean Reddit, YouTube and the forums where your category is discussed. Do it under real names, slowly.
5. Run an audit for inconsistency across every profile and every mention. Fix the founding year. Fix the city. Fix the spelling of the founder's name. This is the cheapest work on the list and the most often skipped.
6. Publish llms.txt only if you have documentation. Half an hour, then move on.
7. Measure the answers, not the rankings. A fixed set of twenty category prompts, run weekly on each platform, logged in a sheet. Citation rate is the number that matters. It will not move for two months. Then it will.

Where the founder is now

I said in the last post that his competitor went from winning seven queries in ten to four in ten. I can add the platform split now, because it tells the story of this post better than I can.

On ChatGPT he went from almost absent to cited in roughly half of category queries, and the Wikidata entry and the trade-press roundup were the two changes that preceded the move. On Perplexity his gains came later and came from somewhere else: a founder-written answer in a subreddit thread about business banking for startups, which still gets cited eight months on. On Google AI Overviews he is barely there, because the category is dominated by comparison sites and he has not yet done the YouTube work. On Claude he is now named, which he was not before, and I attribute that to consistency rather than to any one source.

Same company. Same six months of work. Four different outcomes, because four different readers with four different library cards.

That is the fault line. Not llms.txt, not GEO, not any single tactic. It is that there is no longer one search engine to be found in, and the work of becoming an entity has to be aimed. Aim it at the platform your buyers use, do the unglamorous knowledge-layer work first, and let the rest of the industry argue about the file.