A stakeholder forwarded you a thread and asked whether the site needs this file. The thread was confident, the replies underneath it were not, and now you have to give an answer that will hold up if someone follows the topic more closely than you do.
Here is the shape of an honest answer: the proposal is coherent and cheap to implement, and the claims made about its effect on AI visibility are, as far as we can currently verify, not supported by first-party documentation from the systems it is aimed at. Those two facts sit together comfortably. This post keeps them separate on purpose, because almost everything written about this file collapses them.
Verification date for everything below: July 28, 2026. Every statement about documented use is a dated snapshot. Re-open the linked first-party material before quoting it.
What the file is and what it proposes to solve
llms.txt is a proposed convention: a Markdown file placed at the root of a domain, containing a curated, structured summary of the site with links to the pages that matter most, written for consumption by large language models rather than by people.
The problem it addresses is real and worth stating precisely. When a system retrieves a page to answer a question, it is usually handed HTML built for humans: navigation, cookie banners, interstitials, scripts, related-content modules, and the actual content somewhere in the middle. Context windows are finite. Extraction is imperfect. A hand-written Markdown index that says "here is what this site is, here are the twelve pages that matter, here is what each one covers" is a much cleaner input than a rendered marketing page.
The proposal generally involves two things:
/llms.txt— a curated index of the site in Markdown, with sections and annotated links./llms-full.txtor per-page.mdvariants — clean Markdown versions of the content itself, so a system can fetch the substance without parsing the wrapper.
As a piece of design, this is sensible. It is the same instinct that produced sitemaps and feeds: a machine-readable expression of what a site contains, maintained by the people who know.
What it is not
Four clarifications that resolve most of the confusion.
It is not a robots directive. It does not grant, deny, or condition access to anything. It contains no allow or disallow semantics. A site can have an llms.txt and block every crawler in existence, and the file will have no effect on that. Access policy lives in robots.txt and at your edge, and the decision behind it is a separate conversation covered in allow or block AI crawlers.
It is not a ratified standard. It is a published proposal with an adoption story, not a specification with a governing body and a compliance test.
It is not read at answer time by default. A system only benefits from the file if it fetches the file. There is no mechanism that makes an assistant look for it, and nothing in the format announces its own existence.
It is not a visibility mechanism. Nothing in the proposal claims to make a model recommend you. The file describes; it does not persuade. Treating it as an AI-visibility intervention is a category error, and it belongs to the same family of thinking as the other single-file fixes discussed in the quick-fix trap.
What first-party evidence establishes
This is where discipline matters most. A provider belongs in an adoption table only when its own documentation describes the relevant behaviour. Silence is not proof of non-use, and a server-log request is not proof that every product from that provider automatically discovers or prefers the file.
| Verified observation | What it establishes | What it does not establish | Checked |
|---|---|---|---|
OpenAI's developer-documentation index links to a site-level llms.txt file and says Markdown versions are available by appending .md to a page URL. | OpenAI uses the convention as a publisher for its own developer documentation. | It does not say ChatGPT or OpenAI crawlers automatically discover or privilege llms.txt on arbitrary websites. | 2026-07-28 |
| The original llms.txt proposal describes a voluntary Markdown convention and explicitly leaves processing to each application. | The proposed location, format, and intended inference-time use. | It is not a ratified web standard or a commitment by any model provider to consume the file. | 2026-07-28 |
No broader provider-adoption claim is made here. If a provider later documents automatic discovery or use, add only the exact documented surface and date; do not generalise it to the provider's other products.
Two things that are true and frequently misread as adoption. Developer-documentation platforms and some frameworks generate these files automatically, which produces a lot of files in the wild; that is publisher-side adoption, not consumer-side. And separately, tools that let a user paste a URL into a model will happily fetch whatever path you point them at, including this one; that is a person choosing to fetch a file, not a system doing it on its own.
What we can and cannot observe about its effect
The interesting question is empirical, and it is harder to answer than it looks.
What is observable from your own server logs: whether anything requests /llms.txt, which user agents do, how often, and whether those requests correlate in time with anything else. This is genuinely worth checking and costs you one log query. If nothing has ever requested the file, that is a data point, though a weak one, because the absence of requests to a file that does not exist tells you nothing.
What is much harder to observe: whether the presence of the file changes what an assistant says about you. To test that properly you would need a baseline with a fixed prompt set and adequate samples, the file as the only change, a control site or control section, and enough measurement cycles to separate a real shift from model-side variance. Even done well, a positive result on one site would not generalise, and a null result would not prove the file does nothing.
What is not observable at all: whether content from the file influenced anything internal to a model. There is no mechanism available to you that would show that.
This article makes no controlled-study claim. Any reader evaluating the effect should pre-register the prompt set, model versions, run dates, sample sizes, and variance, then report the result whichever way it falls.
Our current position, stated as uncertainty rather than as a finding: the strongest claims for this file are indirect and hard to distinguish from the general benefit of having clean, well-structured, accurate content. A site that writes a good llms.txt has usually just done the work of deciding what its most important pages are and describing them accurately, which is valuable whether or not any machine reads the artefact.
Cost to implement, and the honest verdict
The costs are small but they are not zero, and the ongoing one is what people underestimate.
| Cost | Scale | Notes |
|---|---|---|
| Initial authoring | An afternoon for a small site | Longer if nobody has agreed which pages matter |
| Ongoing maintenance | Recurring, indefinite | This is the real cost |
| Risk of staleness | Moderate | A file describing pages that moved or changed is worse than no file |
| Opportunity cost | Low, unless it displaces real work | Also low unless it becomes a talking point that substitutes for measurement |
| Technical risk | Low | A static text file at the root |
The verdict, without advocacy in either direction:
- If you can write it in an afternoon and you have a genuine owner who will keep it current, the cost is low enough that "why not" is a reasonable answer. Do it with correct expectations: you are producing a clean description of your site, not an AI-visibility intervention.
- If nobody will own it, do not create it. An abandoned file describing a site that has since changed is an actively misleading artefact, and stale beats absent only in the wrong direction.
- If creating it would displace work on retrievable content, accurate entity information, or measurement, do the other work first. That prioritisation holds regardless of what future adoption looks like.
- Do not present it to a stakeholder as a step that will change AI recommendations. No published first-party documentation currently supports that framing, and a claim like that is the kind that gets remembered when the numbers do not move.
If you do implement it, keep the content honest and descriptive. It is a summary of what you have, not a place to make claims. And if you want to know whether it is being read, check your logs for requests to the path, matched against the tokens in the AI crawler reference.
In Plain English
llms.txt is a proposed Markdown file at your site root that summarises the site for machine readers. It is a sensible idea, cheap to build, and it is not a robots directive or a ratified standard. OpenAI publishes one for its own developer documentation, but that publisher-side use is not evidence that ChatGPT or any crawler automatically discovers or rewards the file on other sites.
This is for whoever has been asked to make the call, and needs a defensible answer rather than a confident one. Use it when the question comes up, and re-check the verification date before quoting it, because this is a fast-moving area and the table above is a snapshot rather than a fact.
Next: decide whether you have an owner for the file. If yes, write it accurately and set a review cadence. If no, skip it and spend the afternoon on structure and accuracy instead. Either way, check your logs before and after so you have your own evidence rather than someone else's opinion.
Download the AI-readiness audit checklist to work through the items that are better supported than this one.
Primary-source verification
Checked July 28, 2026. The operational and factual guidance above was reviewed against the primary references below. Provider features, prices, policies, interfaces, jurisdictions, and account-specific results can change; recheck the controlling first-party document and your own account or measured data before acting.
- The original llms.txt proposal - Defines the voluntary format, location, rationale, and the fact that processing is application-specific.
- Google Search: AI features and your website - How Google describes eligibility, preview controls, and appearance in its AI search features.
- Google: common crawlers and special-case user agents - Current Googlebot and Google-Extended identities and controls.
- OpenAI: overview of OpenAI crawlers - Current GPTBot, OAI-SearchBot, and ChatGPT-User descriptions and publisher controls.
- Anthropic: web-crawler controls - Current ClaudeBot, Claude-SearchBot, and Claude-User descriptions and controls.
- Perplexity: crawler and user-agent documentation - Current PerplexityBot and Perplexity-User descriptions and controls.
- Google Search: structured-data introduction - Supported structured-data behavior and validation guidance.
- Apache HTTP Server: mod_log_config - Authoritative Apache access-log fields and formats.
- nginx: HTTP log module - Authoritative nginx access-log configuration and variables.
- Google Analytics: traffic-source dimensions - Current GA4 source, medium, campaign, and default-channel definitions.
- FTC: Endorsement Guides Q&A - Current U.S. regulator guidance on endorsements, disclosures, and reviews.
- FTC: Consumer Reviews and Testimonials Rule Q&A - Current U.S. rule guidance on deceptive review practices.