AI readiness: make your site retrievable by assistants

Crawler access, rendering, structure and schema decide whether an assistant can reach and quote your pages at all. It is the one layer of AI visibility fully under your control, and the one you can verify from your own server logs.

Use the AI crawler reference · See how to measure the result

User-agent strings, robots directives and provider control mechanisms change without notice. Verify every technical detail against the provider's own documentation on the day you implement it, not against any page's remembered version of it.

In short

AI readiness is the technical precondition for being named or cited by an assistant: the crawlers and agents that fetch your site can reach it, what they receive contains your actual content rather than an empty shell, and that content is structured so a single passage can be lifted without losing its meaning. This hub covers identifying and verifying AI crawlers, the business decision about allowing or blocking them, rendering and what a retriever really fetches, extraction-friendly structure, which schema is worth the effort, an evidence-led read on llms.txt, and how to confirm any of it in server logs.

Who this is for

Technical SEOs and web engineers deciding what to change in robots rules, rendering, or markup
In-house SEO leads who need to check whether a retrievability problem is blocking every other initiative
Platform and infrastructure owners being asked to allow or block AI crawlers and wanting the tradeoffs stated both ways
Publishers weighing content access against referral value, where the answer genuinely differs from a software company's
Analytics people trying to separate training crawls from live user-triggered fetches in log data

When to use this

Before you invest in content or earned media, so you are not publishing into a page a retriever cannot read
When your site is client-side rendered, gated, or behind an aggressive edge protection layer
When someone asks whether to block a named AI crawler and you need a decision framework rather than an opinion
When you are cited for the wrong sentence and suspect a structure problem
When you want log evidence that anything is fetching your pages at all

Crawler and agent access, and how to verify who is asking

Several distinct kinds of automated client may fetch your pages. Some collect content for model training. Some fetch a page because a user in a live session asked a question that triggered a retrieval. Some belong to search indexes whose results feed assistants. Some are agents acting on behalf of a person. They have different purposes, different controls, and different value to you, and lumping them together is the first mistake.

A user-agent string is a claim, not an identity. Anything can send any string. Where a provider publishes verification guidance — a published IP range, or reverse and forward DNS that resolves to their domain — verify before you make an access decision based on it. Treat unverified traffic wearing a well-known name as unverified traffic.

The specific strings, owners, purposes and control mechanisms change often enough that a list is only useful with a verification date attached. Our maintained reference carries one on every row.

Check the provider's current documentation on the day you act. Any list, including ours, is a snapshot.

Allowing or blocking is a business decision, and we do not have a default for you

There is no correct universal answer, and this site will not pretend otherwise. Blocking crawlers that collect content for training reduces the chance of your material being absorbed without compensation, and may reduce the chance of being recalled in answers that never link to anyone. Allowing them makes your content eligible in more places, and blocking live retrieval fetches specifically can mean an assistant answers a question about you from stale or third-party material instead of from your own current pages.

The right decision depends on your business model. A publisher whose product is the content itself weighs this very differently from a software company whose content is marketing for something else, and both differ from a services firm whose main risk is being described inaccurately. There are real costs in both directions, and anyone presenting one side as obvious is selling something.

Whichever way you go, be deliberate. The worst outcome is an accidental configuration — an edge rule, a legacy robots line, or a bot-protection default — that blocks retrieval you would have wanted, and that nobody discovers for a year.

Rendering: what a retriever actually receives

A retrieval fetch is not a browser session. Depending on the client, it may execute no JavaScript, wait less time than a human would, follow fewer requests, and give up on anything behind an interaction. If your content is assembled client-side, revealed on scroll, hidden behind a consent wall, or served only after a challenge, the document that arrives at the other end may contain almost nothing.

Test rather than assume. Fetch your own page without JavaScript, inspect what a plain request returns, and compare it against what you see in a browser. If the substantive text is missing, that is a retrievability bug and it outranks every content idea on your list.

The same applies to error handling. Soft 404s, rate limiting that returns a challenge page, and geo-redirects can all serve something valid-looking and empty.

Structure that survives extraction

Assistants quote passages, not pages. If the sentence that answers a question only makes sense after three paragraphs of setup, it will either be skipped or lifted in a form that misrepresents you. Structure is how you decide which sentence gets quoted.

The pattern that holds up: self-contained sections that each answer one question, headings phrased as the question rather than as a slogan, terms defined on first use inside the section that uses them, a short quotable summary near the top, stable tables with consistent columns, and specifics in place of adjectives. Every one of those also helps a human skimmer, which is a useful test — if a change only makes sense for machines, be suspicious of it.

This site applies the same rules to itself, including on this page.

Schema worth implementing, and schema that is cargo cult

Structured data has a documented role in search systems, and search systems feed some assistant surfaces. That is the honest basis for implementing it. What is not established is that adding more schema types produces better outcomes in conversational answers, and the habit of stacking every type a generator offers is cost without evidence.

The defensible minimum is the markup that describes what a page and an organisation genuinely are, kept accurate and consistent with the visible content: organisation identity, article or product identity, breadcrumbs, and question-and-answer markup where the page genuinely contains questions and answers. Markup that contradicts the visible page is worse than no markup, and misrepresenting content in structured data is a documented policy problem in search regardless of what assistants do with it.

Grade the rest as unproven and fund it accordingly. If a proposed schema addition cannot be tied to a documented behaviour or to something you will measure, it is a preference, not a requirement.

llms.txt, described without advocacy

llms.txt is a proposed convention: a plain-text file at a known path offering a curated map of a site's content for language models. It is a proposal, not a standard, and adoption by the systems that would have to honour it is the entire question.

We take no position for or against it. What we will say is what can be checked: whether any provider documents reading it, and whether your logs show anything fetching it. Those are both answerable at your site, on your date, and they beat any general claim. If nothing fetches it, the file is not doing anything for you yet, which is a fact rather than a verdict.

The cost is small, and so is the evidence. The real risk is opportunity cost — treating the file as the AI-readiness project while product pages remain unrenderable.

Verify it in your logs

Readiness is the rare part of this field where you can get direct evidence instead of inference. Your server logs record what asked for what, when, with which user agent, from which address, and what status code came back. That tells you whether the crawlers and agents you decided to allow are actually fetching the pages you care about.

Two things to separate. A broad, steady crawl pattern across many URLs looks like collection. A single fetch of one specific page, closely following a moment when someone plausibly asked about you, looks like live retrieval in a session. They mean different things for your access decision and for your reporting.

Check status codes as carefully as user agents. A crawler you believe you allowed but which is receiving challenges or rate-limit responses is effectively blocked, and only the logs will tell you.

Writing your own access policy

Document whether named crawlers may fetch your public content, and define narrow exceptions for private tool output and customer data. Verify deployed behaviour against your current robots.txt, edge controls, and server logs rather than assuming it.

Treat llms.txt as an experiment unless current provider documentation demonstrates support. Record the file's version and how you tested its deployment alongside the policy.

The written policy, the deployed configuration, and the observed logs must agree; a prose statement alone is not evidence.

Frequently asked questions

Should we block AI crawlers?

We do not give a default, because the tradeoffs genuinely differ by business model. Blocking collection crawlers reduces uncompensated use of your content; blocking live retrieval fetches can mean assistants answer questions about you from stale or third-party material rather than your own pages. Publishers, software companies and services firms reasonably land in different places. Decide deliberately, document the decision, and re-check it when providers change their controls, which they do without notice.

How do I know a crawler claiming to be a named provider really is one?

You do not, from the user-agent string alone, because any client can send any string. Use whatever verification the provider publishes — typically a documented IP range, or reverse DNS on the requesting address that forward-resolves back to it. If no verification method is published, treat the traffic as unverified and make access decisions on that basis. This matters most when you are allowing something specific rather than blocking broadly.

Our site is client-side rendered. Is that fatal?

Not automatically, but it is the first thing to test. Fetch your own pages the way a simple client would, without executing JavaScript, and see what comes back. If the substantive content is absent, retrievers relying on that path receive an empty page and nothing else you do will matter for them. Server-side rendering, prerendering, or ensuring critical content is in the initial response all solve it; which one is an engineering decision.

Is llms.txt worth adding?

It is cheap, and the evidence that anything honours it is thin. We take no position either way. The two checks worth doing are specific to you: whether any provider documents reading the file, and whether your own logs show anything fetching it. If nothing fetches it, it is not doing anything for you yet. The real cost is not the file, it is treating it as your readiness project while rendering problems go unfixed.

How much schema do we need?

Enough to describe accurately what the page and the organisation are, and no more than you can keep consistent with the visible content. Organisation identity, page or product identity, breadcrumbs, and genuine question-and-answer markup are the defensible core. Markup that contradicts what a reader sees is a liability. Additional types are unproven for conversational answers, so treat them as optional rather than as a checklist to complete.

How can I tell a training crawl from a live retrieval fetch in my logs?

Look at the shape rather than the name. Broad, steady traversal across many URLs over a long window looks like collection. A one-off fetch of a single relevant page, isolated in time, looks like a retrieval triggered by someone's live question. Provider documentation sometimes distinguishes agents by user agent, which helps when it exists and is verifiable. Record what you observe with dates, because these patterns shift as providers change infrastructure.

We fixed the technical issues and our visibility number has not moved. Was it pointless?

No, but you should describe the result accurately. Readiness work buys eligibility, not recommendation. Log evidence that the right clients are fetching the right pages and rendering tests showing your content is present are genuine, verifiable outcomes. Whether that turns into more mentions depends on levers outside this pillar and on sample sizes large enough to detect a change. Report eligibility as eligibility rather than stretching it into a visibility claim.

How often does the information on this page go stale?

Quickly. User-agent strings, published IP ranges, robots directives and provider control mechanisms change without announcement, which is why each row of the crawler reference carries its own verification date and source. Treat any technical detail here as a prompt to check the provider's own documentation on the day you act. A claim found to be outdated should be corrected in place with a dated note rather than quietly edited.

Check the layer you can actually control

Work through access, rendering, structure and logs using the sections above, then use the measurement pillar to judge whether anything downstream changed.

Find AI crawlers in your logs · Learn to measure the result