Skip to main content
RankRift
Menu
AI Readiness

The AI Crawler Reference: Who's Crawling You And What Each Bot Does

A maintained reference of AI crawler and agent user agents — owner, purpose, and how each is controlled.

crawlersuser-agentsreference

You sorted a week of access logs by user agent and found a dozen names nobody on the team recognises. Some of them are hitting a few pages an hour. One is hitting a few hundred. The question in front of you is not philosophical: you need to know which of these things is a search crawler, which is collecting training data, which is a live fetch triggered by a human typing a question, and which is a scraper wearing someone else's name.

This page is that lookup. It is built to be re-verified rather than published once, because crawler names change, get split into variants, and get retired without much warning.

How to use this table, and what "last verified" means here

Every row carries its own verification date and its own source link. That is deliberate. A table where one date at the top covers forty rows is a table that is quietly wrong within a month, because operators do not update their documentation on a shared schedule.

Three rules govern what goes in this reference:

  • A row is only included when the exact token was read from the operator's own published documentation, on the date recorded. Not from another reference table, not from a blog post, not from memory.
  • A row we cannot confirm against first-party documentation is omitted. A shorter verified table is more useful than a comprehensive-looking list built on inference.
  • Behaviour claims are separated from identity claims. "This token exists and belongs to this operator" is a different kind of statement from "this crawler respects that directive," and the second is much harder to verify than the first.

Every row below was checked against the linked first-party documentation on July 28, 2026. It is still a snapshot: re-open the source before implementing a rule, because names, purposes, and controls change. We also publish the same data as a JSON export from resources and downloads so you can diff it against your own log parsing rather than re-reading a web page.

The reference table

Columns are stable so that successive versions can be diffed. Class maps to the three categories explained in the next section.

OperatorUser-agent tokenClassStated purposeControl mechanismStatusLast verifiedSource
OpenAIGPTBotTraining / corpusMay be used to improve OpenAI's generative modelsrobots.txt by product tokenVerified2026-07-28OpenAI crawlers
OpenAIOAI-SearchBotSearch indexSurfaces and links websites in ChatGPT searchrobots.txt by product tokenVerified2026-07-28OpenAI crawlers
OpenAIChatGPT-UserLive fetch (user-triggered)Visits a page in response to a user requestSite controls and access rules; not described as an automatic crawlerVerified2026-07-28OpenAI crawlers
AnthropicClaudeBotTraining / corpusCrawling that may contribute to model trainingrobots.txt by product tokenVerified2026-07-28Anthropic crawler controls
AnthropicClaude-UserLive fetch (user-triggered)Fetches content when a user asks Claude to visit a pagerobots.txt by product tokenVerified2026-07-28Anthropic crawler controls
AnthropicClaude-SearchBotSearch indexSupports search-result qualityrobots.txt by product tokenVerified2026-07-28Anthropic crawler controls
GoogleGooglebotSearch indexGeneral crawling for Google Searchrobots.txtVerified2026-07-28Google common crawlers
GoogleGoogle-ExtendedTraining / corpus control tokenControls whether crawled content may help improve Gemini Apps and Vertex AI generative APIsrobots.txt token; it does not itself crawlVerified2026-07-28Google common crawlers
PerplexityPerplexityBotSearch indexIndexes websites for search resultsrobots.txt plus published IP rangesVerified2026-07-28Perplexity bots
PerplexityPerplexity-UserLive fetch (user-triggered)Fetches a page when a user request cites itSite access controls; Perplexity says it generally ignores robots.txt because the fetch is user initiatedVerified2026-07-28Perplexity bots
AppleApplebotSearch index / Apple featuresPowers search features across Apple products; crawled data may also support model trainingrobots.txt, metadata, and Apple-specific controlsVerified2026-07-28About Applebot
AppleApplebot-ExtendedTraining / corpus control tokenControls training use of data crawled by Applebotrobots.txt token; it does not itself crawlVerified2026-07-28About Applebot
AmazonAmazonbotProduct improvement / trainingImproves Amazon products and services and may be used to train Amazon AI modelsrobots.txt, supported robots metadata, and published IP rangesVerified2026-07-28About Amazonbot
Common CrawlCCBotOpen web corpusBuilds Common Crawl's web corpus for downstream reuserobots.txt and documented opt-out controlsVerified2026-07-28CCBot

Tokens without readable first-party documentation are deliberately absent. If you see an unlisted token in your logs, record it as unidentified and investigate it from the operator's own site; do not infer ownership from the name.

Do not paste this table into a robots.txt file. Several rows are control tokens rather than crawlers, at least one is a third-party corpus builder whose output is reused by parties who never touched your server, and the correct policy differs by row and by business. The decision belongs in allow or block AI crawlers, not here.

Training crawlers, live-fetch agents, and search crawlers are three different problems

Grouping every unfamiliar user agent under "AI bots" is the mistake that produces bad policy. The three classes have different consequences when you deny them.

ClassWhat it doesTypical traffic shapeWhat denying it costs you
Training / corpusBulk-collects pages that may contribute to a model's parameters or to a reusable corpusBroad, steady, crawls deep and wideLittle or no immediate visible effect; the effect, if any, is delayed and hard to attribute
Search indexBuilds an index that an assistant queries to find and cite pagesBroad but more selective, revisits changed pagesRemoval from the index that feeds citations on that surface
Live fetch (user-triggered)Retrieves one page because a person or an agent asked about it right nowSpiky, shallow, often a single URL, sometimes correlated with a referralThe assistant cannot read the page it was about to summarise or cite for a specific user

The practical asymmetry: blocking a training crawler is a slow, mostly invisible decision, while blocking a live-fetch agent has an immediate and observable effect on what an assistant can say about a specific page. A control token is different again, because it is not a fetching agent at all and denying it is a statement about permitted use rather than about access.

Whether any given operator honours any given directive is a separate question from what its documentation says, and it is one you can only answer from your own logs. That is the point of find AI crawlers in your logs: compliance is measured, not assumed.

Verifying that a bot is who it says it is

A user-agent string is a self-declared label. Anyone can send any string. Two things follow: some traffic claiming to be a major crawler is not, and some traffic you would like to identify sends nothing recognisable at all.

There are two standard verification methods, and both belong to the operator, not to you.

Reverse DNS, then forward confirmation. Take the client IP from the log line. Run a reverse DNS lookup to get a hostname. Check that the hostname sits inside a domain the operator publicly claims for its crawlers. Then run a forward DNS lookup on that hostname and confirm it resolves back to the original IP. The forward step is the one people skip, and skipping it is what makes the check worthless, because reverse DNS records can be set by whoever controls the address block. Both directions have to agree.

Published IP ranges. Some operators publish machine-readable lists of the address ranges their crawlers use, usually as a JSON file at a stable URL. Where that exists it is the stronger check: fetch the list on a schedule, keep it fresh, and match on the range rather than the string. Where it does not exist, reverse DNS is what you have.

A few practical notes on running this:

  • Verify a sample, not every request, unless you are enforcing at the edge. A daily sample per token is enough to tell you whether the traffic is broadly genuine.
  • Cache verification results per IP with a short expiry. Doing a DNS round trip on every request is a good way to make your own site slow.
  • Log the verification outcome next to the request. A month later, "unverified GPTBot from a residential range" is a far more useful record than the raw string.
  • Treat a failed check as a claim about that request, not as an accusation about the operator. Impersonation is done by third parties.

If a token consistently fails verification and the traffic is expensive, that is a rate-limiting and abuse question, handled the way you would handle any other unwanted automated traffic. It is not an AI-visibility question, and it should not drive your crawler policy.

When a user agent isn't in this table

Most of the unfamiliar strings in a typical log are not from any of the operators above. They are SEO tools, uptime monitors, security scanners, feed readers, link previewers, headless browsers with a default string, and a long tail of research and scraping projects.

Work through it in this order:

  1. Search the exact token, in quotes, against the operator's documentation site first. If the token names a company, go to that company's site before anything else. Secondary tables are frequently stale.
  2. Look for a URL inside the user-agent string. Well-behaved crawlers usually embed a link to a page explaining themselves. That page is a first-party source and can be cited.
  3. Check the request pattern. One URL, no assets, no repeat visits looks like a live fetch or a link preview. Deep, ordered, sustained crawling looks like corpus collection. The pattern tells you the class even when the name tells you nothing.
  4. Check whether it requested robots.txt at all. A crawler that never reads the file is telling you something about how much your directives will matter to it.
  5. If you still cannot identify it, record it as unidentified with the evidence you have. Unidentified is a legitimate status. Assigning it to a vendor because the name looks similar is how errors propagate through every table that copies yours.

For a token you cannot place, create a provisional row containing the literal token, first-seen date, request pattern, robots.txt behavior, verification outcome, candidate operator, primary-source URL, and status unidentified. Promote the row only after a readable first-party source identifies the operator and purpose; record the verification date and change in the table's changelog.

In Plain English

This is a lookup table for bot names in your server logs, with three things attached to every row: who runs it, what class of client it is, and when we last checked the operator's own documentation. Every included row was checked on July 28, 2026; anything we could not support from a readable first-party document was omitted.

It helps anyone who owns a web server and has been asked "what is this thing and should we block it." Use it when you are triaging log data or drafting a robots.txt change, and pair it with your own verification pass, because a user-agent string is a claim rather than proof. Do not use it as a policy recommendation; identification and decision are separate jobs.

Next: verify a sample of the traffic with the reverse-DNS-plus-forward check above, then take the policy question to the decision framework rather than reaching for a blanket rule. The wider context for structure, rendering, and access sits on the AI readiness hub.

Download the AI-readiness audit checklist to run the crawler-access section of this against your own site.

Primary-source verification

Checked July 28, 2026. The operational and factual guidance above was reviewed against the primary references below. Provider features, prices, policies, interfaces, jurisdictions, and account-specific results can change; recheck the controlling first-party document and your own account or measured data before acting.