Skip to main content
RankRift
Menu
AI Readiness

How To Find AI Crawlers And Assistants In Your Server Logs

Copy-pasteable log queries, plus how to tell a training crawl apart from a live user-triggered fetch.

server-logscrawlersanalytics

Somebody in your organisation has asked whether AI systems are actually reading your website, and every dashboard you own answers with a shrug. Server logs are the one place you hold first-party evidence, so this guide shows you how to pull that evidence out, how to check that a request came from who it claims to be, and where the evidence stops being meaningful.

Everything below assumes the NCSA combined log format, the default access log layout for both Apache (combined) and Nginx (combined), written as:

%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"

If your server writes a different format, the field positions change and the commands will need adjusting. Confirm your format before running anything.

What logs can and can't tell you

A log line records that an HTTP request arrived, what it asked for, and what your server returned. That is genuinely useful, and it is also narrow.

Logs can tell you:

  • Which declared automated clients requested pages, how often, and which paths.
  • What status codes those clients received, which is how you find out you have been quietly returning 403 or 404 to a crawler for six months.
  • Whether requests come from IP space that resolves back to the operator that the user-agent names.
  • Roughly whether a request looks like a bulk sweep or a single page pull triggered by a person in a chat session.

Logs cannot tell you:

  • Whether any of those fetches ended up influencing an answer a real buyer saw.
  • Whether your brand was named, cited, recommended, or contradicted.
  • What a model already learned about you from a training corpus assembled before you started logging.
  • Anything at all about assistants that answer from parametric memory without fetching anything.

That last gap is the important one. Crawl volume is not a visibility metric. A high fetch count means a crawler found your sitemap interesting, nothing more, and treating that curve as a proxy for being recommended is the single most common misreading of this data. If you want to know whether assistants recommend you, you have to sample the assistants themselves. Fetch logs are an access check, not a scoreboard.

Assume the numbers undercount. Requests served entirely from a CDN edge cache never reach your origin log. Clients that fetch through a shared proxy pool arrive with an IP that tells you nothing. Some clients send a generic browser user-agent by design, and you will never separate them from human traffic. Whatever total you produce is a floor, not a count.

Getting access and the fields you need

Before writing queries, find out where the truth actually lives.

StackWhere to lookWatch for
Apache or Nginx on your own host/var/log/apache2/access.log*, /var/log/nginx/access.log*Rotation and compression; you may need zgrep for .gz archives
Behind a CDNThe CDN's own log export, not the origin logThe origin sees only cache misses, so origin-only analysis undercounts badly
Managed hosting or PaaSPlatform log export or drainRetention is often days, not months; export before you need it
Load-balanced originsAll nodes, mergedAnalysing one node silently divides every number

The five fields you need are client IP, timestamp, request path, status code, and user-agent. If your platform strips the user-agent or replaces the client IP with the load balancer's, fix that first. Everything here depends on those two fields being real.

Ask for at least 30 days. Ask for 90 if retention allows, because monthly patterns in automated fetching are common enough that a two-week window will mislead you.

Queries for common log formats

These use awk with the double-quote character as the field separator, which is the reliable way to read combined-format logs without tripping over spaces inside the request line. With -F'"', field 2 is the request line, field 4 is the referrer, and field 6 is the user-agent.

Start by defining the pattern you are matching once, then reuse it:

AGENTS='GPTBot|OAI-SearchBot|ChatGPT-User|PerplexityBot|Perplexity-User|ClaudeBot|Claude-User|Claude-SearchBot|Google-Extended|Amazonbot|Applebot-Extended|CCBot|meta-externalagent|Bytespider'

1. Total requests per declared agent:

awk -F'"' -v a="$AGENTS" '$6 ~ a {print $6}' access.log \
 | sort | uniq -c | sort -rn

2. Requests per agent per day:

awk -F'"' -v a="$AGENTS" '$6 ~ a {
  split($1, t, "["); split(t[2], d, ":");
  print d[1], $6
}' access.log | sort | uniq -c | sort -k2,2 -k3,3

3. Status codes each agent received. This is the query that finds real problems:

awk -F'"' -v a="$AGENTS" '$6 ~ a {
  split($3, s, " "); print $6, s[1]
}' access.log | sort | uniq -c | sort -rn

If a crawler you intended to allow is receiving mostly 403, something in your WAF, bot-management rules, or robots.txt handling is doing something you did not intend. That is worth more than any volume chart on this page.

4. Which paths are actually being fetched:

awk -F'"' -v a="$AGENTS" '$6 ~ a {
  split($2, r, " "); print r[2]
}' access.log | sort | uniq -c | sort -rn | head -50

5. Compressed archives: substitute zcat access.log.*.gz | awk... or use zgrep for the first pass.

For CDN or warehouse logs exported to SQL, the same logic is one statement, with your own column names substituted:

SELECT DATE(request_time) AS day,
       user_agent,
       status_code,
       COUNT(*) AS requests
FROM   web_logs
WHERE  REGEXP_CONTAINS(user_agent, r'(?i)GPTBot|PerplexityBot|ClaudeBot|CCBot')
GROUP  BY day, user_agent, status_code
ORDER  BY day DESC, requests DESC;

Adjust the regex function to your warehouse dialect. Run each command against a small slice first and read the raw output before you trust an aggregate.

Separating training crawl from live fetch

Several operators publish separate user-agent tokens for different purposes: broad corpus collection, search-index building, and a fetch made because a person in a live session asked something that needed your page. Where those tokens exist and are documented, that is your cleanest signal. Where the tokens are ambiguous, the request pattern carries some information:

SignalLooks more like bulk collectionLooks more like a live, user-triggered fetch
Volume shapeSustained sweeps over hours, many URLsOne or a few URLs, isolated in time
Path selectionFollows sitemap or link graph, includes deep archive pagesLands on a specific commercially relevant page
ReferrerEmptySometimes empty, sometimes populated
Repeat behaviourRevisits on a cadenceClusters unpredictably, often around a topic in the news

Treat that table as a heuristic and label it as such in any report. It is inference from shape, not identification, and confident stories built on it fall apart when an operator changes its scheduling. If the distinction matters commercially, say what you can defend: "requests from a token the operator documents as user-triggered rose in this window," rather than "people asked assistants about us more often."

Verifying claimed identity

A user-agent is a string the client chooses. Anyone can send one. Scrapers routinely impersonate well-known crawlers, so an unverified count is a count of claims.

Two verification methods, in order of preference:

Forward-confirmed reverse DNS. Resolve the IP to a hostname, then resolve that hostname back to a set of IPs, and require the original IP to be in that set. A single reverse lookup on its own is not sufficient, because the PTR record is controlled by whoever holds the IP block.

awk -F'"' -v a="$AGENTS" '$6 ~ a {print $1}' access.log \
 | awk '{print $1}' | sort -u > candidate_ips.txt

while read -r ip; do
  ptr=$(dig +short -x "$ip" | sed 's/\.$//')
  [ -z "$ptr" ] && { echo "$ip UNRESOLVED"; continue; }
  fwd=$(dig +short "$ptr" A; dig +short "$ptr" AAAA)
  if printf '%s\n' "$fwd" | grep -qx "$ip"; then
    echo "$ip VERIFIED $ptr"
  else
    echo "$ip MISMATCH $ptr"
  fi
done < candidate_ips.txt

Published IP ranges. Several operators publish a machine-readable list of the addresses their crawlers use. Where such a list exists, matching against it is stronger than DNS because it is an explicit assertion by the operator.

Report verified and unverified counts as two separate numbers, always. Collapsing them hides the exact thing an executive will ask about the moment the figure looks surprising.

If you find sustained impersonation, that is a security and abuse question for whoever owns your edge configuration, not a visibility question. Route it there.

Turning this into a recurring report

A one-off analysis answers a question. A recurring one catches the regression that matters, which is usually access, not volume.

  • Schedule it. A weekly cron job writing the day-by-day, agent-by-agent, status-code table into a CSV is enough. No tooling purchase required.
  • Alert on access, not on volume. The alert worth having fires when a documented crawler's non-2xx rate crosses a threshold, or when an agent that fetched consistently stops entirely for a week. Volume swings are normal and alerting on them trains everyone to ignore the channel.
  • Version your agent list. Keep the regex in source control with a date and the documentation link for each token. Re-verify quarterly, because tokens are added and retired without notice.
  • Record what changed on your side. A robots.txt edit, a WAF rule, a CDN migration, or a redirect change all move these curves. Without a change log you will attribute your own deploys to the model providers.
  • Pair it with the other two sources. Logs, referral analytics, and direct sampling of assistants each miss different things.

Cross-check the log picture against detect AI referral traffic in GA4, which sees a different and equally partial slice, and keep the AI crawler reference open while you build the regex. The wider retrievability picture, including what to do once you find a crawler getting blocked, sits on the AI readiness hub, and the framing for putting any of this in front of a stakeholder is on the reporting hub.

In Plain English

Server logs prove that automated clients requested your pages, and that is worth knowing, mostly because it exposes crawlers you are accidentally blocking. They do not tell you whether an assistant recommended you, and the totals undercount because of caching, CDNs, and clients that do not identify themselves. This helps technical SEOs and web engineers who need first-party evidence of access. Use it when you are auditing retrievability or investigating a suspected block. Run the status-code query first, verify a sample of IPs before believing any user-agent, schedule the whole thing weekly, and report crawl figures as access data with the word "undercount" attached.

Get the AI referral attribution pack to set up the analytics half of this picture alongside your logs.

Primary-source verification

Checked July 28, 2026. The operational and factual guidance above was reviewed against the primary references below. Provider features, prices, policies, interfaces, jurisdictions, and account-specific results can change; recheck the controlling first-party document and your own account or measured data before acting.