Skip to main content
RankRift
Menu
AI Readiness

If A Retriever Can't Render It, It Doesn't Exist

Client-side rendering, gating, and infinite scroll — how to test what a retriever actually sees on your pages.

renderingjavascripttechnical-seo

The engineering lead says the site renders fine, and points at Search Console. The content lead says assistants never surface the documentation, and points at a dozen sessions where the answer came from a competitor's page. Both are working from evidence. Neither has fetched the page the way the thing they are arguing about fetches it.

That fetch takes about ninety seconds and settles most of these arguments. This guide covers how to run it, the four failure modes it usually exposes, and how to decide what to change without creating a worse problem than the one you started with.

Why retrieval parity with search crawlers is not a safe assumption

Search engines invested years in rendering pipelines that execute JavaScript, wait for network activity to settle, and index the resulting DOM. It is tempting to assume every automated fetcher does the same, because for a long time the only fetchers that mattered did.

That assumption is not safe, for three reasons.

Different fetchers, different jobs. A crawler that builds a persistent index has a very different cost budget from a fetcher that retrieves one URL live while a user waits for an answer. A live retrieval path is under latency pressure that an offline indexer is not. What each one actually does with JavaScript has to be established per operator, from that operator's documentation or from your own logs, rather than inferred from the category.

Multiple agents per provider. A single company may operate several distinct automated agents with different purposes: one that gathers content for training, one that fetches pages to answer a live user query, one that acts on a user's behalf inside a browsing session. These are frequently separate user agents with separate documented behaviors and separate opt-out controls. Testing one and generalizing to the others will mislead you.

Behavior changes. Rendering support, timeouts, and agent names are all subject to change without a corresponding change on your side. A test result is a dated observation, not a permanent property of your site.

The workable posture: assume nothing, test per agent, re-test on a schedule, and write the date next to every result.

How to fetch your page the way a retriever would

Start with the harshest realistic case: a plain HTTP GET with no JavaScript execution. If your content is present there, most of this article stops being your problem.

1. Fetch the raw HTML and read it.

curl -sSL -A "Mozilla/5.0 (compatible; RankRiftAudit/1.0; +rendering test)" \
  https://example.com/your-page \
  -o /tmp/page.html

wc -c /tmp/page.html

A response body of a few kilobytes on a page that looks content-rich in a browser is the classic signature of client-side rendering.

2. Check whether your actual content is in that HTML.

grep -c "a distinctive sentence from the middle of your page" /tmp/page.html

Use a real sentence from body copy, not a heading and not the page title. Titles and meta tags are frequently server-rendered even when the body is not, which is exactly how this problem hides.

3. Extract the text a parser would see.

python3 -c "
import sys, html.parser
class T(html.parser. HTMLParser):
    def __init__(self):
        super().__init__(); self.skip=0; self.out=[]
    def handle_starttag(self,t,a):
        if t in ('script','style'): self.skip+=1
    def handle_endtag(self,t):
        if t in ('script','style'): self.skip=max(0,self.skip-1)
    def handle_data(self,d):
        if not self.skip and d.strip(): self.out.append(' '.join(d.split()))
p=T(); p.feed(open('/tmp/page.html', encoding='utf-8', errors='replace').read())
text=' '.join(p.out)
print(len(text.split()), 'words')
print(text[:1200])
"

Read that output as if it were the entire page, because for a non-rendering fetcher it is. Word count is a blunt but effective signal: compare it against what you would expect from the visible page.

4. Check response headers and status.

curl -sSI https://example.com/your-page | head -20

Look for the status code, redirects, Content-Type, and any X-Robots-Tag header. A noindex in an HTTP header is invisible in page source and catches teams out regularly.

5. Repeat per agent.

Run steps 1 to 3 again with each documented user-agent string you care about, one at a time, and record the result separately.

for UA in "AgentStringOne" "AgentStringTwo"; do
  echo "== $UA"
  curl -sSL -A "$UA" https://example.com/your-page | wc -c
done

Get the exact strings from each provider's own documentation rather than from a blog post or an aggregator list, and record the date you read them. If the byte counts differ meaningfully between agents, something in your stack is already treating them differently, which is worth understanding before you change anything else.

6. Compare against a rendered fetch.

If you have a headless browser available, fetch the same URL with JavaScript executed and diff the extracted text against the raw version. The size of that gap is the amount of your page that depends on rendering.

Then cross-check against reality: your server logs will tell you which agents actually reached the URL, what status they got, and how often. The method for that is in find AI crawlers in your logs. A clean local test plus zero log entries is a different problem, and an access problem, not a rendering one.

The four common blockers

1. Client-side rendering

The page ships a shell and fetches content with JavaScript after load. Non-rendering fetchers get the shell.

How it shows up in the test: tiny HTML payload, body sentence not found, extracted word count near zero, page title present.

What to check first: whether the content is genuinely dynamic or just happens to be rendered client-side because that was the framework default. Most marketing and documentation pages are the latter.

2. Gating

Content sitting behind a login, an email form, a regional interstitial, or a paywall.

How it shows up: a 401, 403, or a redirect to a login path in the header check, or a form where the article should be.

What to check first: whether the gate is intentional. Gating is a legitimate business decision and this guide is not arguing against it. What it argues against is discovering by accident that your most useful explainer sits behind a form nobody meant to put there.

A cookie or consent overlay that blocks or delays content delivery until a choice is made. Automated fetchers do not click accept.

How it shows up: HTML that contains the consent framework and little else, or content present in the DOM but hidden behind an overlay that suppresses it.

What to check first: whether the underlying content is in the initial HTML at all. If it is, the wall is a rendering-layer problem. If it is not, the wall is loading content conditionally and that is a content-delivery problem. Both are constrained by your privacy obligations in the relevant jurisdictions, and how you resolve them is a question for counsel and your privacy owner, not a purely technical call.

4. Infinite scroll and lazy loading

Content that only exists after a scroll event or an intersection observer fires. There is no scroll in a headless fetch.

How it shows up: first screen present, everything below missing, often with a stub or skeleton element where the content should be.

What to check first: whether each item has its own canonical URL. Paginated or individually-addressable content survives this. A feed with no addressable items does not.

Deciding what to server-render

You do not need to rebuild the site. You need to work out which URLs carry the substance and make sure the substance is in the initial HTML for those URLs.

Page typePriorityUsual reason
Documentation, guides, explainersHighestDirectly answers the questions assistants receive
Product and pricing pagesHighFactual claims about you that models otherwise get elsewhere
Comparison and FAQ pagesHighDense, extractable, frequently retrieved shapes
Blog and research postsHighThe body of evidence others cite
Dashboards, account, app viewsNoneAuthenticated, and should stay that way
Search results, filtered listingsLowUsually thin and better addressed by the underlying items

For each high-priority URL, the target is simple: the main content, the headings, and the structured data are present in the response to a plain GET, with no script execution required. How you get there is a stack question. Server-side rendering, static generation, incremental static regeneration, and a prerendering layer all reach the same destination, and the right choice depends on how often the content changes and what your framework already supports.

While you are there, check that the served HTML has a sane heading hierarchy and self-contained sections, because a page that renders but reads as one undifferentiated block is only half solved. That is covered in chunk-friendly structure.

Do not solve this by serving agents something different

There is a shortcut available here and it is a bad one: detect the agent by user-agent string and serve it a clean, content-rich version while human visitors get the JavaScript application.

Serving materially different content to automated agents than to human visitors is cloaking. It is prohibited by major search platforms' guidelines, it is detectable, and the downside is a manual action or removal rather than a slower fix. The principled version of the rule is stronger than the policy version. If the content you would serve an agent is better than what you serve people, serve that to people. If it is different, you are asking a model to describe a page your customers never see, and you will be the last to find out when it goes wrong.

Prerendering is not cloaking when the prerendered output is the same content the user ultimately receives, and it is applied on a URL basis rather than a user-agent basis. The line is straightforward: same content for everyone, delivered in whatever way is fastest for each. The moment the content varies by who is asking, you have crossed it.

Re-testing after the fix

A fix is not done when the ticket closes. It is done when the same test that failed passes and the logs agree.

  • Re-run the exact same commands against the same URLs and save the output next to the original. A before/after diff is the artifact.

  • Re-run per agent. A fix that works for one fetcher and not another is a common outcome, particularly when a CDN, WAF, or bot-management layer sits in front of the origin. Those layers are a frequent cause of agent-specific failures and are worth checking directly.

  • Watch the logs for the affected agents over the following weeks. Re-fetch timing is outside your control and varies by operator, so expect the log confirmation to lag the technical fix by an unpredictable amount.

  • Put the test on a schedule. Quarterly is a reasonable default for most sites, and immediately after any framework upgrade, CDN change, or consent-platform change.

  • Record the date and the agent version alongside every result. An undated rendering test is a guess about the past.

Resist the temptation to declare victory based on a visibility measurement moving. Retrievability is a precondition, not a cause. Fixing it removes a blocker; it does not entitle you to an outcome, and nothing here promises one. The honest claim after a successful fix is "the content is now fetchable," verified by a repeatable test.

The rest of the retrievability sequence, including crawler access and structured data, is on the AI readiness hub.

In Plain English

Fetch your own page with curl, with no JavaScript, and look for a real sentence from the body. If it is not there, non-rendering fetchers are not seeing your content. The four usual culprits are client-side rendering, gates, consent walls, and infinite scroll. Fix the highest-value URLs by putting the real content in the initial HTML, test each agent separately, and never solve it by serving agents a different page than people get.

This helps technical SEOs and web engineers who suspect a JavaScript-heavy site is invisible to retrievers and need evidence rather than a hunch. Run it when content that clearly exists never appears in assistant answers, or before any framework or consent-platform migration. Do not run it as a substitute for checking your logs, and do not treat a passing test as proof of anything beyond fetchability.

Next step: pick your five most important URLs, run the fetch on each, and save the output with today's date.

Download the AI-readiness audit checklist and work through the rendering section against those five URLs.

Primary-source verification

Checked July 28, 2026. The operational and factual guidance above was reviewed against the primary references below. Provider features, prices, policies, interfaces, jurisdictions, and account-specific results can change; recheck the controlling first-party document and your own account or measured data before acting.