Skip to main content
RankRift
Menu
Tools & Platforms

API-Based Measurement Vs UI Scraping: Why The Numbers Differ

Two tools, two scores, one methodological reason — plus the terms-of-service question buyers rarely ask.

apisscrapingmethodology

Two vendors measured the same brand, in the same category, in the same week, and handed you two different numbers. Both sets of sales engineers were confident, neither could explain the other's figure, and you now have to decide which one goes in the deck.

Most of that gap is not accuracy. It is collection method. Once you know how a vendor obtains the answers they analyse, the discrepancy usually stops being mysterious and starts being predictable.

Our conflict, stated first, and our method

Scope note. This article is general educational and measurement guidance, not legal, tax, financial, security, or platform-specific advice. Current law, first-party platform rules, the facts of the real case, and qualified professional advice control any consequential decision. RankRift recommends official provider APIs because their documented interfaces make a measurement easier to audit. This article is independent editorial guidance: RankRift does not operate or sell an AI-visibility tracker, and no vendor paid for this coverage. Our disclosure policy explains how commercial relationships would be handled if that changes.

We are not going to tell you that one method is legitimate and the other is fraud. Both approaches produce numbers that describe something real, and each is blind to things the other sees. The argument of this post is narrower: the two methods answer subtly different questions, the difference is systematic rather than random, and a buyer who cannot say which question a vendor is answering cannot interpret the vendor's chart. Our own method's specific blind spots are listed near the end, and one of them is a coverage gap that a competitor using a different approach may not have.

What the two methods actually capture

Official API access means the vendor sends prompts to a provider's documented programmatic endpoint under a commercial agreement, and analyses the responses. The vendor can usually request a specific model identifier, can record the identifier returned with each response, and can control parameters that affect variability. Access is metered and billed, which means the cost of a run is visible and predictable.

Consumer-surface collection means the vendor derives its data from the assistant product that end users interact with, rather than from a documented programmatic endpoint. That surface is the full shipped product: the model plus whatever the provider has layered around it. We are not going to describe how such collection is performed, because that is not information this site publishes, and it does not matter for the buying decision. What matters is what the resulting data represents.

The distinction that makes this consequential is that the assistant product is not the same thing as the model. A consumer assistant is typically an assembly: a model, plus a system prompt, plus some retrieval or live search layer, plus safety filtering, plus formatting and citation rendering, plus account-level state and product features that change without notice. Query the programmatic endpoint and you generally get the model with a much thinner layer around it, and with parameters you control. Observe the product surface and you get the assembly, including layers whose behaviour is undocumented and can change on a Tuesday.

Neither of these is the thing you actually care about, which is what your buyer saw. We will come back to that.

Why the resulting numbers legitimately differ

Five mechanisms, each of which can move a visibility figure on its own.

1. Retrieval and grounding. If the consumer product performs a live web lookup before answering and the programmatic endpoint does not, the two are drawing on different information. The product surface will tend to name brands with strong current web presence and to cite sources. The endpoint, without grounding, leans harder on what the model absorbed in training. Those two conditions can produce genuinely different brand lists for the same prompt, and a vendor comparison that ignores this is comparing a grounded system to an ungrounded one.

2. System prompts and product instructions. The instructions a provider wraps around the model in its consumer product shape answer length, hedging, willingness to name specific companies, and whether recommendations get presented as a list. Those instructions are not published and change over time. A shift in that layer moves consumer-surface numbers while leaving API numbers untouched, which is a discontinuity in one vendor's time series with no corresponding event in the other's.

3. Version pinning. Requesting a specific model identifier programmatically is normally possible. Determining which model produced a given answer on a consumer surface generally is not, because the provider routes requests according to its own logic. The practical consequence is that consumer-surface data may be a blend of several models, and a decline in your score may be a routing change rather than anything about your brand.

4. Sampling and parameters. API access exposes parameters that influence how much the output varies between identical requests. Consumer surfaces apply the provider's defaults, whatever those are that week. Two vendors could run identical prompt sets at identical frequencies and still report different variance because one of them chose a setting the other could not see.

5. Cost structure, and therefore sample size. API calls are billed per use, which pushes vendors toward smaller, more carefully designed prompt sets and fewer repeats. This is often the largest single driver of a discrepancy and the one buyers ask about least, because sample size determines how much of a reported difference is noise. How many runs you need covers the arithmetic.

Official API accessConsumer-surface collection
Model version knownUsually, and recordableUsually not
Product layers includedThin, and statedFull assembly, undocumented
Live retrievalPresent only if the endpoint offers itWhatever the product does that day
Parameters controllableYesNo
Cost per observationMetered and predictableDifferent cost profile, different pressures
Reproducible by a third partyYes, given the prompt set and identifiersDifficult, because conditions cannot be pinned
Access durabilityGoverned by a commercial agreementGoverned by terms the provider sets unilaterally

Personalization, session state, and what each method includes

Here is the part that should stop anyone from claiming their method is the true one.

A real buyer arrives with history. They may have a subscription tier, memory of prior conversations, custom instructions, a location, a language, a device, a logged-in account, and a preceding turn of conversation that framed the question. Some of those can affect what an assistant says. API measurement is typically stateless: a fresh request with no memory, no history, and no account personality. That is excellent for reproducibility, because a stateless request is a controlled experiment you can re-run. It is a poor imitation of a specific human, because no human is stateless.

Consumer-surface measurement is closer to the shipped product, and that is a real advantage. But it is not closer to a specific buyer, because whatever account state the collection runs under is one arbitrary state, and it is not the state of the customer whose behaviour you are trying to understand. Getting near the product surface does not solve personalization. It relocates the problem.

Neither method observes what your buyer saw. Both produce a controlled, artificial, repeatable approximation. The value of an AI-visibility number is comparative, across time and across competitors under identical conditions, and it is not a literal readout of any real person's experience. Any vendor telling you otherwise is overselling.

The useful framing for your executive audience: this is an index, in the sense that a price index is an index. It tracks a fixed basket under stated conditions. It is meaningful when the conditions hold constant and misleading the moment they change silently. That is why our methodology treats the conditions as part of the number rather than as a footnote.

The terms-of-service and continuity risk

Scope note. This article is general educational and measurement guidance, not legal, tax, financial, security, or platform-specific advice. Current law, first-party platform rules, the facts of the real case, and qualified professional advice control any consequential decision. General information, not legal advice. Terms differ by provider, change without much notice, and their application to any particular arrangement depends on facts we cannot see. Have your counsel and your procurement team review this before it becomes a contract question.

Every provider publishes terms governing how its services may be accessed and used, and those terms are not identical across providers or across the programmatic and consumer products of a single provider. We are not going to characterise any specific method as permitted or prohibited, and we would treat any vendor who does so casually in a sales call with caution, in either direction. What a buyer can reasonably do is ask a factual question and read the answer: under what agreement do you access this data, and can you point us to the terms that govern it?

The part that belongs in a business case, separate from any legal question, is continuity. Any measurement programme depends on continued access to a data source, and providers control that access. A commercial API agreement can be changed, repriced, or terminated. Consumer surfaces can change structure, add friction, or restrict automated access at any time and without consultation. The exposure is different in character, and it exists in both cases.

The risk that matters to you is not that a vendor loses access one morning. It is what happens to your time series when they do.

  • The break may be silent. If collection degrades gradually rather than failing outright, the chart keeps drawing and the numbers keep looking plausible. A vendor with no monitoring for this will report a decline in your visibility that is a decline in their collection.
  • The re-baseline may be undisclosed. If a vendor switches method mid-contract, numbers before and after are not comparable. Whether they tell you is a policy question you can ask about now, at no cost.
  • The history may not be yours. If access ends and the relationship ends with it, the value of two years of measurement depends entirely on whether you exported the raw responses along the way.
  • Your reporting commitments outlive the vendor. If you have told a board you will report this quarterly, the continuity risk transfers to you.

Price this the way you would price any single-source dependency. Ask for the vendor's continuity plan in writing, ask what happened the last time a provider changed something material, and make raw-response export a contract term rather than a feature request. Questions to ask any vendor has the full list, and our risk and brand safety hub covers the wider category of things that can go wrong with a measurement programme.

What to ask your vendor, and what to do with the answer

Four questions, and what each answer buys you.

  1. "Under what agreement do you access this data, and which terms govern it?" You are testing willingness, not legality. A vendor who answers plainly and points at a document is a vendor whose method you can evaluate. Evasion is itself information.
  2. "Do you record the model identifier with every response?" If not, they cannot separate a provider-side change from a change in your visibility, and neither can you.
  3. "If your access or the provider's product changes, what happens to my historical data?" Listen for annotation and re-baselining. Silence, or an assurance that it would not affect anything, is the answer to worry about.
  4. "Does your number represent a stateless request or an account state, and which one?" Either is defensible. Not knowing is not.

And what to do with two conflicting scores: stop trying to determine which is right. Ask each vendor to state collection method, model identifiers, sample size, and variance for the run that produced their figure. Given how the category is structured, most gaps should resolve into one of the five mechanisms above once those four facts are on the table. Then pick one instrument and stay with it, because a consistent index you understand is worth more than an accurate-sounding number you cannot reproduce. Switching instruments resets your history.

The API-only method and its blind spots

For reproducible measurement, use official provider APIs where available. That constraint improves auditability but has coverage trade-offs.

  • Coverage gaps. Some assistant surfaces have no official programmatic access. We do not cover those surfaces, and we say which ones rather than approximating them from something else. A vendor using a different collection method may report on surfaces we cannot see. If those surfaces are where your buyers are, that vendor fits you better than we do, and we would rather say so here than after you have signed.
  • Stateless requests. Runs made this way carry no memory, no history, and no account personality. Reproducible, and unlike any real buyer.
  • The product layer is missing. This method measures model behaviour under a published prompt set, not the full consumer assembly. When a provider changes the layer around the model, its numbers may not move even though what users see does.
  • Every operator is exposed to continuity risk. A commercial agreement is not permanence. Record any break in a time series, publish the change with a dated methodology version, and never silently re-baseline. Treat an untested continuity promise as a procurement claim rather than evidence.

When you compare specific products, platforms compared applies stable criteria across the category, and the tools hub holds the criteria themselves.

In Plain English

When two AI-visibility vendors give you different numbers for the same brand, the usual explanation is that one measures the model through a documented programmatic endpoint and the other derives data from the consumer assistant product. Those are different systems: the consumer product wraps the model in retrieval, instructions, and account state that are undocumented and change without notice, while a programmatic request is thinner, version-pinned, and reproducible. Neither shows you what your buyer actually saw, so treat the output as an index tracked under stated conditions rather than as a literal user's-eye view. Ask every vendor which agreement governs their access, whether they record model identifiers, and what happens to your history if access changes, then take those answers to your counsel and procurement team rather than settling the question yourself. Pick one instrument, insist on raw-response exports, and stay with it.

See the evaluation criteria on the tools hub and take the collection-method question into your next vendor call.

Primary-source verification

Checked July 28, 2026. The operational and factual guidance above was reviewed against the primary references below. Provider features, prices, policies, interfaces, jurisdictions, and account-specific results can change; recheck the controlling first-party document and your own account or measured data before acting.