Skip to main content
RankRift
Menu
AI Search Fundamentals

Why Two People Asking The Same Question Get Different Brands

Non-determinism, personalization, region, and session context — why one answer is never a measurement.

variancesamplingai-search

A colleague runs the question on her laptop and your company is the second name in the answer. You run the same words ten minutes later and your company is absent. Nobody in the thread can say who is right, and by the afternoon two people are arguing about a visibility problem that may not exist. This is not a mystery about AI. It is a sampling problem with a known shape, and the fix is method rather than a better screenshot.

Variation is a property of the system, not a fault in your observation. That means it can be measured, reported honestly, and reasoned about. What it cannot be is resolved by one person running the query one more time.

The five things that differ between two people's answers

There are at least five distinct reasons two answers diverge. They have different causes and different remedies, so keep them separate.

1. Sampling variation (the model's own non-determinism). Language models select tokens probabilistically. Run the identical prompt against the identical model twice and the wording can differ, and so can the set of brand names, without anything else changing. Some APIs expose parameters that reduce this, and providers generally do not promise bit-identical output even so. This is the source people forget first and it is often the largest.

2. Personalization and account context. Consumer assistant accounts may carry memory, saved preferences, subscription tier, or prior conversation history that conditions later answers. The precise mechanism differs by provider and by product tier, and it changes. Do not assert how any specific provider personalizes without pointing at its current documentation.

3. Region and language. Location can influence which sources get retrieved and which brands appear, particularly for anything with local availability, regulatory differences, or a language-specific market. Two colleagues in different countries are not running the same test, even with identical words.

4. Session context. What was said earlier in the same conversation conditions what comes next. A question asked cold and the same question asked after three turns about your industry are different inputs. Sessions also pick up whatever the user did before: an uploaded file, a stated role, a follow-up that narrowed the topic.

5. Model and product version. Providers ship changes to models, retrieval behaviour, and interfaces continuously, sometimes without a version bump visible to users. Two people on different tiers or different apps may be on different versions of the underlying system on the same day. An observation without a model version and a date cannot be compared to anything later.

To that list add a sixth thing that is not variation at all but gets mistaken for it: the two people did not actually ask the same question. Word order, a dropped qualifier, plural instead of singular. Prompt text has to be identical and recorded verbatim before any of the above is worth discussing.

Which of them you control and which you don't

Control here means control over your test conditions. None of it means control over what a model says.

Source of variationCan you hold it constant?How
Sampling variationNoCannot be removed; must be averaged out across repeated runs
Personalization and account contextLargely yesMeasure through official APIs rather than consumer accounts, so no user memory is attached
Region and languageYesFix and record the region and language of every run; treat each region as its own measurement
Session contextYesOne question per session, no follow-ups, no carried history
Model and product versionPartlyPin the model version where the API exposes one, and record it on every run either way
Prompt wordingYesA written prompt set, used verbatim, versioned, and changed only deliberately

The pattern is clear: four of the six can be controlled by testing discipline, and only one, sampling variation, is genuinely irreducible. That is why measurement here uses official model APIs. It is not a purity position. It is the only way to hold personalization and session context still, and it keeps the work inside providers' documented interfaces rather than automating consumer accounts.

The irreducible one is handled with statistics, not with effort. You do not remove sampling variation. You run enough times to estimate how often something happens and to state how uncertain that estimate is.

Why a screenshot is not evidence

A single screenshot is one draw from a distribution you have not characterised. It tells you the outcome was possible. It tells you nothing about how likely it is, and likelihood is the entire question when someone asks whether you are visible.

Three specific failures follow from treating a screenshot as a finding.

It cannot distinguish absence from infrequency. A brand that appears in a minority of runs will be missing from any given screenshot fairly often. "We are not in the answer" and "we are in the answer sometimes" look identical at a sample size of one.

It has no comparison point. Even if the observation is real, you cannot say it changed unless you know what it was before, measured the same way. Most screenshot panics are the first measurement anyone has ever taken.

It arrives with selection bias attached. The screenshots that get shared are the alarming ones. Nobody circulates the run where everything looked normal, so the evidence reaching leadership is systematically skewed toward bad news, or toward good news when a competitor's dashboard is doing the sharing.

The rule for anecdotal screenshots: a screenshot is a ticket, not a finding. Log it with the prompt text, assistant, date, region, and account type, then decide whether to run a proper sample. It never goes on a slide, into a board update, or into a client report as evidence of a change.

That rule is worth stating out loud to your team once, because it prevents the most common failure in this category: an expensive reaction to a single draw. It also protects you in the other direction, when a favourable screenshot tempts someone to declare a win.

None of this means the metric is unreliable. Plenty of useful measurements sit on top of noisy processes, from A/B tests to polling. Variation makes the measurement a statistical exercise instead of an observation. It does not make it worthless, and treating it as worthless is just the mirror image of treating a screenshot as proof.

What a defensible answer to "are we visible?" looks like instead

The question "are we visible?" has no defensible answer as posed, because it has no unit. The version that can be answered looks like this:

Across a fixed set of buyer-realistic prompts, run repeatedly against named model versions from a fixed region with no session history, our brand was named in the answer in a measured proportion of runs, with a stated uncertainty range, on stated dates.

That sentence has seven components, and each one is doing work:

  • A prompt set, written down and versioned, representing realistic buyer situations rather than keywords.
  • Repeated independent runs of each prompt, each in a clean session.
  • Named models and versions, recorded per run.
  • A fixed region and language, recorded per run.
  • Run dates, because the system changes underneath you.
  • An outcome definition that separates named in the prose from cited as a source, since these are different events.
  • An uncertainty range around the proportion, not a bare percentage.

The statistical core is unremarkable. Each run either produced the outcome or did not, so what you are estimating is a proportion, and a proportion estimated from a small number of runs carries a wide interval. Precision improves with more runs, but slowly: the interval narrows in proportion to the square root of the number of runs, so cutting your uncertainty in half costs roughly four times the runs. That trade-off is the reason sample size has to be decided before you start rather than negotiated afterwards. The sample-size calculator exists to make that decision explicit, and how many prompt runs you need works through the reasoning.

The related question, whether a change between two measurements is real, is harder than a first baseline and needs its own treatment. Two intervals that overlap heavily are not evidence of movement, however different the midpoints look on a chart. Telling a real change from noise covers how to decide.

The minimum you should do before reporting anything

If you take nothing else from this post, take the following floor. Below it, do not report a number.

  1. Write the prompt set down and use it verbatim. Version it. Note the date you changed it.
  2. Decide the sample size before running, using an explicit target for how precise you need to be.
  3. Run through official APIs, one prompt per clean session, no follow-ups, no consumer-account memory attached.
  4. Record per run: prompt ID, assistant, model version, region, language, date, timestamp, whether the brand was named, and whether it was cited as a source.
  5. Report a range, never a bare number. A percentage with no interval is a claim you cannot defend when someone asks how confident you are.
  6. State what you did not measure. Other regions, other languages, other assistants, other phrasings, logged-in behaviour.
  7. Keep the raw runs. If a result is disputed later, the run log is the only thing that settles it.

Do not compare a number produced this way to a number produced any other way, including a vendor dashboard, unless both methods are on the table. Different prompt sets and different outcome definitions produce different numbers from the same underlying reality, and reconciling them without the methods is guesswork.

In Plain English

Two people get different brands because the system is genuinely variable: the model samples probabilistically, accounts carry personalization, region and session context shape the answer, and versions change underneath everyone. Four of those can be held still by testing through official APIs with a fixed prompt set. The fifth cannot, which is why you run many times and report a range instead of a number.

This matters most to whoever has to answer "are we visible in AI answers?" in front of people who will act on the answer. Use it the moment a screenshot starts circulating, and again before your first baseline. The next step is to decide your sample size before you run anything, then hold the line that no single screenshot reaches a report.

Work out how many runs your question actually needs: try the sample-size calculator, or create a manual snapshot to see a sampled snapshot with its confidence range attached.

Primary-source verification

Checked July 28, 2026. The operational and factual guidance above was reviewed against the primary references below. Provider features, prices, policies, interfaces, jurisdictions, and account-specific results can change; recheck the controlling first-party document and your own account or measured data before acting.