Skip to main content
RankRift
Menu
Reporting & Impact

What You Can And Cannot Claim Causally About AI Visibility

Model updates, seasonality, and competitor moves all confound your result. Language that survives scrutiny.

attributioncausalityreporting

Your measured AI visibility went up in the quarter you did the work, and now the person who approved the budget wants to know whether the work caused it. The honest answer is almost never a clean yes, and the useful skill is saying so in language that keeps the programme funded rather than language that sounds like hedging.

Scope note. This article is general educational and measurement guidance, not legal, tax, financial, security, or platform-specific advice. Current law, first-party platform rules, the facts of the real case, and qualified professional advice control any consequential decision.

The four confounders you can't rule out

A before-and-after comparison establishes that something changed. It does not establish what changed it. In this category, four things move your number independently of anything you did, and at least two of them are usually moving at any given time.

1. Model and system updates

Assistants change underneath you. Providers ship new model versions, adjust retrieval behaviour, change which sources are surfaced, and modify how answers are grounded. Some of those changes are announced in provider documentation and changelogs; many are not, and some are gradual rollouts that reach different accounts at different times. This is the confounder that most damages before-and-after reasoning, and it deserves its own treatment below.

2. Competitor and third-party moves

Your visibility is measured relative to a field. If a competitor stops publishing, gets acquired, loses a review-site listing, or has a prominent piece of coverage age out, your share can rise without you touching anything. The reverse holds too. Because you usually cannot see what competitors did or when, this confounder is rarely rulable-out even in principle.

Third parties matter as much as competitors. If a publication the models cite in your category updates its "best of" roundup, every brand in that category moves at once.

3. Seasonality and demand shifts

Category interest is not flat. Buying questions change shape across a year, and the phrasing people use changes with them. If your prompt set is fixed and the real-world question drifts, your measurement drifts too, even though nothing about your brand changed. Seasonality also affects the content the models retrieve, because publishers produce category content on a calendar.

4. Your own instrument

The least discussed and most controllable one. If you added prompts, changed a prompt's wording, moved sampling from weekly to daily, switched model versions, changed the region you sample from, or altered how you count a mention, then part of your movement is measurement change rather than reality change. Any of these can produce a step change that looks exactly like an effect.

Rule: an instrument change and an intervention must never land in the same period. If they do, the quarter is uninterpretable, and the honest move is to say so.

Why model updates make before/after comparisons fragile

Every before-and-after design carries an implicit assumption: that in the absence of your intervention, the metric would have stayed roughly where it was. Statisticians call that the counterfactual. You never observe it, so you assume it.

That assumption is weak here for a specific reason. The measurement instrument and the thing being measured are the same system. In classic search you could at least argue that the index was relatively stable between two dates. With assistants, a version change can alter retrieval behaviour, source preference, and answer length simultaneously, and any of those can shift whether your brand gets named. The "before" period was measured on a system that no longer exists.

Three practical consequences:

  • Record the model version against every data point. Without it you cannot even detect that this happened, let alone control for it. If your tool does not store version identifiers, that is a defect worth raising with the vendor.
  • Treat any step change coincident with a known model release as unattributable by default. Not as evidence against your work. As unattributable, which is a different and more honest claim.
  • Watch the whole field, not just yourself. If every brand in your tracked set moves in the same direction in the same week, that is a system change, not a marketing result. This is the single most useful diagnostic available to you and it costs nothing beyond tracking competitors you already track.

Distinguishing a real move from ordinary sampling noise is a prior step to any of this. If you have not already set thresholds for what counts as movement, start with reading variance, because a "change" that sits inside your run-to-run spread does not need a causal explanation at all.

What a controlled comparison would require

To claim your work caused the change, you need a comparison group that experienced everything except your work. In practice that means one of the following.

A holdout. You apply the intervention to some units and deliberately not to others, then compare. Units here might be product categories, market regions, or content sections — anything you can measure separately and treat differently.

What a holdout would actually require:

  • Units measured separately, with their own prompt sets and baselines established before the intervention.
  • Enough units that the comparison is not one-versus-one. Two units gives you a difference, not evidence.
  • Genuine withholding, sustained for the full period. Partial contamination — someone updates the holdout pages because it seemed sensible — voids it.
  • Pre-registration of what you will measure and what result would count as an effect, written down before you look.

A staged rollout. You apply the same intervention to different units at different times, and look for the change to appear in each unit when its treatment starts rather than on a common date. This is often more palatable internally than a holdout, because nobody is permanently denied the work.

What a staged rollout would require:

  • At least three or four stages, separated by enough time that the effect could plausibly appear within a stage.
  • Stable measurement across the whole window, with no instrument changes.
  • A pre-specified expected lag. If you do not state in advance how long the effect should take to appear, you can always find a stage boundary that fits.

The limits of both

Neither design is clean here, and you should say so when you propose one.

  • Contamination. Content is public. Changes made for treated units get retrieved, cited, and generalised in ways that can affect untreated units. Entity-level work in particular — clarifying who your company is — cannot be confined to one product line.
  • Small n. Most teams have a handful of measurable units, not hundreds. That means low statistical power, which means a genuine effect can easily fail to reach significance, and a null result is not evidence of no effect.
  • Time. Model-visible changes appear to take weeks to months to propagate through retrieval and citation, and that lag is itself uncertain. A holdout that runs for three weeks tests almost nothing.
  • Organisational cost. Deliberately withholding work from part of the business is a real conversation, and in many companies the answer is no. That is a legitimate answer. It just means you do not get to make causal claims.

The alternative frequently offered — a multi-touch attribution model that assigns fractional credit — does not fix any of this. Attribution models allocate credit according to rules someone chose; they do not establish causation, and they are not more rigorous for producing a decimal. A model that hands your programme 34% of an outcome has told you about its own weighting, not about the world.

Claims that are defensible today

These hold up under a hostile question because each one describes something you actually observed.

  • Descriptions of what was measured, carrying the prompt set, model versions, run dates, and sample size. This is a factual statement about model output on specific dates, and it is the only kind of claim in this category that is unarguable.
  • Statements of change with sampling context attached, including whether the change exceeds your established run-to-run variance.
  • Temporal association, stated as association. "We did X in March; the metric moved in April" is true and useful. It becomes indefensible only when "therefore" is inserted.
  • Statements about the intervention itself, kept separate from its effect. You completed the work, it shipped on a date, and here is what it changed on your site. That is verifiable regardless of outcome.
  • Statements about the field. If your tracked competitors moved with you, saying so is a stronger and more credible observation than claiming a private win.
  • Honest non-attribution, with the specific missing ingredient named. "We cannot separate this from the model release on [date]" tells a stakeholder something real.

Claims that aren't, however tempting

  • Attributing a movement to your work without a control group.
  • Converting a visibility change into pipeline or revenue via a conversion assumption nobody validated.
  • Treating a correlation between your visibility metric and a pipeline metric as evidence that one caused the other. Both are downstream of demand, and demand moves on its own.
  • Claiming an absence of change proves your work failed, when the sample is under-powered.
  • Any statement implying you can control what a model outputs. You cannot, nobody can, and a claim of that shape is the fastest way to lose the room permanently.
  • Any forward-looking promise about placement. There is no guarantee available in this category, from us or anyone, and the word should only ever appear in a report as a refusal.

Sentence templates for each

Pair them. The left column is what people write. The right column is what survives being challenged.

IndefensibleDefensible
"Our work increased AI visibility by 18% this quarter.""Measured mention rate rose between the stated baseline and follow-up windows. The report includes the prompt set, run count, model versions, and run-to-run variance. We have not isolated the cause."
"The entity work is why ChatGPT now recommends us.""We completed the entity consistency work in [month]. Measured mentions rose in the following period. We did not run a holdout, so we cannot separate our work from model or competitor changes in the same window."
"AI visibility drove £X of pipeline.""N sessions arrived from identifiable AI referrers in the period, and M of those are associated with opportunities in CRM. Attribution beyond that association is not something this data supports."
"We now outrank competitor A in AI answers.""Across our tracked prompt set in [period], our brand was named in a higher share of sampled responses than competitor A. This reflects our prompt set, not the whole category."
"The drop means the campaign failed.""The measured decline coincides with a provider model release on [date]. Every brand in our tracked set moved in the same direction. We are treating this period as unattributable and re-measuring after [date]."
"We can get the model to recommend us.""We can improve the inputs models draw on. We cannot control model output, and we do not report as though we can."
"Nothing changed, so the work didn't help.""We observed no change beyond variance. With N runs across M prompts, this test is under-powered to detect an effect smaller than [threshold]. That is not evidence of no effect."

Copy the right-hand column into your report skeleton once and it stops being extra effort. The structure we use for that is in build a monthly report.

Two other habits that hold up in the room:

  • Report the confounders alongside the number, not in a footnote. A slide that names the model release in the same period reads as competence. The same fact discovered later by someone else reads as concealment.
  • Say the uncomfortable thing first. "We cannot attribute this, and here is what would let us" is a stronger position than a claim that collapses under one question.

How to build toward a stronger claim over time

You are not stuck at "we cannot say." Causal confidence accumulates, and it accumulates in a specific order.

  1. Stabilise the instrument. Fix the prompt set, the sample size, the cadence, the region, and the model versions. Version the whole configuration and change one thing at a time, never during an intervention period.
  2. Establish a baseline long enough to know your normal variance. Without that you cannot tell a result from noise, and every subsequent step is guesswork.
  3. Track the field, always. Competitor series are your cheapest control. Field-wide moves get labelled as system changes automatically.
  4. Log everything external. Model releases, competitor launches, major coverage, seasonal events, your own site changes. A dated log turns "unattributable" into "attributable to one of these three things," which is genuinely more useful.
  5. Run one staged rollout when you have four or more separable units. Pre-register it. Expect it to be under-powered and say so in advance, so that a null result does not get read as failure.
  6. Repeat the intervention. A single result is an anecdote. The same intervention producing the same directional movement across three separate categories at three separate times is a real argument, even without a formal control.
  7. Publish your method internally. A stakeholder who can see how the number is made will accept an uncertain number. A stakeholder shown only a chart will demand certainty you do not have.

A related trap worth handling before it arrives: unattributed direct traffic. When it rises, someone will suggest AI assistants sent it, and that suggestion is usually unevidenced. We work through what can and cannot be shown in is dark AI traffic real?.

In Plain English

You can report what you measured and when. You cannot usually report that your work caused it, because model updates, competitor moves, seasonality, and your own measurement changes are all moving at the same time and you have no control group.

This helps you if you are writing a quarterly report and know that a sharp colleague could take your central claim apart. Use the sentence pairs: replace causal language with measured-change language plus an explicit statement of what you have not isolated. It reads as more competent, not less, because it demonstrates you know what your data supports.

Use this before you present, not after you are challenged. And treat causal confidence as something you build across quarters — stable instrument, long baseline, tracked field, dated external log, then a pre-registered staged rollout when you have enough separable units to try one. Our own measurement rules are on our methodology, and the wider reporting material sits on the reporting hub.

Next: put the defensible sentences into a report structure you can reuse every month.

Get the executive report template.

Primary-source verification

Checked July 28, 2026. The operational and factual guidance above was reviewed against the primary references below. Provider features, prices, policies, interfaces, jurisdictions, and account-specific results can change; recheck the controlling first-party document and your own account or measured data before acting.