The monthly number came in twelve percent above last month and three people have already asked what you did differently. You did nothing differently. Somewhere between "we should tell the exec team" and "this is nothing" there is a defensible answer, and getting to it reliably requires deciding what counts as movement before you look at the movement.
This post is about that discipline. It covers where the wobble comes from, how to find your own noise floor, how to set thresholds you cannot talk yourself out of, and how to tell a genuine step change from slow drift.
A note on vocabulary. Statistical words get used loosely in this field and the looseness is where the mistakes hide, so every term is defined at first use and used only in that sense afterwards.
Why your metric moves even when nothing changed
Scope note. This article is general educational and measurement guidance, not legal, tax, financial, security, or platform-specific advice. Current law, first-party platform rules, the facts of the real case, and qualified professional advice control any consequential decision. Ask a model the same question ten times and you may get ten differently worded answers, sometimes naming different companies. That is not a fault. Generation is probabilistic by design, meaning the model samples from a range of possible outputs rather than returning one fixed answer. Your visibility metric inherits that randomness directly.
There are at least five separate sources of movement, and they need different responses:
| Source | What it is | Does it mean anything? |
|---|---|---|
| Generation randomness | The same prompt returning different answers across runs | No. Average it out with more runs. |
| Sampling error | The gap between your measured rate and the true underlying rate, caused by measuring a limited number of runs | No, but it shrinks predictably as runs increase. |
| Retrieval variation | Live-retrieval answers reflecting whatever was indexed at that moment | Sometimes. Investigate if persistent. |
| Model or system updates | The provider changed the model, its version, or the surrounding system | Yes, but the change is theirs, not yours. |
| Real market movement | Competitor activity, new coverage, your own work landing | Yes. This is the only one you can act on. |
Sampling error deserves a plain-English definition because everything downstream depends on it. If the true rate at which a model names you is some fixed percentage, and you only observe a limited number of runs, your observed rate will land near the true rate but rarely exactly on it. Small numbers of runs land further away. That distance is sampling error, and it is not a mistake you made. It is the price of not running the prompt an infinite number of times.
The practical consequence: with a small number of runs per prompt, a metric can move several points in either direction with nothing whatsoever happening in the world. Sizing that properly is the subject of the companion piece on how many runs you need.
Establishing your own noise floor
Your noise floor is the amount of movement your specific measurement produces when nothing has changed. It is not a universal figure. It depends on your prompt set, your runs per prompt, the models you test, and how common your brand is in the category. Someone else's threshold is not yours.
Measuring it takes one deliberate exercise:
- Freeze everything. Same prompt set, same models and versions, same runs per prompt, same parsing rules for what counts as a mention.
- Run the whole set repeatedly over a short window. Short enough that nothing real is plausibly changing, but with a gap between runs. Several repeats across a few days is a reasonable shape.
- Record the headline metric for each repeat. You now have a small set of numbers that should, in principle, all be the same.
- Look at the spread. The difference between the highest and lowest repeat is your rough noise floor. If you want a more stable figure, calculate the standard deviation, which is the typical distance of individual measurements from their own average. A larger standard deviation means a noisier instrument.
- Write it down, with the date and the configuration it applies to.
That last step matters more than it sounds. The noise floor is a property of a configuration, so it expires the moment you change runs per prompt, edit the prompt set, or add a model. Recalculate it when you do.
The uncomfortable finding, and it is common: for many prompt sets, the noise floor is larger than the month-over-month movement people have been reporting as results. Finding that out early is cheaper than finding it out in front of a board.
Setting action thresholds in advance
Here is the rule that does the most work in this entire post. Decide what movement will trigger action before you see the data. Not after. Not "we'll know it when we see it."
The reason is simple and applies to careful people as much as careless ones. Once you have seen a number you like, you will find reasons the threshold should be lower. Once you have seen a number you dislike, you will find reasons it should be higher. Setting the rule in advance removes the negotiation entirely, and it means you can defend your interpretation to someone who suspects you of motivated reasoning.
A workable structure has three bands, all defined relative to your measured noise floor:
- Inside the noise floor: report and do nothing. Show the number, state that it is within expected variation, move on. No investigation, no narrative.
- Beyond the noise floor but sustained for only one period: re-run, do not react. One period past the line is a candidate, not a finding.
- Beyond the noise floor and sustained across consecutive periods: investigate. Now there is something to explain.
Two guardrails on this. First, we are not going to prescribe a universal threshold, because there is no defensible one. A percentage that is meaningless for a high-volume consumer category may be substantial in a narrow B2B category with four credible vendors. Anyone who hands you a fixed "significant change" number without asking about your prompt set and run count is guessing.
Second, be careful with the word significance. In statistics it has a narrow technical meaning: a result is statistically significant relative to a stated test and a stated threshold, meaning the observed difference would be unlikely to arise from chance alone under a specific assumption. It does not mean important, and it does not mean large. A tiny movement can be statistically significant with enough runs, and a large movement can be uninformative with few. If you use the word in a report, state the test and the threshold beside it, or use "beyond our noise floor" instead, which is honest and needs no footnote.
Similarly, a confidence interval is a range calculated from your data using a stated procedure, such that if you repeated the whole exercise many times, the intervals produced would contain the true value a specified proportion of the time. It is not a statement that the true value has a given probability of sitting inside this one particular range. The practical use is what matters: if two periods' intervals overlap substantially, you should be slow to claim they differ.
A worked illustrative example: noise misread as a win
The following numbers are illustrative and invented for teaching purposes. They are not from any real measurement, real brand, or real research run.
An illustrative company, Brand A, tracks presence rate across a 40-prompt set, running each prompt 3 times per month against one model, for 120 observations per month.
- January: named in 42 of 120 responses, a presence rate of 35%.
- February: no changes to the programme. Named in 50 of 120, a presence rate of 42%.
That is a seven-point rise, or twenty percent relative growth. It is reported as an early win for a content programme that shipped in January.
In March, still with no further changes, the figure comes back at 33%. In April it is 38%.
What went wrong is that nobody measured the noise floor first. Had Brand A repeated the January measurement four times in the same week, they would have seen the metric ranging across roughly the same band all four times, with no intervention at all. The February result was inside that band. It was never a finding.
The additional trap: because the February number was attributed to the content programme, the March decline had to be explained too, and the explanation offered was that a competitor had responded. Two invented causal stories, one underlying instrument, zero real events. This is the ordinary failure mode, and it costs credibility slowly rather than dramatically.
The fix in this illustrative case is not a bigger conclusion. It is more runs per prompt, a measured noise floor, and a threshold agreed in advance.
Distinguishing a step change from drift
Once something does clear your threshold, the shape of the movement tells you where to look.
A step change is an abrupt shift to a new level that then holds. The metric was oscillating around one value, then oscillates around a different one. Step changes usually have a discrete cause: a model or system update, a major piece of third-party coverage appearing or disappearing, a competitor launching, a technical change on your site that altered whether you are retrievable at all.
Drift is gradual movement in one direction across several periods, where no single period's change would clear your threshold but the cumulative movement does. Drift usually reflects something accumulating: your content presence in the sources models draw on growing or decaying, an entity description slowly propagating, a competitor's coverage compounding.
The distinction changes your investigation:
| Step change | Drift | |
|---|---|---|
| Shape | Abrupt, then stable at a new level | Slow, consistent direction |
| First thing to check | What changed on that specific date | What has been accumulating over months |
| Common false alarm | A model update mistaken for market movement | A run of same-direction noise mistaken for a trend |
| Sensible response | Investigate the date | Confirm across more periods before acting |
The false alarm for drift deserves emphasis. Random movement produces same-direction runs more often than intuition expects. Three consecutive small rises is not a trend, and treating it as one is the most common way a monitoring programme starts telling people what they want to hear.
When to re-run instead of react
Re-running is cheap. Investigations, meetings, and strategy changes are not. Default to re-running when:
- The movement is close to your threshold rather than clearly past it.
- Only one model moved and the others did not.
- The movement appears on a handful of prompts rather than across the set.
- A model version changed inside the measurement window.
- The result is convenient. Convenient results deserve the same scrutiny as inconvenient ones, and they rarely get it.
Re-run with the identical configuration. Changing anything while investigating a movement means you can no longer tell whether the movement or your change produced the second result. If the second run reproduces the movement, you have something. If it does not, you have saved everyone a week.
And whatever you find, keep the causal language tight. Reproducing a movement establishes that the movement is real. It does not establish what caused it, which is a separate and much harder claim, covered in the piece on what you can and cannot claim causally.
Recording model-version changes as annotations
Every time-series chart you produce should carry annotations for model and system updates, marked on the date they took effect.
The reason is that a provider updating a model changes the instrument you are measuring with. Movement across that date is a mixture of two things: whatever happened in your market, and whatever changed in the model. Without the annotation, a future reader, quite possibly you, will attribute the whole movement to the market. Practical habits that make this work:
- Log the exact model identifier and version with every run, not just the product name.
- Pin versions where the provider's API allows it, so updates arrive when you choose rather than silently.
- Keep a dated changelog of your own configuration changes in the same log, since your edits create discontinuities exactly like theirs.
- Mark a comparison as non-comparable, on the chart itself, when a version changed between the two periods. A visible caveat is worth more than a correct footnote nobody reads.
This is the same provenance discipline we apply to our own published runs, described in our methodology.
In Plain English
What this means: an AI-visibility metric moves on its own. Before any movement can be called a result, you need to know how much your own measurement wobbles when nothing is happening, and you need to have decided in advance what size of movement earns a reaction.
Who this helps: anyone running an ongoing measurement programme who has to say each month whether the number means anything.
When to use it: at the start of a programme, when you set it up, and every time a number moves in a direction someone likes.
What to do next: measure your noise floor with repeated runs on a frozen configuration, write down a three-band action threshold before your next reporting period, and start annotating model versions on every chart. The wider measurement approach that sits around this is on how to measure AI visibility.
Use the briefing template to publish method notes, corrections, and the reasoning behind decision thresholds.
Primary-source verification
Checked July 28, 2026. The operational and factual guidance above was reviewed against the primary references below. Provider features, prices, policies, interfaces, jurisdictions, and account-specific results can change; recheck the controlling first-party document and your own account or measured data before acting.
- Google Search: AI features and your website - How Google describes eligibility, preview controls, and appearance in its AI search features.
- Google: common crawlers and special-case user agents - Current Googlebot and Google-Extended identities and controls.
- OpenAI: overview of OpenAI crawlers - Current GPTBot, OAI-SearchBot, and ChatGPT-User descriptions and publisher controls.
- Anthropic: web-crawler controls - Current ClaudeBot, Claude-SearchBot, and Claude-User descriptions and controls.
- Perplexity: crawler and user-agent documentation - Current PerplexityBot and Perplexity-User descriptions and controls.
- FTC: Endorsement Guides Q&A - Current U.S. regulator guidance on endorsements, disclosures, and reviews.
- FTC: Consumer Reviews and Testimonials Rule Q&A - Current U.S. rule guidance on deceptive review practices.