How to measure AI visibility without fooling yourself

AI answers vary by model, phrasing, region and date, so a screenshot proves nothing. This is the ruler: what you are estimating, how to build a prompt set, which metric answers which question, and what every result must carry to be defensible.

Read the methodology · Size your sample

Measuring a non-deterministic system is a sampling problem. Any number quoted without a prompt set, model versions, run dates, sample size and variance is a claim you cannot defend when someone pushes back.

In short

AI visibility measurement estimates how often assistants name, cite, or recommend a brand for a defined set of buying questions, over a defined window, across named model versions. It is estimation, not observation, because the system is non-deterministic and personalised. Doing it honestly means building a prompt set that reflects how buyers really ask, choosing the metric that matches the question being asked, running enough samples to detect a change worth acting on, reporting the spread rather than a single figure, and attaching a provenance record to every result. You can do all of it in a spreadsheet before you spend anything.

Who this is for

In-house SEO and growth leads who need a defensible metric to replace eroding click data
Agency and consultant strategists standardising measurement across several clients without staking their name on an unexplainable number
Analytics and RevOps people asked to instrument something that resists conventional attribution
Founders who were handed a vendor score and want to know whether it means anything
Anyone about to buy a tracking tool who has not yet defined what they want counted

When to use this

Before you set a baseline, so the baseline is still interpretable in six months
When a report shows movement and you have to decide whether it is a real change or noise
When two tools give you different scores for the same brand and you need to work out why
When you are writing the measurement section of a proposal and every claim has to survive scrutiny
Before buying anything, to find out whether a spreadsheet already covers your needs

What you are actually estimating

You are not reading a value off a dial. You are estimating, from a sample of generated answers, how likely an assistant is to name, cite, or recommend a brand for a defined class of questions, during a defined window, against a named model version.

That sentence contains every constraint that follows. Change the questions and the estimate changes. Change the window and it changes. Change the model version and it changes, sometimes without any announcement from the provider. This is why the measurement object has to be written down before the first run, not reconstructed afterwards.

It also sets the limit of what a result can support. A well-run measurement tells you what came back from a sample of prompts on some dates. It does not tell you why, it does not tell you what a buyer would have done next, and it does not tell you that your work caused a difference.

If you cannot state the population of questions you are sampling from, you do not yet have a measurement — you have a collection of answers.

The prompt set is your sampling frame

A prompt set is the list of questions you will ask, in the phrasings you will ask them, at the buyer stages you care about. It is not a keyword list and it is not a wish list. It is the sampling frame that decides what your number can possibly mean, and it is where most scores go wrong before a single run happens.

The failure mode is self-flattery. Prompts that name your brand, describe your exact differentiator, or use your internal category label will return you at a high rate and tell you nothing. Buyer-realistic prompts describe a problem or a category the way an outsider would say it, and they will often be uncomfortable to read.

Version and date the set. When you change it, you have started a new time series, and pretending otherwise is the quietest way to publish a misleading trend.

The metric family, and what each one gets wrong

Four numbers get shown on dashboards and they answer different questions. Mention rate is the share of runs in which the brand is named at all — runs naming the brand divided by total runs. Citation share is the share of cited sources across runs that point at your own properties. Share of voice compares your mentions against the set of brands named in the same runs. Framing and sentiment describe how you are characterised when you do appear.

Each misleads in its own way. Mention rate treats being named as a warning and being named as a recommendation identically. Citation share can rise while recommendations fall, because being read is not being endorsed. Share of voice depends entirely on which competitors your prompt set summons, so a change in the competitor set moves it without anything about you changing. Framing is the hardest to score consistently and the easiest to grade generously in your own favour.

Pick the metric that matches the question your stakeholder is really asking, report it with its denominator visible, and never let a single composite score stand in for all four.

Sampling and variance: why one run is not evidence

Because the same prompt can produce different answers on consecutive runs, a single run gives you a sample of one from a distribution you have not characterised. Three runs is barely better. How many you need depends on how rare the outcome is and how small a change you want to be able to detect — a rare mention and a small target change together demand a lot of runs.

The practical consequence is that most movement in most reports is noise. Before you act on a difference, you need some idea of the spread you would see if nothing had changed at all. Run a stable prompt set repeatedly during a quiet period and you will have that baseline; without it, every wobble looks like a story.

A screenshot is the weakest form of this. It is a single draw, undated, from an unknown model version, in someone's personalised session. It can be useful to illustrate a point to a colleague. It is not evidence, and it should never appear in a report as one.

Write down the calculation and its assumptions next to any figure that depends on them, so a reader can check the arithmetic and see where it stops applying.

Start in a spreadsheet, not in a purchase order

You can run a credible first measurement with a defined prompt set, official model APIs, a sheet, and a scoring rubric you wrote down before you looked at any answers. It is slower than a platform and it will not scale past a certain number of prompts, brands, and cadences — but it teaches you what the numbers mean, which is exactly the knowledge a dashboard lets you skip.

Do this first and one of two things happens. Either the manual method covers what you need, and you have saved a subscription. Or it visibly breaks under your volume, and you now know precisely which capability you are buying and what to ask a vendor to prove.

Measurement uses official model APIs. Scraping assistant web interfaces, evading rate limits, or automating consumer accounts is off the table here for both terms-of-service and defensibility reasons.

The provenance record every result must carry

A result without provenance cannot be re-run, cannot be defended, and cannot be corrected. Attach the record to the number itself, not to a methodology appendix nobody opens.

Use this record for reader-run research and hold any vendor to the same standard. If a supplier cannot produce these fields for a figure shown in a demo, treat the figure as a marketing artefact.

Prompt set reference and version, with the exact phrasings
Model names and versions, plus the surface each was queried through
Run dates and the window the result covers
Sample size, stated per prompt and in total
Variance, reported as a range or interval, with the method named
Method version and a link to the full specification

How these numbers get used against you

The same metrics that make a programme defensible make a persuasive slide when they are cherry-picked. Single runs presented as findings, prompt sets quietly rewritten between periods, a silent model change absorbed into a trend line, competitor sets chosen to flatter, brand-baiting prompts that guarantee a mention — each of these produces a chart that looks good and means nothing.

Learn the patterns so you can audit a report before you present it, including one produced by your own team.

Frequently asked questions

How many prompts should be in my prompt set?

Enough to cover the buyer stages and phrasings that matter, and no more than you can afford to re-run at your chosen cadence. Coverage beats volume: a set that spans problem-stage, category-stage and comparison-stage questions in the words outsiders use is more informative than a hundred near-duplicates. Whatever you choose, version and date it, because changing the set mid-programme starts a new time series even if the dashboard keeps drawing one line.

Can I just use a screenshot of an answer as proof?

No. A screenshot is one draw from a distribution, taken in a personalised session, usually undated and without a recorded model version. It can illustrate a point in conversation, and it is useful evidence when documenting a factually false statement about your brand for a correction. It is not a measurement, and presenting it as one in a report will eventually be challenged by someone who runs the same prompt and sees something different.

Which single number should I report to leadership?

Resist the single number. If you are forced to one, report mention rate for a named prompt set with its sample size and range visible, and carry share of voice as the competitive context underneath it. A composite visibility score that blends metrics with undisclosed weights is unauditable, which means you cannot explain a movement in it. Anything you report you should also be able to decompose on request.

My tool's number does not match what I see when I ask myself. Who is wrong?

Probably neither. Your personal session carries history, region, account settings and a served model version that a measurement run does not share, and a single ad-hoc question is one sample against the tool's many. The productive question is whether the tool can show you its prompt set, model versions, run dates, sample size and variance. If it cannot, the mismatch is unresolvable and that is itself the finding.

How often should we re-run?

Frequently enough that a model change does not sit undetected between measurements, and consistently enough that periods are comparable. A fixed cadence with a fixed prompt set beats an irregular one with a larger sample, because comparability is what lets you separate signal from noise. Whatever cadence you pick, record run dates on every result and re-baseline explicitly, in public, when a provider ships a change you can observe.

Do I need to pay for a tool to measure this?

Not to start. A defined prompt set, official model APIs, a spreadsheet and a scoring rubric written before you look at any answers will produce a defensible first baseline. Tools earn their place when your prompt count, brand count, cadence or reporting obligations make manual runs impractical. Doing it manually first also means you can interrogate any vendor's numbers rather than accepting them, which is worth more than the time it costs.

Does a mention mean we are being recommended?

Not necessarily. Assistants name brands for many reasons, including listing an option the answer then advises against. Raw mention counts collapse endorsement, neutral listing and explicit warning into one figure. If the decision you are supporting depends on whether buyers are being pointed at you, you need framing captured alongside the count, with a rubric applied consistently and ideally scored by someone who did not build the prompt set.

Is a free one-off check a real measurement?

A one-off check is only a measurement when it exposes the prompt sample, models queried, run date, run count, metric definition, and uncertainty. Treat a small snapshot as a prompt to build a proper prompt set and baseline, not as a standalone board metric.

Build a measurement you can defend

Start with the published methodology, size your sample honestly, and build a versioned prompt set from buyer-stage patterns rather than inventing sixty questions from scratch.

Read the methodology · Open the sample-size calculator · Use the prompt patterns