You send the same prompt three times and get three different lists. Your brand shows up twice. Someone writes down 67% and the meeting moves on. The budget question underneath is simple and unpleasant: every extra run costs money, and nobody has told you whether five runs or fifty is the defensible number.
There is no single answer, and any guide that gives you one is guessing on your behalf. What exists is a method for deciding, based on three things you can estimate before you spend anything. This post walks through that method, shows the arithmetic that makes cost visible, and gives you language for the very common situation where the honest sample size is out of reach.
What you are actually estimating
Start by naming the quantity, because most sampling arguments are really disagreements about the estimand.
When you run one prompt against one model many times, you are estimating an appearance rate: the probability that this model, in this configuration, on these dates, names your brand in response to this prompt. It is a proportion. It has a true value you cannot observe and a sample value you can.
That framing has three immediate consequences.
Two runs of the same prompt are a sample, not a fact. "Your brand appeared" and "your brand did not appear" are two draws from a coin whose bias you are trying to learn.
The estimand is narrow. It is specific to one prompt, one model version, one configuration, one region, one date range. Change any of those and you are estimating something else. This is why prompt-set versioning is not bureaucracy: a rate without a prompt-set version attached has no defined subject.
Aggregating across prompts creates a different quantity. A category-level "visibility rate" across forty prompts is an average of forty different proportions, weighted by however many runs each got. That is a legitimate thing to report, but it is not the same as a proportion estimated from a single population, and the uncertainty around it does not shrink the way a simple sample-size formula suggests. More on that below.
Two related pieces sit underneath this one: why answers vary between people explains where the run-to-run variation comes from, and telling a real change from noise covers what to do with the variation once you have measured it.
The three inputs: baseline rate, detectable change, confidence
Every sample-size decision reduces to three numbers you supply.
1. Baseline rate (p₁). Roughly what proportion of runs currently name you for this prompt? You do not need precision here. A pilot of ten to twenty runs is enough to know whether you are near 5%, near 30%, or near 70%, and that is enough to size the real test. Rates near 50% need larger samples than rates near 5% or 95%, because a proportion's variance is largest in the middle.
2. Detectable change (Δ). How big a change do you need to be able to see? This is the input people skip, and it is the one that dominates the arithmetic. Sample size scales roughly with the inverse square of Δ: halving the change you want to detect multiplies the runs needed by about four. Set this by asking what size of movement would actually change a decision. If a five-point shift would not alter anything you do, do not pay to detect it.
3. Confidence and power. Confidence is your tolerance for calling a change real when it is not. Power is your tolerance for missing a change that is real. Conventional defaults are 95% confidence and 80% power, and they are conventions rather than laws. Lower them deliberately if you want, but say so in the report, because a result at 80% confidence and 60% power is a weaker claim and readers deserve to know.
Note what is missing from that list: how important the brand is, how much you spent on content last quarter, and how confident the agency feels. None of those affect the arithmetic.
A rule-of-thumb table (with its assumptions stated)
Scope note. This article is general educational and measurement guidance, not legal, tax, financial, security, or platform-specific advice. Current law, first-party platform rules, the facts of the real case, and qualified professional advice control any consequential decision. Two tables follow. The first tells you how precise a single estimate is at a given number of runs. The second tells you how many runs each side needs to detect a change between two points in time.
Assumptions behind both tables. Every one of these must hold, at least approximately, for the numbers to mean what they appear to mean. Where one fails, the tables understate the runs required.
- The outcome is binary: your brand is named in the answer, or it is not. A framing or ranking outcome is a different measurement and needs a different design.
- Runs are independent. No shared session, no memory or personalization carrying between runs, no caching returning a stored answer.
- The model version and configuration are fixed and pinned for the whole run window. A version change mid-window means you sampled two different systems and the comparison is void.
- Temperature and any other sampling parameters are held constant across compared groups.
- The prompt string is byte-identical across runs, and the prompt set version is unchanged between the two periods you are comparing.
- Region, account, and any retrieval settings are constant.
- The true underlying rate is stable across the run window. If it drifts within the window, you are averaging over a moving target.
- The figures use the normal approximation to the binomial, a two-sided test at 95% confidence and 80% power. The approximation is unreliable at very low or very high rates, or when the expected count of appearances is small; treat anything under roughly five expected appearances per group as indicative only.
- The numbers are per prompt, per model. They are not a budget for a whole prompt set.
- No correction is applied for testing many prompts at once. If you run forty prompts and go looking for the ones that moved, some will appear to have moved by chance alone. See the multiplicity note below.
- Rounded up to convenient numbers. They are planning figures, not thresholds to defend.
Table A: precision of a single appearance-rate estimate. Approximate 95% margin of error, in percentage points, for a proportion near 50% (the widest case).
| Runs of this prompt on this model | Approximate 95% margin of error |
|---|---|
| 10 | ±31 points |
| 30 | ±18 points |
| 50 | ±14 points |
| 100 | ±10 points |
| 200 | ±7 points |
| 400 | ±5 points |
Read the top row carefully. At ten runs, an observed 60% is consistent with a true rate anywhere from roughly 30% to roughly 90%. The three-run estimate that started this post has an interval so wide it is barely a constraint at all. Margins narrow at rates further from 50%: at a true rate near 10%, thirty runs gives roughly ±11 points rather than ±18.
Table B: runs needed per period to detect a change. Each cell is runs per period, so a before-and-after comparison costs twice the figure shown.
| Baseline rate | Change you want to detect | Approximate runs per period |
|---|---|---|
| 30% | 30 points (to 60%) | ~40 |
| 30% | 20 points (to 50%) | ~90 |
| 30% | 10 points (to 40%) | ~355 |
| 30% | 5 points (to 35%) | ~1,375 |
| 10% | 20 points (to 30%) | ~60 |
| 5% | 10 points (to 15%) | ~140 |
| 50% | 20 points (to 70%) | ~90 |
The shape matters more than any individual cell. Large changes are cheap to detect. Small changes are expensive, and the cost grows fast. Detecting a five-point move at a 30% baseline costs roughly fifteen times what detecting a thirty-point move costs.
The clustering caveat. If you run forty prompts twenty times each, you have 800 answers but nothing like 800 independent observations about your category. Answers to the same prompt resemble each other far more than answers to different prompts do. Prompt-to-prompt variation is typically the larger source of spread in this kind of measurement, which means the effective sample size for a category-level number is closer to the number of prompts than the number of answers. Practical implication: when you are estimating a category-level rate and have a fixed budget, more prompts with fewer runs each usually buys more information than fewer prompts with many runs each. When you are estimating one prompt's rate precisely, the opposite holds. Decide which question you are answering before you allocate. If you want a defensible confidence interval on a clustered estimate, that is a job for someone who can specify the model, not for a lookup table.
The multiplicity caveat. Scanning forty prompts for "the ones that changed" at 95% confidence means you should expect around two false alarms per run even if nothing changed anywhere. Decide in advance which prompts are your primary measures, treat the rest as exploratory, and label them that way in the report.
The sample-size calculator applies the same arithmetic with your own inputs and shows the assumptions on screen rather than burying them. The full set of design choices sits in our methodology.
Cost: how runs multiply across prompts and models
Sampling decisions feel abstract until you write the multiplication out. The unit is one API call.
Calls per cycle = prompts × runs per prompt × models × regions
An illustrative programme, with round numbers chosen for legibility rather than drawn from any real deployment:
| Design | Prompts | Runs each | Models | Regions | Calls per cycle |
|---|---|---|---|---|---|
| Pilot | 20 | 5 | 2 | 1 | 200 |
| Standard monthly | 40 | 20 | 4 | 1 | 3,200 |
| Standard, two markets | 40 | 20 | 4 | 2 | 6,400 |
| Precision test on one prompt | 1 | 355 | 1 | 1 | 355 |
| Precision across the set | 40 | 355 | 4 | 1 | 56,800 |
Then multiply by cadence. The standard monthly design at twelve cycles a year is 38,400 calls. Weekly, it is roughly 166,000. Multiply by your own per-call cost, which depends on the model, the prompt length, and the answer length, and which you should take from the provider's current pricing page rather than from any figure quoted in an article. Add the cost of storing and coding the answers, which is often larger than the API bill once a human is reading any of them.
Three levers cut this without lying about it:
- Cadence. Monthly instead of weekly is a four-fold reduction and usually loses very little, because the underlying rates rarely move fast enough for weekly sampling to be informative.
- Model coverage. Measuring the two surfaces your buyers actually use, rather than all of them, is a defensible choice as long as the report says which were covered.
- Tiering the set. Give twenty core prompts deep sampling and run the long tail shallowly, clearly labelled as indicative.
What to do when you cannot afford the honest sample size
This is the normal situation, not a failure. The wrong response is to run the small sample and report the number as though it were precise. Here are better options, roughly in order of preference.
Widen the detectable change. Accept that you can see thirty-point moves and not ten-point ones. Say so, and stop reporting single-digit movements entirely.
Narrow the scope. Sample a few prompts properly rather than every prompt badly. A well-powered result on eight prompts is more useful than an under-powered result on forty, and it is far easier to defend.
Pool across prompts deliberately. Report the category-level rate, where prompt count carries most of the information, and stop making per-prompt claims. State the pooling rule.
Lengthen the window. Combine two cycles into one reported period. You lose time resolution and gain precision, and the trade is worth stating explicitly.
Report direction and interval, not a point. "Between 25% and 55%, no detectable change since the last run" is a true sentence. "38%, down two points" is not, at that sample size.
Report qualitatively. Read the answers. Note who is being recommended and on what grounds. Observations about content are not sample-size-limited in the same way, as long as you never convert them into a percentage.
What you should not do: run the same prompt until you get a result you like, drop prompts that underperform after seeing the data, or add runs mid-analysis until a difference becomes significant. Each of these invalidates the confidence level you are quoting, and each is easy to do accidentally when the analysis and the collection are the same person's job.
How to report an under-powered result without misleading anyone
Say it plainly, in the same sentence as the number. Use language like this:
Measured across 5 runs per prompt on 20 prompts against 2 models during the run window stated above, using prompt set v1.2. This sample supports statements about differences of roughly 30 points or more. The apparent 6-point decline since the previous cycle is within the range we would expect from sampling variation alone and should not be treated as a change.
Other usable phrasings, each honest about a different limit:
- "Indicative only. This run was sized to establish a baseline range, not to detect change."
- "We cannot distinguish this result from no change. Detecting a movement of this size would need roughly [n] runs per period, which is outside the current budget."
- "Prompt-level figures below are exploratory. The two primary prompts are marked, and only those support a claim."
- "The interval is wide because the sample is small. It will narrow at the same rate whether or not the underlying position improves."
Two habits make the honesty sustainable. First, put the detectable-change statement in the report template so it appears every cycle rather than only in bad quarters. Second, when a number does move beyond the detectable threshold, point back at the standing caveat: a programme that has been consistently modest about precision is far more credible on the occasion when it says something real has happened.
Never present a percentage without the sample size and the run dates attached. A bare percentage in an AI visibility report is the single most common way a measurement programme loses its credibility, usually in the meeting where someone finally asks how it was calculated.
In Plain English
There is no correct number of runs. There is a number that follows from three inputs: your current appearance rate, the size of change you need to see, and how confident you want to be. Big changes are cheap to detect; small ones get expensive fast. Ten runs of a prompt gives you a range about thirty points wide, which is a baseline, not a metric. If your budget cannot buy the precision you want, reduce the precision you claim rather than the honesty of the report.
This is for whoever owns the measurement budget and will have to defend the numbers: in-house search and growth leads, agency strategists, and analysts building the reporting. Use it before you commit to a cadence, and again whenever someone asks why a small movement is not being treated as news.
Next step: run a twenty-run pilot on your five most important prompts to get rough baseline rates, decide what size of change would actually change a decision, then size the real programme from those two numbers and write the detectable-change sentence into your report template before the first cycle.
Try the sample-size calculator to work through your own baseline, detectable change, and cost, with every assumption shown alongside the result.
Primary-source verification
Checked July 28, 2026. The operational and factual guidance above was reviewed against the primary references below. Provider features, prices, policies, interfaces, jurisdictions, and account-specific results can change; recheck the controlling first-party document and your own account or measured data before acting.
- Google Search: AI features and your website - How Google describes eligibility, preview controls, and appearance in its AI search features.
- Google: common crawlers and special-case user agents - Current Googlebot and Google-Extended identities and controls.
- OpenAI: overview of OpenAI crawlers - Current GPTBot, OAI-SearchBot, and ChatGPT-User descriptions and publisher controls.
- Anthropic: web-crawler controls - Current ClaudeBot, Claude-SearchBot, and Claude-User descriptions and controls.
- Perplexity: crawler and user-agent documentation - Current PerplexityBot and Perplexity-User descriptions and controls.
- FTC: Endorsement Guides Q&A - Current U.S. regulator guidance on endorsements, disclosures, and reviews.
- FTC: Consumer Reviews and Testimonials Rule Q&A - Current U.S. rule guidance on deceptive review practices.