Sample-size and confidence calculator

Work out how many runs per prompt you need to detect a change of a given size at a given confidence, and what that costs in API calls, before you spend a budget on a measurement that could never have proved anything.

Read the sampling method · Learn the measurement basics first

The calculator shows its assumptions and its formula on screen. If a tool will not tell you what it assumed, do not trust the number it gives you.

Size your runs

The calculator on this page takes a baseline appearance rate, the change you want to detect, a confidence level of 90, 95, or 99 percent, the number of prompts, models, and measurement periods per year, and optionally the runs per prompt you can afford. Power is fixed at 80 percent. A warning appears when a target needs more than 1,000 runs per prompt per period.

Formula: n = [z(confidence) × √(2·p̄(1−p̄)) + z(power) × √(p1(1−p1) + p2(1−p2))]² ÷ (p2 − p1)², where p1 is the baseline rate, p2 is the baseline plus the change, p̄ is their average, and power is fixed at 80%.

Assumptions: Assumes independent runs, an unchanged model, a fixed prompt set, a yes-or-no appearance outcome per answer, and a comparison between two measurement periods of equal size. No other adjustment is applied.

Worked example: a 30 percent baseline, a 10-point change, and 95 percent confidence need 356 runs per prompt in each period. With 20 prompts and 3 models that is 21,360 runs per period, or 256,320 API calls a year at a monthly cadence.

In short

This calculator answers one question: how many times do you need to run each prompt before a difference you observe means anything. You give it the change you want to be able to detect, the confidence you want in that detection, and roughly how often the brand currently appears. It returns the runs required per prompt, the total runs across your prompt set and model roster, and the resulting number of API calls so you can price the exercise before committing. It is client-side, it shows the assumptions it made, and it will tell you when your target is unaffordable rather than quietly rounding it down.

Who this is for

In-house SEO leads scoping a measurement programme before asking for budget
Agency strategists pricing recurring client measurement across several brands and categories
Analytics and RevOps people checking whether a proposed test could ever have detected the effect it claims to look for
Anyone about to buy a tracking tool who wants to know what sample volume the plan actually implies
Anyone who ran a prompt three times, got three answers, and wants to know how many times is enough

When to use this

Before a first measurement run, so you size it rather than guess it
Before promising leadership you can detect a specific improvement by a specific date
When a figure moved and you want to know whether your sample could have resolved a move that size
When comparing measurement plans or tool tiers on cost per meaningful result

Why sample size matters here more than in normal reporting

Most marketing metrics are counts of things that already happened. Sessions, signups, orders. You do not sample them, you total them. AI visibility is not like that. An assistant's answer is generated fresh each time, and the same question asked twice can name a different set of brands. So what you have is not a total, it is a sample drawn from a process, and every figure you compute from it is an estimate that would come out somewhat differently if you ran it again.

That has a blunt consequence. If you run each prompt a handful of times, the natural wobble in your estimate can easily be larger than the change you are trying to detect. You will see movement every month, and none of it will mean anything. The number of runs is what sets the size of change your programme is physically capable of noticing.

The uncomfortable version: if you have not sized your runs, you do not know whether your report is measuring your work or measuring noise, and neither does anyone reading it.

What the calculator computes

It works in both directions, because the two questions people actually have are different sides of the same arithmetic.

Forwards: given the change you want to detect and the confidence you want, how many runs per prompt does that require, and what does the whole programme cost in API calls once you multiply by your prompt count, your model roster, and your reporting cadence.

Backwards: given the run budget you already have, what is the smallest change that budget could reliably detect. This is the more useful direction for most people, and often the more sobering one.

Runs required per prompt for your target detectable change and confidence level
Total runs across your prompt set, model roster, and cadence
Resulting API call volume, so cost can be estimated against your providers' own current rates
The smallest change your existing budget could detect, if you work backwards
A clear warning when the target you entered is not achievable at any sensible cost

The inputs, and how to choose them

Every input is a judgement call, and changing one at a time in the calculator shows how sensitive the answer is to it. Two of them dominate the result.

The detectable change is the big one. Halving the change you want to detect increases the runs required far more than proportionally, which is why "we want to catch any movement" is not a specification and cannot be costed. Pick a change size that would actually alter a decision, and size for that.

Baseline appearance rate: roughly how often the brand is currently named. A first small run or the manual snapshot is enough to estimate this.
Detectable change: the smallest difference you need to be able to see. Choose the size that would change what you do.
Confidence level: how sure you want to be that an observed difference is not just run-to-run variation.
Prompts in your set: the number of distinct prompts you will run, before phrasing variants.
Models in your roster: each assistant you query multiplies the run count.
Cadence: how often you repeat the programme, which turns a one-off number into a recurring cost.

Illustrative only: if you enter a low baseline appearance rate and ask to detect a very small change at high confidence, expect the tool to return a run count large enough that you will want to reconsider the target. That behaviour is the tool working.

How to read the output

The headline figure is runs per prompt. Multiply it out and you get the programme's true size, which is usually larger than people expect because the model roster and the cadence both multiply rather than add.

If the number is affordable, size for it and freeze your prompt set before you start. If it is not, you have three honest options and one dishonest one. The honest ones: accept a larger detectable change, cut the prompt set or model roster to concentrate your runs, or lengthen the cadence so each report covers more accumulated data. The dishonest one is running the small sample anyway and reporting the point estimates as though they were solid.

One thing the output does not do is tell you a result is real. It tells you what your design is capable of resolving. A well-sized run can still produce an ambiguous answer, and that answer is still worth reporting as ambiguous.

The assumptions, shown on screen

The calculator prints its assumptions next to the result rather than burying them, because a sample-size figure without its assumptions is false precision. If any of the following does not hold for your situation, treat the output as an approximation and widen your own expectations accordingly.

We would rather you distrust the output for the right reason than trust it for the wrong one.

It treats each run as an independent draw. Caching, session reuse, or a provider serving a stable response will violate this and make your effective sample smaller than your run count.
It assumes the underlying model does not change mid-programme. A version update during your window is a confounder no sample size fixes.
It treats appearance as a binary outcome per answer. Framing and position are richer signals that need their own handling.
It assumes your prompt set is fixed for the duration. Editing prompts mid-programme breaks the comparison the sample size was calculated for.
It applies no hidden adjustment factors. The formula in use is shown alongside the result so you can check the arithmetic yourself.
Cost is expressed in API call volume, not currency. Rates change and vary by provider, so price it against your own providers' current published rates.

When this is the wrong instrument

A sample-size calculator answers a narrow question well and several adjacent questions not at all. If your real problem is on this list, sizing runs will not help you.

Sizing is also the wrong first move when your prompt set is not settled. A perfectly powered run against prompts your buyers would never ask is a precise measurement of the wrong thing.

This page claims no specialist review. Before using the result for a consequential decision, record the formula, assumptions, method version, reviewer identity, review scope, and review date; otherwise keep the interpretation within the stated assumptions.

Frequently asked questions

Why can't you just tell me a good default number of runs?

Because the right number depends almost entirely on two things this page cannot know: how often your brand currently appears, and how small a change you need to notice. A brand named in most answers and a brand named in a handful need very different run counts to resolve the same sized shift. Any universal rule of thumb would be over-sized for some readers and useless for others, so the tool asks instead of guessing.

Does running the same prompt many times just give me the same answer repeatedly?

Not usually, and where it does, that is itself worth knowing. Assistant output is generated rather than retrieved from a fixed list, so repeats vary. But caching, sticky sessions, or a provider serving a stable response will reduce the real variation, which makes your effective sample smaller than your run count. The calculator assumes independent draws, so it flags this as an assumption to check on your own setup.

How do I estimate the baseline appearance rate before I have measured anything?

Run a small exploratory batch, or the manual snapshot, and take the resulting rate as a rough input. It does not need to be accurate, it needs to be in the right region, because the run count is not very sensitive to small errors in the baseline. If you are genuinely unsure, enter a range of plausible values and size for the one that demands the most runs.

The calculator says I need more runs than I can afford. Now what?

Three legitimate moves. Widen the change you are trying to detect, because catching a large shift is far cheaper than catching a small one. Narrow the programme, concentrating runs on fewer prompts or fewer assistants. Or lengthen the cadence so each report pools more accumulated data. What you should not do is run the small sample and present the point estimates as though the sample supported them.

Do I need to size each assistant separately?

For a per-assistant figure, yes, because each model is its own sampling problem and they do not share variance. Each assistant you add multiplies your run count rather than splitting it. If your budget only supports a pooled cross-model figure, that is a defensible choice, but say so in the reporting and do not present per-model breakdowns your sample cannot support.

Why is the cost shown as API calls rather than money?

Because provider rates change, differ by model, and depend on how much text each call produces, and quoting a currency figure here would be a number we could not stand behind next month. Call volume is the part that follows from your inputs and does not go stale. Take the volume, apply your own providers' current published rates, and you have a price you can actually defend internally.

Does a properly sized run mean my result is correct?

It means your design could resolve the change you targeted. It does not mean the measurement is right. A well-powered run against a flattering prompt set, or across a model roster that does not match where your buyers are, produces a precise answer to the wrong question. Sizing is the last step of designing a measurement, not the first, and prompt-set quality matters more than run count.

Size it before you spend on it

Most disappointing measurement programmes were under-powered from the day they were scoped. Fix the prompt set, size the runs, then start.

Build a prompt set worth sizing · Read the sampling methodology