Skip to main content
RankRift
Menu
Tools & Platforms

Twelve Questions To Ask Any AI-Visibility Vendor About Their Methodology

Every demo looks the same. These questions don't — including the ones we ask ourselves.

vendorsmethodologybuying

You have three demos booked this month, and all three will show you a line trending upward, a share-of-voice chart, and a table with your competitors in it. Nothing in that thirty minutes tells you whether those numbers are measurements or decoration, and the sales engineer is not going to raise the question for you.

These twelve questions do. They are not scored, and there is no passing grade. They are designed so that the shape of the answer tells you what you need to know: a vendor who has thought carefully about measuring a non-deterministic system answers them quickly and in writing, and a vendor who has not will change the subject to features.

Our conflict, stated first

Scope note. This article is general educational and measurement guidance, not legal, tax, financial, security, or platform-specific advice. Current law, first-party platform rules, the facts of the real case, and qualified professional advice control any consequential decision. RankRift publishes vendor-neutral evaluation criteria for AI-visibility trackers and does not claim an owned product. Those two facts pull in opposite directions, and pretending otherwise would be the first thing you should hold against us. Our standing policy is on our disclosure policy: we never rank ourselves first in our own comparisons, we always say who a competitor serves better, and we do not take affiliate revenue from any tool in this category.

Read this list adversarially. The test of evaluation criteria is whether they expose weak answers instead of rewarding a preferred vendor. Apply all twelve questions to every shortlisted product; if a question appears engineered so that only one product could pass it, treat that as a defect in the list.

Questions 1–4: Data collection and provider terms

These four decide how much of everything else is even meaningful. A metric definition sitting on top of an unstable collection method inherits that instability.

1. Which models and versions do you query, and how is the version recorded at run time?

Model behaviour changes when the underlying model changes. If a vendor cannot tell you which version produced a given data point, they cannot tell you whether last quarter's decline was your visibility moving or the model moving. A good answer names the surfaces covered, states which specific model identifiers are requested, and confirms the identifier is stored with every response. A weak answer says "we cover ChatGPT" and stops there, because "ChatGPT" is a product with several models behind it, not a model.

2. How are the answers obtained: official provider APIs, a documented interface, or the consumer web product?

This single fact explains most of the discrepancy when two vendors give you two different numbers for the same brand. We wrote the long version in API-based measurement vs UI scraping. For now, the question is simply whether the vendor will tell you, plainly, in writing. A vendor who describes their collection method as "proprietary" is telling you they would rather you not evaluate it.

3. What does a single run represent: which region, which language, which account state?

An answer generated for a logged-out request from one region is not the answer your buyer in another region sees while logged into a paid account with memory turned on. Neither is wrong. Both are specific. Ask which specificity you are buying, and ask whether you can change it.

4. What happens to your numbers if a provider changes or withdraws the access you rely on?

This is a continuity question, not a legal one. Any collection method depends on access that a provider controls and can alter. The answer you want describes what the vendor would do to your historical time series if that happened: annotate the break, re-baseline, or quietly carry on as if nothing changed. The third option is the one that damages you, because it damages you invisibly.

Questions 5–8: Sampling, metrics, and definitions

An AI answer is not stable. Ask the same question twice and you can get two different brand lists. Everything in this group exists because of that.

5. How many times is each prompt run, and how was that number chosen?

The number matters less than the reasoning. "Three runs per prompt per week, because that is what our infrastructure costs allow" is an honest answer you can work with. "Enough to be statistically significant" is not an answer. If you want the underlying logic, how many runs you need walks through how sample size relates to the size of change you can actually detect.

6. What variance do you observe, and does the interface show it?

A dashboard that reports a single number with no spread is asking you to treat noise as signal. Ask to see the run-to-run variation behind one of the numbers in the demo. A vendor who has the data will show you. A vendor who has to go and ask engineering has told you the answer already.

7. How was the prompt set built, can I see all of it, and can I change it?

Prompt sets are where measurement quietly becomes marketing. A prompt set that leans on your brand name will show you flattering mention rates that mean nothing about buyers who have never heard of you. Ask for the full list, not a sample. Ask who approves additions. Ask whether prompts your team adds are pooled with other customers' prompts.

8. Give me the formula behind every metric on this screen, especially any composite score.

Composite scores are the most common place for unpublished weighting to hide. If a headline number combines mention rate, citation share, and sentiment, the weights are an editorial judgement someone made, and you are reporting that judgement to your executives as if it were a measurement. Ask for the arithmetic. If the reply is that the scoring model is proprietary, you have your answer, and it is a bad one.

Questions 9–12: Retention, exports, and change management

The first eight questions are about whether the numbers are real. These four are about what happens after you sign.

9. Where do my prompts and my results live, how long are they kept, and who else can see them?

Your prompt set is a document that describes how your buyers think about your category, and for agencies it carries client names. Ask about retention periods, sub-processors, whether your data trains anything, and what a support engineer can see. What your data leaves behind covers the procurement version of this in detail. If a vendor cannot answer this from a written policy, they have not written one.

10. Can I export everything, including raw model responses, and what happens to it if I cancel?

Exports of aggregate charts are not exports. The valuable artefact is the raw response text with its timestamp, model identifier, and prompt, because that is the only form your data survives a vendor change in. Ask about format, ask about API access to your own history, and ask what the deletion timeline looks like after cancellation.

11. Do you publish a dated changelog of methodology changes, and what happens to historical numbers when the method changes?

Methods should change. Models change, surfaces appear and disappear, and better sampling designs get discovered. What separates a measurement company from a dashboard company is whether that change is announced, dated, and versioned, and whether prior data is recomputed or clearly annotated as pre-change. Silent method changes make your quarter-over-quarter comparison meaningless without telling you.

12. What does your tool not measure, and which kind of customer should not buy it?

Ask this last, and let the silence sit. Every instrument has a domain outside of which it says nothing useful. A vendor who cannot name a customer they are wrong for either has not thought about it or has decided not to tell you.

The answers that should end the conversation

Some responses are not weak answers. They are the answer.

What you hearWhat it actually means
"Our methodology is proprietary."You cannot audit the number, and neither can they, in any way you could check.
"Our AI determines the score."The weighting is unpublished and probably unstable.
"It's statistically significant." (no sample size)Nobody has computed a detectable effect size.
"We cover all the major AI platforms." (no model versions)Coverage is a marketing claim, not a measurement parameter.
"We can improve your placement in ChatGPT."Nobody can control a model's output. Treat this as disqualifying.
"We can't share the full prompt set."The prompt set was built to produce a demo, or it is shared across customers.
"No customer is a bad fit for us."They will happily sell you something you do not need.

An unpublished method is itself the answer. This is worth stating on its own, because it is the point of the whole list. When a vendor declines to publish how a number is produced, you are not left uncertain about the number. You are certain about one thing: it cannot be reproduced, disputed, or defended, by you or by anyone else. That makes it unusable for its main job, which is defending a decision in a room full of people who are entitled to ask how you know. A method you cannot inspect is not a trade secret you are respecting. It is a claim you are agreeing to repeat.

Use the twelve questions as a scorecard

RankRift does not operate or sell a tracker, so there is no owned product to score. Put every candidate in a table, record the vendor's written answer and evidence for each question, and mark an answer "unknown" when it cannot be verified. A spreadsheet is a candidate too: score its coverage, repeatability, retention, and labour honestly rather than treating it as free.

A vendor with stronger, documented answers should beat one with a better demo. If several answers remain unknown after a sales call, pause the purchase rather than filling the gaps with assumptions. When you are ready to compare product categories, platforms compared applies the same criteria.

In Plain English

Every AI-visibility demo looks the same because demos are designed to. What distinguishes vendors is not the interface, it is whether they will state in writing which models they query, how they get the answers, how many times they ask, what the formulas are, what happens to data, and who they are a bad fit for. Ask these twelve questions before the call so the answers can be compared. If a vendor will not put the method in writing, you have learned the most important thing about the number. Apply the same list to any owned implementation.

See the evaluation criteria on the tools hub and hold every vendor, including this one, to them.

Primary-source verification

Checked July 28, 2026. The operational and factual guidance above was reviewed against the primary references below. Provider features, prices, policies, interfaces, jurisdictions, and account-specific results can change; recheck the controlling first-party document and your own account or measured data before acting.