How to Measure AI Visibility Without Fooling Yourself

Measuring AI visibility means running a fixed set of buyer questions through each engine several times, logging every answer, and reporting rates with their denominators and an error range. Without the error range, the number cannot tell progress from noise. Your brand showing up in five of twelve answers this month against three last month looks like a big jump, and a significance test cannot tell it apart from noise.

What a usable AI visibility number contains

Four things: a rate, the denominator it was computed over, an error range around it, and the date window it covers. Drop any one of them and the number can no longer answer the question you are paying for, which is whether anything changed since last month. The rest of this post shows how to produce all four.

Our published definition of a baseline holds it to four conditions: fixed and recorded prompt wording, multiple runs per prompt, the engine and product mode and location and timestamp recorded for every run, and the denominator stated explicitly. Lose one and next month's comparison loses most of its value.

Pew Research Center tracked 68,879 Google searches from 900 US adults and found people clicked a traditional result in 8% of visits where an AI summary appeared, against 15% where none did. What being named inside that AI answer is worth to you is covered below.

How wide your error bar actually is

Run twelve prompts once each and see your brand in three answers, and your mention rate is somewhere between 9% and 53%. Calling it 25% is the kind of single figure one study calls misleadingly precise. The table below shows how the range shrinks as the sample grows, and how differently it behaves at low rates.

The bounds come from a Wilson score interval, the standard way to bound a proportion measured on a small sample. We computed every row for this article. The last two rows are there because your rate will rarely be one in four.

Look at the low-rate row: the top of its interval is more than four times the bottom, so a low rate that doubles can still be noise.

One assumption sits under every row: that each run is independent. Three runs of the same prompt on the same engine usually agree, so they carry less information than three separate prompts, and an interval over pooled runs is a best case. For a range you can defend, score each prompt once per engine by its majority result and use prompt-engine pairs as the total. Ten prompts across four engines is 40 pairs, which is the 40 row above, and it is a lot wider than the 120 row.

Run those twelve prompts again a month later, see your brand in five answers instead of three, and the rate reads 41.7% against 25%. Your agency puts a green arrow next to it. A two-proportion test returns p = 0.39, and Fisher's exact test, the better choice at samples this small, returns p = 0.67. Either way, that is what measuring the same thing twice looks like. At a hundred independent observations a comparable move, 25% to 40%, returns p = 0.02 and is worth acting on. Call a change real only when p is below 0.05. We report volatility beside every rate for the same reason, and our claims audit marks "AI citations are stable once earned" as false.

A volatility number to be wary of

Visiblie defines volatility as the share of prompts whose answer changes meaningfully across runs, which is worth measuring. Its worked example instead reports a brand appearing in seven of ten runs of one prompt as 30% volatility. For a yes-or-no outcome that minority share is always the smaller of the rate and its complement, so it restates the mention rate and adds nothing. Use the definition, ignore the example, and remember that what you actually need is two rates, two intervals and the test above. Our rule is that a 40% mention rate with 5% volatility and a 40% mention rate with 30% volatility are different findings, and only one of them supports a decision.

How many runs do you actually need?

Enough that the interval is narrower than the change you hope to detect, and that differs by engine and topic. A 2026 preprint sampling Perplexity, SearchGPT and Gemini repeatedly found Gemini reaching an interval five points wide on citation share at roughly 40 to 50 queries for two of its three test topics, and Perplexity at about a hundred.

SearchGPT is the instructive case. The paper puts it at 150 or more queries, and warns that interval width can narrow and then widen again as queries are added, because the citation distribution shifts across the sequence instead of settling. Its advice is to fix the sample size in advance from earlier measurements of that platform and topic, and never to stop because the interval happens to look narrow. A second 2026 preprint makes the practical point: answers vary across runs, prompts and time, so visibility has to be characterized through repeated measurements as a distribution. Both are preprints, tested on a handful of product topics. They are still better evidence than the vendor blogs covered below.

Do not borrow the paper's 40 to 50 for your own mention rate. It measures citation share, the split of all citations across every domain an engine cites, which is a different quantity from how often your brand is named. Its queries also allowed repeats.

Then the question a founder actually asks: how many prompts before I can prove the work did something? To detect a move from a one-in-four to a two-in-five mention rate at conventional statistical power (80 in 100 chance of catching a real change) you need about 152 independent observations in each window. Counted as prompt-engine pairs, that is at least 38 prompts across four engines if you pool the engines, and about 152 prompts to prove it on one engine alone. Treat both as floors. A prompt that one engine answers in your favor may well do so on another, so the independence assumption is optimistic across engines too, and 50 or more prompts is the safer plan for a pooled rate. Per engine, 38 prompts is thin: the 38 row in the table above is 27 points wide, so only a large per-engine move in a set that size counts as proof.

What the pages ranking for this query leave out

On 28 September 2026 we scored the seven readable pages a US English search returned for this query against six checks fixed before reading. Three publish a formula, two give a repeat-run count and one asks for an error range. None shows how to compute it or says to measure logged out.

Positions 3 and 5 are missing because we could not read them: one served a bot challenge, one returned a 403. One search on one day in one locale is not a reproducible ranking, so treat the positions as an anecdote. The columns are the finding, and every page is linked in the sources so you can check each cell.

On the last column: Visiblie notes that personalization changes outputs, but none of the seven tells the reader to log out. A measurement taken inside your own logged-in account is partly a measurement of your account. We capture from fresh, logged-out sessions with the product mode named exactly, and from consumer interfaces instead of APIs, because the consumer interface is what a buyer sees.

AuthorityTech publishes a weighted composite with a denominator under every component, an evidence floor of ten observations across seven distinct run dates, and an instruction not to hide volatility inside the aggregate. It is the most rigorous page in the set, and it still asks for a confidence interval without showing how to produce one.

Presence and accuracy fail differently

A mention-rate chart records only one of the two failures. An engine that names your brand and gets your price wrong has produced a mention, which every presence metric scores as a success. The customer who reads that answer and arrives expecting a different number is counted nowhere.

We run this protocol on our own brands first. The field notes on LuckyPaw, our dog walking, daycare and boarding business in Wassenaar, come from a short pass: twelve prompts, four engines, a single run per prompt. By the standard of this post that makes them observations, and they should be read that way.

For a store, the same failures look like a price from last season, a discontinued variant recommended as current, or a shipping region you do not serve. We have not published store data of this kind yet, so the scoring format is what we can give you. Each error gets one row: the engine and product mode, the exact prompt, the claim as the engine stated it, the correct fact and the URL where you publish it, and the date.

Every prompt in our protocol is scored against five binary criteria fixed before measurement begins. The fifth, not confused with another entity or group company or country, is the one that fails most often in our client work. AuthorityTech shows it can be measured: entity resolution is the largest component of its composite, a quarter of the score on weights it calls an operating proposal and not a calibrated model. None of the seven pages above, AuthorityTech included, describes the check scored per run against a fixed written rubric, which is the form that tells you which answers to go and fix.

What a mention rate is worth, honestly

Nobody has shown that gaining AI visibility causes revenue, us included: every dataset we have audited linking the two is observational. Our claims audit marks the claim that AI-referred traffic converts better than organic as unsupported, because AI users arrive later in their decision process and the AI did not cause that.

That is a selection effect. It is why we publish 24 claims with verdicts and no conversion multiplier. An agency putting a euro figure on a mention-rate gain is doing something nobody has yet shown how to do.

What you can get is a directional read on whether the channel exists for you, and the cheapest instrument is a one-question checkout survey asking how customers found you. Analytics only attributes the visits that arrive with a referrer; the survey also catches people who read an AI answer and then typed your name. One founder we wrote about was sure AI search was not a channel for her brand: GA4 showed eleven ChatGPT sessions in ninety days, while the checkout survey on the same store said one in nine customers had found them through an AI assistant. Run that before commissioning a baseline. If almost nobody is finding you through an assistant, the measurement can wait and the hours belong in creative.

Five things a vendor has to state before their number means anything

Ask for these five things before accepting any AI visibility report, whether it comes from an agency, a paid tracking tool or your own spreadsheet. A vendor who will not supply them is asking you to trust a number you cannot audit.

1. The prompt set, verbatim. Every prompt, frozen and dated, with identity prompts kept apart from commercial ones. Average "does the engine know you exist" together with "does it recommend you" and the result describes neither.
2. The denominator for every rate. Mention rate over valid runs is a different quantity from mention rate over runs that named any brand. Both are defensible. Only one is comparable to last month.
3. Runs per prompt and the date window. In our audit each prompt runs three times per engine, the majority becomes that prompt's result, and any disagreement is logged. Whoever runs yours, ask what the interval counts: if repeat runs are counted as independent, the range is a best case. A refusal scores zero, because for the buyer it is the same as absence.
4. Session state and product mode. Logged out, no history, location and language fixed and stated. ChatGPT Search is a different surface from the ChatGPT API, and Google AI Mode is a different surface from AI Overviews. A report that does not name the surface has not told you what it measured.
5. The error range, and the errors. An interval on every rate, plus the register of factual mistakes. A report with neither can track your own movement roughly and cannot support any claim about a competitor.

The cheap version, and what it can and cannot do

Ten questions your best customer would ask before buying, phrased the way people type them. Four engines. Three runs each from a logged-out session with location and language fixed. That is 120 runs, one row and one screenshot each. At two to four minutes per run to capture and log, plan on four to eight hours.

For a store, spread the ten questions across the intent types our audit uses. A store selling merino base layers might write:

1. Category: "best merino base layer for hiking"
2. Category: "warmest base layer that does not smell after a week"
3. Category: "where to buy merino base layers in Europe"
4. Alternatives: "[competitor brand] alternatives"
5. Alternatives: "cheaper alternative to merino for a base layer"
6. Comparison: "[your brand] vs [competitor brand]"
7. Comparison: "merino vs synthetic base layer for skiing"
8. Problem: "I sweat and then freeze on winter hikes, what should I wear"
9. Brand: "is [your brand] any good"
10. Brand: "[your brand] shipping and returns"

The first eight tell you whether you get recommended. The brand prompts tell you whether the engine gets your facts right. When a category answer names a marketplace or a review site instead of a brand, log that as the competitor.

Be clear about what the cheap version buys. Scored as 40 prompt-engine pairs, a one-in-four rate carries an interval about 26 points wide, and proving a fifteen-point move needs several times as many prompts per window as this set has. It finds factual errors and establishes a rough level. It cannot prove a trend.

Log seven fields per run: the exact prompt, the engine and product mode, the timestamp, whether your brand was named, whether your domain appeared in the sources, whether what the engine said was correct, and every competitor named instead. Keep the full response text so it can be re-scored if your rubric changes. Collect weekly if that suits you, compare 30-day windows, and call a change only when the test above says it is real.

Fix what the accuracy column shows before trying to move the presence column: a wrong fact is cheap to correct and does more damage the more visible you become. Then put the five questions above to whoever produced your number, starting with yourself.