How Often Should You Re-Measure AI Visibility?

Run your frozen prompt set every week, but only compare four-week windows, and only call a change real when it is bigger than the noise band your own panel produces. For a store tracking ten buyer questions on four engines, where about a third of the answers flip between runs, a week-over-week swing of 10 points can be pure chance. The same swing between two four-week windows usually is not.

Why do weekly AI visibility reports mislead small brands?

Weekly reports mislead because each reading is one draw from an engine that answers the same question differently on every run. On a small prompt set, one draw per question is not enough to separate a real gain from a lucky week, and most dashboards compare exactly that: this week's single draw against last week's.

SparkToro and Gumshoe had hundreds of volunteers run a dozen prompts through ChatGPT, Claude and Google's AI thousands of times, and found the lists of recommended brands almost never repeated. The same study found something steadier underneath: in one category, a single agency appeared in 85 out of the 95 responses Google's AI gave. Rand Fishkin, who started out expecting tracking to be pointless, ended up calling visibility a reasonable metric when it is measured "across dozens to hundreds of prompts run multiple times". The study is not peer reviewed, and Gumshoe sells tracking software.

A ten-question panel sits at the small end of that range, which is why the number of runs behind each comparison decides what you can conclude. As our GEO claims audit puts it: "A single observation is not a state."

Collecting and comparing are two different jobs

How often you run your prompts and which periods you compare are separate decisions. Run weekly, because four readings a month give each prompt four draws to average. Compare four-week windows, because the average of four draws moves far less by chance than any single one, and a weekly dashboard invites you to read every single draw.

Most cadence advice names one rhythm. Averi tells readers to investigate trends over several weeks rather than single weeks, and Tenten asks for "two to three consecutive readings" before calling a signal, so both hint at comparing several runs. Neither says how many runs a comparison needs.

Our guide to getting recommended by ChatGPT already tells founders to write ten real buyer questions, run them through four engines, log everything and "Repeat weekly." The same guide says "Six to eight weeks of runs is a baseline you can defend."

Our measurement protocol counts disagreement the same way. Each prompt runs three times per engine, the majority result is that prompt's score, and where the runs disagree, "the disagreement is recorded as volatility rather than averaged away." You will use that count of disagreeing pairs in the formula below.

The one number that sets your cadence: volatile pairs

Your noise band depends mostly on how many prompt-engine pairs are volatile, meaning their repeat runs disagree. A pair that names you on every run, or on none, adds no chance movement. A pair that names you only sometimes is a coin you flip each time you measure, and the band grows with those coins.

To find the count, run each question on each engine on three different days in your first week, for example Monday, Wednesday and Friday, and mark every pair whose three runs did not all agree. Back-to-back runs tend to agree, so spreading them out catches more of the flipping pairs. Then use this formula, with V for volatile pairs, k for runs per pair in each window and N for total pairs:

=1.96*SQRT(V/(2*k))/N

Multiply by 100 for percentage points. The result is the gap between two windows that chance alone exceeds only about one time in twenty when nothing has changed. A smaller difference is not evidence of a change.

Three runs will still miss some volatile pairs. A pair that names you one time in five shows three matching runs 52% of the time, so your first count will run low. After your first four-week window, re-count V as the pairs whose four weekly runs disagreed, and use whichever count is larger.

The formula rests on three simplifying assumptions. It treats every volatile pair as a 50/50 coin, every stable pair as fixed, and the runs inside a window as independent draws. The first makes the band wider than reality; the other two make it narrower. Treat it as the smallest gap worth investigating, and re-count V every window.

A tracking sheet needs one row per prompt-engine pair and one column per weekly run:

A worked example: one skincare store, 40 pairs

Take a skincare brand selling mineral sunscreen online. It tracks ten buyer questions, such as "What is the best mineral sunscreen for sensitive skin under €30? List brands and cite your sources," on ChatGPT, Perplexity, Gemini and Google AI Overviews: 40 prompt-engine pairs. In week one, 12 of the 40 pairs give mixed results across their three runs.

The table puts that panel, and two others with fewer and more volatile pairs, through the formula. "Weekly" compares one run per pair against one run per pair. "Four-week" compares the average of four weekly runs per pair against the next four.

Say the brand is named in 12 of its 40 pairs, a 30% mention rate. With 12 volatile pairs, that rate going to 40% in one week, 12 mentions to 16, sits inside the 12.0-point band and supports no conclusion. If the four-week average then moves from 30% to 38%, an average of 15.2 mentions per weekly run, that is outside the 6.0-point band, and it is worth asking what changed. Averaging four runs per pair halves the band, because the band shrinks with the square root of the runs.

Then read the pairs, not just the rate. A real gain shows up as pairs that were stable at "absent" becoming stable at "present". If the rate rose only because volatile pairs landed heads more often, you have not moved anything you can point to.

What weekly collection costs by hand

Weekly runs of 40 pairs are 160 captures in a four-week window. LLM Pulse plans for "two to three minutes per prompt-engine combination", which puts the month at 5.3 hours to 8 hours of logging answers. The first month costs more: the two extra volatility runs in week one add 80 extra captures, another 2.7 to 4 more hours.

A tracking tool can do the collecting instead, as long as the prompt set stays frozen and you apply the same band to its numbers. The formula does not care who runs the prompts. Daily collection is the wrong trade at this scale: LLM Pulse's own help page notes that daily tracking generates about seven times as many executions as weekly. In the formula, more pairs and more runs per pair narrow the band equally, but runs on neighbouring days tend to agree, so on a small panel the budget usually buys more as extra questions than as extra days.

What the pages ranking for this question recommend

On 12 October 2026 we read the eight pages a US English search returned for this question and noted each one's cadence and whether it says what size of change is real. Only Averi gives a numeric noise threshold. Only Meikai tested anything, and found one run per day enough for its pooled 500-prompt panel.

"No" means we found no numeric threshold for a real change anywhere in the page text. RankAI comes closest to linking noise to panel size, warning that with fewer than 20 buyer-intent prompts the noise floor swamps the signal, but it gives no threshold. GrowthRocks' answer, "Monthly for most companies.", is the closest to what the arithmetic supports for a small panel, though it does not show the arithmetic.

Averi's rule, that "Weekly shifts within 4-5 points are noise.", shows why a fixed threshold misleads. On the 40-pair panel above with 12 volatile pairs, a weekly comparison needs 12.0 points before it means anything, while the same 4–5 points could be a meaningful move on a panel of several hundred pairs. Work out your threshold from your own panel with the formula, not from someone else's rule of thumb.

Why vendor cadence advice does not transfer to your panel

Vendors measure on panels far larger than a founder can run by hand, so a cadence that works for them can be noise for you. The size of the panel, not the calendar, decides how small a change you can detect, and most cadence advice never says which panel size it assumes.

Meikai's test is the clearest case, and the only page in the set that publishes its method and its limits. It ran a fixed portfolio of 500 prompts across five platforms for ten days. At one run per day, 95% of its ten-day citation-share estimates landed within 0.17 percentage points of the full estimate. One run per day is 25,000 responses per ten-day arm. A founder's four-week window of weekly runs on 40 pairs is 160 captures, so the vendor's estimate rests on 156 times as many draws, and it measures citation share pooled across the portfolio, not one brand's mention rate. Meikai says itself that the results apply to its 500-prompt portfolio and that smaller portfolios may need more repetition.

The same arithmetic applies to the panels on our pricing page. The Core retainer re-tests 38 buyer queries monthly on 4 engines at 1 run each, 152 logged captures a month; Authority runs 3 per query, 456 captures a month. If 45 of Core's 152 pairs were volatile, close to the example's share, a month-over-month move under about 6.1 points would be within chance. That band is close to the small store's four-week band because the panel is nearly four times larger, not because monthly is a better cadence. With three runs per pair, the Authority band would be about 3.5 points if the runs behaved as independent draws, which runs taken together only partly do. Core runs each query once a month, so its volatile pairs can only be counted by comparing months; Authority's three runs let them be counted inside each month. The baseline audit protocol, not the retainer page, is where DeviLabs states that "Volatility is reported alongside every rate."

When to break the cadence

Check outside the schedule when something happens, such as a fix, a launch or an engine change, not when a number moves. Each of those calls for a targeted check of the prompts it touches. A rate that jumped in one week calls for nothing except letting the four-week window close.

• A factual error fixed. If an engine stated the wrong price or the wrong founder and you corrected the source, re-check those specific prompts the following week. That is a yes-or-no question, not a rate, and it does not need a window.
• An engine or product change. When an engine changes its interface or product mode, close the current window and start a new one. Comparing across the change mixes two systems.
• The baseline ages. In Google's AI Overviews, Authoritas found in US data from late 2024 that over two to three months you could expect 70% of the pages cited there to change. Other engines were not measured. Assume similar drift: at least once a quarter, repeat the three-day volatility count, re-count V, and treat the latest four-week window as your baseline instead of one from months ago.

What to do this week

Run your ten buyer questions on each engine on three different days this week and count the pairs whose runs disagree. Put that count into the formula and write the band at the top of your tracking sheet. Then run once a week, close the first four-week window, and judge the second one only against the band.

If you would rather have the panel and the error ranges built for you, that is what the AI Visibility Audit on our pricing page is for, and our guide to measuring AI visibility explains the error ranges behind it.