Asking Once Tells You Nothing: Your AI Visibility Score Is a Distribution

Ask ChatGPT for the best providers in your category, screenshot the answer, and you have learned almost nothing. SparkToro and Gumshoe.ai tested this at scale in late 2025: across nearly 3,000 runs of identical prompts, the odds that any two responses contained the same brand list were under 1 in 100. An AI visibility score built on a single check is a coin flip dressed up as a metric.

The honest version of the number is a rate. Ask the same question 50 or 100 times, count how often your brand appears, and read the percentage the way a pollster reads a sample. Measured that way, the figure holds up. The same research that demolished one-shot checks found certain brands appearing in a steady 60% to 90% of responses for a given intent, run after run.

Search demand for the metric is small but climbing: Semrush’s US database shows “ai visibility score” at 390 searches a month as of August 2026, its 12-month peak, carrying an $11.31 cost per click. People want a number. This piece settles what the honest number looks like, how many runs it takes to earn one, and which claims a screenshot can never support.

What is an AI visibility score?

An AI visibility score is the percentage of times your brand appears when an AI engine is asked a relevant question repeatedly and independently. It is a rate across many runs, not a position in one answer. The single number on a vendor dashboard is a compression of a distribution that spreads across prompts, engines, and question types.

The distinction matters because the two readings support different decisions. A rate tells you whether AI engines associate your brand with an intent often enough to influence buyers. A one-shot reading tells you what one sample from a probability distribution happened to contain at 2:14pm on a Tuesday.

Appearing is also only half the game. A brand can be named in answers without ever being the cited source behind them, and the two are earned differently; we broke down that split in mentioned but not cited: the two scoreboards of AI visibility. This article is about the sampling problem underneath both scoreboards, and it is the statistical foundation of every AI audit we run.

Why does ChatGPT give a different answer every time you ask?

Because large language models are probability engines: each response is sampled fresh, so the brands named, their order, and even how many appear all change between runs. In the SparkToro and Gumshoe.ai data, two runs of the same prompt produced identically ordered lists roughly 1 time in 1,000.

The study behind those numbers, published in January 2026 by SparkToro CEO Rand Fishkin and Gumshoe.ai CTO Patrick O’Donnell, had 600 volunteers run 12 identical prompts through ChatGPT, Claude, and Google’s AI a combined 2,961 times during November and December 2025, with the analysis modeled on Carnegie Mellon’s LLM-consistency research. Response length alone swung from 2 or 3 recommendations to more than 10 for the same question.

This is not a defect the engines will patch. Variation is how generative models work.

Treating an AI answer like a Google ranking, a stable position you can hold and defend, imports an assumption from traditional search that the underlying system does not honor. Everything else in this article follows from that one difference.

Rankings persist; samples scatter.

How inconsistent are AI brand recommendations, really?

Predictably inconsistent, and the predictor is the type of question, not your industry. Conductor’s study, updated July 17, 2026 (vendor research: 14,000 API calls across 10 industries, 7 intent types, and 4 engines), found purchase prompts returned only 40% brand overlap between any two runs, while comparison prompts reached 63% with the same brand leading 91% of the time.

Brand overlap is a plain-language reading of pairwise Jaccard similarity: out of every 10 unique brands that appeared across any two runs of the same prompt, how many showed up in both. Conductor ran each prompt 50 times per engine, stateless, which yields 1,225 run-pair comparisons per cell. At 40%, the consideration set your buyer sees is mostly reshuffling underneath them at the exact moment they are ready to spend.

The gaps hide inside the averages, too. In the same dataset, 20% of pricing prompts returned no brands at all, and education prompts came back with zero brand names in 45% to 72% of responses depending on the engine. High agreement about who the authorities are, near silence about saying so.

Industry moves your baseline without changing the ranking: Conductor measured insurance at 62% overall overlap and cloud computing at 36%, even though both are concentrated markets dominated by a handful of names. The difference was the prompt mix each category attracts, not the number of competitors. If your category is asked about mainly through purchase and open-ended recommendation prompts, your visibility is structurally noisier than a category living on comparison questions, and no amount of content volume changes that.

How many times should you run a prompt to measure AI visibility?

At least 60 to 100 runs per prompt before averaging, based on the SparkToro and Gumshoe.ai research; Conductor’s study used 50 stateless runs per prompt per engine. Below that range, the difference between a brand appearing 40% of the time and one appearing 70% of the time is hard to distinguish from noise.

Independence is the part most tracking setups quietly get wrong. Each run has to be a fresh session with no conversation history, the way a new buyer would ask, because carried-over context contaminates the sample. Conductor’s runs were stateless for exactly this reason.

Fishkin is candid that the precise statistical floor is still an open research question; 60 to 100 is the working guidance from the people who gathered the data, not a law of nature. Directionally, though, more runs, more prompts, and per-engine separation all move you toward a number you can defend.

Two practical implications follow. First, cost scales with honesty: one prompt run 100 times across 4 engines is 400 calls, so a 25-prompt panel is 10,000 calls per refresh, which is why this is an API-budget line item rather than an intern with a browser. Second, the runs are wasted if the prompts are wrong. Building the prompt panel itself, which questions, in whose phrasing, across which intents, is its own discipline, and we covered it in AI prompt tracking: build the prompt set before you measure.

Which AI engine gives the most consistent answers?

ChatGPT, in Conductor’s July 2026 data: it returned the fewest brands per response, an average of 5, and led on knowledge-style prompts like comparison, education, and support. Gemini was the least consistent engine tested and averaged 9.2 brands per answer, while Perplexity was steadiest on transactional prompts such as pricing and purchase.

A 9-brand list that reshuffles every run is worth less than a stable slot on a 5-brand shortlist, even though the longer list feels like more exposure. And because no single engine led across all intent types, an aggregate score can show you healthy while you are close to invisible on the one engine your buyers actually use.

Track engines separately or the average will lie to you.

Personalization is now adding variance from a second direction: who is asking. At I/O on May 19, 2026, Google expanded Personal Intelligence in AI Mode to nearly 200 countries and territories across 98 languages with no subscription required, letting people connect Gmail and Google Photos, with Calendar to follow. Google also said AI Mode had passed 1 billion monthly users with queries more than doubling every quarter. The same prompt from two different people was already two different samples; increasingly it is two different questions.

What can a one-time AI visibility screenshot actually prove?

Only that the brand appeared, or failed to appear, in that single sample. In the SparkToro data, City of Hope hospital showed up in 69 of 71 ChatGPT responses to a cancer-care prompt, a 97% rate, yet it was the top mention in just 25 of them. A screenshot taken on one of the 2 missing runs would have called a 97% brand invisible.

Rank position is worse than incomplete; it is noise. The SparkToro report is blunt that ranking positions inside AI answers are so unstable they are effectively meaningless, and that any product selling movement in an “AI rank” is selling fiction. With an estimated $100 million or more already spent annually on AI visibility tracking (a Search Engine Land estimate cited in the January 2026 report), that is an expensive fiction to buy into.

So put the burden of proof on whoever hands you a score. A defensible AI visibility report states the run count per prompt, the full prompt list, the engine split, the intent segmentation, and the refresh cadence. If a number arrives without a run count, it is a screenshot with extra steps.

One honest edge case: tight categories stabilize faster. In small consideration sets like regional service providers or niche B2B tools, SparkToro found answers clustering around a few familiar names, and top headphone brands hit 90% to 100% visibility across runs while leading brand-design agencies managed only the 30s and 40s in a wide-open field. If you operate in a narrow space, fewer runs may already show you a stable picture; in a broad one, the sample size requirement climbs.

Doing nothing does not exempt you from the variance. With 68.01% of US Google searches ending without a click from January through April 2026 (SparkToro analysis of Similarweb clickstream data, published June 2026) and click-through rates dropping nearly 60% when an AI Overview appears, the AI answer increasingly is the customer touchpoint. Skipping measurement just means you never learn which half of the coin flips you are losing.