AI Prompt Tracking: Build the Prompt Set Before You Measure

AI prompt tracking only works if the prompt set is right, and the prompt set is the step almost every AI-visibility program skips. Teams buy a tool, accept whatever questions it generates, and start watching a percentage move up and down without ever asking the obvious thing: are these the questions our buyers actually ask? A visibility score built on the wrong prompts is not a weak signal.

It is a confident number about a market you do not serve.

This piece covers what a prompt set is, where the questions come from, how many you need, how many times each has to run before the number means anything, which prompt intents deserve priority, and how often the whole thing should be rebuilt. Every figure below is real, current and attributed. Nothing is modeled.

What is AI prompt tracking, and why does the prompt set come first?

AI prompt tracking is the practice of running a fixed list of buyer questions through AI assistants on a schedule and recording whether your brand appears in the answers and which sources get cited. The prompt set is that list. It comes first because it is the only part of the measurement you control. You cannot influence how the model generates an answer, but you decide entirely which questions you are graded on.

The gap this fills is real and widely admitted. Semrush’s 2026 AI Visibility Index, released June 26, 2026 and built on 126 million U.S. AI search prompts collected from January through April 2026 across ChatGPT, Gemini, Google AI Mode and Google AI Overviews, reports that 45% of marketing leaders cannot accurately measure their brand’s visibility within AI-generated answers, and only 9% have the tools to track all the relevant metrics across platforms. The market is not short on dashboards.

It is short on agreement about what should be on them.

It matters commercially because the traffic behind these answers has stopped being marginal. Adobe data cited in the same release found AI-referred traffic to U.S. retail sites up 1,324% between October 2024 and May 2026, with travel up 2,215%. Meanwhile SparkToro’s June 2026 analysis of Similarweb clickstream data found 68.01% of U.S. Google searches ended without a click in the first four months of 2026, up from 60.45% in 2024. Buyers are getting more answers and clicking less to get them. If you are not present in the answer, the visit that used to tell you so never happens. That is the same problem our guide to measuring AI search ROI when the click disappears deals with on the reporting side.

This piece deals with the input side.

Where do you find the questions buyers actually ask AI?

From your own sales conversations first, your search data second, and generated prompt suggestions last. The order matters, because the further you get from a real buyer’s words, the more the prompt set drifts toward what a tool finds convenient to generate.

There is hard evidence that prompts are far messier than keywords. In the SparkToro study, 142 volunteers were asked to write, in their own words, the prompt they would use to solve a single shared task: choosing headphones for a family member who travels. The average semantic similarity between any two of those prompts was 0.081. Rand Fishkin’s summary was that people do not reduce intent to two to five logical keywords the way they learned to with Google; they get creative, specific and strange. A keyword list is a poor proxy for that. A transcript of your last twenty sales calls is a much better one.

Four sources are worth mining before you write a single prompt. Sales and support recordings give you objections in the buyer’s own phrasing. Your site search box and chat logs show what people ask when nobody is watching. Search Console still shows real demand, especially the long, question-shaped queries that read like prompts already. And win-loss notes tell you which competitors buyers name unprompted: those names belong in the set too, because comparison prompts behave differently from open ones.

One caution about generated prompts. The same research found synthetic prompts, ones an AI writes for you, produced results broadly in line with human-written ones for the same intent, so they are not useless. They are a reasonable way to add volume once the core set is grounded. They are a bad way to decide what the core set is.

How many prompts does a prompt set need?

Enough to cover every intent your buyers move through, which in practice lands most B2B and considered-purchase categories in the 30 to 50 range. The number matters less than the coverage: a set of 60 prompts that are all variations of “best [category] software” measures one narrow moment repeatedly, while 30 prompts spread across research, comparison, pricing and purchase questions measures the actual journey.

Coverage is not a stylistic preference. Conductor’s brand recommendation study (14,000 API calls structured as 10 industries by seven intent types by four large language models by 50 runs each, last updated July 17, 2026) found that consistency varies enormously by intent type, and that the rank order of intents held across all four engines and all ten industries. If your prompt set only contains one kind of question, your visibility number inherits that intent’s consistency profile and you will never know it.

A workable structure is to allocate roughly by intent rather than by keyword. Six to ten unbranded category questions (“what’s the best way to handle X”), six to ten comparison prompts naming you and your two or three real competitors, six to ten problem-and-solution prompts phrased the way a frustrated buyer would phrase them, four to six pricing and cost questions, four to six purchase-ready prompts with real constraints in them (budget, team size, timeline), and a handful of educational questions where you expect no brand mention at all but want to know who owns the explanation. That last group is the one most teams cut, and it is diagnostic rather than decorative: more on that below.

How many times should you run each prompt before trusting the number?

Dozens of times, not once. The SparkToro and Gumshoe.ai research put a number on it: usually at least 60 to 100 runs of the same prompt before the pattern stabilises enough to average. A single manual check tells you nothing, and it is the check most teams actually perform.

The underlying instability is severe. In that study, 600 volunteers ran 12 prompts through ChatGPT, Claude and Google’s AI a combined 2,961 times over November and December 2025. The finding: there is less than a 1 in 100 chance that any two of a hundred responses return the same list of brands, and closer to 1 in 1,000 that two return the same list in the same order. Responses varied in three ways at once, which brands appeared, the order they appeared in, and how many were listed, sometimes two or three and sometimes ten or more.

Conductor arrived at the same conclusion by a different route, running each prompt 50 times per engine and measuring the proportional overlap between every possible pair of runs. Both studies point at the same operating rule: measure appearance frequency across many runs, never position in a single answer. What you are looking for is a share, this prompt returns our brand 62% of the time, and that share only exists once you have run the prompt enough times to compute it.

Which prompt intents should you prioritize?

Prioritize by how much the intent moves and how close it sits to revenue, which means purchase-intent prompts need the most runs and the most attention, not the fewest. Conductor’s data shows the type of question, not the industry, is what predicts whether an answer holds still.

Two numbers describe each intent. Brand overlap is the share of brands two runs of the same prompt have in common: 40% means that out of every ten distinct brands appearing across two runs, only four appeared in both. Lead brand stability is how often the same brand comes first. The two can diverge sharply, and where they diverge tells you what kind of position you actually hold.

Three practical readings come out of that curve. Purchase prompts are the least stable intent in the study (the brands recommended to someone ready to buy change on almost every ask) so they need the largest number of runs to produce a trustworthy share, and they are exactly where an unstable position costs money. Comparison prompts are the opposite: 63% overlap and 91% lead brand stability, meaning the engine picks a winner and keeps picking it. Holding that spot is durable; taking it from someone else is the hardest job in the set.

And educational prompts need a different metric entirely. Conductor found that depending on the engine, between 45% and 72% of education-intent prompts returned no brand name at all. Track those for mention volume and you will log a flat, stable line that is really just consistent silence. Track them for whose material the answer is built from instead, which is the distinction between being named and being the source, covered in our piece on brand mentions versus citations. Semrush found the same split from the platform side: on Gemini, the overlap between brands mentioned in answers and domains cited as evidence can be as low as 30%.

How often should you refresh the prompt set?

Run the set on a fixed schedule and rebuild it quarterly, with an out-of-cycle rebuild whenever your product, category or competitive set changes. The distinction between running and rebuilding is the one that gets lost: running the same prompts frequently is what gives you a trend, and changing them is what destroys it.

Treat the set as versioned. Keep a stable core that does not change between quarterly reviews. That core is your trendline. Add a smaller rotating group for new questions, competitor launches and seasonal demand, and report it separately so a change in the mix never reads as a change in performance. When you retire a prompt, note the date, because a share that appears to collapse is sometimes just a question nobody kept asking.

Two forces argue for a quarterly floor rather than an annual one. Engine behavior keeps moving: Semrush found ChatGPT cites an average of 15 sources per response while Gemini cites about 3, and that only 36 global brands held top-100 visibility across all four platforms in every month of the study. And the surfaces are still scaling: SparkToro found Google AI Mode accounted for just 0.34% of searches from January to April 2026, while Google said at its 2026 I/O conference that AI Mode had passed one billion monthly users with queries more than doubling every quarter.

Cadence beats volume here. A 35-prompt set run every two weeks and rebuilt each quarter will tell you more than a 200-prompt set run once and never repeated, because the first produces a trend and the second produces a snapshot of a probability distribution. That principle, disciplined, repeatable, boring measurement, is what our AI audits practice is built on, and the prompt set is where it starts.