Rolling The AI Dice: The Truth About Probability And Consistency In AI Search Answers

We may never know exactly how the AI sausage is made.

But for those who love a walk-through of how good ol’ fashion research is still conducted — in this case, on whether AI search recommendations are consistent, through the beauty of brute-force experimentation — SparkToro has a step-by-step breakdown like none other.

Highlighting only does so much justice, but essentially, some curious minds wrangled hundreds of volunteers to ask dozens of questions to ChatGPT, Claude, and Google’s AI Overview and AI Mode. Getting lost in these weeds is part of the fun, but for those looking for highlights:

• No matter what the question or which tool was answering, the same list of brands or recommendations was given less than 1 in 100 times. (“AIs do not give consistent lists of brand or product recommendations,” writes SparkToro’s Rand Fishkin of the randomness. “If you don’t like an answer, or your brand doesn’t show up where you want it to, just ask a few more times.”)

(Source: SparkToro)

• Despite the randomness, the repeat-appearance of many answers “indicates a set of brands that the AI’s system generally associates more (or less) as a good answer for the prompt intent.” So ask a question enough times, and you’ll see some patterns of recommendations. (Average visibility for the top-three mentioned brands was 73% for Claude, 68% for Google’s AI, and 64% for ChatGPT.)

• When asked to phrase their own prompts with specific constraints, volunteers “used wildly different prompts.” (“The variation of brands/recs in AI answers around a space in the messy wilds of AI prompting is likely much higher than what our controlled experiments revealed here.”)

“Measuring your brand’s presence in AI answers with precision is a fool’s errand,” writes Fishkin of these AI engines that are perhaps better regarded as probability engines than as “sources of truth or consistency.”

“Any tool that gives a ‘ranking position in AI’ is full of baloney.”