No, ChatGPT does not give everyone the same answer. We tested this directly at Honeyb (that is us, so read the method before the numbers). On 13 July 2026 we sent 20 buyer-intent prompts, each three times, to four AI engines via API: ChatGPT (gpt-5-mini), Gemini (gemini-2.5-flash), Claude (claude-haiku-4-5) and Perplexity (sonar). That is 240 answers to identical questions, asked minutes apart, with nothing else changed.
ChatGPT swapped its number-one recommended brand between identical consecutive runs 35% of the time. Gemini was the least stable at 44%. If you asked ChatGPT "what is the best CRM" twice in a row, roughly one time in three you would get a different top pick.
How much the same question changes the answer
We tracked two things per engine. Top-pick change rate: how often the first-named brand differed between two identical back-to-back runs. Brand-set overlap: how much of the full recommended brand list survived from one run to the next.
| Engine | Top-pick change rate | Brand-set overlap between runs | Avg brands per answer |
|---|---|---|---|
| Gemini | 44% | 54% | 4.8 |
| Perplexity | 43% | 61% | 5.2 |
| ChatGPT | 35% | 42% | 5.1 |
| Claude | 28% | 67% | 5.1 |
Read that middle column carefully. ChatGPT kept its top pick more often than Gemini, but its full brand list was the least stable of the four: only 42% of the brands it named in one run reappeared in the next identical run. Claude was the most consistent on both measures.
Top-pick change rate
How often the top recommendation changes between identical runs
| Item | Top-pick change rate |
|---|---|
| Gemini | 44.1% |
| Perplexity | 42.5% |
| ChatGPT | 35% |
| Claude | 27.5% |
This matches what others have found at larger scale. SparkToro ran 2,961 prompts through ChatGPT, Claude and Google's AI tools and found less than a 1-in-100 chance that two runs return the same list of brands, although the leading brands still turned up in most runs. Our figures look calmer because we measured only the top pick and the overlap between brand sets, rather than an exact match of the whole list, and we ran everything within a single session on fixed models.
The four engines also disagree with each other
Run-to-run variance is only half the problem. The engines rarely agree among themselves either. Across our 20 prompts, the share of prompts where two engines named the same top brand ranged from 20% to 53%.
| Engine pair | Same top brand | Mean brand-list overlap |
|---|---|---|
| Gemini and Claude | 53% | 44% |
| ChatGPT and Gemini | 47% | 29% |
| ChatGPT and Claude | 40% | 27% |
| Gemini and Perplexity | 37% | 47% |
| Claude and Perplexity | 35% | 37% |
| ChatGPT and Perplexity | 20% | 25% |
ChatGPT and Perplexity agreed on the top brand for just 4 of 20 prompts. So "does ChatGPT give everyone the same answer" understates the issue. Even if it did, your buyers are spread across four engines that recommend different brands for the same question. Our guide to what AI visibility actually means covers why each engine behaves like a separate channel.
Want to get recommended by AI?
Check your AI search visibility, then let the Honeyb agent write, fix, and earn what gets you recommended. Free to start.
Why the answers keep changing
Three mechanisms drive this, and none of them is a bug.
Sampling. These models generate text probabilistically. Two brands with similar internal weight can swap places from one generation to the next. That alone explains much of Claude's 28% top-pick churn.
Retrieval variation. Every engine in our test ran live web retrieval, and the pages fetched differ between runs. ChatGPT averaged 15.0 sources per answer, drawn from 445 distinct domains across 900 total citations, with its top three domains accounting for just 7.6% of citations. When the evidence pool is that long-tailed, a different fetch produces a different shortlist. Perplexity leaned hardest on community content: Reddit alone was 14% of its citations and YouTube another 8%, consistent with Reddit being the most-cited domain in Semrush's June 2025 study of about 150,000 AI citations.
Model updates. Providers ship silent model and retrieval changes. A reading taken before an update does not describe the system after it.
One caveat on sources: via API, every engine returned sources on 100% of answers. Consumer apps often show fewer or none, so what your buyer sees cites less than what we measured. Gemini added its own wrinkle by wrapping 429 of its 580 source URLs (74%) in grounding redirects that mask the real source.
What this means if you checked ChatGPT once
If you asked ChatGPT about your category last month and saw your brand, that reading had roughly a one-in-three chance of flipping on the very next run. It says almost nothing about what the engine tells buyers this week, on this model version, in this retrieval mood.
The practical consequence is that AI visibility is a series, not a snapshot. A single check cannot distinguish "we are reliably recommended" from "we got lucky on that generation". We wrote up the statistics of this in more detail in why spot-checking AI answers fails, and if you want to run repeated checks without doing it by hand, see our comparison of AI model tracking dashboards, which covers Honeyb (our product) alongside the alternatives.
A sensible baseline: track a fixed prompt set across all four engines on a schedule, and treat share of runs where you appear as the metric, not any individual answer. Ahrefs' research suggests the lever that moves that share is third-party mentions and video rather than on-page changes, which is a different playbook from classic SEO.
Want to know what the engines say about your brand right now, measured properly across repeated runs rather than one lucky prompt? Start with a free AI visibility check.
