If you ask what is one major risk associated with generative AI models, the honest answer is confident fabrication. A generative model can produce a fluent, well-structured, entirely plausible answer that is simply not true, and it presents that answer with the same calm authority it uses for facts. The industry calls this hallucination. It is the risk that sits underneath almost every other problem these systems cause, because a false output that looks right is far more dangerous than one that looks wrong. This piece answers the literal question first, then widens out to the other material risks of generative AI, and ties the whole thing back to something brand and marketing teams now have to manage directly: what AI models say about you when nobody from your company is in the room.
The single biggest risk: confident fabrication
Ask the question plainly and the field largely agrees on the answer. The one major risk associated with generative AI models is that they fabricate, and they do it without any visible signal that they have stopped reporting and started inventing. A search engine that cannot find a result returns nothing, or an error. A generative model that does not know an answer will often write one anyway, in clean prose, with the confidence of a textbook.
This is not a bug that a future patch removes. In September 2025 a pair of OpenAI researchers published a paper titled Why Language Models Hallucinate. Their argument is blunt. Hallucinations, they write, originate simply as errors in binary classification. If a model cannot reliably separate true statements from false ones during training, then hallucinations in pretrained language models will arise through natural statistical pressures. In other words, fabrication is a structural feature of how these models learn, not a stray defect.
The paper goes further and blames the way the industry grades its models. Most benchmarks score only whether an answer is correct, with no credit for admitting uncertainty. So a model that guesses scores better than a model that says it does not know, because a guess is sometimes right and an abstention never is. The authors describe an epidemic of evaluations that penalise honest uncertainty, and conclude that models are optimized to be good test-takers, and guessing when uncertain improves test performance. The machine is, in a real sense, trained to bluff.
How big the problem actually is
Fabrication rates have fallen as retrieval grounding and better training have matured, but they remain measurably non-zero, and they climb fast on hard, specialised questions. Two numbers frame the range.
On the easy end, when a model is handed a document and asked only to summarise it, hallucination is rare but not absent. Vectara's hallucination leaderboard, which scores models on grounded summarisation of provided text, puts the best performer (Gemini 2.5 Flash-Lite) at around a 3.3% hallucination rate on its harder current benchmark, while several widely used reasoning models, including GPT-5 and Claude Sonnet 4.5, exceed 10% on the same longer documents. That is the floor, on the task most favourable to the model.
On the hard end, the numbers are sobering. A Stanford RegLab and Institute for Human-Centered AI study, Large Legal Fictions, found that when asked specific, verifiable questions about random US federal court cases, leading models hallucinated between 69% and 88% of the time. The pattern is consistent across research: the more specialised and verifiable the question, the more the model invents, precisely the situations where a confident wrong answer does the most harm.
| Setting | Approximate hallucination rate | Source |
|---|---|---|
| Grounded summarisation of supplied text (best model, Gemini 2.5 Flash-Lite) | Around 3.3% | Vectara leaderboard, harder current benchmark |
| Grounded summarisation (GPT-5, Claude Sonnet 4.5) | Over 10% | Vectara leaderboard |
| Specific questions about random US court cases | 69% to 88% | Stanford RegLab |
The takeaway is not that generative models are useless. It is that the error rate is a function of the question, the rate is never zero, and the model gives you no reliable signal for which answer falls into the wrong column.
Why fabrication is the risk that matters most
Plenty of technologies make mistakes. What makes generative fabrication uniquely risky is the combination of three things: the errors are fluent, they are confident, and they are invisible until verified. A typo signals its own unreliability. A hallucinated legal citation, complete with a plausible case name, court and year, signals nothing at all. The reader has to do the work of catching it, and fluent prose actively discourages that work.
The legal profession has become the clearest public record of what happens when nobody does the checking. The first widely reported case was Mata v. Avianca in 2023, where a New York lawyer submitted a brief citing six cases that ChatGPT had invented wholesale. The court fined the attorneys 5,000 dollars. That was treated as a cautionary one-off. It was not. A public database maintained by researcher Damien Charlotin now tracks more than 1,300 court filings worldwide that contained AI-generated fabrications, updated daily, with new cases added almost every day, a pace corroborated by Bloomberg Law reporting. The technology improved across that period. The fabrication problem did not go away. It scaled with adoption.
The lesson generalises well beyond law. Anywhere a generative answer is treated as a finished fact rather than a draft to verify, fabrication converts directly into real-world consequences: a wrong dosage, a misquoted contract clause, a citation that does not exist, a competitor recommended in place of you.
The other material risks of generative AI
Want to see this in action?
Check how AI models talk about your brand — free, instant, no signup required.
Confident fabrication is the headline, but it is not the only risk, and an honest look has to name the rest. The US National Institute of Standards and Technology built its Generative AI Profile around twelve risks unique to or amplified by these systems. NIST even uses its own term, confabulation, for what most people call hallucination. The risks brand and marketing teams are most likely to meet, in plain terms:
- Embedded bias. Models learn from human-produced text, and they reproduce and sometimes amplify the skews in it. An answer can be fluent, confident and quietly unfair, which is harder to spot than an answer that is simply wrong.
- Data privacy and leakage. Anything typed into a consumer chatbot may be retained or used to improve the model. Sensitive customer data, unreleased strategy and internal documents do not belong in a prompt box without an enterprise agreement that says otherwise.
- Prompt injection and manipulation. Because models follow instructions found in the content they read, a malicious or strategically worded page can change what a model says. This is a genuine security surface, not a hypothetical.
- Synthetic media and deepfakes. Generative models produce convincing fake images, audio and video, which raises the cost of trusting any single piece of content at face value.
- Over-reliance. The most underrated risk. The smoother the output, the more people stop checking it, which is exactly when fabrication does its damage.
Notice how many of these loop back to the first risk. Bias is a skewed picture of the world presented as neutral. Prompt injection is fabrication induced on purpose. Over-reliance is what turns an occasional fabrication into a systemic failure. The common thread is a system that produces output indistinguishable in tone from the truth, whether or not it is true.
Where this becomes a brand problem
For a marketing or communications team, the risks of generative AI models stop being abstract the moment a buyer opens ChatGPT, Gemini, Claude or Perplexity and asks for a recommendation in your category. The engine answers in confident prose. It names a handful of brands, describes them, and cites a few sources. If it gets your brand wrong, or invents a weakness you do not have, or omits you entirely, that fabrication lands in front of a prospective customer with the full authority of the interface. Nobody from your company reviewed it. Most buyers will not check it.
Two experiments we have covered make the exposure concrete. The first is search analyst Lily Ray's gullibility test, where she published a clearly satirical blog post naming fictional winners of absurd categories. Within 24 hours, Google AI Overviews, AI Mode, Gemini and ChatGPT were repeating her invented rankings as if they were established fact. The engines indexed a single new page on a personal blog and surfaced its claims to users within a day, with little real verification. If satire propagates that fast, so does a competitor's spin, or a stale negative review.
The second is sharper still. Two Harvard researchers showed that adding a short strategic text sequence, a roughly 30-token string, to a product page could move it from never recommended to top recommendation inside an LLM-driven shopping tool. That is fabrication on demand, induced by an adversary, in a system that is supposed to be giving the buyer a neutral answer. Two different attack surfaces, one in content and one in code, both exploiting the same underlying weakness: the model treats what it reads as true.
Why these models fabricate about brands specifically
The selection logic compounds the risk. Generative engines do not read your homepage and form a balanced view. They lean heavily on third-party sources, and they lean on a narrow set of them. We have written before about why AI models cite Reddit more than your own site: community discussion, review platforms and listicles carry disproportionate weight, because they read as consensus. That means a fabricated or outdated claim about your brand on a forum can be laundered into a confident first-person answer that an engine presents as settled.
Multimodality widens the surface further. As we covered in multimodal AI models, engines now read images, video and voice as well as text, which means more inputs that can be misread, mislabelled or manipulated into a wrong conclusion about your product. More modalities, more places for a confident misreading to start.
What an honest response looks like
The right reaction to fabrication risk is neither panic nor dismissal. It is the same discipline that careful users already apply to a generative answer: treat the output as a confident draft, not a verdict, and verify anything that carries consequences. For the engines you control, this means open the citations and read them. For the engines that talk about your brand, it means watching what they say.
That is the part most teams miss. You cannot manage fabrication about your brand if you never see it. And spot-checking by asking one engine one question on one day proves almost nothing, because generative answers shift with phrasing, differ across engines, and change as the underlying sources change. The same prompt can name you on Monday and a competitor on Thursday. The practical response is to monitor how the major engines describe your brand across a representative set of buyer questions, repeatedly, and to trace each description back to the source feeding it, so that when a fabrication appears you can find the page it came from and correct the record at the source.
- Treat every generative answer about anything consequential as unverified until you have checked the citation.
- Keep sensitive data out of consumer chatbots, and use enterprise agreements where retention and privacy actually matter.
- Assume third-party pages, not your own site, shape what engines say about you, and invest where the engines actually look.
- Monitor the major engines on a schedule rather than spot-checking, because answers drift and a one-off screenshot proves little.
The takeaway
What is one major risk associated with generative AI models? Confident fabrication. The model writes a fluent, authoritative answer that may be wrong, gives you no signal which is which, and the cost lands on whoever trusted it without checking. Research suggests this is structural rather than a passing defect, the error rate climbs on exactly the specialised questions where mistakes hurt most, and the public record of legal sanctions shows what happens at scale when verification is skipped. The other risks, bias, privacy, manipulation, synthetic media and over-reliance, are real and mostly orbit this same core problem. For brands, the exposure is direct: these systems now describe you to buyers in confident prose, drawn from sources you do not own, and the only way to manage that is to watch what they say and fix the inputs they read.
