A hallucination is when an AI model produces something that sounds confident and plausible but is not true. It might be a statistic nobody published, a report that does not exist, a quote nobody said, a product feature you do not offer or opening hours that are simply wrong.
How hallucination works
A large language model writes by predicting which words are most likely to come next, based on patterns in its training. It does not look facts up unless it has been connected to a source, and it has no built-in sense of when it does not know something. When there is a gap, it fills it with text that fits the pattern, and fitting the pattern is not the same as being correct.
Some requests make this far more likely:
- Specific numbers, dates, prices and percentages.
- Citations, links, report titles and author names.
- Facts about small businesses, local places or niche products that barely appear in training data.
- Anything that happened after the model’s training ended.
- Legal and regulatory references, where a plausible-sounding rule number is easy to generate.
Connecting the model to real documents through grounding or retrieval-augmented generation cuts the rate considerably, but does not remove it. The model can still misread a source, merge two sources or add a detail that neither contains.
A typical marketing example: you ask an AI tool for a blog post on UK small business spending and it cites a named government survey with a precise figure. The survey title sounds real, the figure sounds reasonable, and neither exists.
Why it matters
Once you publish a claim, it is yours. The ASA applies the CAP Code to your marketing regardless of who or what drafted it, and the Code expects you to hold evidence for objective claims before they go out. Under the DMCC Act 2024 the Competition and Markets Authority can now fine businesses directly for misleading consumers. “The AI wrote it” is not a defence in either case.
There is a search cost too. Invented facts undermine the trust signals your content depends on, and readers who spot one error tend to doubt the rest. UK courts have already criticised lawyers who put AI-invented case citations in front of judges, which shows how convincing these errors can look to a busy professional.
Hallucination also happens about you. AI assistants and AI search answers sometimes state wrong prices, services or locations for real businesses, usually because the facts on the web about that business are thin or inconsistent.
Common mistakes
- Trusting a citation without opening it and finding the exact figure on the page.
- Asking an AI tool for statistics instead of giving it the statistics.
- Checking only the parts that look suspicious. Hallucinations look exactly like the accurate text around them.
- Assuming a paid or newer model no longer makes things up.
- Letting AI draft health, financial or legal claims without specialist review.
How to act on it
- Give the tool your source material and tell it to answer only from that material, and to say so when the answer is not there.
- Check every number, name, date, quote, link and legal reference against the original source before publishing.
- Name the person responsible for fact-checking each piece, so the step is never assumed.
- Keep the facts about your business clear and consistent across your website, Google Business Profile and directories, so AI answers have an accurate source to draw on.
- When you find an AI answer that is wrong about you, fix the source pages it is likely drawing on, then check again later.
Building checking into the writing process is part of how I run content SEO and strategy for clients who use AI in drafting.