AI hallucination: why chatbots make up confident facts
An AI hallucination is a false or misleading statement generated by an artificial intelligence model and presented as true fact. Large language models produce these errors while maintaining a confident, fluent tone, often inventing details like fabricated citations. This happens because models rely on statistical pattern completion rather than genuine fact-checking or comprehension.
By the edgi team We find the most surprising true thing about an idea and build a 60-second lesson around it.
A common fact, such as France's capital, appears in training text again and again. A person's obscure birthday may appear once, or not at all. Repetition gives a model more evidence for a fact. With only one example, it may not learn the fact reliably.
Asked for a rare fact anyway, the model can still generate a plausible continuation: a date in the right format and the same confident tone. That is a hallucination: a plausible but false statement generated by a language model.
A screenshot shows a ChatGPT hallucination where the chatbot generates a seemingly valid summary of an article from a fake URL, despite having no internet connection. ChatGPT, Public domain, via Wikimedia Commons
A 2025 OpenAI analysis formalized a lower bound: when a fact appears only once and the model cannot abstain, some errors are unavoidable. It is a theoretical floor, not a count of real-world birthday mistakes.
The exam problem
A model should be able to say when it is unsure. Many evaluations instead score it only as right or wrong. In that setup, a right answer earns one point, while a wrong answer and “I don't know” both earn zero.
Guessing can raise the expected score. Abstaining cannot, even when abstaining would be more honest. OpenAI's analysis found this right-or-wrong framing in many influential evaluations. It encourages optimization for guessing rather than calibrated uncertainty.
Who pays, and what to do
In 2024, a Canadian tribunal ordered Air Canada to compensate a passenger who relied on a refund policy the airline's chatbot had invented. Air Canada argued that the chatbot was a separate entity. The tribunal held that it was part of the airline's website and awarded about 800 Canadian dollars.
If an obscure answer changes across fresh chats, treat that as a warning signal. The disagreement tells you the model is not giving a stable answer. Matching answers are not proof either. Check an important factual claim against a reliable source.
Sampling several answers can help automated systems flag instability, but it cannot turn a chatbot into a fact checker.
Why language models generate false facts
A language model learns facts from repetition across its training data. Common facts appear constantly, while rare facts may appear only once. When prompted for rare facts, a model without enough data to know the answer generates a plausible-sounding continuation anyway, using the correct format and a confident tone.
Evaluation methods also push models to guess. Standard benchmark tests often score answers with a simple binary metric, awarding one point for a correct answer and zero points for both an incorrect answer and an admission of uncertainty. Under this scoring system, guessing yields a higher expected score than admitting uncertainty, discouraging the system from abstaining.
Symbolic artificial intelligence models generally avoid these errors, but neural networks and statistical models remain vulnerable. In high-stakes settings like medical diagnostics, chip design, and supply chain logistics, these generated errors present severe operational challenges.
History and debate over the term
The term hallucination entered computer science in Eric Mjolsness's 1986 PhD thesis. In 1990s and 2000s computer vision, face hallucination referred to algorithms that added detail to construct high-resolution facial images from low-resolution inputs. Researchers later used the word to describe failure modes in statistical machine translation, recurrent neural network citation errors, and object detection mistakes.
Many scientists and engineers criticize the word hallucination for anthropomorphizing software by borrowing concepts from human psychology. Critics like Mary Shaw, Gary Smith, and Usama Fayyad argue that large language models do not understand word meanings and that psychological framing obscures statistical pattern completion. Alternative proposed terms include mirage, confabulation, fabrication, and factual error.
Test yourself
Why does an AI give a fluent, confident answer to a question it has barely seen?
It mimics the pattern of valid facts. Language models predict fluent text based on statistical patterns, so a made-up answer sounds just as confident as a real one.
A test gives zero points for saying 'I don't know' and zero for a wrong answer. What does this encourage?
Guessing even when completely unsure. When silence and error score the same, guessing offers the only mathematical chance at a point.
Why can an accuracy-only evaluation encourage a model to guess?
A wrong answer and “I don't know” both score zero. If abstaining and being wrong both score zero, a guess has some chance of earning a point. That incentive favors guessing over admitting uncertainty.
Claim the Hallucination (artificial intelligence) card
Play the lesson in edgi and the card is yours. It lands on your Map next to the ideas it connects to, and turns from matte to foil to gold as you learn more around it.
Are companies legally liable for chatbot hallucinations?
Yes, automated outputs can carry legal liability. In 2024, a Canadian tribunal rejected Air Canada's argument that its chatbot was an independent entity, ordering the airline to pay roughly 800 Canadian dollars after the bot invented a fake refund policy.
How can you detect if an AI answer is a hallucination?
Generating multiple responses to the same prompt across fresh chats can expose instability if the answers contradict each other. However, consistent answers do not guarantee truth, meaning factual claims must be verified against reliable external sources.
