Ask a chatbot for a source and it may give you a book that was never written, a court case that never happened or a statistic nobody measured, all in the same assured tone it uses for things that are true. The industry calls these errors hallucinations. They are not a rare glitch. They follow from how the technology works, and understanding why is the best protection against being caught out.

What a hallucination is

A hallucination is a statement produced by an AI model that sounds plausible and is wrong. The model is not lying, because it has no intention to deceive, and it is not malfunctioning in the usual sense. It is doing what it was built to do, which is to produce fluent text.

Hallucinations are different from ordinary mistakes in one important way. A person who does not know something usually shows it, by hesitating or saying so. A chatbot’s false statements look exactly like its true ones.

Why it happens

The model predicts words. It does not look things up. A large language model, the kind of system behind chatbots, generates text one fragment at a time by working out what is most likely to come next, based on patterns in the enormous amount of text it was trained on. It has no built-in list of facts to check against. When the right answer appeared often in its training material, the likely words and the true words are the same. When it did not, the model still produces something likely-sounding.

It is rewarded for guessing. A 2025 paper by researchers at OpenAI and Georgia Tech argues that this is the heart of the problem. Models are trained and then scored on tests, and most tests give marks for a right answer and nothing for “I don’t know”. A confident guess therefore scores better, on average, than an honest admission of uncertainty. In the authors’ words, models are “optimized to be good test-takers”. They propose changing how the main tests are scored, so that guessing wrongly costs more than declining to answer.

Rare and specific facts are the weak point. The less often something appeared in the training text, the less the model has to go on. Names, dates, quotations, page numbers and citations are exactly the kind of detail it is most likely to invent.

A chatbot’s false statements look exactly like its true ones.”

How often it happens

There is no single hallucination rate. It depends on the model, the subject and how the question is asked. Law is the field where it has been measured most carefully, because legal claims can be checked against real records.

StudyWhat was testedShare of answers that were hallucinations
Dahl and others, 2024General chatbots asked specific questions about US federal court cases58% (GPT-4) to 88% (Llama 2)
Magesh and others, 2024Specialist legal research tools from LexisNexis and WestlawBetween 17% and 33%
Two studies of AI and legal questions, by a group including Stanford researchers

The first study, published in the Journal of Legal Analysis, asked models verifiable questions about randomly chosen cases. The second tested commercial products that search a database of real legal documents before answering, a design their makers had promoted as avoiding the problem. Those tools did hallucinate less than a general chatbot. They still got between one in six and one in three answers wrong.

Both studies date from 2024, and the models have been updated many times since. The figures should not be read as current scores. They do show that the problem was severe in a demanding field, and that connecting a model to real documents reduced it without removing it.

How often AI tools got legal questions wrong% of answers that were hallucinations, 2024 studies
How often AI tools got legal questions wrong
Item% of answers that were hallucinations, 2024 studies
Llama 2, general chatbot88%
GPT-4, general chatbot58%
Specialist legal tools, worst33%
Specialist legal tools, best17%

The two studies used different questions, so compare within a study.Source: Dahl and others; Magesh and others, Stanford. Chart: The Sidelight.

What it has cost people

The clearest examples come from courts, where lawyers have filed documents citing cases that a chatbot invented.

In the United States, a judge in New York sanctioned two lawyers in June 2023 in a case called Mata v. Avianca. They had submitted a filing that cited non-existent court decisions generated by ChatGPT, and then stood by them when challenged. The judge found they had acted in bad faith and imposed a penalty of $5,000.

In England, the High Court dealt with two such cases together in a judgment in June 2025. In one, five cases cited to the court did not exist. In the other, 18 of 45 citations were to cases that did not exist. The judges said plainly that generative AI tools “are not capable of conducting reliable legal research”, and that lawyers who use them have a professional duty to check the results against authoritative sources. They referred lawyers in both cases to their regulators.

The lesson of both judgments is the same. The person who relies on the output is responsible for it. “The AI said so” is not a defence.

What reduces it

  • Giving the model the source. A model asked to summarise or answer from a document you supply has much less room to invent than one answering from memory. Tools that search first and then answer work on the same principle. The legal study shows this helps and does not cure the problem.
  • Asking for sources you can open. A link or a quotation can be checked. Check it. An invented reference often looks more convincing than a real one.
  • Letting it say it does not know. Models can be instructed, and increasingly are trained, to decline when unsure. The 2025 paper argues this will only become normal when tests stop penalising it.
  • Using a second check. For anything that matters, confirm the claim somewhere that does not depend on the chatbot.

When to be most careful

Hallucinations are most likely, and most costly, in the same places:

  • specific facts: names, numbers, dates, quotations and references
  • niche subjects and anything recent
  • questions with a false assumption built in, which a model may accept and build on
  • law, medicine and money, where a wrong answer has consequences

They are least likely when the model is working on text you gave it: rewriting, summarising, translating or reorganising.

Can it be fixed?

Researchers disagree. The authors of the 2025 paper think a large part of the problem comes from how models are scored, which can be changed. Others argue that any system that generates likely text will sometimes generate likely falsehoods, so the rate can be pushed down but not to zero. On either view, the practical advice for now is the same. Treat a chatbot’s answer as a draft from a well-read assistant who never admits to being unsure, and check anything you intend to rely on.