AI hallucinations: why language models make things up and who is liable
Short answer
An AI hallucination is a convincingly worded but incorrect answer: an invented fact, source or ruling. It happens because a language model predicts likely text, not true text. Whoever uses AI output stays responsible for it, as two US lawyers found in 2023 when they were sanctioned for case law ChatGPT had made up. You limit the risk with sources from your own documents and human review.
Why do language models make things up?
A language model is trained to predict the most likely next word. It has no database of facts it consults, and it does not know when it does not know something. Ask it for a source and it produces something that looks like a source.
Hallucinations are more frequent with:
- specific facts: names, dates, amounts, article numbers, rulings;
- topics that appeared little in the training data, such as local regulation;
- requests for sources or quotations;
- long answers, where errors are less noticeable.
What it cost in practice
The best-known example is Mata v. Avianca (2023). Lawyers in New York filed a brief citing case law that ChatGPT had invented. In June 2023 the judge imposed sanctions and ordered them to notify every judge falsely named as the author of a non-existent ruling.
It has also reached the courts in the Netherlands. In 2024 a sub-district judge in Gelderland used ChatGPT as a source for an estimate in a neighbours’ dispute about solar panels. The court stressed that it played no role in the outcome, but it sparked a broad debate on whether that was responsible.
For organisations the risk is less visible but similar: a wrong amount in a quote, a non-existent clause in an opinion, a wrong dosage in a patient letter.
Who is liable?
Whoever uses the output. A lawyer remains responsible for their brief, a doctor for their letter, an organisation for what it sends to clients. AI vendors’ terms generally exclude liability for the accuracy of the output.
Professional bodies say the same. The Dutch bar, the German BRAK and the French CNB all stress that the professional remains ultimately responsible. See AI for law firms. And the European Commission explicitly names the risk of hallucinations as a topic for AI literacy. See AI literacy.
How to manage the risk
- Have the model work from your own sources. With RAG the model answers from your documents and names the source, so an employee can check it. See AI on your own documents.
- Do not ask for facts from the model’s memory. Use AI to summarise, rewrite and structure text you supply yourself.
- Human review of anything that goes out or on which a decision rests, with extra attention to names, numbers and quotations.
- Choose the model per task. A larger model for precision work; a smaller one is enough for classifying.
- Train staff to recognise and report hallucinations.
With werqly the models and the connections to your own documents run on a node of your own. Answers refer to the source in your organisation, and the audit log records which source went with which answer. See What is private AI?
Frequently asked questions
Questions on this topic
What is an AI hallucination?
An answer from a language model that sounds convincing but is wrong: an invented fact, source, quotation or ruling. It arises because the model predicts likely text, not true text.
Can hallucinations be prevented completely?
No, but they can be greatly reduced. Have the model answer from your own documents with sources, use it mainly to work on text you supply, and check output on which a decision rests.
Is the AI vendor liable for errors?
Usually not: AI vendors’ terms generally exclude liability for the accuracy of the output. Whoever uses the output, such as a lawyer, doctor or organisation, remains responsible.
Which tasks are most prone to hallucinations?
Questions about specific facts, sources, quotations, case law and figures, especially on topics that appear little in the training data. Summarising and rewriting supplied text is far more reliable.
Sources
- Mata v. Avianca, Inc., 678 F.Supp.3d 443 (S.D.N.Y. 2023)
- Advocatie — Judge cites ChatGPT in ruling (2024, Dutch)
- European Commission — AI Literacy: Questions & Answers
This article is general information, not legal advice. Rules and guidance change; check the sources listed or ask your lawyer or DPO if in doubt.