Short answer

AI chatbots sometimes state false things confidently, including made-up court cases, sources and software packages. The cause is how they work: they predict plausible text, not verified facts. Check any legal, medical or financial claim against the original source, and never install a package name an AI suggested without confirming it exists.

Key takeaways

  • A hallucination is a confident, fluent answer that is simply wrong or invented.
  • Fake citations look real: proper formatting, plausible titles, even quotes. Always open the source.
  • In Mata v. Avianca (2023), New York lawyers were fined $5,000 for filing a brief with fake ChatGPT cases.
  • A 2025 study found about 19.7% of packages recommended by code-writing models did not exist, which attackers can exploit (slopsquatting).
  • For law, health and money, use AI to prepare questions, then confirm with a professional or official source.

Confident does not mean correct

AI chatbots write smoothly, and that is the problem. A wrong answer reads exactly like a right one. Researchers call these errors hallucinations: output that sounds authoritative but is false or invented. It happens because a chatbot predicts likely text, rather than checking a database of facts.

In 2023, lawyers in Mata v. Avianca filed a brief citing court cases that ChatGPT had made up. The cases had realistic names, quotes and citations. When questioned, the lawyers asked the chatbot whether the cases were real, and it said they were. Judge P. Kevin Castel fined the lawyers and their firm $5,000 in total and noted that using AI is not improper in itself, but lawyers must check their filings.

The problem is not limited to general chatbots. A 2024 Stanford study of commercial legal research tools found that Lexis+ AI and Ask Practical Law AI produced incorrect information more than 17% of the time, and Westlaw’s AI-Assisted Research more than 34% of the time. The vendors disputed parts of the study, and the tools did beat a general-purpose model. Still, no tool was error free.

Where it hurts ordinary people

Health. A chatbot might invent a study, mix up two drugs with similar names, or give a dose that is wrong for your age. Use it to learn vocabulary, then check an official health site and ask a clinician.

Money and tax. Tax rules, deadlines and fee rates change, and chatbots mix up countries and years. A wrong figure can cost you a penalty. Confirm on the regulator’s site.

Fake references. Ask for sources and you may get plausible author names, journals and links that lead nowhere. Students and writers have been caught out by this.

Scam confirmation. If you ask a chatbot “is this investment legitimate?”, it may produce a reassuring answer built on nothing. It cannot see company registers in real time unless it searches, and even then it can be wrong. See our guide to phishing messages for checks that do not depend on AI.

Slopsquatting: fake package names

Developers now ask AI to write code, and the code often imports software packages. Researchers at three US universities tested 16 models across 576,000 code samples. They found that on average at least 5.2% of packages suggested by commercial models and at least 21.7% by open-source models did not exist, with 205,474 unique invented names overall. Across all 30 tests, the paper counts 440,445 hallucinated packages out of 2.23 million recommended, about 19.7%.

Attackers can register those invented names and fill them with malware, hoping a developer installs without checking. The tactic has been called slopsquatting. Even if you are not a developer, the lesson is the same: do not trust a name just because an AI said it.

A 60 second verification routine

  1. Spot the claims that matter: names, numbers, dates, quotes, citations.
  2. Open the original source yourself. Do not accept a summary of it.
  3. Search the exact phrase elsewhere.
  4. For anything that affects your health, money or legal position, get a human professional.
  5. When unsure, say so in your prompt and ask the model to list what it cannot confirm.

Treat the chatbot as a fast, sometimes careless research assistant. It drafts. You verify. If a fake answer led you to send money or details, see what to do after a data breach. For privacy habits to use alongside this, read what not to tell AI chatbots.

What you can do

  1. 01Ask for sources, then open each link yourself. If you cannot find the case, paper or page, treat the claim as unverified.
  2. 02Search for the exact title or quote in a normal search engine or official database. Fake references usually return nothing.
  3. 03For health, check NHS, CDC or similar official pages and talk to a clinician before acting on dosage or diagnosis advice.
  4. 04For money or tax, confirm figures on the regulator or tax authority site, and never move money on a chatbot's say-so.
  5. 05Before installing any package an AI suggests, search the official registry (PyPI, npm), check its age, downloads and maintainer.
  6. 06Ask the chatbot to say when it is unsure, and cross-check important answers in a second, independent source.

Related reading: AI voice clone scams and the data breach tracker.

Frequently asked questions

Why do AI chatbots make things up?

They generate the most plausible next words based on patterns in training data. They do not look facts up by default, so when they lack a reliable answer they can still produce a confident, invented one.

Do AI tools with web search still hallucinate?

Yes, less often, but they can misread a page, cite a source that does not support the claim, or pick a poor source. Stanford researchers found even specialised legal research tools produced errors in a meaningful share of queries.

What is slopsquatting?

Slopsquatting is when attackers register software package names that AI models tend to invent, so a developer who installs the suggested name gets malicious code. The term describes the risk found in package-hallucination research.

Can I trust AI for medical or legal advice?

Use it to understand terms and prepare questions, not as the final word. Verify with official sources and a qualified professional before acting.

Sources

  1. ChatGPT phony legal filing case gets lawyers a $5,000 fine, Bloomberg Law, 2023-06-22
  2. AI on Trial: Legal Models Hallucinate in 1 out of 6 (or More) Benchmarking Queries, Stanford HAI, 2024-05-23
  3. We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs, arXiv / USENIX Security 2025, 2025-03-02

This article reflects public information as of 2026-11-05. AI tools and rules change quickly; we update articles when new facts are published. Spot an error? See our editorial and corrections policy.