// Glossary · technical

Hallucination

Also: AI hallucination · confabulation · fabrication

An AI model confidently generating false information, the single biggest production risk for customer-facing AI, mitigated by grounding, guardrails, and supervision.

A hallucination is what happens when a language model generates an answer that sounds correct but is not true. The model invents a citation that does not exist, quotes a price that is wrong, references a policy the company never wrote, or attributes a quote to a person who never said it. The output reads with the same confidence as a correct answer because the model has no internal signal for "I do not actually know this." Probability over the next token is the only thing the model optimizes, and probable text is not the same as true text. Hallucination is the defining failure mode of language models in production.

Hallucinations cause real damage in customer-facing roles. A support bot that invents a refund policy creates a refund obligation the company never agreed to. A sales agent that quotes pricing the model imagined leads to deals priced wrong. A medical-adjacent assistant that invents a drug interaction creates legal liability that no insurer wants to underwrite. This is why every production AI system at funded scale layers three defenses. Grounding via RAG forces the model to answer from retrieved source documents instead of training-data recall. Guardrails filter outputs against hard rules before they reach the user. Human supervision catches what the first two layers miss and feeds corrections back into the loop.

No production stack eliminates hallucinations entirely. The honest goal is to push the rate low enough that the remaining failures are caught by review before reaching the customer. The AI Support Department and AI Sales Department both ship with citation-required output, confidence scoring on retrieval, and human review queues for low-confidence answers. Teams that skip these layers and deploy raw LLM output to customers learn the lesson the expensive way. The hallucination problem is the reason "we built an AI chatbot in a weekend" rarely survives contact with paying customers.

// Examples
  • A support bot without RAG tells a customer they qualify for a refund under "section 4.2 of our policy," a policy section that does not exist in the company terms.
  • A research assistant cites a 2021 Stanford paper that the model invented, complete with plausible author names and a fake DOI, before a reviewer catches the fabrication.
  • A sales engineering copilot quotes a competitor integration the product does not actually support, because the model pattern-matched on similar SaaS feature sets in training data.
// Common questions
Why do language models hallucinate at all?
They optimize for probable text, not true text. Training rewards predicting the next token correctly, which often coincides with truth but is not the same goal. When the model encounters a question outside its reliable knowledge, the most probable continuation is a confident-sounding answer, because confident answers are what the training data contains.
Does RAG eliminate hallucinations?
No, but it reduces them substantially. The model can still misinterpret retrieved context, ignore it in favor of training-data recall, or generate over-confidently when retrieval returns weak matches. Production RAG systems add retrieval quality scoring, citation requirements, and evaluation infrastructure on top to catch the failures that grounding alone misses.
How do I detect hallucinations in production?
Three layers help. Citation tracing checks whether claimed facts appear in retrieved sources. LLM-as-judge runs a second model to score the answer for grounding. Human review samples a fraction of outputs and feeds corrections into the eval set. Funded teams budget for at least one of these layers from day one.
Which models hallucinate less?
Larger models hallucinate less than smaller ones on average, and instruction-tuned models hallucinate less than base models. Claude 3.5 Sonnet and GPT-4o post the lowest hallucination rates on most benchmarks, with smaller open-source models like Llama 3.1 70B close behind. Model choice alone does not solve the problem, but it sets the floor.
// Related terms
// Ready to ship?

EOI runs fractional AI departments for funded teams under 50. Sales, Content, Ops, Support. Live in 14 days on a monthly retainer.

Apply for a sprint →See all departments →