All glossary terms
A Reliability Foundations

AI hallucination

AI hallucination is fluent, confident, false output. It is not a malfunction; it is the same mechanism that produces correct answers. In an agent, the worst case is not a wrong fact. It is the agent reporting an action it never took.

Definition

AI hallucination is output that is fluent, confident, and false. It is a property of how language models work rather than a defect in them: the model generates the most plausible continuation, and plausibility and truth come apart. In an agentic system the consequence changes, because a hallucinated premise becomes an action rather than a sentence.

What is AI hallucination?

AI hallucination is output that sounds right and is not. The word implies a malfunction, which is the most misleading thing about it, because the model is not doing something different when it hallucinates. It is doing exactly what it always does.

A language model generates the most probable continuation of the text in front of it. When the training data supported a correct answer, the probable continuation is true. When it did not, the probable continuation is a plausible-looking answer rather than an admission of ignorance. Same mechanism, different luck.

That reframing matters because it changes what you can expect. If hallucination were a bug, a better model would fix it. Since it is a property of generating plausible text without a model of truth, the practical question is not how to eliminate it but where you are willing to let its consequences land.

The term is also under challenge, and for good reason. Hallucination implies perception gone wrong, which anthropomorphises a statistical process. Fabrication and confabulation are both more accurate. The industry has settled on hallucination anyway, so this page uses it, while noting that the metaphor is responsible for a fair amount of muddled thinking about what can be fixed and by whom.

Why do language models hallucinate?

Because the training objective rewards plausible answers and does not reward saying "I do not know." Confident wrong answers turn out to be locally optimal under standard training and evaluation, which means the behaviour is produced by the incentive we set rather than emerging despite it.

Research published by OpenAI in 2025 formalised this: hallucinations arise because the training and evaluation setup makes a confident wrong answer a better bet than an abstention. A benchmark that scores accuracy and treats a refusal as a miss teaches the model to guess.

  • No representation of its own uncertainty. The model produces a distribution over next tokens, not a belief about whether it knows. Fluency is uniform across confident and unfounded output, which is why the two are indistinguishable to a reader.
  • Refusal is penalised. If evaluation rewards answering and treats abstention as failure, guessing scores better. The behaviour is trained, not accidental.
  • Gaps get filled with the shape of an answer. Asked something absent from the training distribution, the most probable continuation still looks like an answer, because answers are what follow questions in the data.
  • More capable does not mean better calibrated. This is the counterintuitive part. Successive frontier models have sometimes scored worse on factual reliability about specific entities while scoring better on capability, because knowledge expansion has outpaced calibration.

The last point is worth sitting with, because it undermines the most common plan. Waiting for a better model is not a hallucination strategy. Capability and calibration are different axes, and they have not moved together.

What do agents hallucinate that chatbots do not?

Three things that have nothing to do with facts: invented parameters, actions never taken, and completions that never happened. When a chatbot hallucinates it tells the user something false. When an agent hallucinates it can do something wrong, or claim to have done something at all.

This is the part of the subject that general hallucination coverage misses, and it is the part that matters once a model can call tools.

Six hallucination types arranged by escalating consequence. Factual and contextual hallucination produce a wrong sentence. Citation hallucination produces a fabricated source. Parameter hallucination produces an invented identifier passed to a tool. Action hallucination is a claim to have performed a step absent from the trace. Outcome hallucination is a claim of task completion contradicted by the evidence. The last three are marked as agent-specific, with a note that only the last three can be checked deterministically.
Six hallucination types, what each produces, whether it is specific to agents, and how each can be detected
Type What the model produces Agent-specific? How you catch it
Factual A claim about the world that is untrue No Compare against a source. Requires knowing the truth
Contextual A claim contradicting the content it was given No Check the answer against the retrieved passage
Citation A source, case, or reference that does not exist No Resolve every citation programmatically. Cheap and rarely done
Parameter An invented identifier, timestamp, or field value passed to a tool Yes Schema validation, and checking the value appears in the trace or the input
Action A claim to have performed a step that does not appear in the trace Yes Compare the claim against the actual tool calls. Deterministic
Outcome A claim that the task is complete when the evidence says otherwise Yes Verify against system state rather than against the summary

Why an agent says it finished when it did not

The mechanism is mundane and that is what makes it dangerous. An agent has been given a task, has the context, knows what it was supposed to do. The most natural-sounding continuation of that conversation is a sentence reporting success. Nothing in the architecture checks the claim against what happened, so the system is optimised to produce completion-shaped output with no mechanism that could contradict it.

It shows up in a subtler form too. An agent that hits a refusal partway through, corrects itself, and finishes properly can still omit the refusal from its own summary of the work. The account is not false about the outcome and is incomplete about the path, which is exactly the sort of gap that matters when somebody later asks what happened.

Why is a hallucination rate not a property of a model?

Because it is a property of a model and a benchmark together. The same model in the same configuration can score under two percent on one factuality test and over ten percent on another, so any rate quoted without its benchmark and its date tells you nothing usable.

This matters commercially, because hallucination rates appear in vendor material and board papers as though they were specifications. They behave more like exam results: informative about the exam as much as about the candidate.

  • Benchmarks measure different things. Summarisation faithfulness, factual recall about people, closed-book knowledge, and domain accuracy are separate skills. A model can be well calibrated on one and poorly calibrated on another.
  • The spread within one model is large. Published results have shown the same model producing single-digit and double-digit hallucination rates depending on which test it faced. Neither number is wrong. Neither is a property of the model.
  • High capability and high hallucination coexist. Some recent models have posted record accuracy alongside high hallucination rates on the same evaluation, because they answer more questions rather than declining more of them.
  • Your workload is not any of these benchmarks. The only rate that governs your risk is the one measured on your tasks, with your data, under your prompts.

The practical rule is the same one that applies to any statistic in this field. Ask which benchmark, on what date, measuring what. A rate presented without those three is being used to persuade rather than to inform. See agent evaluation for measuring your own.

Does grounding fix hallucination?

It reduces factual hallucination and introduces a subtler problem. A model given source material can still misread it, over-generalise from it, or contradict it while citing it, and the citation makes the answer considerably more persuasive without making it more correct.

Retrieval is the right first move and it is not a solution. Understanding what it does and does not address prevents a common and expensive misplacement of confidence.

On the left, an agent's summary claiming that a record was updated and a notification sent. In the middle, a verification step. On the right, the execution trace showing only one tool call, with the second action absent. The verification step is marked deterministic, with a note that comparing the claim against the trace requires no judgement and no second model, unlike checking whether a factual claim is true.
  • What grounding fixes. Claims about things absent from the training data. If the answer has to come from a document you supplied, the model has less room to invent one.
  • What it does not fix. Misreading the passage, drawing an unsupported conclusion from it, blending two sources into a claim neither makes, or attributing a statement to a document that does not contain it.
  • What it makes worse. Perceived reliability. An answer with a citation reads as verified, and readers check citations far less often than they trust them. Grounding raises trust faster than it raises accuracy, and the gap between those two curves is where the damage happens.
  • The cheap control almost nobody implements. Resolve citations programmatically. If the answer names a source, confirm the source exists and contains the claim, by rule rather than by judgement. This catches an entire hallucination class at negligible cost.

There is also a compounding case specific to multi-step systems. A hallucinated fact from step one becomes a retrieved premise at step four, and by then it carries the appearance of established context rather than of a guess. That is the same mechanism as context poisoning, covered at context engineering.

How do you reduce hallucination in an agentic system?

Stop trying to detect false statements and start verifying claims you can check mechanically. Whether a fact is true requires judgement. Whether the tool call the agent claims to have made actually appears in the trace does not, and that is where the leverage is.

The ordering below is deliberate: the deterministic checks come first because they are cheap, provable, and cover the agent-specific failures that matter most.

  1. Verify completion against the trace, not the summary

    If the agent says it updated a record, confirm the tool call happened and returned success. This single check eliminates action and outcome hallucination as a class, requires no model, and is the highest-value control on this page. It also depends on having per-step traces, covered at AI agent observability.

  2. Enforce tool schemas strictly

    Reject malformed arguments and invented field names at the boundary rather than passing them through. Schema validation is deterministic and catches most parameter hallucination before it reaches a system that will act on it.

  3. Resolve every citation

    Confirm programmatically that each cited source exists and contains the claim attributed to it. Cheap, mechanical, and it removes the failure mode that has produced the most public embarrassment in professional settings.

  4. Check answers against the source they claim to use

    Faithfulness scoring compares an answer to the retrieved passage. This one is probabilistic rather than deterministic, so treat it as a signal, and remember the judge doing the comparison can also be wrong.

  5. Make abstention a valid outcome, and reward it

    If your evaluation treats "I could not determine this" as a failure, you are training the same incentive that produces hallucination in the first place. Score a correct refusal as a success, and instrument how often it happens.

  6. Constrain where a false premise can land

    The last line, and the one that holds when everything above misses. Scoped permissions, approval gates on the irreversible, and no outbound path an error could exploit. See AI guardrails.

Read that list in order and the shape becomes clear. Two of the six require a model to judge anything. The rest are rules. Most teams start with the probabilistic layer because it addresses the failure they read about, and leave the deterministic checks that would catch the agent-specific failures unbuilt.

Frequently asked questions about AI hallucination

If your question is not here, our team will answer it directly.


Talk to a Specialist →
What is an AI hallucination?

Output that sounds right and is not. The important thing is that the model is not doing anything different when it happens. It generates the most probable continuation of the text in front of it; when the training data supported a correct answer, that continuation is true, and when it did not, the continuation is a plausible-looking answer rather than an admission of ignorance. Same mechanism, different luck, which is why a better model does not simply fix it.

Why do AI models hallucinate?

Because the training objective rewards answering and does not reward abstention. Research published by OpenAI in 2025 formalised this: confident wrong answers are locally optimal when evaluation scores accuracy and treats a refusal as a miss, so guessing scores better than admitting ignorance. The model also has no representation of its own uncertainty. It produces a distribution over next tokens, not a belief about whether it knows, and fluency is identical either way.

Can AI hallucination be eliminated?

Not by better models, and that is the practical point. Since hallucination is a property of generating plausible text without a model of truth rather than a defect to be patched, the useful question is not how to remove it but where you are willing to let its consequences land. Frontier models have also shown that capability and calibration are separate axes: some newer models have scored better on capability while scoring worse on factual reliability about specific entities.

What is tool-use hallucination?

An agent claiming to have taken an action it did not take. It is not about getting a fact wrong; it is about reporting completion of something that never happened. The mechanism is mundane: the agent has the task and the context, so the most natural-sounding next sentence is one reporting success, and nothing in the architecture checks that claim against what actually occurred. It also covers invented parameters and calls to tools that do not exist.

What is the difference between action and outcome hallucination?

Action hallucination is a claim to have performed a step that does not appear in the execution trace. Outcome hallucination is a claim that the overall task is complete when the evidence shows otherwise. They often occur together and they are checked differently: action hallucination is caught by comparing the claim against the recorded tool calls, and outcome hallucination by verifying the state of the target system rather than reading the agent's summary of it.

Why does this failure have so many names?

Because research groups, platform teams and practitioners named it independently without reading each other. Tool-use hallucination, execution hallucination, action hallucination, outcome hallucination, procedural hallucination and functional hallucination all describe overlapping versions of the same thing. The practical consequence is worse than untidy: an engineer who hits it cannot search for it, because they do not know which phrase to type and whoever solved it wrote it up under a different one. Procedural and action hallucination are the two with formal published definitions.

Does RAG stop hallucination?

It reduces one kind and introduces another. Grounding an answer in supplied documents removes much of the room to invent facts absent from training data. It does not stop the model misreading the passage, drawing an unsupported conclusion, blending two sources into a claim neither makes, or attributing something to a document that does not contain it. It also raises perceived reliability faster than actual reliability, because an answer with a citation reads as verified and readers check citations far less often than they trust them.

What is a good hallucination rate?

The question does not have an answer as asked, because a hallucination rate is a property of a model and a benchmark together rather than of a model. The same model in the same configuration has scored under two percent on one factuality test and over ten percent on another. Neither figure is wrong and neither is a specification. Ask which benchmark, on what date, measuring what; a rate presented without those three is being used to persuade rather than to inform.

How do you detect hallucination in production?

Split the problem. Whether a factual claim is true requires judgement, needs a source of truth you often lack, and is best handled by sampled faithfulness scoring treated as a signal. Whether the tool call an agent claims to have made appears in the trace requires no judgement at all. That second class covers the agent-specific failures, is deterministic, is cheap enough to run on every request, and is the check most teams have not built.

Why should abstention be rewarded?

Because if your evaluation treats "I could not determine this" as a failure, you have recreated the exact incentive that produces hallucination during training. A system scored purely on answering will learn to answer. Count a correct refusal as a success in your own evaluation suite, instrument how often it happens, and treat a rising abstention rate on ambiguous inputs as a sign of good calibration rather than of declining performance.

Are hallucinations worse in agents than in chatbots?

The rate is not necessarily different; the consequence is. A chatbot's hallucination is a false sentence a person reads and can disregard. An agent's hallucination can become an action: a refund to the wrong customer, a record changed on an invented identifier, a fabricated software dependency introducing a vulnerability. Agents also chain steps, so a hallucinated fact at step one arrives at step four looking like established context rather than a guess.

Should hallucination stop you deploying AI agents?

No, but it should determine where you deploy them and what they can reach. Since the rate cannot be driven to zero, deploy where a false premise has bounded consequences: scoped permissions, approval on anything irreversible, deterministic verification that claimed actions actually occurred, and a trace complete enough to reconstruct what happened. An agent that drafts for review is a different risk proposition from one that executes with broad access, and the difference is design rather than model choice.

Check the trace, not the summary
Your agent says it did it. Did it?

CAMS records every model call and tool call per agent, so a claimed action can be checked against what actually executed, alongside the guardrails, spend ceilings and approval gates that bound what a false premise can reach.

Book a demo Explore CAMS