To reduce hallucinations in AI agents, find which layer is inventing things (retrieval, tool calls, planning or final generation), measure it on a fixed test set, then add the guardrail that fits that layer: grounded answers, validated tool calls, verified completion and permission to say "I don't know". Published studies show these fixes work. Chain-of-Verification more than doubled answer precision in one benchmark, and in OpenAI's tests a model that abstained when unsure made 26 percent errors against 75 percent for one that almost always guessed.
Agents hallucinate differently from chatbots. A chatbot that invents a fact gives a wrong answer. An agent that invents a fact may act on it: it books the wrong date, emails a made-up figure or reports a task as done when it failed. This guide is for teams building agents with LLMs and RAG. It breaks hallucination down by layer, shows how to build a test set and measure a hallucination rate you can trust, lists guardrail patterns with published before and after numbers, and sets out when an agent should hand off to a person.
Key Takeaways
- Diagnose by layer before fixing. Retrieval, tool calls, planning and generation fail in different ways and need different fixes.
- Measure with a fixed test set. A few hundred real cases, scored at the claim and trajectory level, show whether a change helps or hurts.
- Reward abstention. OpenAI's SimpleQA results show a model that abstains on 52 percent of hard questions makes 26 percent errors, against 75 percent for one that almost never abstains.
- Verification steps pay off. Chain-of-Verification raised list-question precision from 0.17 to 0.36, and better retrieval cut failed retrievals by up to 67 percent in Anthropic's tests.
- Some tasks need a person. Legal, medical, financial and irreversible actions should route to a human when evidence is thin.
Why AI Agents Hallucinate
A language model predicts likely text. When it lacks the right information, the likeliest text is often a confident guess. OpenAI's research on why language models hallucinate argues that training and evaluation reward this guessing, because accuracy-only scoreboards give no credit for "I don't know".
Agents add three more sources of error on top:
- More steps. Each retrieval, tool call and plan step is a chance to go wrong, and errors compound.
- More context. Long tool outputs and conversation history crowd out the facts that matter.
- Pressure to finish. Agents are prompted to complete tasks, which nudges them to claim success.
The US National Institute of Standards and Technology lists "confabulation" as a core risk in its Generative AI Profile of the AI Risk Management Framework, and the real-world cost is visible. A researcher-maintained database of court decisions involving AI hallucinations listed 2,190 cases as of October 2026, mostly fake legal citations.
Root Causes by Layer
This is the core diagnostic. Trace each bad output back to the first layer where the truth was lost.
| Layer | What goes wrong | Typical symptom | First fix to try |
|---|---|---|---|
| Retrieval | The right evidence is not found or not ranked high enough | Fluent answer built on the wrong document | Hybrid search, reranking, better chunking |
| Tool calls | Wrong tool, invented arguments, misread results, ignored errors | Action on a non-existent record, or "success" after a failed call | Strict schemas, argument validation, error surfacing |
| Planning | Skipped steps, wrong order, loops, goal drift, false completion | "Done" when a step never ran | Explicit plans, state checks, verify side effects |
| Generation | Claims not supported by context, invented numbers and citations | Correct evidence, wrong summary | Grounded-answer contract, citation checks, abstention |
Retrieval layer
Pixel's guide to RAG chatbots covers retrieval failures and the four levers that fix most of them, so this section stays brief. For agents the extra risk is that retrieved text drives an action, not just an answer. Anthropic's contextual retrieval experiments cut the top-20 retrieval failure rate by 35 percent with contextual embeddings, 49 percent when combined with contextual BM25, and 67 percent with reranking added. Fix retrieval first when the agent acts on documents.
Tool call layer
Tool hallucinations are the agent-specific problem. Common forms:
- Invented arguments. The agent calls
get_order(id="A1234")with an ID that never appeared in the conversation. - Wrong tool. It searches the knowledge base when it should query the order system.
- Fabricated results. After a timeout, it describes what the tool "would have" returned.
- Swallowed errors. A 404 or permission error becomes "I could not find any issues".
Better tool definitions help measurably. Anthropic reported that adding tool use examples raised accuracy on complex parameter handling from 72 to 90 percent in internal tests, and that its tool search feature lifted one model from 49 to 74 percent on large tool libraries.
Planning layer
Planning errors are hard to spot because every individual step can look fine. The agent skips a check, does steps out of order or loops between two tools. The most damaging form is false completion: claiming a task is done without confirming the result. The tau-bench study of tool-using agents found that even strong function-calling agents succeeded on fewer than 50 percent of tasks and were inconsistent across repeated runs, with pass^8 below 25 percent in its retail domain.
Generation layer
Even with the right evidence in context, the final answer can drift: a rounded number becomes a precise one, two sources merge into one claim, or a citation points to the wrong page. Model choice matters here. Vectara's hallucination leaderboard, which measures how often models add unsupported facts when summarising documents, put the best listed model at 1.8 percent, while widely used models such as DeepSeek-R1 scored 11.3 percent, as of its 22 September 2026 update. Domain matters even more: Stanford researchers found leading legal AI research tools hallucinated on more than 17 percent of queries, and one on more than 34 percent.
Build a Test Set and Measure Your AI Agent Hallucination Rate
You cannot reduce what you do not measure, and generic benchmarks do not predict your agent's behaviour. Build your own test set.
What goes in the test set
- 200 to 500 real tasks taken from logs, support tickets or user interviews, not invented ones.
- Known-answer cases where the correct output and correct tool calls are recorded.
- Unanswerable cases (10 to 20 percent) where the right behaviour is to abstain or ask.
- Adversarial and edge cases: missing records, tool errors, conflicting documents, long histories.
- Recent failures, added every week from production.
How to score it
Score at three levels, because each catches different hallucinations:
| Level | Question | How to score |
|---|---|---|
| Claim | Is each factual claim in the output supported by the evidence the agent had? | Split the answer into claims; check each against sources, by reviewers or a judge model calibrated on human labels |
| Tool call | Were the tool, arguments and interpretation of the result correct? | Compare the trace with the reference trace; flag any argument value not found in context |
| Trajectory | Did the task actually get done, and did the agent report the outcome truthfully? | Check the end state in the real system, not the agent's summary |
Then use an asymmetric score. Give +1 for a correct answer, 0 for an appropriate abstention and a penalty, say -2, for a wrong answer or false completion. This mirrors the argument in OpenAI's research: if wrong answers cost more than abstentions, the agent you choose will guess less.
Worked example: is the improvement real?
Your baseline agent hallucinates on 18 of 300 test cases, a rate of 6.0 percent. After adding a verification step, it hallucinates on 9 of 300, or 3.0 percent. Is that real or noise?
- Standard error at 6.0 percent: square root of (0.06 x 0.94 / 300) = 0.0137, so the 95 percent margin is 1.96 x 0.0137 = plus or minus 2.7 points
- Standard error at 3.0 percent: square root of (0.03 x 0.97 / 300) = 0.0098, so the margin is plus or minus 1.9 points
- The difference is 3.0 points. The standard error of the difference is the square root of (0.0137 squared + 0.0098 squared) = 0.0168, so the margin on the difference is 1.96 x 0.0168 = plus or minus 3.3 points
The 3-point gain sits inside the 3.3-point margin, so 300 cases cannot prove it. You either need more cases, or a paired comparison on the same cases, which is usually more sensitive. As a rough guide, detecting a drop from 6 to 3 percent with independent samples needs about 750 cases per arm. Many teams ship "improvements" that are noise.
Also run each case several times. Agents are not deterministic, and tau-bench's pass^k metric (success on all k attempts) shows reliability falling sharply as k rises.
Guardrail Patterns That Work
The practical answer to how to reduce hallucinations in AI agents is a set of guardrails, chosen by layer. Use the layer diagnosis to pick patterns. Most production agents need several.
Grounded-answer contract
Tell the agent to answer only from provided evidence, to quote or cite the source for each factual claim, and to say what is missing when the evidence is incomplete. Anthropic's guide to reducing hallucinations recommends explicitly allowing the model to say it does not know and asking it to extract direct quotes before answering.
Citation verification
Check every citation in code: the cited document must exist, the quoted text must appear in it, and the claim must match the quote. Reject or regenerate any answer that fails.
Tool argument validation
Validate every argument against a schema and against context. An ID the user never gave and no tool ever returned is a hallucination by definition, and the call should be blocked before it runs.
Error surfacing
Return tool errors to the agent in plain language ("order not found", "permission denied") and require the agent to report them, never paraphrase them away.
Verify side effects before claiming success
After an action, read the result back: fetch the booking, query the updated record, confirm the email was queued. Only then report completion.
Chain-of-Verification and self-checks
Have the agent draft an answer, write verification questions about its own claims, answer them independently and revise. The original Chain-of-Verification paper used this pattern.
Deterministic tools for maths and facts
Send arithmetic, dates, currency conversion and lookups to code or databases. Models should not do sums that a calculator can do.
Confidence and abstention thresholds
Route low-confidence cases to a person or a clarifying question. Tune the threshold on your test set using the asymmetric score.
Measured Before and After Results from Published Sources
These figures come from different tasks and models, so compare the size of each effect, not the absolute numbers.
| Technique | Setting | Before | After | Source |
|---|---|---|---|---|
| Abstention-aware model behaviour | SimpleQA, OpenAI models | 75 percent error, 1 percent abstention (o4-mini) | 26 percent error, 52 percent abstention (gpt-5-thinking-mini) | OpenAI, September 2025 |
| Chain-of-Verification | Wikidata list questions, Llama 65B | Precision 0.17, 2.95 hallucinated items per answer | Precision 0.36, 0.68 hallucinated items | Dhuliawala et al., 2023 |
| Chain-of-Verification (factor and revise) | Long-form biographies, FActScore | 55.9 | 71.4 | Dhuliawala et al., 2023 |
| Contextual retrieval with reranking | Top-20 chunk retrieval | 5.7 percent failure rate | 67 percent fewer failures | Anthropic, September 2024 |
| Tool use examples | Complex tool parameters | 72 percent accuracy | 90 percent accuracy | Anthropic, November 2025 |
| Tool search | Large tool libraries, Opus 4 | 49 percent | 74 percent | Anthropic, November 2025 |
Two patterns stand out. First, letting the model abstain cuts errors more than any prompt trick. Second, verification and better tool definitions each deliver double-digit gains. Notably, abstention lowered OpenAI's accuracy slightly (24 to 22 percent) while cutting errors by about two thirds, which is the right trade for most business agents.
Hallucinations in RAG vs Agentic AI
The fixes overlap but the risks differ.
| Aspect | RAG assistant | Agentic AI |
|---|---|---|
| Main harm | Wrong answer shown to a user | Wrong action taken in a real system |
| Where errors start | Retrieval and generation | Retrieval, tool calls, planning and generation |
| Key metric | Faithfulness and answer correctness | Task success, false completion rate, invalid tool calls |
| Most useful guardrail | Grounded answers with citation checks | Argument validation and side-effect verification |
| Human role | Review samples and escalations | Approve risky or irreversible actions |
For RAG systems, Pixel's RAG development services start from a retrieval assessment. For agents, attacks such as prompt injection can also cause false outputs, and those are covered separately in Pixel's guide to AI agent security.
When to Hand Off to a Human
No technique reduces hallucinations to zero, so plan the handoff. Use this checklist when you design an agent's escalation rules.
Hand off when any of these is true:
- The evidence does not support an answer, or sources conflict.
- The action is irreversible: payments, cancellations, deletions, legal notices.
- The topic is regulated advice: legal, medical, tax, investment.
- A tool returned an error the agent cannot resolve in one retry.
- The user disputes the agent's answer or asks for a person.
- Confidence falls below the threshold you set on your test set.
- The task exceeds a cost or step budget, which often signals a loop.
Make the handoff useful. Pass the person a short summary, the evidence found, the tools called and what remains unknown, so they do not start from scratch.
The cost of skipping this is real. In Moffatt v. Air Canada (2024), a Canadian tribunal held the airline responsible for incorrect refund information its website chatbot gave a customer, rejecting the argument that the chatbot was responsible for its own statements.
A 30-Day Plan to Reduce Hallucinations in AI Agents
- Week 1: measure. Build the test set, label 200 or more cases and record the baseline at claim, tool call and trajectory level.
- Week 2: fix the worst layer. Use the layer table to find where most failures start. Fix retrieval or tool definitions first, because those gains are large and cheap.
- Week 3: add guardrails. Add argument validation, error surfacing, side-effect checks and an abstention path.
- Week 4: verify and govern. Rerun the test set with paired comparisons, set release thresholds, and add the agent to your model register under your AI governance framework.
Repeat the test run on every model, prompt, tool or data change. Hallucination rates regress silently when any of those move.
How Pixel Web Solutions Can Help
Pixel Web Solutions builds and hardens production AI agents, with test sets, guardrails and human handoff designed in from the start. Its AI agent development services include task-level evaluation, tool schema design and supervised rollout, while its AI consulting services suit teams that want an independent review of an existing agent's failure rates. Pixel operates from Ras Al Khaimah, UAE and Madurai, India. To have your agent's hallucination rate measured against your own tasks, contact the Pixel Web Solutions team.
Frequently Asked Questions
Can AI hallucinations be eliminated completely?
No. Language models generate likely text, so some error always remains. You can make hallucinations rare and harmless by grounding answers, validating tool calls, verifying results, rewarding abstention and handing risky cases to people.
What is a normal AI agent hallucination rate?
There is no universal figure, because rates depend on the task and how you count. Document summarisation benchmarks range from under 2 percent for the best model to over 10 percent for some widely used ones, while specialist legal research tools were measured above 17 percent. Measure your own rate on your own tasks.
How do you reduce hallucinations in LLM outputs without changing the model?
Give the model the right evidence, require citations and quotes, let it say "I don't know", ask it to verify its own claims, use code for maths and lookups, and check outputs before users see them. Each of these works with any model.
Does RAG stop hallucinations?
RAG reduces them when retrieval finds the right evidence, but it does not stop them. Models can still misread or override correct context, and poor retrieval produces confident wrong answers. Measure retrieval and generation separately.
How do you reduce hallucinations in agentic AI specifically?
Add controls that chatbots do not need: validate every tool argument against context, surface tool errors, verify side effects before claiming success, cap steps and cost, and require human approval for irreversible actions.
How do you measure hallucinations in an AI agent?
Use a fixed test set of real tasks, including unanswerable ones. Score claims against evidence, tool calls against reference traces, and outcomes against the real system state, then compare versions on the same cases with enough samples to rule out noise.
Which model hallucinates least?
Rankings change with every release and differ by task. Public leaderboards such as Vectara's are useful for summarisation, but the best choice for your agent is the model that scores best on your own test set at an acceptable cost.
Knowing how to reduce hallucinations in AI agents comes down to discipline more than clever prompts: diagnose the failing layer, measure on real tasks, add the guardrail that fits, reward honest uncertainty and keep a person in the loop for the decisions that matter. Rerun the test set on every change, and the numbers will tell you whether the agent is getting safer.
