Your support bot confidently quotes a refund policy that was retired last quarter. The customer expects you to honor it, the support team has to untangle the promise, and engineering gets a request to replace the model.
That replacement might change the wording without fixing the mistake. If you're working out how to prevent AI hallucinations, start with the system that supplied the evidence, approved the answer, and decided whether the bot should respond at all. Zero hallucinations isn't a realistic production target. Detecting unsupported answers and containing their consequences is.
Table of Contents
- How to Build a Grounded Answer Pipeline Step by Step
- How to Measure Hallucinations With a Domain Test Set
- Diagnosing Where a Wrong Answer Came From
- Setting an Abstention Policy for High-Risk Answers
- Confirm the Pipeline Passes Before Launch and Monitor After
How to Build a Grounded Answer Pipeline Step by Step
For the refund bot, the fix starts before generation: exclude retired policies, retrieve the applicable version, and stop the answer if the evidence doesn't establish eligibility. A stronger prompt helps, but it can't make an obsolete document current.
Retrieval-augmented generation, or RAG, gives the model external evidence at answer time. A review published through ACM identifies more than 32 mitigation techniques across the model lifecycle, reinforcing why teams should select controls by risk rather than expect retrieval alone to solve factuality. The practical starting point is a retrieval-augmented generation architecture with explicit checks between stages.
Build the Evidence Path
Classify risk and answerability. Separate routine lookups, consequential advice, and requests the agent must never handle. Classify by the consequence of a wrong response, not just the topic. A billing question about opening hours and a billing dispute shouldn't receive the same treatment because both mention billing.
Retrieve from an authoritative, versioned corpus. Attach the document identifier, effective date, version, and access permissions to each passage. Index approved content, not an entire shared drive. Otherwise, drafts, duplicates, and superseded policies become indistinguishable from valid evidence.
Rerank before generation. Inspect whether the selected passages answer the request, including its product, location, and customer eligibility conditions. Don't treat the initial similarity score as an approval decision. Sample queries manually and read the leading passages, especially when similar policies apply to different groups.
Constrain generation to the evidence. In the prompt, require every factual claim to be supported by the supplied passages, prohibit filling gaps from memory, and instruct the model to identify missing coverage. Avoid demanding a complete answer regardless of evidence. Completeness is useful only when the sources support it.
Verify Before Releasing the Answer
Extract the draft's checkable claims. Create a structured list of policy conditions, entities, amounts, dates, and citation targets. Checking only the prose makes it easy to miss a plausible fee or an invented exception embedded in an otherwise correct response.
Verify each claim. Check support, applicability, and source freshness, using an independent authoritative record where available. Reusing the same context can establish whether the draft follows that context, but it can't independently establish whether the context is current or complete. Mark unsupported and contradictory claims explicitly.
Abstain or hand off. Hold the answer when required evidence is missing or conflicts. Say a bank's card-fee draft relies only on a superseded schedule. Route the question for review rather than presenting the old fee as current, and attach the conflicting evidence so the reviewer can resolve it.
The hallucination evaluation survey supports evaluating factuality across domains and treating evidence quality as part of the problem. A passage appearing in search results is not proof that the answer is correct.
Six hallucination controls, what each prevents and where it runs
| Control | Failure it prevents | Where it runs |
|---|---|---|
| Risk classification | Answering questions the system should not handle | Intake, before generation |
| Grounded retrieval | Answers drawn from stale or invented policy | Retrieval from versioned sources |
| Reranking | Relevant evidence buried under irrelevant passages | After retrieval, before generation |
| Evidence-only generation | Facts supplied from the model's memory | During generation |
| Claim verification | Unsupported names, numbers and citations | After the draft is written |
| Abstention and handoff | Forced answers when evidence is missing | Final output stage |
For manual review, Model Diplomat's guide to spotting errors in AI research answers is a useful companion to tracing claims back to their sources. Apply that discipline to support answers, not just research reports.
Run this check this week: Take a small sample of live answers and map every consequential claim to a supporting passage. Record missing support separately from incorrect support.
How to Measure Hallucinations With a Domain Test Set
A headline accuracy score hides the failures that matter. A 2024 study of legal question answering tested randomly selected, verifiable U.S. federal court cases and found hallucination rates ranging from 58% for ChatGPT-4 to 88% for Llama 2, while also finding that the models were poor at recognizing their own hallucinations.
Those results describe that legal evaluation, not your support agent's expected error rate. The operational lesson is to measure performance on your own questions and never substitute a model's self-assessment for verification.
Metrics for grading grounded answers
| Metric | What it measures | How to calculate it |
|---|---|---|
| Grounded accuracy | Share of answers whose claims match the evidence | Graded answers with all claims supported divided by total answers |
| Unsupported-claim rate | How often a claim has no backing in the evidence | Unsupported claims divided by total claims |
| Citation precision | Whether cited passages actually support the claim | Supporting citations divided by total citations |
| Abstention rate | How often the system declines or escalates | Abstentions divided by total requests |
| Latency | Time from question to answer | Final response timestamp minus request receipt timestamp |
Define the grading rules before comparing builds. Treat an answer as grounded only when its consequential claims are supported, and separately label whether it resolves the request. Otherwise, an empty but harmless response can look deceptively successful.
Build Cases That Expose Your Weak Spots
Start with anonymized questions from support logs, employee-service tickets, and escalation queues. Have domain owners write the expected answer, acceptable evidence, applicable policy version, and required route for each case.
Include deliberately difficult cases:
- Ambiguous requests. Omit the account, location, or product information needed for a reliable answer.
- Policy transitions. Ask questions where an old document gives a plausible but no longer applicable response.
- Adversarial requests. Pressure the agent to ignore evidence, reveal restricted information, or invent an exception.
- Channel and language differences. Include voice transcripts and multilingual phrasing, not just polished English chat.
Segment results by language, channel, intent, and risk. A multilingual voice failure shouldn't disappear inside an otherwise strong aggregate score. Broad public benchmarks can help identify missing categories, but they don't replace your operational cases.
Make conversational AI testing a release requirement. Freeze the approval set, keep it separate from prompt tuning, and rerun it after every prompt, index, or model change so regressions are visible before customers encounter them.
Diagnosing Where a Wrong Answer Came From
Did the model, the knowledge base, or the retrieval pipeline produce the error? That question should come before any proposal to upgrade the model.
NIST-hosted guidance on AI evaluation emphasizes evaluating the retriever and generator separately, then testing the complete pipeline, including ingestion, APIs, model integration, monitoring, and outputs. That separation gives engineers a repair target instead of a vague instruction to improve accuracy.
Hallucination symptoms, likely sources and first checks
| Symptom | Likely source | First check |
|---|---|---|
| Confident answer quoting an old policy | Stale knowledge base | Compare the passage's version date with the current policy |
| Correct document exists but is never cited | Retrieval ranking | Confirm the passage appears in the top results for that query |
| Detail appears in the answer but in no document | Generation ignoring evidence | Check whether supporting text was present in the model's context |
| Correct in English, wrong in Spanish | Multilingual retrieval gap | Rerun the query translated into English |
| Wrong answer only on voice calls | Transcription error | Compare the transcript with the audio |
| Two answers contradict each other | Duplicate or superseded documents | Search the index for duplicate or retired files |
| Errors start after an update | Model version change | Replay the regression suite on the previous version |
Treat these as hypotheses, not automatic diagnoses. Errors after an update could also follow an index rebuild or prompt edit deployed alongside it.
Preserve the Trace Before Changing Anything
Capture the original request, identity and permission context, retrieved passages, passages supplied to the model, tool results, and final answer. For voice, retain the relevant transcript and an appropriately protected audio reference.
Then follow the evidence. If the correct document exists but wasn't retrieved, inspect filters, chunking, query rewriting, and ranking. If it was retrieved but excluded from context, inspect context assembly; if it reached the model and the answer contradicts it, inspect generation and verification.
Check source applicability as well as the date. A current policy for the wrong jurisdiction is still the wrong evidence, and access-control failures require investigation even when the exposed answer is factually correct.
A larger model won't repair missing policy clauses or resolve contradictory documents unless the system establishes which source governs.
For financial workflows, Artul.ai's discussion of the limits of automated earnings summaries provides a useful parallel: reviewers need to inspect what happened between source material and summary, not judge fluency alone.
Correlate error reports with document refreshes, index changes, prompt edits, model versions, latency, and user corrections. Change the implicated layer, then replay the failing request before widening the rollout.
Setting an Abstention Policy for High-Risk Answers
Contained failure beats a promise of zero hallucinations. An abstention policy defines what the agent should do when it can't establish a safe, supported response, rather than leaving the decision to generated prose.
A human handoff should preserve the customer's progress, not force them to start again.

An Applause report announced through Business Wire reports that 40% of users experienced AI hallucinations in the measured year, up from 32% the prior year. Adoption alone isn't a quality control.
Define the Allowed Routes
- Clarify an ambiguous request. Ask which account or policy period applies before interpreting eligibility. Ask for the missing fact, not a generic request for more information.
- Answer from approved evidence. When the policy is clear, cite the applicable source and its version date. For a refund request, state the documented window without inventing exceptions.
- Use a deterministic workflow. For a rebooking, validate availability, permissions, and user confirmation through application logic. The model can explain the action, but it shouldn't invent successful execution.
- Transfer with context. For suspected fraud or unresolved high-risk evidence conflicts, attach the request, collected facts, and reason for escalation to the human queue.
Set the routing thresholds with the business owner. Banking, financial services, insurance, healthcare, utilities, and HR require different decisions because the consequences include financial loss, safety, privacy, and employment harm.
The same review boundary applies outside support. Teams using government proposal software, such as SamSearch, should distinguish drafting assistance from approval of consequential factual commitments.
Use the demonstration below as a discussion prompt for where an automated interaction should stop and a controlled workflow should begin.
Build human-in-the-loop automation around explicit reasons for intervention. Our policy is that users shouldn't rely on the factual accuracy of GenAI Services output and should evaluate it, including through human review where possible.
Measure unsafe actions, escalation precision, false refusals, and time to human resolution by intent and severity. An agent that refuses answerable questions also fails customers; safe routing must remain useful.
Confirm the Pipeline Passes Before Launch and Monitor After
A successful demonstration isn't a release decision. Approve the candidate build only when evidence shows that it handles ordinary requests and fails safely on consequential ones.
Require Observable Release Conditions
- The frozen test set passes. Report unsupported-claim rate, citation precision, and abstention rate by language, channel, and intent. Agree on acceptance thresholds before seeing the candidate results, and don't let strong routine lookups compensate for unsafe high-risk answers.
- Every high-risk route has been exercised. Test clarification, controlled transactions, and escalation using realistic requests. Verify that a handoff reaches a person with the conversation context, rather than merely checking that the bot displays a transfer message.
- Every document source has an accountable owner. Record who approves changes, how freshness is checked, and what happens when a source becomes overdue. A refresh schedule without an owner isn't an operational control.
Use a release record that identifies the prompt, model, index, corpus snapshot, verification rules, and routing policy. Without that record, the team can't reliably reproduce an approval result or restore a known-good configuration.
Replay, Review, Then Monitor
Run the frozen set against the candidate build and the last approved version. Have a domain expert review failed answers, prioritizing newly introduced failures and cases with serious consequences. Distinguish genuine improvements from changes that replace wrong answers with excessive refusals.
For the engineering workflow, our AI Copilot lets users preview, debug, and optimize AI agent performance, and we offer Automated Testing for AI agents. Keep the release decision tied to domain-specific evidence, regardless of the tools used to produce it.
Release gate: A failed consequential case needs an explicit disposition: repair it, disable the affected route, or require human handling before launch.
After launch, review a recurring sample of live answers and retain the same grading definitions used before release. Include escalations and user corrections in the sample so reviewers don't inspect only apparently successful conversations.
Set alerts against an agreed deterioration margin for unsupported claims, with separate treatment for severe incidents. Assign an incident owner and a containment action, such as disabling an affected intent or reverting a changed index. Log document refreshes, index changes, prompt edits, and model versions alongside the measurements.
Your launch rule should be straightforward: release only when the test set passes the agreed gates, and keep monitoring after deployment. Approval establishes a baseline, not permanent permission to operate unchanged.
Yellow.ai offers AI Copilot for previewing, debugging, and optimizing agent performance, alongside Automated Testing for AI agents. Visit Yellow.ai to explore how these tools can support an evidence-grounded testing and release workflow for your customer and employee service agents.