A refund agent replies in a calm, correct tone, then the customer sees two refunds on the card. The second one came from a retry that fired after the first write had already succeeded. Your scripted test still passes, because the final message looks right and the ticket closes.
That gap is the whole problem with agentic AI testing. The same request can take a different path on the next run, pick a different tool, or write twice. A single pass or fail on the last reply hides the bad step in the middle.
Table of Contents
- The Six Layers of Agentic AI Testing at a Glance
- How to Plan Scenarios and Set Risk Tiers
- Unit and Functional Tests for Tools, Permissions and Decisions
- Conversational End-to-End Testing Across Paraphrases and Repeated Runs
- RAG and Knowledge Validation Checks
- Failure Injection, Recovery and Metrics That Gate Release
- Choose Test Tools and Enterprise Practices
- Set Up Regression and Drift Checks Before You Ship
The Six Layers of Agentic AI Testing at a Glance
Traditional suites assume one input maps to one path. Agents don't work that way. They plan, call tools, and change state, so a polite close can sit on top of a duplicate write, a wrong record, or a skipped permission check.
AgentBench, introduced in 2023, treats models as agents across 8 distinct environments, including operating-system commands, databases, knowledge graphs, and web browsing. Its overall success rate is tasks completed successfully divided by total tasks. Its step success rule is stricter. A step counts only when both the selected target and the predicted operation match the ground truth. A bad intermediate action is a failure even when the final response reads well.
| Layer | What It Checks | Core Metric | When It Runs |
|---|---|---|---|
| 1. Tool and unit tests | Tool arguments, schemas and permission checks | Invalid-call rate | Every commit |
| 2. Functional task tests | Whole task outcome against each scenario | Clean task-success rate | Every commit and nightly |
| 3. Conversational end-to-end | Multi-turn paths, paraphrases and channels | Pass rate across repeated runs | Before each release |
| 4. RAG and knowledge validation | Retrieved evidence and grounded answers | Grounded-answer rate | Each knowledge base update |
| 5. Failure injection and recovery | Timeouts, bad data and permission gaps | Recovery rate and unsafe-side-effect rate | As a release gate |
| 6. Regression and drift | Behavior after model, prompt or tool changes | Pass-rate change against baseline | Each change and weekly in production |
Tool and unit tests catch bad arguments and permission misses before a conversation starts. Functional task tests ask whether the scenario actually finished. Conversational end-to-end checks that paraphrases and channels don't wander off the allowed path. RAG and knowledge validation asks whether the cited policy is the current one. Failure injection asks whether the agent recovers without a harmful write. Regression and drift asks whether last week's safe behavior is still safe after a model, prompt, or tool change.
Ship layers 1 to 3 before any launch. Add layers 4 to 6 when the agent retrieves knowledge or writes to a system of record. How those layers sit on tools and channels is easier to see against an agentic AI platform architecture.
Practical rule: If the trace shows a wrong tool, a second write, or a skipped check, fail the run even when the customer-facing text is fine.
How to Plan Scenarios and Set Risk Tiers
A refund can show the right amount and still be a failed run if the agent wrote the ledger before the cap check. Start from that workflow: the user, the system of record, and every action that is allowed before the final reply.
NIST's AI evaluation guidance puts scenario definition and KPI selection first, and its agent work calls for adversarial probes that leave a machine-readable audit trail.

Six steps before you write a case
- Write the use case. Name the sector, the user types, and the intended outcome, so a refund flow is not mixed with a status lookup.
- List impacts and KPIs. Note who is helped and who is harmed, then attach resolution rate, escalation quality, and compliance violations.
- Assign a risk tier. Severity of a bad write sets the tier, not how often the intent appears.
- Build the matrix. Cover normal requests, no-tool controls, ambiguous intent, wrong-tool temptation, missing permissions, partial tool failure, conflicting data, and repeated execution.
- Vary the run. Execute each scenario across several seeds, paraphrases, and user-authority levels, plus controlled faults such as timeouts, malformed responses, stale records, and duplicate requests.
- Record the trajectory. Keep prompts, model decisions, tool calls with arguments, ordering, retrieved evidence, state changes, latency, and the final response.
Score outcome correctness, action and tool correctness, policy compliance, and recovery quality separately. A final answer can look right while the agent made an unauthorized write. Tool-using agents can satisfy the goal and still perform unauthorized writes, and that risk rises on ambiguity-heavy tasks.
I do not use one global accuracy target for that reason. A wrong tool call, a second write, or a skipped check fails the run on its own. High-risk customer and employee actions need authorization-boundary tests and a human-approval gate before anything irreversible. Tie the tiers to your enterprise AI governance rules, then agree the starting thresholds below with the business owner. Frequency of the intent does not buy a looser bar.
| Risk Tier | Example Workflows | Retry Tolerance | Unauthorized-Action Limit | Rollback Requirement |
|---|---|---|---|---|
| Low: informational chat | FAQs and product details | Retries allowed | Read-only, zero writes | Logging only |
| Medium: account changes | Rescheduling and address updates | One controlled retry | Zero unauthorized writes, dry run first | Verified rollback |
| High: money, health or HR | Refunds, claims and leave requests | No silent retries | Zero tolerance, human approval required | Tested rollback and audit trail |
Informational chat may retry because a bad answer does not change a record, and it must never write. Account changes get one controlled retry, a dry run, and a verified rollback: a second address write is a fail even if the customer later sees the correct street. Refunds, health, leave, and claims get no silent retry. A person approves the action before it commits, and the audit trail has to show the check that blocked or allowed it.
The walkthrough below turns a live workflow into that matrix.
Unit and Functional Tests for Tools, Permissions and Decisions
Component tests fail fast, before a conversation harness is involved. Build them in the order a developer would add a write tool.
Schema, then temptation, then identity
Schema checks come first. Valid arguments pass. A missing field, or a string where a number belongs, is rejected, and the agent must recover instead of guessing a value.
Wrong-tool tests offer a tempting but incorrect tool. If someone requests a booking cancellation, a status-lookup tool sitting in the same registry should not be called.
Permission boundaries need at least two identities. A customer must not reach another customer's booking. A junior agent must not approve a refund above their limit, even when the prompt insists.
Writes you can see without committing them
Put every write tool behind a dry-run mode so the test sees the intended action and the arguments without executing them. Then test idempotency. Send the same refund request twice and assert that only one write is planned.
Rollback is the next case. Force a failure after a partial write and confirm the record returns to the starting state. Parser tests feed malformed tool output, such as JSON with a missing closing bracket or a null balance, and expect a controlled stop rather than a guessed amount.
Say the policy caps a refund at a fixed limit. The user demands more. The test should assert both the block and the explanation, not only a friendly sentence.
Keep this suite under a few seconds per case so it runs on every commit. A slow permission test gets skipped the week you need it.
Conversational End-to-End Testing Across Paraphrases and Repeated Runs
One clean transcript is a demo, not a release signal. A delivery agent can answer "Where is order 4412?" correctly, then mishandle "hey wheres my stuff, order was 4412 i think, can you send it to my office." The intent is the same. The path is not.

ReliabilityBench, published in 2025, ran 1,280 episodes across scheduling, travel, customer support, and e-commerce, using models and architectures including Gemini 2.0 Flash, GPT-4o, ReAct, and Reflexion. With no perturbation, agents reached a 96.9% pass@1 rate. At perturbation level 0.2, that fell to 88.1%, an 8.8-percentage-point drop. A script that looks production-ready on the happy path gets worse when wording, data, or the environment shifts.
How to build the variation layer
- Paraphrase the intent. Write 10 to 20 variants, including typos, partial requests, slang, and reordered details.
- Script the branches. A customer asks about delivery, changes the address, then asks for a refund. Assert state after every turn, not only at the end.
- Use personas and channels. A frustrated repeat buyer and a non-native speaker should run the same flows on every channel and language you support.
- Repeat the run. Ten executions of one scenario, then report the pass rate and the spread. One green result isn't the metric.
Add incomplete requests, altered tool outputs, timeouts, and dependency failures. In CX and EX work, a small change in wording, identity data, permissions, or backend availability can send an autonomous agent toward the wrong action. If the spread between runs is wide, you don't have a stable agent yet, whatever the best run scored.
RAG and Knowledge Validation Checks
A fluent answer can quote the wrong policy. Conversation tests often miss that, because the sentences still sound helpful.
Retrieval, freshness, conflict, and silence
Retrieval. On a labelled question set, check whether the right document lands in the top results. Track recall at the top five and the top ten, and review misses by hand. A miss here becomes a confident wrong answer later.
Freshness. Say the refund window moved from 30 days to 14 days and the old page is still indexed. The agent should cite the new policy. Fail the case if it quotes 30 days, even when the tone is perfect.
Conflict. Two documents disagree on a fee or an eligibility rule. The agent should apply the newer authoritative source, ask a clarifying question, or escalate. Picking one without explanation is a fail.
Unanswerable questions. If the knowledge base has no answer, the correct behavior is an honest refusal or a handoff, not an invented rule.
Grounding. Trace each claim in the reply back to a retrieved chunk. For every knowledge update, report a grounded-answer rate, a citation accuracy rate, and a stale-source rate. Block the release if any of those worsen.
A retrieval-augmented generation setup only stays trustworthy if those three rates are release criteria, not a quarterly audit.
If you can't point at the chunk, the sentence doesn't ship.
Failure Injection, Recovery and Metrics That Gate Release
A booking agent reschedules the right flight and writes a confirmation the customer can read. The trace still charged the old fare and never called the refund tool. I treat that run as a fail. The final sentence looked fine. The intermediate action did not.
Put failure injection on the release gate. A stratified suite should cover normal requests, no-tool controls, ambiguous intent, wrong-tool temptation, missing permissions, partial tool failure, conflicting retrieved data, and repeated execution. Inject timeouts, malformed responses, stale records, duplicate requests, and missing permissions, then score the full trajectory, not only the last reply.
Replay each recorded tool trace on its own, against a clean environment. A contaminated benchmark state inflates the score. A weak evaluator can mark a run successful while an independent replay of the same trace shows the agent never got back to a safe path. Harness integrity is part of the test.
Count recovery only when the agent diagnoses the fault and takes a valid alternative path. An apology that leaves the customer on the broken path still fails. Under basic execution faults, recovery is often the weak point, even when the model is strong. On a deterministic tool-misuse benchmark, parser-and-repair logic moved success from 15.8% without recovery to 76.7% with that logic, and conditional recovery success was 73.6%, as reported in this failure-injection study.
Gates for money, health, HR, and account changes
Informational chat can absorb a retry. Financial, healthcare, HR, and account-mutating workflows cannot. Those flows need near-zero unauthorized actions and a verified rollback. I would not relax the zero on unsafe writes to chase a higher task-success number. Set a separate pass line for each risk tier instead of one global accuracy target.
| Metric | Definition | Suggested Gate |
|---|---|---|
| Clean task-success rate | Episodes that pass when the recorded trace is replayed independently | Track against baseline before each release |
| Tool-call precision | Correct tool calls divided by all tool calls | Track against baseline per workflow |
| Invalid-call rate | Calls with malformed or disallowed arguments | Below 2% |
| Recovery rate | Correct diagnosis plus a valid alternative path after a fault | 90% or higher on high-risk workflows; a lower bar only for informational chat |
| Unrecoverable-failure rate | Episodes ending without a safe outcome | Below 1% |
| Unsafe-side-effect rate | Runs with unauthorized or irreversible writes | Zero |
| Step success | Correct target and operation at each step, per AgentBench | Trend per workflow, no fixed gate; a wrong step fails the case even if the final reply is correct |
Use the 2%, 90%, 1%, and zero figures as starting points on high-risk workflows. Informational flows can sit lower on recovery and still ship. Trend step success per workflow. A correct final sentence does not clear a wrong intermediate action. Agree any exception with the business owner in writing before the pipeline will pass it.
Choose Test Tools and Enterprise Practices
Pick one harness and one trace store before you add more tools. Extra dashboards don't fix a contaminated replay.
| Tool | Role | When It Runs |
|---|---|---|
| pytest | Test harness for tool, permission and rollback tests | Every commit |
| Playwright | Drives the chat widget in a real browser | Before each release |
| promptfoo | Prompt-level regression | Each prompt change |
| DeepEval or Ragas | Output quality and grounding scores | Each knowledge update and release |
| Langfuse or LangSmith | Trace storage and review | Continuous in production |
| Yellow.ai Automated Testing | Single-interface testing before deployment | Before deployment |
pytest belongs on the commit path because schema, permission, idempotency, and rollback checks are deterministic and fast. Playwright drives the chat widget before release, which catches channel bugs a unit test never sees. promptfoo diffs prompt behavior on each prompt change. DeepEval or Ragas score output quality and grounding after a knowledge update. Langfuse or LangSmith stores the full trajectory so a reviewer can open a failed run and see the tool arguments, not only the reply.
Yellow.ai's Automated Testing is one way to run these checks in a single interface before deployment, next to the open-source tools above. AI agents can be optimized through automated testing in that interface, and AI Copilot lets you preview, debug, and optimize performance. Use whichever set your team can run and audit.
Practices that make the numbers believable
- Name an owner for each risk tier and its gates, so a red metric has a person, not a shared inbox.
- Keep replay separate from the environment that produced the traces. Shared state is how a contaminated run looks healthy.
- Block the pipeline on the gates. A dashboard someone might check is not a gate.
Don't buy a fifth scorer until those three are in place. Another model-as-judge won't catch a second refund if nobody owns the unsafe-write gate.
Set Up Regression and Drift Checks Before You Ship
A workflow can pass today and fail after a prompt edit, model switch, CRM schema change, knowledge-base update, integration outage, or policy revision. Keep a regression suite of production failures and rerun it after each change.
A lifecycle study identifies six distinct drift categories and tests a production SRE agent swarm spanning 108 tools and 53 alerts across nine categories. The lifecycle evaluation study shows why a fixed golden set is not enough: static validation, sub-agent auditing, and full-trace observation produced materially different pass rates. Each evaluator can miss a different failure mode.
What to watch after launch
Correlate traces with resolution rate, escalation quality, compliance violations, latency, tool failures, and cost. These signals can expose drift that scripted chats miss, such as a changed authorization scope, altered API behavior, or incomplete customer data.
Release rule: Set separate pass thresholds by risk tier. Block a release if a high-risk pass rate falls below its gate, or if any unsafe side effect appears. A correct final answer does not make a wrong intermediate action safe.
One week, one workflow
- Pick one workflow that already has real traffic.
- Write its scenario matrix and assign risk tiers with the business owner.
- Run tool tests, functional task tests, and conversational repeats before adding more layers.
- Set the gates, record the baseline, and freeze it before the next model or prompt change.
Use that baseline to assess the next change. Without one, teams cannot distinguish a real regression from ordinary variation.
Yellow.ai offers Automated Testing to test AI agents in a single interface before deployment. AI Copilot lets users preview, debug, and optimize agent performance. Explore Yellow.ai.