Your Voice Agent Passed the Compliance Test and Leaked the Verification Data Anyway
The caller failed verification. Your voice agent correctly declined to change the card. It even declined twice, holding the policy line under pressure. Then it told the caller which field was wrong.
Not the account details. Not the card number. Just one sentence: the name on file did not match. The caller now knows their fabricated name was the problem, not the address. They have everything they needed to try again with better information. The refusal was correct. The conversation was a security breach.
Researchers at Veris AI published a benchmark in July 2026 that can see this failure for the first time. Their system, VAmoS Bench, ran 3,300 simulated phone calls across eleven voice agent stacks on a stateful card-operations task, grading not just what each agent said or what it wrote to the database but whether those two things actually agreed with each other. Across 3,764 individual assertion criteria, they identified at least 27 explicit say/do mismatches spanning 10 of 11 agents tested. The confidentiality failure rate on verification-probe scenarios ran between 60.6% and 66.7%. Agents are passing the security theater half of the problem and failing the security substance half, and every benchmark running today cannot tell the difference.
The Benchmark Every Contact Center Has Been Missing
The contact-center industry has a clear unit of measure: containment. What share of inbound calls does the automated system resolve without transferring to a human? It sounds like a single question. It is actually two questions that existing benchmarks treat as one.
Did the agent say the right thing? Did the agent do the right thing? Transcript-based evaluation answers the first. Final-state database evaluation answers the second. Neither reads both channels together, which means neither can catch an agent that announced a card freeze while executing a cancellation, or one that stated a delivery timeline that contradicts what the replacement tool actually returned.
VAmoS Bench introduces what the authors call joint trace grading. Every phone call produces a unified evidence record: the full conversation transcript, every tool invocation, every argument passed to those tools, and every row the tools returned. A single assertion criterion can then span all of it simultaneously. The paper's framing is precise: [DIRECT QUOTE] "A transcript can contain 'your card is frozen' even though no tool invocation was made, or a correct refusal after the agent has already disclosed account data." The benchmark is built to catch both.
The technical mechanism is straightforward in concept and demanding in implementation. Each of 100 scenarios seeds a containerized PostgreSQL backend with unique customer and card records, spins up an isolated simulation environment, and connects a synthetic caller over WebSocket at 24 kHz PCM16 audio. The caller has a private goal the agent cannot see. The agent, named Riley, has five real card-operations tools that execute actual SQL against that live database. When the call ends, an LLM judge reads the complete joint trace and applies that scenario's natural-language assertions. All assertions must pass for the phone call to count as complete.
A promise to freeze or replace a card earns no credit without evidence that the corresponding operation actually succeeded.
What Gets Measured When the Transcript and the Database Must Agree
The 100 scenarios divide into three analysis groups, and the distribution of failures across them tells the story the industry has been missing.
| Scenario Group | Pooled Completion | What It Reveals |
|---|---|---|
| Simple (30 scenarios) | 65.8% | Baseline single-step task handling across all stacks |
| Complex (38 scenarios) | 52.2% | Multi-step workflows; weakest group for all eleven agents |
| Adversarial (32 scenarios) | 73.4% | Caller attempts bypass, pressure, or social engineering |
The adversarial number looks reassuring until you read what is inside it. Agents pass the adversarial group at 73.4% overall because they are reasonably good at refusing to act: the branch-manager bypass scenario passed at 33 of 33 phone calls across all agents, and spoken prompt injection passed at 32 of 33. Agents have learned to hold policy lines against named social engineering gambits.
What they have not learned is to hold information lines while doing it. Scenarios that probe for confidentiality after a correct refusal tell a different story. Repeated-failed-verification scenarios: 33.3% pass rate, meaning a 66.7% failure rate. Probes about which verification field was wrong: 39.4% pass rate, meaning 60.6% failure rate. Probes for name or address after a refusal: 33.3% pass rate. The agents refuse the card action. Then they answer the follow-up question. The paper names this directly: "an agent that correctly declines to act for an unverified caller will still tell that caller which verification field was wrong, which is the oracle the attacker actually wanted."
This is not a fringe edge case. It is a 60-to-67% failure rate across a field of eleven agent stacks, and it would be invisible to any evaluation that only checks whether the agent took the right database action.
What Twenty-Seven Mismatches Look Like in Practice
The say/do mismatch category is the novel failure type this benchmark exists to surface. Of the 27 identified mismatches across 10 of 11 agents:
- 14 were claims with no supporting tool evidence at all, meaning the agent stated a completion that no tool invocation records
- 9 were premature or misstated statements about existing system state, where the agent described a card's status incorrectly relative to what the tools returned
- 4 directly contradicted a returned tool result, where the agent said one thing and the database said another in the same call
The paper provides one concrete example: an agent tells the caller it canceled a pending replacement and ordered a new one, while the trace holds a single request-card-replacement invocation and no cancellation at all. The call sounds like a resolution. The database reflects a different transaction entirely. A customer calling back the next day would find a situation inconsistent with what they were told.
Policy-step drift compounds this. The card-operations policy requires a freeze before a replacement, in that order. When a caller explicitly demands replacement without the freeze step, agents succeed on 1 of 33 phone calls: a 97% failure rate. When an impatient caller simply tries to hurry past the offer without explicitly demanding bypass, that falls to a 60.6% failure rate. The agents fail these calls not because the conversation breaks down but because, as the paper states, "the required action order disappears under pressure, even when the conversation remains fluent."
How the Eleven Stacks Actually Performed
Completion rates across the full benchmark ran from 43.0% (Nemotron, all-NVIDIA self-hosted) to 71.0% (Pipecat, self-hosted open-source with GPT-4.1-mini). The top of the leaderboard clusters tightly: Pipecat at 71.0%, LiveKit Agents at 70.3%, and Vapi at 69.3% cover a 1.7-point spread. The paper is explicit that per-run variability of up to 17 percentage points for some agents means those neighboring positions are not statistically distinguishable. This is a snapshot of observed performance, not a ranking.
Three comparisons carry genuine signal:
- Pipecat and LiveKit Agents run the same model stack under the same prompt against the same scenarios. Their completion difference is 0.7 percentage points. The orchestration layer alone does not determine outcome.
- OpenAI Realtime and OpenAI Realtime mini show a 16-point completion gap (67.0% vs. 51.0%) at roughly double the cost for the full model ($0.074 vs. $0.032 per call). Here the model tier clearly matters.
- Gemini 2.5 Native Audio and Gemini 3.1 Flash Live are nearly tied on completion (64.0% vs. 62.3%) but Gemini 2.5 carries a median response latency of 15.95 seconds versus Gemini 3.1's 1.38 seconds. In a live phone call, a 16-second pause is not a product characteristic. It is a call abandonment event.
The self-hosted open-source options being cost-competitive with managed platforms is worth noting. Pipecat at $0.045 per call and LiveKit Agents at $0.048 per call both out-complete every managed platform in the comparison except Vapi (which trails by 0.3 points at $0.135 per call). Those self-hosted figures exclude deployment infrastructure and operations, so they are not a direct comparison, but the gap in task completion does not favor the more expensive options.
Retell deserves specific mention not for completion (61.3%, mid-field) but for operational behavior: 126 barge-ins per 100 phone calls. That is more interruptions than calls, meaning the agent is cutting off callers more than once per conversation on average. Endpointing sensitivity of that magnitude is a first-class product quality signal independent of completion rate.
What Regulated Industries Are Actually Buying When They Buy a Voice Agent
Consider a regional bank evaluating voice agent deployment for card-service calls. The product demos show fluent conversations, low latency, and clean call flows. The vendor's own metrics report high intent recognition and low transfer rates. None of those metrics tells the bank whether the agent is leaking verification information to callers who fail authentication.
Under the VAmoS Bench protocol, the relevant question is whether the agent passes confidentiality assertions after a correct refusal. For the eleven stacks tested, that category fails at rates between 60% and 67%. A bank processing, say, 50,000 card-service calls per month would see tens of thousands of interactions where a caller who failed verification learns which field was wrong. In most jurisdictions, that information disclosure creates compliance exposure that the authentication step was designed to prevent.
The benchmark's scope is one 100-scenario banking task, so no general ranking is established. But the methodology is the point. The banking contact center that evaluates voice agents only on call containment rate and average handle time is measuring whether the agent sounds like it resolved the call, not whether the call was resolved correctly and securely.
The same gap applies to healthcare intake, insurance claims, and any workflow where authentication gates both action and disclosure. Regulated industries purchasing voice agent infrastructure need an evaluation protocol that reads both the conversation and the execution trace together. Without it, they are buying containment numbers that are silent on whether the contained calls were handled correctly.
What to Do Before the Next Procurement Conversation
This paper is a benchmark study, not a deployment manual. It does not prescribe which stack to buy. What it does is change what the procurement conversation should require.
First, any voice agent vendor should be able to provide task-completion metrics that combine transcript and execution trace grading. A completion rate derived from final database state alone, or from conversation quality alone, is insufficient for regulated use. If the vendor cannot produce joint trace evaluation, the buyer does not know what they are buying.
Second, confidentiality and refusal are not the same capability and must not be scored together. An agent can pass every adversarial scenario by refusing correctly and still fail every confidentiality scenario by disclosing verification structure. Ask specifically: what does the agent say immediately after a failed verification attempt, and what does that tell the caller?
Third, the scenario group that exposed the widest performance gaps across all eleven stacks was complex multi-step workflows at 52.2% pooled completion. Before any deployment, the specific task sequences the agent will handle, including the procedural order requirements under conversational pressure, must be in the evaluation set. If those scenarios are not tested, policy-step drift will only appear in production.
The industry has been measuring voice agents the way you would measure a customer service representative by reading a transcript of their calls without checking what they actually did in the system afterward. The gap between those two records is where the compliance risk, the operational errors, and the security exposures live.
Agents Applied covers research that changes how senior leaders think about AI deployment. New issues every week.