How to Test a Voice AI Agent (When the Transcript Isn't Ground Truth)
Every voice agent testing platform grades how well the agent talked. A second model decides what the conversation was worth, and nobody tests that one.
Somewhere in your conversation logs is a prospect who asked a real question. They named the system they were replacing, said the budget was locked until the next quarter, and asked what your team actually does. Nobody on your side has ever seen it.
The voice agent handled that conversation well. Latency stayed low, it did not talk over anyone, the transcription was clean. Then a second model read the transcript, decided the exchange was thin, and filed it under an outcome that notifies nobody.
Every voice agent testing platform you could buy would have graded that conversation a pass.
What voice agent testing measures today
The tooling in this space has matured fast, and it measures real things. Hamming's evaluation framework covers eight categories: latency percentiles, word error rate, interruption handling, task completion, containment and escalation, hallucination, compliance, and production health. Coval, Cekura and Maxim grade broadly the same surface.
These are hard problems and the vendors solve them properly. Turn-level latency, barge-in detection within a couple hundred milliseconds, recovery after a caller cuts the agent off: none of that is easy to instrument yourself.
Look at what the eight have in common, though. Every one of them grades the agent's conduct during the conversation. How fast it responded, how cleanly it heard, whether it stayed inside its instructions.
None of them grade what happened after the conversation ended.
The second model
A production voice pipeline runs two models, not one.
The first is the agent itself, the model holding up the real-time exchange. That is the one every testing platform is built to examine.
The second runs after the audio stops. It reads the transcript, assigns the conversation an outcome, and a routing layer acts on that outcome. In our own setup a short list of outcomes pings the team, a couple more land silently in a follow-up queue, and everything else ends there.
That second model is doing the commercially consequential work. The first one determines whether the conversation went well. The second determines whether anyone acts on it.
It is also, in most stacks, the one nobody tests.
Why this error is invisible by construction
Consider the two ways that classifier can be wrong.
When it flags a conversation that turns out to be nothing, someone opens it, sees an empty exchange, and is mildly irritated. The system made a claim and a human disagreed. That is a loud error, and it self-corrects because a person looked.
When it decides a real conversation was nothing, no alert fires. No row appears in a queue. Nobody opens anything, because the entire point of the classifier is deciding what deserves opening.
The error and the alert are the same decision. There is no dashboard panel titled "discussions we wrongly ignored," and there structurally cannot be one, because the system would have to already know the answer to populate it.
This is not a quirk of our pipeline. Any classifier tuned to avoid embarrassing false alarms skews the same way: it discards correct items rather than admits incorrect ones, and ends up high-precision at the expense of recall.
The standing advice is to treat the judge as one more prediction rather than as ground truth. That advice is rarely applied to the model grading your conversations.
You can watch the industry make the choice explicitly. Hamming publishes an LLM grader rubric for voice conversations, the nearest published thing to what we are describing. Its holdout gate is that "high-risk false passes are near zero." It measures human-to-model agreement without separating precision from recall, and it says nothing about conversations wrongly graded as low signal.
That is a deliberate, defensible trade. It is tuned to catch the agent saying something it should not have said. It is not tuned to catch a genuine discussion the pipeline decided was noise.
The shape of this will be familiar if you have watched AI-generated code sail through a green test suite and break in production. A verification step that only knows how to confirm what it was built to look for will keep reporting that everything is fine.
Four ways a pipeline loses a real conversation
We went looking for that second error in our own system. These are the mechanisms we found, and none of them look like bugs when you read the code.
A category with no way out. Our classifier prompt stated that the low-signal bucket is never high-potential, which as an instruction to a model forbids the very reading that would rescue a subtle but genuine discussion. A rule written to suppress noise had quietly become a guaranteed blind spot.
A cheap path that skips the judge. When billing reported zero on a conversation that had in fact connected, the transcript never reached the model at all and was sorted by rules alone. The saving and the blind spot were the same line of code.
A fallback that fails quietly. On a double model failure, the handler defaulted to "not interested," which is a silent outcome. The failure mode of the error handler was indistinguishable from a real negative result, so an outage and a genuine rejection produced identical records.
The weakest model on the largest pile. Short exchanges were routed to Claude Haiku rather than Claude Sonnet, on the reasonable theory that a ten-second exchange needs less reasoning. They are also the most common thing our outbound agent produces, so the bucket needing the most nuance to judge got the least capable judge.
The transcript is not ground truth
The obvious way to find real conversations without reading all of them is to measure how much the human said. Long human speech means a real exchange. It is the first proxy anyone reaches for, and it does not work.
Voicemail greetings transcribe as long stretches of human speech. So do IVR menus, and so do the virtual receptionists we hit constantly on business lines. The proxy inflates hardest on precisely the conversations that are worth nothing.
A high human-speech figure is a candidate signal. It is never a verdict. Something with judgment has to read the thing.
The same caution applies to your agent's own account of itself. Recent work on silent failure in LLM agents describes systems confidently reporting completion of work they did not finish. Short conversations are where we trust our own summaries least, and where we go back to the raw transcript rather than the summary of it.
Building the harness the other way
If you want to know what your classifier is missing, the audit has to be built against its grain. Four rules make it work.
Re-judge independently, from the transcript. Read the raw exchange and decide what you would have done with it, before looking at what the pipeline decided. Judging from the pipeline's own summary just reproduces the pipeline's blind spots at one remove.
Join to what production actually did. For each conversation, pull the outcome the live system assigned and how far that outcome surfaced: notified a human, logged quietly, or ended there. The finding is not your judgment, it is the delta between your judgment and the system's.
Bias hard toward recall. When an exchange sits on the line between thin and real, flag it. Over-flagging spends a few minutes of review, while under-flagging spends a discussion you will never learn you had.
Quote the line, or it is not a miss. Every flagged conversation has to carry an actual sentence the human said that justifies the flag. This one rule is what keeps a recall-biased audit from decaying into noise, and it is the reason the output stays trustworthy enough that people keep reading it.
The output should be a list of conversations to open, not a written account of each. You are building a routing correction, not a report.
When precision is the right choice instead
This approach has a price, and it is not the right default everywhere.
Tuning for recall spends human review time on conversations that turn out to be nothing. That is the correct trade only when a missed discussion is worth more than a wasted review, which is true in outbound sales, where a single missed buying signal outweighs a stack of pointless reviews.
It inverts for compliance. If the expensive error is your agent saying something it was not allowed to say, then a false pass is the failure that matters and precision-first calibration is right. Hamming's rubric is not making a mistake for its use case, it is making the correct choice for a different one.
Work out which error is expensive in your setting before you tune anything. Most teams never make the choice deliberately, and inherit precision by default because that is what the tooling ships with.
The judge is a prediction too
Your voice agent gets tested against adversarial callers, background noise, accents and interruptions, and it should be. That work is real and the platforms that do it are worth what they charge.
Then a second model reads the transcript, decides what the conversation was worth, and quietly determines whether a human ever finds out. Ask what its recall looks like. Nobody in the stack is going to raise a hand and tell you.
AnAr Solutions is an AI-native software engineering company. We run voice agents in production across sales and recruitment, and this is one of the practices that came out of our AI-assisted development work on verification. If you are putting a voice agent or an agentic system into production, talk to our AI engineering team.
