The Voice AI Silence Problem Nobody Is Shipping Around
Your voice agent picks up the question. The user waits. One second. Two. Four. Eight. The frontier model is reasoning, retrieving, calling a tool. The answer, when it arrives, is accurate. But the pause has already done its damage. The user has already felt, at a visceral level, that they are talking to something slow. They will rate the interaction poorly, not because the answer was wrong, but because the silence told them the system was not really listening.
This is the core failure mode of every current voice AI deployment. Fast local models respond instantly but answer poorly. Powerful frontier models answer accurately but pause long enough to break conversational flow. The research on this is unambiguous: Levinson and Torreira document that humans experience prolonged silence as conversational rupture. Skantze's work on dialogue shows that initial acknowledgement of a speaker is essential for natural interaction, not a nice-to-have. The silence IS the product failure, even when the eventual answer is correct.
Researchers at the Paul G. Allen School of Computer Science and Engineering at the University of Washington published a system paper in July 2026 that attacks this problem directly rather than working around it. The system, ConvFill, introduces a task they call "conversational infill": a small on-device language model begins speaking immediately when a user finishes talking, generates contextually grounded filler responses while a frontier model reasons in the background, and then seamlessly incorporates that frontier model's knowledge into its ongoing reply as it arrives in real time. In a live user study with 18 participants, ConvFill achieved a latency rating of 4.24 out of 5 compared to 3.46 for frontier models, was preferred by 12 of 18 participants on retrieval-heavy tasks versus 5 of 18 for the frontier model alone, and scored statistically equivalent to frontier models on satisfaction, task completion, coherence, clarity, and fluency. A model as small as 135 million parameters, trained for $3.07, can relay frontier-model knowledge so faithfully that users cannot distinguish the experience from talking directly to that frontier model on most dimensions that matter.
Why Silence Cannot Be Patched With Generic Filler Phrases
The instinct to hide latency with "I'm thinking about that..." or "Give me a moment..." is wrong in a specific, measurable way. Those phrases are not grounded in what the user just said. They are conversational dead air dressed in words. Research cited in the paper shows that repeated corrections and inconsistencies in spoken dialogue harm user trust, confidence, and engagement. Generic filler scores marginally better than silence but does not solve the problem. Heuristic-selected phrase sets, used in systems like Lucy (Gao et al., 2025), are a modest improvement but still not conditioned on the actual conversation.
The other available architecture is full-duplex speech models, where a single large model handles audio input and output simultaneously. Systems like Moshi and Sesame reduce latency by bypassing speech-to-text and text-to-speech separately. The cost is significant: a single LLM backbone means the model responsible for speaking is also the model constrained by real-time audio generation. You cannot swap in a more capable reasoner without retraining the entire system. And the computational requirements make these systems expensive to run and difficult to adapt.
ConvFill does not try to patch either of these approaches. It formalises a different task entirely.
What Decoupling the Speaker From the Thinker Actually Enables
The architecture has two distinct roles. The Reasoner is a frontier model: Claude, GPT, Gemini, any capable API model. It receives the user's utterance, performs multi-step reasoning, retrieval, or tool calls, and produces a stream of concise "knowledge chunks," sentence-level summaries of what it has worked out. It does this asynchronously, at whatever speed it needs.
The Talker is a small language model, 135 million to 1.7 billion parameters, running locally on the user's device. It receives the user's utterance at the same moment the Reasoner does. It begins generating immediately. If the Reasoner has not yet sent anything, the Talker generates a contextually grounded filler phrase, something specific to what the user asked, not a generic holding pattern. When the first knowledge chunk arrives from the Reasoner, the Talker incorporates it into its response naturally, as if it had known all along.
The silence token is the mechanical key. When the Reasoner's output queue is empty and the text-to-speech queue is also empty, a <sil> token is inserted into the Talker's input stream. This triggers filler generation. When a knowledge chunk arrives, it triggers conditioned response generation. The user hears a continuous, coherent spoken reply that shifts from conversational acknowledgement to factually grounded content as the Reasoner's work completes.
| ConvFill Mechanism | What It Enables | What Breaks Without It |
|---|---|---|
| Silence token triggering | Talker generates contextual filler the instant Reasoner queue is empty | User hears silence; conversation feels broken or system feels slow |
| Streamed knowledge conditioning | Talker incorporates Reasoner output mid-response without corrections | System either waits (slow) or answers without frontier knowledge (inaccurate) |
| Decoupled Talker/Reasoner roles | Either model upgrades independently; Reasoner can be swapped without Talker retraining | Capability improvements require retraining the entire voice system |
The broader implication, stated directly by the authors, is a clean separation between the model responsible for conversational behaviour and the model responsible for advanced capabilities and reasoning.
The Numbers That Make This Real
ConvFill was tested across seven Talker models ranging from 135M to 1.7B parameters, paired with three frontier Reasoners: Claude, GPT-5.5, and Gemini 3.1 Pro. The accuracy results on SimpleQA, a factual benchmark adversarially filtered for difficulty, show what the architecture actually delivers. The base small language models, operating alone, scored between 0.4% and 2.5% accuracy on SimpleQA. Those same models, operating as Talkers with a Gemini 3.1 Pro Reasoner, scored between 60.7% and 65.1%. Gemini 3.1 Pro operating as a standalone frontier system scored 73.1%. The gap between ConvFill and its Reasoner operating alone ranged from 0% to 6.3% across models and Reasoners, with one exception: Gemini 3.1 Pro on SimpleQA showed a 9.8% gap, which the authors attribute to that model's sensitivity to the Reasoner output format rather than a Talker failure.
On time-to-first-response, the speedup over waiting for the Reasoner was 7.38 times faster on direct queries, 9.20 times faster on retrieval tasks, and 19.12 times faster on tool-calling tasks involving email access via the Model Context Protocol. The 19x figure on tool-calling reflects how long MCP operations take, not an efficiency trick: when the Reasoner needs 7,242 milliseconds on average to complete a tool call, a Talker that starts speaking within 478 milliseconds is buying substantial time.
The user study finding on RAG tasks is the sharpest signal in the paper. When users evaluated a direct-answer task, no significant preference emerged between ConvFill and the frontier model (8 vs 10 rank-1 votes, p = 0.740). That result is expected: when the Reasoner responds quickly, the filler phrases have less time to differentiate the experience. On retrieval tasks, where the Reasoner takes longer and the Talker's ability to bridge the gap matters more, ConvFill received 12 rank-1 votes versus 5 for the frontier model, a statistically significant result (p = 0.044). RAG is precisely where most enterprise voice deployments sit today.
The Honest Trade the Paper Does Not Bury
ConvFill scored below frontier models on naturalness: 3.83 versus 4.41 on a five-point scale. This is a real and acknowledged limitation, not a footnote. Some users, particularly those less familiar with filler phrases in spoken conversation, found the pattern of bridging responses less natural than a frontier model that simply waits and then delivers a polished answer. On MCP tasks, where Reasoner latency is highest, the system averaged 2.35 filler phrases per turn. The authors note that at high infill counts, the filler may begin to detract from the experience rather than improve it.
The current system does not implement latency-adaptive filling. It inserts filler uniformly whenever the Reasoner queue is empty. A future version, which the authors propose explicitly, would anticipate how long a given Reasoner task is likely to take and calibrate accordingly, holding off on filler for fast responses and generating more deliberate bridging language, including conversational pauses, for longer ones. That version does not exist yet. The current system trades a meaningful degree of naturalness for responsiveness on complex tasks, and any deployment should account for that.
There is also a safety gap. The Talker operates independently before the Reasoner responds. Fillers are generated before the guardrailed Reasoner output is available. In domains where safety constraints are critical, the Talker's independent operation is an exposure that the current architecture does not fully close.
What This Looks Like Deployed on a Laptop
The University of Washington team ran the live user study on an Apple M2 SoC with 16 GB of unified memory. The smallest Talker model, SmolLM2 at 135 million parameters, occupies 136 MB in INT8 quantization. The 1.7 billion parameter version occupies 1.69 GB. The entire suite of seven Talker models was trained in 122 GPU hours at a total compute cost of $133.70 on previous-generation hardware. Dataset generation using Claude Opus 4.6 cost approximately $2,400 as a one-time expense.
The practical shape of a RAG deployment looks like this. A user asks a question that requires retrieving from a document corpus. The frontier Reasoner begins retrieving and reranking. Its average time to first response on this task in the study was 4,852 milliseconds. The ConvFill Talker begins speaking within 976 milliseconds, generating a contextually grounded acknowledgement while the retrieval completes. When the Reasoner's first knowledge chunk arrives, the Talker incorporates it. The user experience is of a system that heard them immediately and is sharing what it knows as it finds it, rather than one that went silent and returned with a document.
The authors note that smartphone SoCs now include dedicated neural hardware with native INT8 support, and the Talker weights occupy a fraction of available RAM on any recent device. The inference infrastructure already exists in the hardware most users carry. The training cost to adapt a Talker for a new application domain is measured in dollars, not in the infrastructure investments that frontier model fine-tuning typically requires.
What the Latency-Capability Tradeoff Looks Like After This Paper
For the past three years, the operating assumption in enterprise voice AI has been that you choose a position on a single axis: more capable means slower, faster means less capable. Teams deploying voice interfaces have navigated this by either accepting slow frontier responses and trying to set user expectations, or deploying faster local models and accepting capability ceilings that limit the use cases they can support.
ConvFill's results suggest that axis is not as fixed as it appeared. The architecture reaches a point on the latency-capability curve that neither approach can access independently: frontier-level accuracy on factual and retrieval tasks, millisecond-level response initiation, and a total hardware footprint that fits on a consumer laptop. The decoupling of Talker and Reasoner also means that as frontier models improve, the Talker does not need retraining to benefit. The capability ceiling rises automatically. The conversational behaviour stays consistent, which matters more than most voice AI roadmaps currently account for: users who experience a model's personality shift after an API update notice, and the trust cost is real.
The naturalness gap and the safety exposure from independent Talker operation are the live constraints. Neither is architectural in the sense that would require redesigning the system, but neither is resolved in the current implementation. The question for any organisation evaluating voice AI architecture now is not whether to wait for those gaps to close. It is whether the current capability-responsiveness combination already fits enough of their use cases to justify moving, and whether their RAG deployment is the right place to start.
The silence that broke the conversation was never just a latency problem. It was a design assumption.
— Agents Applied Editorial