The Voice Agent Pipeline Nobody Needs to Build

The Voice Agent Failure That Happens Before the AI Speaks

The smart-home assistant mishears "heat down" as "feet down." The ASR transcript looks plausible. The downstream model, trained on text, has no reason to question it. It generates a structured command: action: deactivate, object: feet, location: none. The thermostat does nothing. The user repeats themselves, louder, in a noisier room. The cascade fails again.

This is not a hallucination. Every component worked exactly as designed. The ASR model transcribed what it heard. The language model parsed what it received. The failure lives in the seam between them, where a confident but wrong transcript becomes an unrecoverable input for everything downstream. Under clean acoustic conditions, that seam is manageable. Under the noise levels of real environments, it compounds fast.

Researchers at Maum AI, with affiliates at Seoul National University and KAIST, published a framework in June 2026 that addresses this seam directly. CORTIS, which stands for COnveRsational agents via Text-only supervised Instruction tuning of SLMs, eliminates the ASR module at inference time entirely, while requiring no speech annotations at training time. On a proprietary smart-home function-calling benchmark, a 7B CORTIS model beat an equivalent ASR-LLM cascade by 10.49 absolute exact-match points at 0 dB signal-to-noise ratio, the kind of acoustic environment found in a kitchen with a television on. The model trained only on text outperformed the pipeline that actually processed the audio through dedicated speech recognition. The results are promising, though they come without statistical significance tests and from a relatively small test set of 591 examples, so they warrant careful interpretation rather than immediate extrapolation.

Why Cascades Break Under Load and CORTIS Doesn't

The standard architecture for a voice agent looks like this: a speech recognition model converts audio to text, a language model converts text to a structured output (a JSON frame, a function call, an intent-entity annotation), and downstream systems act on that structure. The logic is modular and intuitive. You can swap components, upgrade ASR independently, and reuse text-LLM investments across multiple interfaces.

The problem is that modular pipelines propagate errors without flagging them. A corrupted transcript arrives at the language model with the same formatting and confidence as a clean one. The language model has no way to know the input was wrong. It parses faithfully and produces a structured output that looks correct and isn't.

CORTIS sidesteps the propagation problem by removing the transcript entirely. A spoken language model (SLM), which is a speech-capable LLM that takes audio directly as input, is fine-tuned using only text input-output pairs for the target task. At inference time, the audio occupies the same prompt position the text query held during training. The model's pretrained speech-text alignment, already baked in before fine-tuning began, transfers the learned task schema from text to speech automatically.

The speech encoder and modality adapter are frozen during CORTIS fine-tuning. Only the LLM component is trained. This is not an incidental implementation choice. Those modules were not supervised on text during adaptation, and updating them on text-only data risks corrupting the pretrained speech pathway that CORTIS depends on entirely.

What CORTIS Gets Right and Where It Quietly Loses

The headline results are on three datasets: FSC (smart-home command understanding, exact match over a three-slot semantic frame), SLURP (home assistant interactions, evaluated on intent accuracy, entity F1, and combined SLU-F1), and a proprietary in-house dataset of 42 smart-home functions evaluated on exact match over the full function call and arguments.

On FSC, CORTIS wins consistently. At 7B scale, it beats the cascade by 1.00 absolute exact-match point under clean speech, and that gap widens to 5.59 points at 0 dB. FSC is a constrained benchmark with a small output vocabulary, and the authors acknowledge it may appear in SLM pretraining data, so the 100% oracle scores should be read with that caveat in mind.

On SLURP intent accuracy, the pattern holds. The 7B CORTIS model outperforms the cascade by 4.05 percentage points under clean conditions, 8.78 points at 5 dB, and 7.15 points at 0 dB. When the task is predicting what the user wanted to do (weather query, music playback, event recommendation), CORTIS is consistently more reliable under noise.

But on SLURP SLU-F1 at 3B scale, the cascade wins across most noise conditions. At 5 dB, the cascade scores 50.81 against CORTIS's 49.35. At 2.5 dB, the gap is 42.90 to 40.38. When exact slot values matter and the model is smaller, the cascade's ability to pass a correct transcript directly to the downstream model is an advantage CORTIS cannot overcome. The authors state this directly: direct speech-based prediction can capture high-level task semantics but struggles with exact lexical grounding for slot values. A user says "rap" and CORTIS, unable to recover the precise word from degraded audio, returns "rock."

The in-house results follow a more nuanced pattern. At 3B scale under clean and mild-noise conditions, the cascade is marginally better (90.86 vs 89.51 exact match under clean speech). The crossover happens at 5 dB, where CORTIS moves ahead and stays ahead, reaching an absolute EM gain of 4.57 at 0 dB. At 7B scale, CORTIS wins across all conditions, with the 0 dB gain of 10.49 absolute points being the strongest single result in the paper.

The Architecture Dependency No Benchmark Reveals

Appendix results on the Qwen2-Audio-7B backbone complicate the picture significantly. On FSC and SLURP, CORTIS with this architecture performs comparably to the cascade. On the in-house dataset, CORTIS loses: 87.82 versus 91.54 under clean conditions, 39.76 versus 44.67 at 0 dB. The authors attribute this to a mismatch between Qwen2-Audio's pretrained speech pathway and the product-level audio preprocessing applied to the in-house data.

This is the finding most likely to be missed in a promotional reading of the paper. CORTIS is not a model-agnostic technique. It depends entirely on the pretrained speech-text alignment of the SLM backbone it is applied to. If that alignment does not generalise to the audio characteristics of your production environment, the approach degrades, and it can degrade below a simple ASR cascade with no additional sophistication. The main results, reported on Qwen2.5-Omni, should not be assumed to transfer to other SLM families without empirical validation.

The robustness evaluation also uses synthetic babble noise from the MUSAN dataset, not the reverberation, far-field microphone pickup, or device-specific noise profiles that real voice deployments encounter. The 0 dB SNR results are striking, but 0 dB babble noise is not the same acoustic challenge as a vehicle interior at highway speed or a hospital ward with overlapping conversations.

How the Training and Inference Pipelines Actually Work

The mechanism is simpler than the results might suggest:

1. Assemble a text-form dataset of input-output pairs for the target task. These are standard NLP annotation artifacts: a text query ("turn the kitchen lights on") paired with a structured output (a JSON frame or function call). No microphone required.

2. Wrap each pair in a fixed prompt template using a ChatML-style format. The text query occupies a specific slot in the template. This slot position is fixed and will not change at inference time.

3. Fine-tune the LLM component of the SLM on next-token prediction over the structured outputs. The speech encoder and modality adapter are frozen throughout. Three epochs on 4 x NVIDIA A100 GPUs.

4. At inference, replace the text query in the same prompt slot with the audio input. The speech encoder converts the audio to representations aligned with the LLM input space. The fine-tuned LLM generates the structured output directly, with no transcription step.

The critical constraint is prompt-position consistency. The model learns during training that structured outputs follow from whatever appears in the query slot of the prompt template. At inference, audio appears in that slot instead of text. The pretrained alignment ensures the LLM receives speech-conditioned representations that correspond functionally to what text representations would have provided. If the prompt structure changes between training and inference, the alignment does not transfer cleanly.

A Production Scenario Where the Seam Costs Money

Consider a smart-building management platform with voice control across 200 meeting rooms. The system handles lighting, climate, AV setup, and privacy mode activation from spoken commands. The acoustic environment varies by room geometry, occupancy, and ambient noise from HVAC systems.

The current architecture uses Whisper large-v3 plus a fine-tuned text LLM. Under clean conditions (quiet, near-field microphone), the cascade performs reliably. During active sessions, with background conversation and HVAC running, ASR word error rates climb. A command to "set the screen brightness" becomes "turn off this room privately" in the transcript. The downstream LLM, seeing plausible text, activates privacy mode. Facilities management logs the error but has no systematic way to audit it, because the transcript looked correct to any human reviewer.

Adapting to CORTIS in a scenario like this requires text-form examples of the 42 target functions paired with representative natural-language commands. That annotation exists already in most deployments as intent-labelling work done during initial LLM fine-tuning. The speech annotation pipeline (recording, transcription, alignment, slot labelling per audio file) is eliminated. What changes is not the annotation budget but the annotation type, from speech-paired data that requires a recording studio and specialised labelling to text pairs that any NLP team already knows how to produce.

The caveat is real: if the production audio pipeline applies proprietary preprocessing (noise cancellation, gain normalisation, room correction), the SLM backbone must be validated against that processed audio before deployment. The Qwen2-Audio results in the paper are a warning, not an edge case. Architecture and audio pipeline compatibility are empirical questions that benchmarks cannot answer in advance.

What This Changes for Organisations Building Voice Interfaces

The conventional assumption in enterprise voice deployment is that structured task execution requires either: (a) expensive paired speech-target annotation to fine-tune an end-to-end model, or (b) a cascade architecture with a good ASR module absorbing the acoustic complexity. CORTIS offers evidence that assumption is incomplete.

Text-only SLM fine-tuning can match or exceed a text-only fine-tuned cascade on command understanding and intent prediction, particularly as acoustic conditions deteriorate. That finding is specific: the comparison is against a cascade using the same text supervision, not one given additional speech data. A cascade with paired speech-supervised fine-tuning might outperform CORTIS. The paper does not test that.

For organisations where voice interfaces handle high-level command semantics (turn this on, play that, schedule this), CORTIS represents an annotation efficiency argument worth evaluating. For organisations where exact slot values are operationally critical (medication dosage from a voice pharmacy interface, financial transaction amounts, precise database lookup fields), the entity grounding limitations documented in the paper are not an acceptable trade-off at current performance levels.

The broader implication is about where the cost of voice agent deployment actually lives. It is not primarily in model architecture or compute. It is in the data annotation pipeline required to adapt a general-purpose model to a specific task schema in a specific acoustic environment. CORTIS reduces the cost of that pipeline significantly for the right class of tasks. The question for any deployment team is whether their tasks belong to that class, and the answer is not the same for every function call their voice agent is expected to make.

The pipeline you build around a voice agent shapes what it can recover from. Eliminating the transcript does not eliminate the noise. It changes where the model meets it.

Agents Applied covers research that matters for practitioners. If a paper changes how we think about something already in production, we cover it honestly, including the parts that complicate the headline.