A single 11B end-to-end model replaces the traditional ASR-LLM-TTS chain, delivering about 448 ms turn-taking latency and enabling tool calling via a separate side channel with operator-defined on-hold messages.

Ideal use cases include real-time multi-turn customer service agents, tool-calling during conversation, multilingual or speaker-diverse applications, full-duplex speech interaction research, and low-latency voice applications that benefit from end-to-end processing.

The project trained with roughly 550,000 hours of real and synthetic audio from sources like Fisher, LibriVox, LibriTTS, VCTK, and synthetic data from Nemotron 5.5, Ultrachat, and PromptTTS.

Its architecture combines a Fast Conformer encoder, the Nemotron Nano v2 9B LLM backbone, an NVIDIA TTS decoder/codec for 22.05 kHz speech, and a separate tool-calling output path, built on SALM-Duplex and Audio Flamingo 3.

The FAQ clarifies commercial use restrictions, required hardware, comparison to cascaded systems, failure modes in tool calling, language support, input length, inference speed, maintenance status, and fine-tuning possibilities.

Deployment is partial—usable for pilots but not production; weights and container are public under a permissive license, but checkpoints are labeled research-only with documented failures like limited audio context, gibberish after long turns, runaway self-talk, and occasional dropped words.

Performance benchmarks place VoiceChat among the top open full-duplex models, with detailed metrics on pause handling and turn-taking latency, and outputs including agent speech, agent text, user transcriptions, and tool-calling scripts.

TOR-based references show turn-taking around 448 ms and related metrics, with NVIDIA ranking as the second among open full-duplex models on major benchmarks.

Recommended deployment requires at least one high-end GPU with hefty VRAM (80 GB+ on Linux x86_64), and there is no hosted API or inference service available.

Model inputs and outputs include 16 kHz input, 22.05 kHz output, text prompts, on-hold tool messages, with vLLM deployment limited to constrained Linux environments and no documented CPU inference.

VoiceChat offers a unified, low-latency alternative to cascaded pipelines, enabling real-time interaction but with notable production deployment and tool orchestration limitations.

Limitations include research-only deployment, variable tool-calling performance across scenarios, modest accuracy metrics, strict hardware requirements, potential synthetic-data biases, and limited multilingual clarity.