NVIDIA has released Audex (Nemotron-Labs-Audex-30B-A3B), a unified audio-text large language model. It understands and generates both audio and speech. It also keeps the text intelligence of its backbone. The checkpoints, along with a smaller Audex-2B, are released under a noncommercial license.
Most multimodal models pay a text tax. When labs add audio or vision output, text benchmarks often drop. NVIDIA research team reports this even for speech-only output models. Audex is designed to avoid that regression.
TL;DR
Audex is a single 30B-A3B MoE model that handles audio in and out.
Audio inputs enter the text embedding space; audio outputs are treated like text tokens.
Text scores match the backbone, with small gains and small losses per benchmark.
Multi-stage SFT plus text-only Cascade RL avoids the usual multimodal text regression.
It is few of the open models that generate general audio beyond speech.
What is Audex?
Audex is a single Mixture-of-Experts (MoE) Transformer decoder. It has 30B total parameters and 3B activated per token. The backbone is Nemotron-Cascade-2-30B-A3B, a text-only MoE LLM. That backbone is a hybrid Mamba-Transformer with 52 layers. It uses 128 routable experts and 6 activated experts.
The design is deliberately simple. Audio inputs are encoded and projected into the text embedding space. Text tokens and quantized audio tokens are then treated uniformly during generation. There is no thinker-talker split and no stacked cascade of models.
Because the design stays simple, Audex runs on standard LLM stacks. These include Megatron-LM for training and vLLM for inference. It supports both an instruct mode and a thinking mode. Context length reaches 1M tokens.
How the Unified Design Works
Three components sit around the LLM backbone:
An audio encoder reads sound. Audex uses AF-Whisper from Audio Flamingo 3. It shares the Whisper Large-v3 architecture and handles 16kHz input.
Two-layer MLP adapters map audio features into the model dimension.
An extended vocabulary holds discrete audio output tokens. The original 131,072 tokens grow to 205,312.
Audex uses two codecs for output. Speech uses X-Codec2 at 50 tokens per second. It applies single-layer finite scalar quantization (FSQ) with a 65,536 codebook.
Non-speech sound uses X-Codec at 200 tokens per second. It uses four flattened residual vector quantization (RVQ) layers. Complex sound gets a larger token budget than speech. The interactive demo below computes these token counts for any duration.