Alibaba’s Qwen team has released Qwen3.8-Flash-Next, an open-weight multimodal Mixture-of-Experts model built for cost per token. The checkpoint pairs a 125B backbone with a 51B N-gram embedding table and a 4B multi-token prediction module. Only 6B parameters activate per token. The team positions it as an early preview of the architecture that will underpin Qwen4, the same role Qwen3-Next played for Qwen3.5. Four changes carry the release: a Gated DeltaNet and Qwen Sparse Attention hybrid, Gated Residual, N-gram Embedding, and the Muon optimizer. Qwen team reports training cost at roughly one-ninth that of Qwen3.7-Plus.
Is it deployable?
Yes but not on a workstation. The FP8 checkpoint is 172.78 GiB and the BF16 checkpoint is 335.28 GiB. Per vLLM recipes, TP2 is the minimum validated FP8 configuration on GB300 and TP4 is recommended. On an 8×H200 node, use TEP8; plain TP8 is incompatible with the checkpoint’s 128-wide quantization blocks. Sparse activation cuts compute, not storage.
https://qwen.ai/blog?id=qwen3.8-flash-next
What is actually new
Qwen3.8-Flash-Next pairs a 125B main model with 51B N-gram embedding parameters and a 4B multi-token prediction module, totaling 180B on disk. Only 6B parameters activate per token. Four changes drive this:
Hybrid attention (GDN + QSA): Three of every four layers use Gated DeltaNet, a linear-attention layer that compresses history into a fixed-size recurrent state. The fourth layer runs Qwen Sparse Attention (QSA), which uses a lightweight indexer to select context at micro-block granularity rather than per token. The layer layout is 12 × (3 × GDN → 1 × QSA) across 48 layers, with a QSA budget of 512 blocks or 2048 tokens.
Gated Residual: The residual stream widens into 4 parallel branches, with an element-wise read gate and a per-branch scalar write gate, at bottleneck rank 320.
N-gram Embedding: A 20,000,000-entry bigram/trigram table at layer 2 adds capacity through deterministic lookups. It can be offloaded to host memory with asynchronous prefetch — though offload currently runs only on NVIDIA devices.
Training recipe.:The Muon optimizer is applied alongside AdamW to specific weight categories, with batch-size warmup eliminated and scaling laws refitted.
The MoE layer carries 512 experts, activating 10 routed plus 1 shared, at expert intermediate dimension 640.
#mtp-qwen38-x7k2{background:#0D0A1B!important;border:1px solid #2B2154!important;border-radius:14px!important;padding:0!important;margin:22px 0!important;overflow:hidden!important;box-sizing:border-box!important}
#mtp-qwen38-x7k2 *{box-sizing:border-box!important}
#mtp-qwen38-x7k2 p:empty,#mtp-qwen38-x7k2 hr,#mtp-qwen38-x7k2 del,#mtp-qwen38-x7k2 s{display:none!important}
#mtp-qwen38-x7k2 iframe{width:100%!important;border:0!important;display:block!important;background:#0D0A1B!important;min-height:600px!important}