QUASAR-QAT/Qwen3.5-4B-QUASAR-NVFP4

Qwen3.5-4B — QUASAR NVFP4

Project page · Code

2.3–2.7× lower KL to BF16 than the leading public 4-bit builds we tested. Best 4-bit AIME'25, GPQA-Diamond, MMLU-Pro and GSM8K-Platinum in our same-harness comparison.

4.2 GB · all 200 decoder projections in NVFP4 · vision + MTP retained · 98.2% hard-task recovery · +35% single-stream decode vs BF16 on H100 · serves directly with vLLM.

Also in this family: W4A16 / vLLM (this repo) · W4A4 / Blackwell · Q4_0 GGUF / llama.cpp · collection · paper

A weight-only NVFP4 (W4A16) version of Qwen/Qwen3.5-4B trained with QUASAR quantization-aware training: the quantized weights are learned by distilling from the frozen BF16 original on its own outputs, so this is a drop-in lower-precision replacement, not a fine-tune. Companion to Qwen3.8-27B-QUASAR-NVFP4 in the Qwen3.5-4B & Qwen3.8-27B — QUASAR 4-bit QAT collection.

Same-harness comparison with leading 4-bit builds

One harness for every row, following RedHatAI's published protocol (0-shot chat, thinking on, 65,536-token budget, vLLM 0.26 on H100). AIME'25 is the mean of 8 seeds, GPQA-Diamond of 3. Bold = best among the compared 4-bit builds.

	QUASAR (this model)	RedHatAI W4A16	cyankiwi AWQ-4bit	cosmicproc NVFP4	BF16 original
Size	4.2 GB	5.5	4.0	3.3	9.3
Decoder projections @ 4-bit	200 / 200	128 / 200	200 / 200	200 / 200	—
Vision / MTP	✓ / ✓	✓ / ✓	✓ / quantized	✗ / ✗	✓ / ✓
KL to BF16 ↓	.0181	.0487	.0419	.0429	—
MMLU-Pro ↑	78.5	77.5	77.6	77.9	78.9
AIME'25 ↑	74.2	65.0	72.5	65.8	79.2
GPQA-Diamond ↑	77.9	71.2	74.6	73.6	77.3
GSM8K-Platinum ↑	94.7	94.0	93.9	94.2	94.0
H100 decode, c=1 ↑	298 tok/s	239	306	230	221

Full benchmark table, seeds, confidence information, fidelity methodology and serving measurements: EVAL.md.

• Closest to BF16. 2.3–2.7× lower per-token KL than the compared public 4-bit builds, with 96.2% top-1 agreement.
• Strongest reasoning results. Best compared 4-bit result on AIME'25, GPQA-Diamond, MMLU-Pro and GSM8K-Platinum; GPQA-Diamond is within seed noise of BF16.
• Full decoder scope, full feature set. All 200 major decoder projections in NVFP4 while preserving the vision tower and MTP drafter; 3.29 GiB of weights in VRAM and +35% single-stream decode over BF16 on H100.

Run it

pip install "vllm>=0.26"

vllm serve QUASAR-QAT/Qwen3.5-4B-QUASAR-NVFP4 --max-model-len 262144 --reasoning-parser qwen3

Loads straight from the Hub, no conversion (compressed-tensors NVFP4, weight-only / W4A16). Validated with vLLM 0.26 on H100 for text, image input, 262K context (needle retrieved from a 200k-token prompt) and MTP speculative decoding (--speculative-config '{"method":"mtp","num_speculative_tokens":2}': +27% single-stream decode, 80% draft acceptance). For Blackwell, use the W4A4 build; for llama.cpp / Ollama, the Q4_0 GGUF.

Fidelity vs standard quantization at the same scope

Per-token forward KL(BF16 ‖ quantized) and top-1 agreement on 734 held-out prompts of the BF16 model's own responses (934k response tokens), all methods at the same 200-projection NVFP4 scope (same data and schedule where trained):

	KL ↓	KL p99 ↓	top-1 agree ↑
QUASAR (this model)	.0181	.126	96.2%
Standard QAT (STE)	.0228	.170	95.6%
GPTQ NVFP4 (llmcompressor)	.0227	.167	95.7%
RTN NVFP4 (no training)	.0423	.326	93.8%

At identical NVFP4 scope, QUASAR cuts RTN KL by 2.3× and improves ~20% over both GPTQ and standard QAT.

Quantization details

NVFP4 (E2M1 weights, group 16, FP8-E4M3 group scales under one fp32 tensor scale) on every large projection in every decoder layer: MLP gate/up/down in all 32 layers, q/k/v/o in the 8 full-attention layers, and in_proj_qkv / in_proj_z / out_proj in the 24 gated delta-net layers — 200 of 200. Kept in BF16: embeddings (tied), norms, the 24 depthwise conv1d kernels, the 48 small delta-net gate projections (0.11% of the decoder), the vision tower and the MTP drafter (bit-identical to the original). Full-decoder NVFP4, with vision and MTP retained.

The packed NVFP4 weights decode to exactly the weights that were trained and evaluated (222,822,400 / 222,822,400 groups, 0 flipped elements — export_ct_report.json); fused q/k/v, gate/up and in_proj_qkv/z groups share their global scale by construction, so vLLM's fused-layer loading changes nothing.

Training. QUASAR QAT with self-distillation from the frozen BF16 checkpoint over ~120M tokens; QUASAR performs loss-aware NVFP4 reconstruction inside the training loop. The exported packed weights are bit-exact to those trained and evaluated. Details in EVAL.md and the paper.

Related

• QUASAR-QAT/Qwen3.8-27B-QUASAR-NVFP4 — the 27B sibling: 496/496 linears in NVFP4 (W4A4), GPQA-Diamond 0.909 vs 0.914 BF16.
• QUASAR-QAT/Qwen3.5-4B-QUASAR-NVFP4-W4A4 — the same weights with calibrated activation scales for native W4A4 on Blackwell.
• QUASAR-QAT/Qwen3.5-4B-QUASAR-Q4_0-GGUF — native Q4_0 QAT for llama.cpp / Ollama / LM Studio.
• Qwen3.5-4B & Qwen3.8-27B — QUASAR 4-bit QAT — the collection; Gemma 4 and Muse Glimmer builds under QUASAR-QAT.
• Paper: QUASAR: Lowering the Loss Floor of Quantization-Aware Training with Loss-Aware Reconstruction.

Citation

arxiv.org/abs/2608.13966

@article{counathe2026quasar,
  title={QUASAR: Lowering the Loss Floor of Quantization-Aware Training with Loss-Aware Reconstruction},
  author={Counathe, Vincent and Athiwaratkun, Ben and De Sa, Christopher and Zhang, Tianyi},
  journal={arXiv preprint arXiv:2608.13966},
  year={2026}
}