hermitdave/LFM2.5-24B-A2B-MLX-DWQ-3bit

LFM2.5-24B-A2B — MLX 3-bit DWQ

An MLX 3-bit quantization of LiquidAI/LFM2.5-24B-A2B built with Distillation-Aware Weight Quantization (DWQ), distilled against Liquid AI's own 8-bit MLX release.

LFM2.5-24B-A2B is a 24B-parameter reasoning MoE with ~2B active parameters per token: 40 layers (30 double-gated short-convolution blocks + 10 grouped-query attention), 64 experts with top-4 routing, 128k context, 128k vocabulary.

Results

Measured on held-out text (32 sequences of 1025 tokens from the TULU-3 sft mixture, held out from calibration), against the 8-bit teacher:

Model	Perplexity ↓	KL vs teacher ↓	top-1 vs teacher ↑	Size
8-bit teacher (reference)	7.05	—	—	24.6 GB
3-bit RTN (baseline)	13.62	0.2393	59.4%	—
3-bit DWQ (this model)	8.12	0.1458	70.7%	10.5 GB

Lower KL and higher top-1 agreement mean the quantized model stays closer to the 8-bit reference. Perplexity is a teacher-free cross-check.

DWQ over RTN at 3-bit (positive = better):

Metric	RTN	DWQ	Improvement
KL vs teacher (lower better)	0.2393	0.1458	+39.0%
Perplexity (lower better)	13.62	8.12	+40.4%
top-1 agreement (higher better)	59.4%	70.7%	+11.3 pts

Reading the top-1 number

A raw top-1 figure invites the reading "29% of predictions are wrong", which overstates it. Most disagreements are positions where the teacher's own distribution is nearly flat, so either token is a plausible continuation and the two models hold almost the same belief.

Bucketing every held-out position by how decisive the teacher was (gap between its top-1 and top-2 logits), over 32,768 positions:

Teacher confidence (top1 − top2 gap)	Share of positions	top-1 agreement
<0.5 (near-tie)	20.0%	39.0%
0.5-1	16.3%	52.9%
1-2	20.4%	69.4%
2-4	19.1%	87.6%
4-8	16.0%	95.9%
>8 (decisive)	8.2%	98.3%

• 68% of all disagreements are near-ties (teacher gap < 1) — coin flips, not errors.
• On positions where the teacher is decisive (gap > 2), agreement is 93.0% (13,772 positions).

So treat 70.7% as a lower bound on fidelity. This is also why top-1 and KL must be read together: KL understates damage (a confidently wrong token barely moves it), while top-1 overstates it (it counts coin flips as errors).

A better variant exists

We rebuilt the 3-bit allocation to follow Liquid AI's recipe (6-bit embed_tokens, router gate left at bf16) and then ran DWQ on it, so it is the same method on a better base.

hermitdave/LFM2.5-24B-A2B-MLX-DWQ-3bit-v2

KL 0.1343 vs 0.1458 here, top-1 72.4% vs 70.7%. On the metric that maps to behaviour — how often a decisive token is predicted differently from the teacher — the two differ by about 94 errors per 10,000 positions (701 vs 607).

Both are our own work built from the bf16 base, so there is no provenance trade-off here — the recipe variant is simply better. If you want the exact uniform-allocation baseline that isolates the DWQ effect, take this one.

Sibling quants

Bits	Perplexity ↓	KL vs teacher ↓	top-1 ↑
4-bit	6.97	0.0746	81.4%
3-bit	8.12	0.1458	70.7%	← this model
2-bit (not published — see Limitations)	120.80	0.5653	32.0%

Method

DWQ refines the quantization grid — the per-group scales and biases — by distilling a higher-precision teacher, rather than rounding weights to the nearest grid point (RTN). The integer weights themselves are never modified; only the scales/biases move. This typically recovers a meaningful fraction of a bit of quality, and the effect is largest at low bit widths.

The pipeline:

1. RTN baseline — mlx_lm convert -q --q-bits 3 --q-group-size 64 from the bf16 base.
2. Teacher targets — Liquid AI's 8-bit MLX release is run over 1024 calibration sequences; the top-1024 logits per position are written to disk.
3. DWQ — only the student is loaded. Adam (lr 1e-6) optimises the scales and biases against the stored teacher logits with a KL loss, batch size 1, with gradient checkpointing. 1024 samples at sequence length 1025.

Calibration data: allenai/tulu-3-sft-mixture, chat-templated with the model's own template, 1024 train / 32 held-out sequences.

Quantization details

• base: 3-bit, group size 64, mode affine
• feed_forward.gate overridden to 8-bit (38 layers)
• effective 3.501 bits/weight, 2 shard(s), 10.49 GB

Usage

from mlx_lm import load, generate
from mlx_lm.sample_utils import make_sampler

model, tokenizer = load("hermitdave/LFM2.5-24B-A2B-MLX-DWQ-3bit")

messages = [{"role": "user", "content": "What is the capital of France?"}]
prompt = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)
print(generate(model, tokenizer, prompt=prompt, max_tokens=256,
               sampler=make_sampler(temp=0.1, top_k=50)))

# CLI
mlx_lm.generate --model hermitdave/LFM2.5-24B-A2B-MLX-DWQ-3bit --prompt "Explain Rayleigh scattering."

Recommended generation settings from the upstream model card: temperature 0.1, top_k 50, repetition_penalty 1.05.

LFM2.5 is a pure reasoning model — the chat template inserts a thinking tag so it always reasons before answering. Expect reasoning tokens before the final answer.

⚠️ Requires mlx-lm ≥ 0.32.0

Load this model with mlx-lm 0.32.0 or newer. Earlier versions compute LFM2 MoE routing incorrectly and output quality will be degraded.

mlx-lm ≤ 0.31.3 routes with softmax over all experts, adding expert_bias to the weights used for both top-k selection and weighting. The HuggingFace reference — and mlx-lm ≥ 0.32.0 — uses sigmoid per expert, adds expert_bias only for selection, gathers weights from the unbiased scores, and scales by routed_scaling_factor.

Measured on the 8-bit teacher, softmax vs sigmoid routing:

Metric	Value
max abs logit difference	17.13
KL(sigmoid ‖ softmax)	0.33 – 1.03
top-1 agreement	54.5% – 83.5%

For scale, the entire effect of 4-bit quantization is a KL of about 0.09 — the wrong routing swamps it several times over. Fixed upstream in PR #1354, first shipped in v0.32.0.

pip install -U "mlx-lm>=0.32.0"

Also: DWQ cannot run at all on ≤ 0.31.3

mlx_lm.dwq fails with

ValueError: [gather_axis] Cannot calculate VJP with respect to indices.

on those versions — the same function omitted mx.stop_gradient on the top-k router indices, so any mx.value_and_grad through the model fails. Also fixed in v0.32.0 (PR #1795).

Reproduction

python3 scripts/download.py            # bf16 base + 8-bit teacher
python3 scripts/build_calibration.py   # TULU-3 -> chat JSONL
bash   scripts/convert_rtn.sh          # RTN baselines
bash   scripts/run_dwq.sh 3            # DWQ (checkpointed, resumable)
python3 scripts/benchmark.py ...       # perplexity + KL/top-1

scripts/dwq_checkpointed.py is a checkpointed DWQ driver. Stock mlx_lm.dwq crashed with a Metal out-of-memory error partway through a run because it never releases cached buffers and only saves at the very end; the driver clears the cache every step and writes a rotating checkpoint every step.

Limitations

• Not a fine-tune. This is a lossy compression of the upstream model. For knowledge-heavy work, pair it with retrieval — the upstream card makes the same recommendation.
• 3-bit is not free. See the Results table for the measured gap to the 8-bit teacher.
• 2-bit is not recommended for this model. At 2-bit, RTN collapses (perplexity ≈ 4782, top-1 agreement 5.3%) and even with DWQ the model is degenerate — it emits repeated thinking tokens instead of answering. 3-bit is the practical floor.
• Not evaluated on agentic benchmarks. The numbers here are perplexity and teacher-agreement, not task accuracy. No claim is made about downstream agent or tool-use performance.

Attribution

• Base model: Liquid AI — LFM2.5-24B-A2B, LFM Open License v1.0.
• 8-bit teacher: LiquidAI/LFM2.5-24B-A2B-MLX-8bit.
• Quantization, DWQ pipeline, calibration, benchmarking and this card: Hermes Agent (Nous Research).

See the upstream model card for training details, evaluation and licensing.