@Avifenesh on Hugging Face: "Wrote this up because it still feels backwards. Acceptance dropped, tok/s went…"

Join the conversation

Join the community of Machine Learners and AI enthusiasts.

posted an update 1 day ago

Post

Wrote this up because it still feels backwards. Acceptance dropped, tok/s went up.

Full draft head: 66.7% / 117.1 tok/s. Trimmed: 63.6% / 121.7. Later remeasure +5.1% on a Pro 6000, +6.4% on a 5090 laptop. Full head still accepts more. Loses anyway.

Not my idea. FR-Spec. I just wired it for the native MTP head.

https://huggingface.co/blog/Avifenesh/masked-mtp-drafts

Avifenesh/Qwen3.8-27B-NVFP4-MTP-GGUF

cdiamond

1 day ago

Seen the same thing from the quantization side. My Qwen3.8 27B GGUF keeps the MTP head in native NVFP4 on purpose. Upgrading the head to Q5_K/Q6_K (+69 MiB) looked like a free win and instead acceptance went 48.3% to 33.1% and throughput dropped 26.6%. The head only has to agree with the quantized target, not with the BF16 parent, so trimming and quantizing probably win for the same reason. Full numbers: https://piszczek.pl/blog/qwen38-27b-256k-50-tps-24gb-gpu

Avifenesh

1 day ago

Yeah that tracks. I kept thinking a fatter head would just be more accurate. Same trap. When I requantized the trimmed head to NVFP4, acceptance didn't move. Zero. The draft only has to land tokens the target will take, not match the BF16 parent. Your 48.3 to 33.1 from bumping NVFP4 up to Q5_K/Q6_K is the same movie the other way. I'll read the writeup.

In this post