Instructions to use ddark-il/Qwen3.8-Flash-Next-Uncensored with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use ddark-il/Qwen3.8-Flash-Next-Uncensored with MLX:
# Make sure mlx-vlm is installed # pip install --upgrade mlx-vlm from mlx_vlm import load, generate from mlx_vlm.prompt_utils import apply_chat_template from mlx_vlm.utils import load_config # Load the model model, processor = load("ddark-il/Qwen3.8-Flash-Next-Uncensored") config = load_config("ddark-il/Qwen3.8-Flash-Next-Uncensored") # Prepare input image = ["http://images.cocodataset.org/val2017/000000039769.jpg"] prompt = "Describe this image." # Apply chat template formatted_prompt = apply_chat_template( processor, config, prompt, num_images=1 ) # Generate output output = generate(model, processor, formatted_prompt, image) print(output) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Pi
How to use ddark-il/Qwen3.8-Flash-Next-Uncensored with Pi:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "ddark-il/Qwen3.8-Flash-Next-Uncensored"
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "mlx-lm": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "ddark-il/Qwen3.8-Flash-Next-Uncensored" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Hermes Agent
How to use ddark-il/Qwen3.8-Flash-Next-Uncensored with Hermes Agent:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "ddark-il/Qwen3.8-Flash-Next-Uncensored"
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default ddark-il/Qwen3.8-Flash-Next-Uncensored
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use ddark-il/Qwen3.8-Flash-Next-Uncensored with OpenClaw:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "ddark-il/Qwen3.8-Flash-Next-Uncensored"
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id "ddark-il/Qwen3.8-Flash-Next-Uncensored" \ --custom-provider-id mlx-lm \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Qwen3.8-Flash-Next-Uncensored โ MLX mixed-precision 5.97 bpw (v2.2, oQe rounding + INT8-prefill lattice)
Mixed-precision affine quantization of
orcarouter/Qwen3.8-Flash-Next-Uncensored
(qwen4_exp, 180B) sized for an M5 Max 128 GB at up to 262k context.
134.30 GB ยท 5.969 bpw ยท 180.0B params ยท 2.68ร vs bf16 (360 GB)
This is v2.2, the revision after v2: same family map, but every per-group quantization range is chosen against the importance-weighted objective (oQe), the n-gram table moves to 8-bit, the whole artifact sits on a single 4-or-8-bit group-size-64 lattice โ and therefore on the runtime's INT8-prefill (A8) grid. It is +25.01 GB (+22.9 %) over v2 and โ0.2 GB resident, and it leads v2 on the output-gated comparison: 72.3 vs 70.8 overall, 16/20 vs 15/20 at the 96-token budget.
What changed vs. v2
- oQe rounding: ranges chosen for the importance-weighted error. The v2 recipe searched
per-group ranges against uniform grids scored on plain rel-L2. Here each group's
(scale, bias)โ and the search that picks it โ is driven by an importance matrix accumulated over 65,536 calibration tokens (128 sequences ร 512) of thedsh_verifyRAG corpus, then refit by weighted least squares. On identical tensors this trades plain error for weighted error: experts measure 0.09130 plain / 0.07662 weighted against v2's 0.08538 plain full-sweep figure, i.e. 10.3 % lower on the objective the quantizer optimises, 6.9 % higher on the one it does not. The L1 gate is therefore read onweighted. - N-gram PLE table 4/32 โ 8/32. 51.2B parameters, the largest accuracy term of the v2 build:
rel-L2 0.07335 โ 0.00484 (โ93.4 %), table 32.0 โ 57.6 GB. It stays
mmap, so the extra 25.6 GB costs 0 resident memory. 8/32 is not a preference but the ceiling โ the runtime ships a Metal dequantize kernel only for gs 32 at 8 bits on 160-wide rows; gs 8/16/20/40/80/160 have no kernel at all. - One lattice: 4 or 8 bits, group size 64. The 6-bit tier and the 8/128 groups are gone. MTP head 8/8/6/6 โ 8/64 throughout, shared experts 8/128 โ 8/64 (+7.5 MB; 8/128 is off the A8 grid and measures 0.76ร bf16), vision tower 8/128 โ 8/64, hyper-connections bf16 โ 8/64. Promotions and re-groupings together cost 0.004 GB per decode step and nothing on disk for the 8-bit families; only the PLE table grows.
- INT8 prefill (A8) compliance. Every quantized family is on the A8 grid (Q4/Q5/Q8 at gs64 for dense projections and experts; the PLE table is a gather, not a matmul). Measured on the artifact: a 4,096-token prefill runs 987 โ 1,066 tok/s, 1.08ร with the INT8 path enabled, and a live A/B on ~2,560-token prompts measures ~+7 % TTFT (medians 2.25 vs 2.42 s).
- Stored in the runtime's own expert layout. The routed experts are written as stacked
switch_mlp.{gate,up,down}_projrather than v2's fused source layout โ 294 fused tensors repacked into 441 stacked ones, byte-identical payload โ so the artifact loads through the currentqwen4_exppath (and throughmlx-vlm's ownsplit_expert_projection) with no runtime patch.
Files
131 safetensors shards (model-00001โฆ00131-of-00131) plus the MTP sidecar
(model-mtp.safetensors), config.json, generation_config.json, chat_template.jinja,
tokenizer.json / tokenizer_config.json / vocab.json / merges.txt,
preprocessor_config.json, video_preprocessor_config.json. 3,781 tensors:
133.77 GB quantized + 0.54 GB exact bf16. The vision tower is inline in the main shards; the
MTP head is the sidecar. Routed experts are stored stacked (switch_mlp.*).
Measured accuracy
The reference is the bf16 weights of the same source checkpoint for tensor-level error, and the same hosted bf16 oracle + fixed 21-case corpus that scored v2 for output-level numbers, so the v2 column is directly comparable.
| metric | v2 | v2.2 (this model) |
|---|---|---|
| Artifact | 109.29 GB ยท 4.857 bpw | 134.30 GB ยท 5.969 bpw |
| Quantized / exact modules | 743 / 761 | 1037 / 466 |
Resident (PLE mmap) |
76.9 GB | 76.70 GB |
| N-gram table rel-L2 (per shard) | 0.07335 (4/32) | 0.00484 (8/32, โ93.4 %) |
| Routed experts rel-L2 | 0.08538 (plain, full sweep) | 0.09130 plain / 0.07662 weighted |
Dense rel-L2 (lm_head, 8/64) |
โ | 0.00600 |
| L2 oracle, 96-token budget | 70.77 / 100, 15/20 | 72.27 / 100, 16/20 |
| Top-1 agreement with the bf16 oracle | 90.7 % | 86.2 % |
| A/B against v2, same captures | โ | 16/21 identical, prefix agreement 0.9902, mean top-1 0.8868, mean JSD 0.05355 |
| INT8 prefill (4,096 tok) | not on the A8 grid | 1.08ร (987 โ 1,066 tok/s) |
Metric discipline. The experts row is not a like-for-like plain-error comparison: v2's 0.08538 is a full 1,707-tensor sweep; v2.2's 0.09130/0.07662 come from the installed artifact on a 2M-element slice. The gate v2.2 is accepted on is the importance-weighted one (0.07662 vs the 0.085 gate), which is the objective oQe optimises; its plain figure is above the older plain gate by design. The 96-token row is the like-for-like pair โ both captures were scored against the same oracle, same corpus, same budget. Top-1 agreement with the oracle is lower than v2's even though the task score is higher: agreement is baseline-relative, so a build that rephrases correctly scores lower on it.
Quantization map
Base format: MLX affine โ packed U32 weights + bf16 scales + bf16 biases, i.e.
2 ร 16 / group_size bits of overhead per weight on top of the nominal width. 1,121 explicit
per-module config entries; 1,037 quantized modules.
| Module | Bits | Group | Params | Size | Rationale |
|---|---|---|---|---|---|
| Routed experts MLP (48 ร 512) | 4 | 64 | 120.8B | 67.95 GB | dominant weight mass; the only grid with an expert-A8 prefill path and the fused MoE decode kernel |
| N-gram PLE table (128 shards) | 8 | 32 | 51.2B | 57.60 GB | mmap โ 0 resident; gs32 is structural (rows are 160 wide) and the only loadable 8-bit grouping |
Dense GDN + attention (in_proj_*, out_proj, q/k/v/o_proj) |
8 | 64 | 2.70B | 2.89 GB | W8A8 prefill path, A8-eligible |
| MTP draft head (experts, attn, shared, FC, indexer) | 8 | 64 | 2.61B | 2.78 GB | the whole head on one lattice; the trunk verifies every token, so its precision costs acceptance |
lm_head, embed_tokens |
8 | 64 | 1.27B | 1.35 GB | A8-eligible; lm_head is the largest single matvec per step |
| Shared experts | 8 | 64 | 0.24B | 0.25 GB | A8 grid (v2's 8/128 was off it); costs +7.5 MB |
| Hyper-connections (per-layer matmuls) | 8 | 64 | 0.64B | 0.69 GB | quantized โ the fused HC kernels require it; v2 kept these bf16 |
| Vision tower | 8 | 64 | 0.45B | 0.60 GB | one lattice with everything else; linear_fc2 and patch_embed admit no group size at any width and stay exact |
| Everything else (norms, routers, convs, trunk indexer, PLE machinery) | exact bf16 | โ | 0.10B | 0.19 GB | routing is discrete, norms are cheap |
Byte accounting measured from the checkpoint: 133.77 GB quantized + 0.54 GB exact bf16.
NAX (M5 tensor unit) eligibility
100 % of quantized bytes are eligible โ 133.77 of 133.77 GB, verified against the kernel grid
that ships inside the app's mlx.metallib rather than from documentation, with the artifact's
own module list:
| path | format | modules | GB | NAX kernel |
|---|---|---|---|---|
| gather | affine 4/64 (trunk experts) | 144 | 67.95 | yes |
| dense | affine 8/32 (n-gram PLE shards) | 128 | 57.60 | yes |
| dense | affine 8/64 (attn/GDN/lm_head/vision/HC/MTP-dense) | 762 | 5.55 | yes |
| gather | affine 8/64 (MTP experts) | 3 | 2.67 | yes |
Two caveats, unchanged from v2. Eligibility is not engagement โ the dispatch heuristic lives
in compiled host code. Measured: the quantized dense NAX kernels reach 31.7 / 34.6 / 33.3 TFLOP/s
at gs32-b8 / gs64-b4 / gs64-b8 against a 50.9 TFLOP/s bf16 reference, i.e. uniformly
1.47โ1.61ร slower โ the expected small-tile penalty at compute-bound sizes, with no SIMD
fallback outlier. The gather path is engaged in the served runtime (sorted gather_qmm reroute
installed, MoE routed-expert decode engaged, routed A8 gate/up on 48 layers, 35โ57 tok/s at
14kโ25k-token prompts with MTP drafting). And at prefill sizes quantization is still the slower
option: the A8 path is what recovers part of that (1.08ร over the same artifact's own A8-off
run), not a bf16 parity claim โ a bf16 prefill baseline does not fit on this machine.
How it was made
Streaming, tensor-by-tensor quantization; the map is applied from a header-only preflight
(export_v5.py --validate-map) that reads every source tensor and asserts the result before any
bytes are written. 1658 source tensors, 0 unmatched.
- Calibration. The imatrix comes from the
dsh_verifyRAG database: 128 sequences ร 512 tokens, expert coverage capped at 1,024 sequences (~6โ9 min of collection on this machine). It measures per-input-channel importance per module, which is what the weighted refit and the oQe range search consume. - oQe. Per group, candidate
(scale, bias)pairs are scored against the importance-weighted error ofmx.dequantizeoutput on the stored bf16 parameters โ the runtime's own dequantizer โ and the built-in min/max result is always among the candidates, so no group can come out worse than v2's rule on the objective being scored. - Layout repack. After the build, the 294 fused expert tensors were rewritten as 441 stacked
switch_mlptensors (weights, scales and biases moved together; byte-identical payload, verified bymx.array_equalon a gate/up pair and a down projection) and the config entries were re-keyed. This is what makes the artifact loadable on the current loader without patches. - MTP head. Quantized by this build (v2 grafted it byte-identically) and merged as the
sidecar; the stock graft helper does not accept a
qwen4_exprecipient, so the merge is explicit. - Tokenizer, chat template, preprocessor and generation configs are the source's own files.
- No training, fine-tuning, distillation or weight editing: this is quantization only. The QLoRA error-correction path is implemented but blocked โ the fused kernels have no VJP and a bf16 training copy cannot be co-resident with the quantized model on 128 GB.
Requirements and serving
Needs oMLX 0.8.0rc1 (the release that resolves its own vendored qwen4_exp) and an M5-class
GPU. Verified on an M5 Max 128 GB: loads in 11.0 s, 75.01 GB resident, generates through
the OpenAI API, MTP drafting at 3.2 tok/cycle with 9/10 accepted on a smoke test, and
92.3 tok/s with 0.12 s TTFT on the recorded session.
- Set PLE to
mode="mmap"(qwen4_ple_ssd_offload). Residency with the 57.6 GB table in RAM does not fit; the engine forces the table to SSD and serves it through page cache. a8_prefill_enabled: true,a8_prefill_min_tokens: 128. These are model settings, not request parameters. Below the floor, prefill stays on the bf16 path.mtp_enabled: true. Lightning MTP decode; the"enable_mtp"request field is ignored.- Raise the Metal wired limit:
sudo sysctl iogpu.wired_limit_mb=120000. - Capping served context at 128k halves KV; only 12 of 48 layers cache KV (2-head GQA, 24 KB/token) and GDN state is context-independent.
- Speculative-prefill token pruning does not apply: GDN state needs every token and pruning breaks n-gram hash adjacency.
Verification
| rung | tool | result |
|---|---|---|
| L0 structure | audit_mlx_quant.py --profile v5 |
all structural checks passed โ 3,781 tensors, 1,037 quantized + 466 exact modules, complete weight/scales/biases triplets, index โ disk parity |
| L1 weights | l1_v5.py |
dense 0.00600, PLE 0.00484, experts 0.09130 plain / 0.07662 weighted โ all within their gates, experts on the weighted gate |
| L2 outputs | compare_captures.py, oracle.py |
A/B vs v2: 16/21 identical, prefix 0.9902, top-1 0.8868; oracle 72.27/100, 16/20 |
| A8 prefill | bench_a8_v5.py |
4,096-token prefill 987 โ 1,066 tok/s (1.08ร) with INT8 on |
| NAX | nax_compat.py |
100 % of quantized bytes eligible; dense engagement measured, gather path engaged in the serving logs |
| Serving | oMLX API | loads in 11.0 s (PLE mmap, no Hadamard patch), generates, MTP active |
Limitations
- The experts' plain rel-L2 is above the older gate by construction. oQe buys importance-weighted accuracy at the cost of plain error (+6.9 % here). Reading this build on plain rel-L2 alone measures the wrong objective.
- The output-level reference is a hosted bf16 endpoint, not this checkpoint's own bf16 weights.
Agreement/JSD therefore include whatever the
-Uncensoredvariant changed relative to Qwen's official model. The v2 โ v2.2 comparison is sound because both were measured the same way against the same reference; the absolute scores are not a quantization-error measurement. - The evaluation corpus is small (21 cases) and three of its checks are known to be uninformative (two expectations are provably wrong, one cannot be satisfied by any continuation). Treat the pass count as indicative only.
- MTP acceptance and long-context behaviour are smoke-tested, not swept. 3.2 tok/cycle at 90 % acceptance came from a 16-token request; the 32k/128k needle was not re-measured on this build.
- The fused hyper-connection kernel is engaged per the post-load layout check, but its speed contribution was not isolated in a measurement; the HC family is 0.69 GB of the file.
- This is an uncensored derivative of the upstream model, inherited from the base checkpoint; see the base model card for its behaviour and safety considerations.
Attribution
Quantization recipe and measurement tooling by the qwen3.8-next-flash project. Base weights:
orcarouter/Qwen3.8-Flash-Next-Uncensored. Architecture and the original model: Qwen.
Quantizer/runtime: oMLX and MLX. Licensed Apache-2.0,
following the base model.
- Downloads last month
- 1,938
Quantized