JRamirez-UAB/Qwopus3.8-27B-Coder-Flash-EXL3-2.25bpw-24GB

Qwopus3.8-27B-Coder-Flash — EXL3 2.25 bpw: two full 262k sessions on one 24 GB GPU (int4 KV, served with vLLM)

Install from this card, not from generic snippets. This is an EXL3 checkpoint: vLLM loads it only together with exllamav3 and our vllm-quantpack plugin. Follow Requirements & install and Quick start; a plain pip install vllm, Transformers or SGLang cannot load it.

An unofficial community EXL3 quantization of Jackrong/Qwopus3.8-27B-Coder-Flash, the coding- and agent-oriented 27B dense model built on Qwen3.8-27B, made with exllamav3 1.6.5: 2.25 bpw on the linear layers, a 6-bit lm_head, a 6-bit vision tower and a 4-bit MTP head. The calibration data includes agent steps written on our sysadmin prompts (see Quantization recipe).

This is the capacity build, not the quality build. Its weights are 11.19 GB instead of 14.99 GB, and that difference buys a second full-context session: on one RTX 3090 (24 GB) with vLLM's int4 KV cache it serves two sessions of the full 262,144-token context at the same time (KV pool 586,300 tokens). The price is quality: its KL divergence against BF16 is about 7 times that of our 3.50 bpw build, it fails some agent tasks the 3.50 passes, and it loops on long answers even at temperature 1.0 (8 of 30). Read Which one should you use? before choosing it.

⚠️ Long answers can loop or come back empty. At the recommended temperature 1.0, 8 of 30 long Spanish answers fell into a repetition loop and 10 of 30 returned no answer text (only reasoning) within 16,000 tokens. Keep a max_tokens limit, and retry or shorten the question when an answer loops. For tasks where every answer matters, use the 3.50 build.

⚠️ Do not use temperature 0 (greedy decoding). The base model loops inside its reasoning at temperature 0 and never writes the answer: we measured repetition 0.93-0.98 with the original BF16 model on a long question (3,000 and 12,000 tokens, no answer text), and 0.87 with the 3.50 build. It is a trait of the base fine-tune, not of the quantization. Use the default sampling (temperature 1.0, top_p 0.95, top_k 20 from generation_config.json).

License: Apache-2.0, inherited from the base model. This is not an official release of Jackrong or Qwen.

Tested setup: RTX 3090 (24 GB, Ampere), vLLM 0.31.0 + exllamav3 1.6.5 + vllm-quantpack 0.1.4, --max-model-len 262144, int4 KV cache, text only (--language-model-only), 2026-11-02. The tests read our local copy of these files (same bytes, SHA256SUMS). Only this setup has been tested.

Highlights (one RTX 3090 24 GB, full 262,144 context, int4 KV)

• Two full-context sessions at once: 2 real concurrent sessions of 235k tokens (8/8 facts found in each), 1,796 MiB really free at peak.
• KLD against BF16: 0.0888 on an opencode agent trace, 0.1087 on our sysadmin trace and 0.0798 on turboderp's public trace (3.50 build: 0.0127 / 0.0148 / 0.0104).
• Agent tasks (opencode run, 10 sysadmin tasks × 3 runs): 8/10, 9/10, 8/10 = 25/30 (3.50 build: 30/30).
• Tool calls: 29/30 in the battery, 0 errors.
• Programming tasks: 11/12, 12/12, 12/12 = 35/36.
• Loops: 14/30, 13/30 and 8/30 at temperature 0.6, 0.8 and 1.0 on long Spanish answers (3.50 build: 12 / 6 / 0).
• Needle: 8 facts in one prompt at 64k / 128k / 241k-258k tokens: 8/8 each time, on two different machines.
• Speed: prefill / decode with one request near 15k · 64k · 128k · 260k tokens: 1,134 / 48.7 · 980 / 41.8 · 822 / 35.5 · 618 / 27.2 tok/s, about 7-13 % faster decode than the 3.50 build.

Our focus. We quantize and evaluate models for one use case: a local Linux system-administration agent (opencode with tool calling) for a university IT department, used in Spanish. Our calibration traces include real sysadmin/agent conversations in Spanish, and our quality checks (tool calls, agent tasks with real command execution, long-context retrieval, long answers in Spanish) reflect that use. KL divergence is also reported on turboderp's public trace, so it can be compared and reproduced. Numbers for other domains or languages may differ.

Which one should you use?

Both builds come from the same model, the same calibration data and the same exllamav3 version; only the bit rate differs.

	2.25 bpw (this repo)	3.50 bpw
Weights	11.19 GB	14.99 GB
Full 262k sessions at once on one RTX 3090	2	1
KV pool on one RTX 3090	586,300 tokens	377,712 tokens
KLD vs BF16 (opencode / public / sysadmin)	0.089 / 0.080 / 0.109	0.013 / 0.010 / 0.015
Agent tasks (3 × 10)	25/30	30/30
Tool calls (30)	29	30
Programming tasks (3 × 12)	35/36	36/36
Loops at temperature 1.0 (30 long answers)	8	0
Decode near 260k, one request	27.2 tok/s	25.4 tok/s
Vision and MTP measured	no	yes

Pick this 2.25 build when two people (or two agents) must work with very long contexts on the same 24 GB card at the same time, and an occasional failed task or looping answer is acceptable because a human reviews the result and can retry.

Pick the 3.50 build when one long session is enough, or when correctness matters more than concurrency: it passed every agent task, did not loop at temperature 1.0 and is about 7 times closer to BF16. If you have two GPUs, two copies of the 3.50 build are better than one 2.25.

Shorter sessions share the pool in both builds: by vLLM's count this build holds 4 sessions of up to 146k or 8 of up to 73k tokens at once (not load-tested at those sizes).

What is in this repository

File	What it is
model-00001-of-00002.safetensors, model-00002-of-00002.safetensors, model.safetensors.index.json	EXL3 weights (11.19 GB) and their index
quantization_config.json	exllamav3's per-tensor storage record (bits per tensor; calibration size only, no text)
config.json, generation_config.json	model config; sampling defaults of the base model
chat_template.jinja, tokenizer.json, tokenizer_config.json	chat template and tokenizer of the base model, unchanged
preprocessor_config.json, processor_config.json, video_preprocessor_config.json	image/video processor of the base model
SHA256SUMS	SHA-256 of every file above

Not included: the BF16 reference, the calibration data (internal), the plugin (install it from PyPI, see below).

Results

All tables: one RTX 3090 24 GB (no other use), vLLM 0.31.0 + exllamav3 1.6.5 + vllm-quantpack 0.1.4, --max-model-len 262144, --kv-cache-dtype int4_per_token_head, text only, the Quick start configuration, measured on 2026-11-02 unless stated otherwise.

Quantization quality (exllamav3 qbench, against BF16)

Trace	KLD mean	KLD median	KLD p90	Perplexity (BF16)
opencode agent trace (ours)	0.0888	0.0290	0.245	2.7367 (2.5982)
sysadmin trace (ours)	0.1087	0.0351	0.303	1.8631 (1.6332)
public (qbench_prompts_gen.json, turboderp)	0.0798	0.0099	0.244	1.5668 (1.4128)

How to read it: the median is low, so most tokens are predicted almost as BF16 does, but the p90 is about 7 times that of the 3.50 build: one token in ten differs clearly from BF16. Those tokens are where the extra loops and failed tasks come from.

Capacity on one RTX 3090 (24 GB)

--kv-cache-memory-bytes	KV pool reported by vLLM	Real concurrent sessions of 235,929 tokens	Peak used	Really free at peak
10705205985	586,300 tokens	2 (8/8 facts found in each; a third waits)	22,320 MiB	1,796 MiB

The token embeddings are stored in BF16 (2.54 GB); the KV of this model is about 18 KB per token in int4. Aggregate prefill with 1 / 2 sessions of 235k: 636 / 1,250 tok/s.

A 16 GB card does not fit the full context: with a 5.0e9-byte KV pool (1.04 × 262k) the peak was 16.5 GB.

Test battery

Test	Result
Tool calls, battery	29/30 valid and correct, 0 errors
Agent tasks, opencode run, 10 sysadmin tasks × 3 runs, temperature 1.0	8/10, 9/10, 8/10 (479-725 s per run)
Programming tasks (12 × 3)	11/12, 12/12, 12/12 (2,107-2,542 s per run)
Needle, 8 facts in one prompt	64k 8/8 · 128k 8/8 · 241k 8/8 (second machine: 64k · 128k · 245k · 258k, 8/8 each)
Prefill / decode near 15k · 64k · 128k · 260k	1,134 / 48.7 · 980 / 41.8 · 822 / 35.5 · 618 / 27.2 tok/s
Decode with 1 / 2 / 4 / 8 short requests	50 / 72 / 132 / 244 tok/s in total
Agents at once, tasks passed (runs that reached the 600 s limit)	4: 8/8 (1) · 8: 14/16 (7) · 16: 27/32 (14)
Time to first token of a new 32k prompt (no cache / cached), with 4 · 8 · 16 agents running	29.9 / 1.6 s · 30.2 / 2.0 s · 33.9 / 7.7 s

With sampling on, the same checkpoint varies by about ±2 of 30 between runs; the gap with the 3.50 build in agent tasks (25 against 30) is larger than that.

Comparison: the five Qwopus3.8-Coder-Flash builds we tested

Same model, two bit rates and three calibration choices; same card, settings and tests as above.

Build	KLD opencode / public / sysadmin	Real 236k sessions	Programming	Loops /30 at T 0.6 / 0.8 / 1.0	Tools (30)	Agent (3 × 10)
2.25, no calibration data	0.0964 / 0.0860 / 0.1205	2	35/36	24 / 16 / 21	27	27/30
2.25, calibration as this repo (this repo)	0.0888 / 0.0798 / 0.1087	2	35/36	14 / 13 / 8	29	25/30
2.25, steps written by Qwopus	0.0869 / 0.0810 / 0.1081	2	36/36	26 / 16 / 10	30	27/30
3.50, calibration as this repo (repo)	0.0127 / 0.0104 / 0.0148	1	36/36	12 / 6 / 0	30	30/30
3.50, steps written by Qwopus	0.0126 / 0.0105 / 0.0152	1	35/36	18 / 7 / 2	30	28/30

Why this 2.25 and not the others: the three 2.25 builds are within about 10 % in KL divergence, and the agent and tool differences between them are inside the run-to-run noise, but this one loops clearly less (8 of 30 at temperature 1.0, against 21 and 10). At 2.25 bpw the calibration data lowers the KL divergence by 8-10 % against no calibration.

Sampling and repetition loops

5 long Spanish sysadmin questions × 6 runs per temperature, up to 16,000 tokens, top_p 0.95 and top_k 20; a loop = more than 30 % repeated word 8-grams.

Temperature	0.6	0.8	1.0
Loops / 30	14	13	8
Answers that finished before 16,000 tokens / 30	19	19	22
Answers with no text (only reasoning) / 30	13	13	10
Mean tokens per answer	9,767	9,885	10,039

Use temperature 1.0 (the default in generation_config.json); lower temperatures loop more, and temperature 0 does not work with this model (see the warning at the top). Even at 1.0 about a quarter of long answers loop: in an agent this shows up as a turn that keeps reasoning until max_tokens, so keep a limit and let the agent retry.

Known model behaviour and discarded options

• Speed: a dense 27B; one request decodes at 49 tok/s near 15k tokens and 27 tok/s near 260k on a 3090, so long agent tasks take minutes. With 8 or 16 agents at once on one card many tasks reach a 10-minute limit.
• Temperature 0 loops: greedy decoding loops inside the reasoning on long answers; we measured it in BF16 and in the 3.50 build (see the warning at the top).
• Not measured with this build: vision and MTP speculative decoding. The vision tower (6 bits) and the MTP head (4 bits) are in the files, but every test above used --language-model-only and no speculative decoding. The 3.50 build, with the same vision tower and MTP head bits, passed vision 15/15 with MTP 3.

Requirements & install (vLLM 0.31 + exllamav3 1.6.5 + vllm-quantpack 0.1.4)

Before you install (the same stack as our other cards, checked on a clean Ubuntu 24.04 host on 2026-10-28):

• An NVIDIA driver that supports CUDA 13.2 and uv.
• The Python 3.12 headers and a C compiler: Triton compiles a small C helper every time vLLM starts; without them every vllm serve stops with fatal error: Python.h: No such file or directory. On Debian/Ubuntu: sudo apt-get update && sudo apt-get install -y python3.12-dev build-essential.
• No CUDA toolkit is needed: the FlashInfer line below installs prebuilt kernels.
• Disk: about 20 GB for the venv, as much again for uv's cache (uv cache clean frees it), plus 12 GB for the model.

EXL3 is not a native vLLM format. vllm-quantpack (ours, on PyPI) loads it through vLLM's plugin entry point: EXL3 linear and embedding layers through exllamav3's kernels, the vision tower, the MTP head and a fixed, faster int4 KV cache.

uv venv --python 3.12 venv
uv pip install --python venv/bin/python vllm==0.31.0 --torch-backend=auto        # torch 2.13.0
uv pip install --python venv/bin/python flashinfer-cubin==0.7.0.post1 flashinfer-jit-cache==0.7.0.post1+cu130 \
  --extra-index-url https://flashinfer.ai/whl/ --extra-index-url https://flashinfer.ai/whl/cu130/ --index-strategy unsafe-best-match   # FlashInfer kernels
W=exllamav3-1.6.5+cu132.torch2.13.0-cp312-cp312-linux_x86_64.whl
curl -L -o /tmp/$W "https://github.com/turboderp-org/exllamav3/releases/download/v1.6.5/${W/+/%2B}"
uv pip install --python venv/bin/python /tmp/$W "vllm-quantpack>=0.1.4"
venv/bin/python -c "import torch; from exllamav3.ext import exllamav3_ext; print(torch.__version__, torch.cuda.get_device_name(0))"

Quick start (single RTX 3090, 24 GB)

The configuration of every table above (text only, no speculative decoding, two full-context sessions):

export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True VLLM_USE_FLASHINFER_SAMPLER=0
vllm serve JRamirez-UAB/Qwopus3.8-27B-Coder-Flash-EXL3-2.25bpw-24GB \
  --served-model-name qwopus-coder \
  --language-model-only \
  --max-model-len 262144 \
  --kv-cache-dtype int4_per_token_head \
  --kv-cache-memory-bytes 10705205985 \
  --max-num-seqs 8 \
  --max-num-batched-tokens 8192 \
  --enable-prefix-caching \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder

On a card that also drives a display, lower --kv-cache-memory-bytes by what the desktop uses (each GB less is about 55k tokens less in the pool; below about 9.6e9 bytes the second full-context session no longer fits).

Using it with opencode (how we tested)

limit.context must match --max-model-len (262144 above).

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "local-vllm": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "local vLLM",
      "options": { "baseURL": "http://127.0.0.1:8000/v1" },
      "models": {
        "qwopus-coder": {
          "name": "Qwopus3.8-27B-Coder-Flash EXL3 2.25 (local)",
          "tool_call": true,
          "temperature": true,
          "limit": { "context": 262144, "output": 32768 }
        }
      }
    }
  },
  "agent": {
    "build": { "model": "local-vllm/qwopus-coder", "temperature": 1.0 },
    "plan":  { "model": "local-vllm/qwopus-coder", "temperature": 1.0 }
  }
}

• Declare "temperature": true in the model, otherwise opencode sends no temperature to a custom model.
• With output: 32768 opencode compacts the history at about context − 32k tokens of input, before vLLM rejects the request.

Important notes

• Temperature 1.0, never 0. Lower temperatures loop more on long answers; at temperature 0 the reasoning loops and no answer is written, as with the original BF16 model.
• Review the results. With 25/30 agent tasks and 8/30 looping long answers, this build makes mistakes the 3.50 build does not; keep a human in the loop.
• Memory. The driver reserves about 453 MiB of a 24 GB card that vLLM cannot use: read the real headroom with nvidia-smi --query-gpu=memory.free. Size the KV cache with --kv-cache-memory-bytes, not --gpu-memory-utilization.
• Prefix caching matters for agents: opencode sends the same tools and system prompt in every request, and each turn repeats the whole conversation (a new 32k prompt takes about 30 s, a cached one about 2 s).
• Long prompts: the first turn of a 258k-token prompt takes about 7 minutes to prefill on a 3090; two such prompts at once share the card (1,250 tok/s in total).
• Tool calls work with the base model's own template and --tool-call-parser qwen3_coder; no extra template is needed.
• Hugging Face page warnings: the "Model size" shown by Hugging Face counts packed storage elements, not parameters; the Configuration Parsing Warning about quantization_config.bits not being an integer is expected (EXL3 bits is an average).

Quantization recipe

Step	Setting
Source	Jackrong/Qwopus3.8-27B-Coder-Flash, downloaded on 2026-11-01; chat template and tokenizer unchanged
Tool	exllamav3 1.6.5 convert.py, one RTX 3090
Linear layers	2.25 bpw (-b 2.25)
lm_head / vision tower / MTP head	6 / 6 / 4 bits (-hb 6 -vb 6 -mb 4)
Token embeddings	BF16, unchanged
Calibration	585 rows built from real opencode conversation prefixes of our sysadmin work plus agent steps written by Swift-1.5 (a Qwen3.8-27B fine-tune) in BF16 on our internal prompts; the prefixes are model-independent (same tokenizer). Not published, and no text is stored in the weights

Integrity and validation

Artifact integrity

• SHA256SUMS lists every file; the verifier compares the repo with our local copy byte for byte before publication.
• The safetensors files carry no __metadata__; quantization_config.json holds only bit widths, tensor layout and the calibration size (rows and columns).

Quantization

• KL divergence against the BF16 model with exllamav3's qbench on three traces (Results).

Runtime

• Load, capacity with two full-context sessions, 262k needle, tool calls, agent and programming tasks, agents at once, loops and long prompts: see Results; engine errors during the runs: 0.

Scope: results apply to the stack (vLLM 0.31.0 + exllamav3 1.6.5 + vllm-quantpack 0.1.4), the RTX 3090 and the settings stated above; other setups are not tested.

Limitations

• Review the commands before they run. As an agent, the model proposes shell commands and opencode can execute them: give it a non-privileged account and a test machine where you can.
• Lower quality than the 3.50 build: about 7 times its KL divergence, 25/30 agent tasks against 30/30, and loops on long answers even at temperature 1.0.
• Will not load with stock vLLM, Transformers or SGLang: it needs exllamav3 and vllm-quantpack.
• Not measured: vision, MTP speculative decoding, the real opencode request replay, reasoning problems at xhigh, GPUs other than the RTX 3090. The calibration and our two traces are internal, so those KLD numbers cannot be reproduced outside; the public-trace KLD can.

Feedback welcome

Reports of installations, both working and failed ones, are very welcome in the Community tab: GPU(s), driver, vLLM version, the full command line and the result (speed, context, or the error).

Credits & license

• Base model: Jackrong/Qwopus3.8-27B-Coder-Flash (Apache-2.0), built on Qwen3.8-27B.
• exllamav3 by turboderp (quantization, kernels, qbench and the public trace); EXL3 is based on QTIP (Tseng et al.).
• vLLM; vllm-quantpack (ours, Apache-2.0).
• This quantization is released under the same license as the base model (Apache-2.0).