Instructions to use prism-ml/Ternary-Bonsai-2-9B-gguf with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use prism-ml/Ternary-Bonsai-2-9B-gguf with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf prism-ml/Ternary-Bonsai-2-9B-gguf:Q8_0 # Run inference directly in the terminal: llama cli -hf prism-ml/Ternary-Bonsai-2-9B-gguf:Q8_0
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf prism-ml/Ternary-Bonsai-2-9B-gguf:Q8_0 # Run inference directly in the terminal: llama cli -hf prism-ml/Ternary-Bonsai-2-9B-gguf:Q8_0
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf prism-ml/Ternary-Bonsai-2-9B-gguf:Q8_0 # Run inference directly in the terminal: ./build/bin/llama cli -hf prism-ml/Ternary-Bonsai-2-9B-gguf:Q8_0
Use Docker
docker model run hf.co/prism-ml/Ternary-Bonsai-2-9B-gguf:Q8_0
- Llama new
- LM Studio
- Jan
- vLLM
How to use prism-ml/Ternary-Bonsai-2-9B-gguf with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "prism-ml/Ternary-Bonsai-2-9B-gguf" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "prism-ml/Ternary-Bonsai-2-9B-gguf", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/prism-ml/Ternary-Bonsai-2-9B-gguf:Q8_0
- Ollama
How to use prism-ml/Ternary-Bonsai-2-9B-gguf with Ollama:
ollama run hf.co/prism-ml/Ternary-Bonsai-2-9B-gguf:Q8_0
- Unsloth Desktop
- Pi
How to use prism-ml/Ternary-Bonsai-2-9B-gguf with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf prism-ml/Ternary-Bonsai-2-9B-gguf:Q8_0
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "prism-ml/Ternary-Bonsai-2-9B-gguf:Q8_0" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use prism-ml/Ternary-Bonsai-2-9B-gguf with Docker Model Runner:
docker model run hf.co/prism-ml/Ternary-Bonsai-2-9B-gguf:Q8_0
- Lemonade
How to use prism-ml/Ternary-Bonsai-2-9B-gguf with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull prism-ml/Ternary-Bonsai-2-9B-gguf:Q8_0
Run and chat with the model
lemonade run user.Ternary-Bonsai-2-9B-gguf-Q8_0
List all available models
lemonade list
- Hermes Agent
How to use prism-ml/Ternary-Bonsai-2-9B-gguf with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf prism-ml/Ternary-Bonsai-2-9B-gguf:Q8_0
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default prism-ml/Ternary-Bonsai-2-9B-gguf:Q8_0
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use prism-ml/Ternary-Bonsai-2-9B-gguf with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf prism-ml/Ternary-Bonsai-2-9B-gguf:Q8_0
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "prism-ml/Ternary-Bonsai-2-9B-gguf:Q8_0" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Prism ML Website | Demo & Examples | Discord
Bonsai 2 Ternary 9B: GGUF
A 9B-class vision-language model with ternary transformer weights, for llama.cpp (CUDA, Metal, CPU).
~2.42 GB language model (PQ2_0), down from ~17.9 GB at FP16
Highlights
- Ternary language weights across embeddings, attention projections, MLP projections and the LM head, stored with a group-wise FP16 scale.
- Three GGUF packings of the same weights, consumed directly by custom kernels and never expanded back to FP16: PTQ1_0 (dense trits, 2.00 GB), PQ2_0 (2-bit slots, 2.42 GB) and Q2_0 (2.56 GB).
- Vision input through a separate mmproj pack (Q8_0, or BF16 for reference precision), loaded only when an image arrives.
- 262K-token context, inherited from the Qwen3.5-9B hybrid-attention backbone (24 linear-attention and 8 full-attention layers).
Model Overview
| Item | Specification |
|---|---|
| Base model | Derived from Qwen3.5-9B (architecture unchanged) |
| Parameters | ~8.95B language model (32 blocks, hidden 4096, separate embedding and LM head) + ~0.46B vision tower (27 blocks) |
| Architecture | Hybrid attention (3 linear-attention : 1 full-attention), SwiGLU MLP, RoPE, RMSNorm |
| Context length | 262,144 tokens (inherited from the base model) |
| Weight format | Ternary {β1, 0, +1} with FP16 group-wise scales, packed as PTQ1_0, PQ2_0 or Q2_0 |
| Weight basis | Blockwise Hadamard rotation folded into the stored weights; the runtime applies the matching transform to activations |
| Higher-precision tensors | Normalization weights and the linear-attention state path (ssm_alpha, ssm_beta) stay F32 |
| Vision tower | Optional mmproj pack: Q8_0 (0.62 GB, recommended) or BF16 (0.92 GB) |
| Backends | llama.cpp (CUDA, Metal, CPU): every file runs on the PrismML fork; Q2_0 also runs on upstream llama.cpp b11556 or newer |
| License | Apache 2.0 |
Files
| File | Contents | Size | Bits/weight (language model) |
|---|---|---|---|
Ternary-Bonsai-2-9B-PTQ1_0.gguf |
language model, dense trits | 2.00 GB | ~1.78 |
Ternary-Bonsai-2-9B-PQ2_0.gguf |
language model, 2-bit slots | 2.42 GB | ~2.16 |
Ternary-Bonsai-2-9B-Q2_0.gguf |
language model, Q2_0 | 2.56 GB | ~2.28 |
Ternary-Bonsai-2-9B-mmproj-Q8_0.gguf |
vision tower | 0.62 GB | |
Ternary-Bonsai-2-9B-mmproj-BF16.gguf |
vision tower (reference precision) | 0.92 GB | |
Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf |
PQ2_0 model with its MTP draft head built in | 2.68 GB | |
Ternary-Bonsai-2-9B-DFlash2-Q8_0.gguf |
DFlash2 block drafter (optional) | 1.57 GB |
Bits/weight is file size over the ~8.95B language-model parameters, so it includes the F32 tensors and metadata. The three language-model packs hold the same ternary weights; they differ only in storage layout and kernel speed. The MTP file is the PQ2_0 model plus one extra draft layer, so you load it instead of the plain PQ2_0, not alongside it.
Quickstart
PrismML-Eng/Bonsai-demo is the source of truth for running these models. It has the tested setup for every backend and pins a known-good binary release.
Use the PrismML llama.cpp build
The ternary kernels live in the PrismML-Eng/llama.cpp fork (branch prism). It runs every file here, including MTP and DFlash2. Upstream llama.cpp runs the Q2_0 file from release b11556 onward (see Use upstream llama.cpp below). It doesn't have the PTQ1_0 and PQ2_0 types yet, so it rejects those files as unknown types.
# prebuilt: pick the archive for your platform
# https://github.com/PrismML-Eng/llama.cpp/releases/latest
tar -xzf llama-<tag>-bin-<platform>.tar.gz -C bin --strip-components=1
# or build it
git clone https://github.com/PrismML-Eng/llama.cpp && cd llama.cpp
cmake -B build -DGGML_CUDA=ON && cmake --build build -j # drop -DGGML_CUDA=ON on macOS; Metal is the default
Text
./bin/llama-cli -m Ternary-Bonsai-2-9B-PQ2_0.gguf -ngl 99 -fa on \
-p "Explain in three sentences why the sky is blue."
Image + text
./bin/llama-mtmd-cli -m Ternary-Bonsai-2-9B-PQ2_0.gguf \
--mmproj Ternary-Bonsai-2-9B-mmproj-Q8_0.gguf -ngl 99 \
--image photo.jpg -p "Describe this image."
Vision projector: use the Q8_0 mmproj. With the current fork build, the BF16 mmproj can crash llama-server when several image requests run at once; Q8_0 is not affected and quality is nearly the same.
Small text in images: for small or dense text, raising the image token budget with --image-min-tokens 1024 may help.
OpenAI-compatible server
./bin/llama-server -m Ternary-Bonsai-2-9B-PQ2_0.gguf \
--mmproj Ternary-Bonsai-2-9B-mmproj-Q8_0.gguf -ngl 99 -fa on -c 32768 --port 8080
curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
"messages": [{"role": "user", "content": [
{"type": "text", "text": "What is in this image?"},
{"type": "image_url", "image_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/4/47/PNG_transparency_demonstration_1.png"}}
]}],
"chat_template_kwargs": {"enable_thinking": false}
}'
Use upstream llama.cpp
The Q2_0 file also runs on stock llama.cpp from release b11556 onward, which added the Hadamard activation runtime these weights need. Older builds lack that runtime and give wrong output without any warning, so use a current release. Text and image input both work:
hf download prism-ml/Ternary-Bonsai-2-9B-gguf Ternary-Bonsai-2-9B-Q2_0.gguf Ternary-Bonsai-2-9B-mmproj-Q8_0.gguf --local-dir .
llama-cli -m Ternary-Bonsai-2-9B-Q2_0.gguf -ngl 99 -fa on \
-p "Explain in three sentences why the sky is blue."
llama-mtmd-cli -m Ternary-Bonsai-2-9B-Q2_0.gguf \
--mmproj Ternary-Bonsai-2-9B-mmproj-Q8_0.gguf -ngl 99 \
--image photo.jpg -p "Describe this image."
llama-server takes the same -m and --mmproj arguments as above.
Best Practices
Generation Parameters
We recommend these sampling settings. They are the same for every packing and for MLX:
- Thinking mode (default):
temperature=1.0,top_p=0.95,top_k=20,min_p=0.0,presence_penalty=1.0,repetition_penalty=1.0- Non-thinking mode:
temperature=0.7,top_p=0.80,top_k=20,min_p=0.0,presence_penalty=1.5,repetition_penalty=1.0
Apply the penalties over the last 64 tokens (llama.cpp's default window). Presence penalty has no GGUF metadata key, so pass it on the command line or per request.
# thinking on
./bin/llama-server -m Ternary-Bonsai-2-9B-PQ2_0.gguf -ngl 99 -fa on \
--temp 1.0 --top-p 0.95 --top-k 20 --min-p 0 --presence-penalty 1.0 --repeat-penalty 1.0 --repeat-last-n 64
# thinking off: start the server with these values, and send
# "chat_template_kwargs": {"enable_thinking": false} with each request
./bin/llama-server -m Ternary-Bonsai-2-9B-PQ2_0.gguf -ngl 99 -fa on \
--temp 0.7 --top-p 0.8 --top-k 20 --min-p 0 --presence-penalty 1.5 --repeat-penalty 1.0 --repeat-last-n 64
Avoid greedy decoding (temperature=0) with thinking on; it can get stuck reasoning on some prompts.
Output Length
Give thinking mode room to finish. Even simple prompts can use 3,000+ thinking tokens, so allow at least 8,192 output tokens with thinking on. For short answers, turn thinking off.
System Prompt
You can use a simple system prompt such as:
You are a helpful assistant
MLX
On Apple Silicon you can also use the MLX build of the same weights, Ternary-Bonsai-2-9B-mlx-2bit. It loads natively in mlx-vlm 0.7.8 and newer (the released mlx-lm cannot load it yet). Pass the values above explicitly: mlx-vlm's Python generate() is greedy by default. Its model card has the commands.
Choosing a Packing
- PQ2_0: the default; cheaper to unpack, and the pack measured on Apple Silicon.
- PTQ1_0: the smallest file; use it where memory is tightest.
- Q2_0: an alternative layout, and the one format upstream llama.cpp can also run (b11556 or newer).
Speculative Decoding
Two optional drafters ship with the model. Both need the PrismML fork.
Recommended: MTP. Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf is the PQ2_0 model with a multi-token-prediction head built in. Draft depth 2 works best:
./bin/llama-server -m Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf -ngl 99 -fa on \
--spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-n-min 0 --spec-draft-p-min 0
KV cache with MTP on CUDA: quantized KV (--cache-type-k q8_0 --cache-type-v q8_0) works with MTP from release prism-b10776-d00e252 onward. On older CUDA builds, keep the default f16 KV cache (or quantize only K) when MTP is on. Metal is not affected.
Optional: DFlash2. A block drafter for Ternary-Bonsai-2-9B-PQ2_0.gguf. It helps on copy-heavy work (code edits, rewrites) and is slower than plain decoding on general chat, so prefer MTP by default:
./bin/llama-server -m Ternary-Bonsai-2-9B-PQ2_0.gguf -md Ternary-Bonsai-2-9B-DFlash2-Q8_0.gguf \
-ngl 99 -ngld 99 --spec-type draft-dflash
On CUDA, speculative outputs can differ slightly from plain decoding where two tokens are nearly tied, so exact-match comparisons against non-speculative runs will show small differences.
License
Apache 2.0, following the base model.
- Downloads last month
- 1,597,946
1-bit
2-bit
8-bit
Model tree for prism-ml/Ternary-Bonsai-2-9B-gguf
Base model
Qwen/Qwen3.5-9B-Base