Ternary Bonsai 2 9B

Ternary Bonsai 2 9B is the mid-size model in the family, for laptops and desktops with 8 GB or more of memory. It is derived from Qwen3.5-9B, a hybrid-attention vision-language model, with the architecture unchanged. Every language-model weight is ternary: {−1, 0, +1} in a fixed rotated basis, with an FP16 scale per group of 128 weights. It takes images as well as text, carries the base model’s 262K-token context, and thinks before it answers by default.

Before you run it: the recommended sampling settings differ between thinking and non-thinking mode, and the 9B wants a presence penalty that the 27B does not. See Best practices.

Specifications


Base model	Qwen3.5-9B (architecture unchanged)
Parameters	~8.95B language model (32 blocks, hidden 4096, separate embedding and LM head) + ~0.46B (27 blocks) vision tower
Architecture	Hybrid attention, 3 linear-attention : 1 full-attention (24 linear-attention and 8 full-attention layers), SwiGLU MLP, RoPE, RMSNorm
Weight format	Ternary g128 with FP16 group-wise scales, blockwise Hadamard rotation folded into the stored weights
Higher-precision tensors	Normalization weights and the linear-attention state path stay F32
Max context	262,144 tokens
Modalities	Text + image in, text out
License	Apache 2.0

Artifacts

Both repositories are in the Bonsai 2 collection on Hugging Face. GGUF: prism-ml/Ternary-Bonsai-2-9B-gguf

File	Contents	Size	Bits/weight
`Ternary-Bonsai-2-9B-PTQ1_0.gguf`	Language model, dense trits	2.00 GB	~1.78
`Ternary-Bonsai-2-9B-PQ2_0.gguf`	Language model, 2-bit slots (the demo’s default)	2.42 GB	~2.16
`Ternary-Bonsai-2-9B-Q2_0.gguf`	Language model, upstream `Q2_0` type	2.56 GB	~2.28
`Ternary-Bonsai-2-9B-mmproj-Q8_0.gguf`	Vision tower (recommended)	0.62 GB
`Ternary-Bonsai-2-9B-mmproj-BF16.gguf`	Vision tower, reference precision	0.92 GB
`Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf`	`PQ2_0` model with its MTP draft head built in (optional)	2.68 GB
`Ternary-Bonsai-2-9B-DFlash2-Q8_0.gguf`	DFlash2 block drafter (optional)	2.54 GB

The three language-model files hold the same ternary weights and differ only in storage layout and kernel speed. Bits/weight is file size over the ~8.95B language-model parameters, so it includes the F32 tensors and metadata. The same model is ~17.9 GB at FP16. MLX: prism-ml/Ternary-Bonsai-2-9B-mlx-2bit, 3.46 GB on disk. model.safetensors carries the language model in MLX’s 2-bit grouped format (group 128) plus the vision tower in FP16. mtp/ holds the optional multi-token-prediction draft head (0.14 GB, 4-bit). MLX stores a scale and a bias per group, so the container costs a little more than the GGUF packings; the weights decode to exactly the same ternary values.

Which runtime runs which file. Upstream llama.cpp (build b11555 or newer) runs the Q2_0 file. PQ2_0 and PTQ1_0 need the PrismML llama.cpp binaries, which the demo’s setup.sh installs. The MLX pack loads natively in mlx-vlm 0.7.8 and newer. See Formats & runtime support.

How to run it

Through the demo repo, which applies the right sampling profile, loads the vision projector, and enables tool calling:

BONSAI_MODEL=9B ./setup.sh                          # PQ2_0 + the Q8_0 projector, about 3.0 GB; plus the MLX pack on Apple Silicon
BONSAI_MODEL=9B ./scripts/start_llama_server.sh     # OpenAI-compatible API + chat UI on :8080
BONSAI_MODEL=9B ./scripts/run_llama.sh -p "Explain KV cache growth in one paragraph."
BONSAI_MODEL=9B ./scripts/run_mlx.sh -p "Explain KV cache growth in one paragraph."   # Apple Silicon

Or directly with llama.cpp / MLX:

./llama-cli -m Ternary-Bonsai-2-9B-PQ2_0.gguf -ngl 99 -fa on -c 32768 \
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0 \
  --presence-penalty 1.0 --repeat-penalty 1.0 --repeat-last-n 64 \
  -p "Explain KV cache growth in one paragraph." -n 16384

./llama-mtmd-cli -m Ternary-Bonsai-2-9B-PQ2_0.gguf \
  --mmproj Ternary-Bonsai-2-9B-mmproj-Q8_0.gguf -ngl 99 \
  --image photo.jpg -p "Describe this image."

# hf download prism-ml/Ternary-Bonsai-2-9B-mlx-2bit --local-dir bonsai2-9b-mlx
# pip install "mlx-vlm>=0.7.8"
from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template

model, processor = load("bonsai2-9b-mlx")
prompt = apply_chat_template(processor, model.config, "What is in this image?", num_images=1)
print(generate(model, processor, prompt, ["photo.jpg"], max_tokens=8192,
               temperature=1.0, top_p=0.95, top_k=20,
               presence_penalty=1.0, presence_context_size=64))

-n 16384 (or max_tokens) leaves room for the thinking trace; a small cap ends generation mid-thought, before any answer. The demo repo is the source of truth for running these models; tested setups and current implementation details live there.

Best practices

These apply to every GGUF packing and to the MLX pack.

Generation parameters

Use the set that matches the mode you’re running in. Thinking and non-thinking mode want different sampling, and changing the thinking mode alone does not change the sampling for you.

Parameter	Thinking (default)	Non-thinking
`temperature`	1.0	0.7
`top_p`	0.95	0.80
`top_k`	20	20
`min_p`	0.0	0.0
`presence_penalty`	1.0	1.5
`repetition_penalty`	1.0	1.0

Apply the penalties over the last 64 tokens: --repeat-last-n 64 in llama.cpp (its default window), presence_context_size=64 in MLX (whose default is 20). The presence penalty has no GGUF metadata key, so pass it on the command line or in each request. Avoid greedy decoding (temperature=0) with thinking on; it can get stuck reasoning on some prompts. The demo’s launchers apply the thinking-mode profile, and BONSAI_THINKING=0 switches both the sampling and the chat template to non-thinking.

Output length

Give thinking mode room to finish. Even simple prompts can use 3,000+ thinking tokens, so allow at least 8,192 output tokens with thinking on. For short answers, turn thinking off.

System prompt

A plain system message works well:

You are a helpful assistant

MLX sampler

mlx-vlm’s Python generate() decodes greedily unless you pass the sampler, and mlx_vlm.server reads temperature, top_p, and top_k from generation_config.json but not min_p or the presence penalty. Send the full set with every request. Matching these values does not make output identical across runtimes: sampler ordering, kernels, and RNG differ.

Choosing a packing

PQ2_0: the default. Cheaper to unpack, faster prompt processing, and the pack measured on Apple Silicon.
PTQ1_0: the smallest file. Use it where memory is tightest.
Q2_0: the one packing upstream llama.cpp runs. Pick it for tools that bundle a stock llama.cpp.

Vision and tool calling

Send an image as an OpenAI-style image_url content part, and tools as a tools array; both come back through the standard API. Use the Q8_0 projector: quality is nearly identical to BF16, and the BF16 file can crash llama-server under concurrent image requests on current builds. For small or dense text in images, raise the image token budget with --image-min-tokens 1024.

curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
  "messages": [{"role": "user", "content": [
    {"type": "text", "text": "What is in this image?"},
    {"type": "image_url", "image_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/4/47/PNG_transparency_demonstration_1.png"}}
  ]}],
  "max_tokens": 8192,
  "temperature": 1.0, "top_p": 0.95, "top_k": 20, "min_p": 0,
  "presence_penalty": 1.0, "repeat_penalty": 1.0, "repeat_last_n": 64
}'

See Tool calling & MCP in the demo repo for MCP server setup and the agentic walkthrough.

Thinking mode

The 9B reasons by default. Over the API, turn it off per request with "chat_template_kwargs": {"enable_thinking": false} on llama-server, or a top-level "enable_thinking": false on the mlx-vlm server, and switch to the non-thinking sampling at the same time. Cap it server-wide with --reasoning-budget N, or start the demo’s launchers with BONSAI_THINKING=0. The built-in chat UI has a Reasoning effort picker that overrides the server default per conversation.

Speculative decoding

Two optional drafters ship in the GGUF repo. Both need the PrismML binaries, and the demo’s launchers don’t select them automatically, so run llama-server yourself. MTP (recommended). Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf is the PQ2_0 model with its multi-token-prediction head built in: load it instead of the plain PQ2_0 file, not alongside it. Draft depth 2 works best:

./llama-server -m Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf -ngl 99 -fa on \
  --spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-n-min 0 --spec-draft-p-min 0

On CUDA, a quantized KV cache (--cache-type-k q8_0 --cache-type-v q8_0) works with MTP from release prism-b10776-d00e252 onward; on older CUDA builds keep the default F16 cache, or quantize only K. Metal is not affected. DFlash2 (optional). Ternary-Bonsai-2-9B-DFlash2-Q8_0.gguf is a block drafter for the PQ2_0 file. It helps on copy-heavy work such as code edits and rewrites and is slower than plain decoding on general chat, so prefer MTP by default:

./llama-server -m Ternary-Bonsai-2-9B-PQ2_0.gguf -md Ternary-Bonsai-2-9B-DFlash2-Q8_0.gguf \
  -ngl 99 -ngld 99 --spec-type draft-dflash

The MLX pack carries the same MTP head under mtp/ as an mlx-vlm drafter; MTP.md in that repo has the code. Speculative output matches plain decoding up to near-ties, so exact-match comparisons against non-speculative runs can show small differences.

Get started

Models

Download

Run Locally

Integrations

Resources

Ternary Bonsai 2 9B

Specifications

Artifacts

How to run it

Best practices

Generation parameters

Output length

System prompt

MLX sampler

Choosing a packing

Vision and tool calling

Thinking mode

Speculative decoding

​Specifications

​Artifacts

​How to run it

​Best practices

​Generation parameters

​Output length

​System prompt

​MLX sampler

​Choosing a packing

​Vision and tool calling

​Thinking mode

​Speculative decoding

Specifications

Artifacts

How to run it

Best practices

Generation parameters

Output length

System prompt

MLX sampler

Choosing a packing

Vision and tool calling

Thinking mode

Speculative decoding