> ## Documentation Index
> Fetch the complete documentation index at: https://docs.prismml.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Ternary Bonsai 2 2B

> The smallest Bonsai 2: a 2B-class vision-language model in 0.52 GB for always-on tasks and the tightest memory budgets.

Ternary Bonsai 2 2B is the smallest model in the family, for always-on background tasks and the tightest memory budgets. It is derived from Qwen3.5-2B, a hybrid-attention vision-language model, with the architecture unchanged. Every language-model weight is ternary: `{−1, 0, +1}` in a fixed rotated basis, with an FP16 scale per group of 128 weights. It takes images as well as text, carries the base model's 262K-token context, and thinks before it answers by default.

<Tip>
  **Before you run it:** the recommended sampling settings differ between thinking and non-thinking mode, and the 2B wants a presence penalty that the [27B](/models/bonsai-2-27b) does not. See [Best practices](#best-practices).
</Tip>

## Specifications

| | |
| - | - |
| Base model | Qwen3.5-2B (architecture unchanged) |
| Parameters | \~1.88B language model (24 blocks, hidden 2048, tied embedding/LM head) + \~0.33B (24 blocks) vision tower |
| Architecture | Hybrid attention, 3 linear-attention : 1 full-attention (18 linear-attention and 6 full-attention layers), SwiGLU MLP, RoPE, RMSNorm |
| Weight format | Ternary g128 with FP16 group-wise scales, blockwise Hadamard rotation folded into the stored weights |
| Higher-precision tensors | Normalization weights and the linear-attention state path stay F32 |
| Max context | 262,144 tokens |
| Modalities | Text + image in, text out |
| License | Apache 2.0 |

## Artifacts

Both repositories are in the [Bonsai 2 collection](https://huggingface.co/collections/prism-ml/bonsai-2) on Hugging Face.

**GGUF:** [`prism-ml/Ternary-Bonsai-2-2B-gguf`](https://huggingface.co/prism-ml/Ternary-Bonsai-2-2B-gguf)

| File | Contents | Size | Bits/weight |
| - | - | -: | -: |
| `Ternary-Bonsai-2-2B-PTQ1_0.gguf` | Language model, dense trits | 0.43 GB | \~1.83 |
| `Ternary-Bonsai-2-2B-PQ2_0.gguf` | Language model, 2-bit slots (the demo's default) | 0.52 GB | \~2.20 |
| `Ternary-Bonsai-2-2B-Q2_0.gguf` | Language model, upstream `Q2_0` type | 0.55 GB | \~2.33 |
| `Ternary-Bonsai-2-2B-mmproj-Q8_0.gguf` | Vision tower (recommended) | 0.36 GB | |
| `Ternary-Bonsai-2-2B-mmproj-BF16.gguf` | Vision tower, reference precision | 0.67 GB | |

The three language-model files hold the same ternary weights and differ only in storage layout and kernel speed. Bits/weight is file size over the \~1.88B language-model parameters, so it includes the F32 tensors and metadata. The same model is \~3.8 GB at FP16.

**MLX:** [`prism-ml/Ternary-Bonsai-2-2B-mlx-2bit`](https://huggingface.co/prism-ml/Ternary-Bonsai-2-2B-mlx-2bit), 1.34 GB on disk. `model.safetensors` carries the language model in MLX's 2-bit grouped format (group 128) plus the vision tower in FP16. MLX stores a scale and a bias per group, so the container costs a little more than the GGUF packings; the weights decode to exactly the same ternary values.

<Note>
  **Which runtime runs which file.** Upstream llama.cpp (build `b11555` or newer) runs the `Q2_0` file. `PQ2_0` and `PTQ1_0` need the [PrismML llama.cpp binaries](https://github.com/PrismML-Eng/llama.cpp/releases), which the demo's `setup.sh` installs. The MLX pack loads natively in mlx-vlm 0.7.8 and newer. See [Formats & runtime support](/download/formats).
</Note>

## How to run it

Through the [demo repo](/get-started/quickstart), which applies the right sampling profile, loads the vision projector, and enables tool calling:

```bash theme={null}
BONSAI_MODEL=2B ./setup.sh                          # PQ2_0 + the Q8_0 projector, about 0.9 GB; plus the MLX pack on Apple Silicon
BONSAI_MODEL=2B ./scripts/start_llama_server.sh     # OpenAI-compatible API + chat UI on :8080
BONSAI_MODEL=2B ./scripts/run_llama.sh -p "Explain KV cache growth in one paragraph."
BONSAI_MODEL=2B ./scripts/run_mlx.sh -p "Explain KV cache growth in one paragraph."   # Apple Silicon
```

Or directly with [llama.cpp](/run/llamacpp) / [MLX](/run/mlx):

<CodeGroup>
  ```bash llama.cpp (text) theme={null}
  ./llama-cli -m Ternary-Bonsai-2-2B-PQ2_0.gguf -ngl 99 -fa on -c 32768 \
    --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0 \
    --presence-penalty 1.0 --repeat-penalty 1.0 --repeat-last-n 64 \
    -p "Explain KV cache growth in one paragraph." -n 16384
  ```

  ```bash llama.cpp (image + text) theme={null}
  ./llama-mtmd-cli -m Ternary-Bonsai-2-2B-PQ2_0.gguf \
    --mmproj Ternary-Bonsai-2-2B-mmproj-Q8_0.gguf -ngl 99 \
    --image photo.jpg -p "Describe this image."
  ```

  ```python MLX (Apple Silicon) theme={null}
  # hf download prism-ml/Ternary-Bonsai-2-2B-mlx-2bit --local-dir bonsai2-2b-mlx
  # pip install "mlx-vlm>=0.7.8"
  from mlx_vlm import load, generate
  from mlx_vlm.prompt_utils import apply_chat_template

  model, processor = load("bonsai2-2b-mlx")
  prompt = apply_chat_template(processor, model.config, "What is in this image?", num_images=1)
  print(generate(model, processor, prompt, ["photo.jpg"], max_tokens=8192,
                 temperature=1.0, top_p=0.95, top_k=20,
                 presence_penalty=1.0, presence_context_size=64))
  ```
</CodeGroup>

`-n 16384` (or `max_tokens`) leaves room for the thinking trace; a small cap ends generation mid-thought, before any answer. The [demo repo](https://github.com/PrismML-Eng/Bonsai-demo) is the source of truth for running these models; tested setups and current implementation details live there.

## Best practices

These apply to every GGUF packing and to the MLX pack.

### Generation parameters

<Warning>
  Use the set that matches the mode you're running in. Thinking and non-thinking mode want different sampling, and changing the thinking mode alone does not change the sampling for you.
</Warning>

| Parameter | Thinking (default) | Non-thinking |
| - | - | - |
| `temperature` | 1.0 | 0.7 |
| `top_p` | 0.95 | 0.80 |
| `top_k` | 20 | 20 |
| `min_p` | 0.0 | 0.0 |
| `presence_penalty` | 1.0 | 1.5 |
| `repetition_penalty` | 1.0 | 1.0 |

Apply the penalties over the last 64 tokens: `--repeat-last-n 64` in llama.cpp (its default window), `presence_context_size=64` in MLX (whose default is 20). The presence penalty has no GGUF metadata key, so pass it on the command line or in each request. Avoid greedy decoding (`temperature=0`) with thinking on; it can get stuck reasoning on some prompts. The demo's launchers apply the thinking-mode profile, and `BONSAI_THINKING=0` switches both the sampling and the chat template to non-thinking.

### Output length

Give thinking mode room to finish. Even simple prompts can use 3,000+ thinking tokens, so allow at least 8,192 output tokens with thinking on. For short answers, turn thinking off.

### System prompt

A plain system message works well:

```text theme={null}
You are a helpful assistant
```

### MLX sampler

mlx-vlm's Python `generate()` decodes greedily unless you pass the sampler, and `mlx_vlm.server` reads temperature, `top_p`, and `top_k` from `generation_config.json` but not `min_p` or the presence penalty. Send the full set with every request. Matching these values does not make output identical across runtimes: sampler ordering, kernels, and RNG differ.

### Choosing a packing

* **`PQ2_0`**: the default. Cheaper to unpack, faster prompt processing, and the pack measured on Apple Silicon.
* **`PTQ1_0`**: the smallest file. Use it where memory is tightest.
* **`Q2_0`**: the one packing upstream llama.cpp runs. Pick it for tools that bundle a stock llama.cpp.

## Vision and tool calling

Send an image as an OpenAI-style `image_url` content part, and tools as a `tools` array; both come back through the standard API. Use the Q8\_0 projector: quality is nearly identical to BF16, and the BF16 file can crash `llama-server` under concurrent image requests on current builds. For small or dense text in images, raise the image token budget with `--image-min-tokens 1024`.

```bash theme={null}
curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
  "messages": [{"role": "user", "content": [
    {"type": "text", "text": "What is in this image?"},
    {"type": "image_url", "image_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/4/47/PNG_transparency_demonstration_1.png"}}
  ]}],
  "max_tokens": 8192,
  "temperature": 1.0, "top_p": 0.95, "top_k": 20, "min_p": 0,
  "presence_penalty": 1.0, "repeat_penalty": 1.0, "repeat_last_n": 64
}'
```

See [Tool calling & MCP](https://github.com/PrismML-Eng/Bonsai-demo/blob/main/TOOLS.md) in the demo repo for MCP server setup and the agentic walkthrough.

## Thinking mode

The 2B reasons by default. Over the API, turn it off per request with `"chat_template_kwargs": {"enable_thinking": false}` on llama-server, or a top-level `"enable_thinking": false` on the mlx-vlm server, and switch to the non-thinking sampling at the same time. Cap it server-wide with `--reasoning-budget N`, or start the demo's launchers with `BONSAI_THINKING=0`. The built-in chat UI has a Reasoning effort picker that overrides the server default per conversation.

## Speculative decoding

The 2B ships without a draft model. At this size a decode step is already cheap, and in our tests speculative decoding with a draft head was slower than plain decoding on everyday prompts, so plain decoding is the recommended setup.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.