> ## Documentation Index
> Fetch the complete documentation index at: https://docs.prismml.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Ternary Bonsai 2 27B

> The flagship Bonsai: multimodal, a 262K-token context, in ternary weights at 1.75 bits.

Ternary Bonsai 2 27B is the largest model in the family and the demo repo's default. It is derived from Qwen3.8-27B, a hybrid-attention causal language model, with the architecture unchanged. Every language-model weight is ternary: `{−1, 0, +1}` in a fixed rotated basis, with an FP16 scale per group of 128 weights. It takes images as well as text, carries a 262K-token context, calls tools natively, and reasons before it answers.

> [!TIP]
>
> **Before you run it:** the recommended sampling settings differ between thinking and instruct mode, and `low` reasoning effort is not supported. See [Best practices](#best-practices).

## Specifications

| | |
| - | - |
| Base model | Qwen3.8-27B (architecture unchanged) |
| Parameters | 27.36B total (24.35B language backbone + 2.54B embedding/LM head + 0.46B vision tower) |
| Architecture | Hybrid attention (\~75% linear / \~25% full), SwiGLU MLP, RoPE, RMSNorm |
| Weight format | Ternary g128 with FP16 group-wise scaling, blockwise Hadamard rotation folded into the stored weights |
| Effective bit width | 1.72 bpw true ternary; 1.75 bpw as shipped in `PTQ1_0` |
| Max context | 262,144 tokens |
| Modalities | Text + image in, text out |
| License | Apache 2.0 |

On the model card's 14-benchmark thinking-mode suite it averages 84.78, retaining 98.2% of the Qwen3.8-27B FP16 baseline's 86.32 at roughly a ninth of the size; the whitepaper's 20-benchmark suite gives 83.9 against 85.4, also 98.2%. Math stays within half a point of full precision and coding is level with the baseline. The full tables are on the [model card](https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf#benchmarks) and in the [whitepaper](https://github.com/PrismML-Eng/Bonsai-demo/blob/main/bonsai-2-27b-whitepaper.pdf).

## Artifacts

All Ternary Bonsai 2 repositories are in the [Bonsai 2 collection](https://huggingface.co/collections/prism-ml/bonsai-2) on Hugging Face.

| Format | Repository | Weights on disk |
| - | - | - |
| GGUF (`PTQ1_0`, 1.75 bpw) | [`prism-ml/Ternary-Bonsai-2-27B-gguf`](https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf) | 5.95 GB |
| GGUF (`PQ2_0`, 2.13 bpw) | [`prism-ml/Ternary-Bonsai-2-27B-gguf`](https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf) | 7.21 GB |
| GGUF (`Q2_0`, upstream type) | [`prism-ml/Ternary-Bonsai-2-27B-gguf-dev`](https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf-dev) | 7.63 GB |
| MLX (2-bit, 2.25 bpw) | [`prism-ml/Ternary-Bonsai-2-27B-mlx-2bit`](https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-mlx-2bit) | 8.60 GB (incl. FP16 vision tower) |

The demo downloads `PQ2_0` by default; see [choosing a packing](#choosing-a-packing) below. The GGUF repo also ships the vision tower separately, needed for image input: `mmproj` Q8\_0 (0.63 GB, what the demo uses) or the BF16 reference (0.93 GB), plus the 53.8 GB F16 reference weights for comparison work.

> [!NOTE]
>
> **Which runtime runs which file.** Upstream llama.cpp (build `b11555` or newer) runs the `Q2_0` packing; its file still carries a historical `prism-fork-required` suffix, but compatibility depends on the build, not the name. `PQ2_0` and `PTQ1_0` need the [PrismML llama.cpp binaries](https://github.com/PrismML-Eng/llama.cpp/releases), which the demo's `setup.sh` installs. The MLX pack loads natively in mlx-vlm 0.7.8 and newer. See [Formats & runtime support](https://docs.prismml.com/download/formats.md).

## How to run it

Through the [demo repo](https://docs.prismml.com/get-started/quickstart.md), where Ternary Bonsai 2 27B is the default, so no flags are needed:

```bash theme={null}
./scripts/start_llama_server.sh   # OpenAI-compatible API + chat/vision UI on :8080
./scripts/run_mlx.sh -p "Explain KV cache growth in one paragraph."   # Apple Silicon
```

Or a single generation directly with [llama.cpp](https://docs.prismml.com/run/llamacpp.md) / [MLX](https://docs.prismml.com/run/mlx.md):

```bash llama.cpp theme={null}
./llama-cli -m Ternary-Bonsai-2-27B-PQ2_0.gguf -ngl 99 -fa on -c 32768 \
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.05 \
  -p "Explain KV cache growth in one paragraph." -n 16384
```

```bash llama.cpp (image + text) theme={null}
./llama-mtmd-cli -m Ternary-Bonsai-2-27B-PQ2_0.gguf \
  --mmproj Ternary-Bonsai-2-27B-mmproj-Q8_0.gguf -ngl 99 \
  --image photo.jpg -p "Describe this image."
```

```python MLX (Apple Silicon) theme={null}
# hf download prism-ml/Ternary-Bonsai-2-27B-mlx-2bit --local-dir bonsai2-27b-mlx
# pip install "mlx-vlm>=0.7.8"
from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template

model, processor = load("bonsai2-27b-mlx")
prompt = apply_chat_template(processor, model.config, "What is in this image?", num_images=1)
print(generate(model, processor, prompt, ["photo.jpg"], max_tokens=16384,
               temperature=1.0, top_p=0.95, top_k=20, min_p=0.05))
```

`-n 16384` (or `max_tokens`) leaves room for the thinking trace: the template defaults to `xhigh` reasoning effort, so a small cap ends generation mid-thought, before any answer. The [demo repo](https://github.com/PrismML-Eng/Bonsai-demo) is the source of truth for running these models; tested setups and current implementation details live there.

## Best practices

These apply to both the GGUF and MLX releases.

### Generation parameters

> [!WARNING]
>
> Use the set that matches the mode you're running in. Thinking and instruct mode want different sampling settings, and mixing them degrades output quality.

| Parameter | Thinking (default) | Instruct / non-thinking |
| - | - | - |
| `temperature` | 1.0 | 0.7 |
| `top_p` | 0.95 | 0.80 |
| `top_k` | 20 | 20 |
| `min_p` | 0.05 | 0.0 |
| `presence_penalty` | 0.0 | 1.5 |
| `repetition_penalty` | 1.0 | 1.0 |

`min_p = 0.05` drops tokens far less likely than the top choice. In our tests it scored at least as well as `0.0` and followed instructions more reliably. It is also llama.cpp's default: the GGUF carries `top_k`, `top_p` and temperature but not `min_p`, so llama.cpp applies `0.05` without being told. Other runtimes may default to `0.0`, so set it explicitly there. Presence and repetition penalties, and all instruct-mode values, are not in the GGUF; pass them yourself. The published [benchmarks](https://github.com/PrismML-Eng/Bonsai-demo/blob/main/bonsai-2-27b-whitepaper.pdf) used the thinking-mode values with `min_p = 0.0`.

### Reasoning effort

> [!NOTE]
>
> The model uses **`xhigh`** by default. **`medium`** thinks noticeably less and, at moderate output limits, is about as accurate, so it is a good default when responses feel slow; the whitepaper's Appendix C quantifies the cost on the hardest benchmarks. **`low` is not supported**: it does not reduce thinking. To bound reasoning, use a token budget instead (see [Thinking mode](#thinking-mode)).

Send `reasoning_effort` per request, or make `medium` the server-wide default while letting a request still ask for `xhigh`:

```bash theme={null}
./scripts/start_llama_server.sh --chat-template-kwargs '{"reasoning_effort":"medium"}'
```

### System prompt

A plain system message works well:

```text theme={null}
You are a helpful assistant
```

### Choosing a packing

| | `PQ2_0` (2.13 bpw, 7.21 GB) | `PTQ1_0` (1.75 bpw, 5.95 GB) |
| - | - | - |
| Decode | Faster on H100, A100, and Blackwell | Faster on Ada-generation cards and L4 |
| Prompt processing | Faster on every platform | — |
| Apple Silicon | Demo default; the pack measured on Apple Silicon | — |
| Footprint | — | Smallest; denser trit packing |

Pick by hardware target and whether footprint or decode speed matters more. The `Q2_0` file is the one upstream llama.cpp runs; pick it for tools that bundle a stock llama.cpp.

## Vision and tool calling

27B is trained for both. Send an image as a normal OpenAI-style `image_url` content part, and a `tools` array for function calling. Both come back through the standard API, no prompt hacks required:

```bash theme={null}
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [{"role": "user", "content": "What is the weather in Lisbon?"}],
    "tools": [{
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Get current weather for a city",
        "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}
      }
    }]
  }'
```

The response's `choices[0].message.tool_calls` carries the call. Use the Q8\_0 projector for images, which is what the demo downloads; the BF16 file is the reference-precision alternative. See [Tool calling & MCP](https://github.com/PrismML-Eng/Bonsai-demo/blob/main/TOOLS.md) in the demo repo for MCP server setup and the full agentic walkthrough, and [Open WebUI](https://docs.prismml.com/run/open-webui.md) for a point-and-click demo of the same capabilities.

## Thinking mode

27B reasons by default. Toggle it per request with `thinking_budget_tokens` (`0` to disable, `-1` for unlimited), cap it server-wide with `--reasoning-budget N`, or turn it off entirely by passing `--reasoning-budget 0` through the launcher (`./scripts/start_llama_server.sh --reasoning-budget 0`); `--no-think` does the same for `run_mlx.sh`. The built-in chat UI has a Reasoning effort picker (Off / Low / Medium / High / Max) that overrides the server default per conversation. The UI's levels are fixed token budgets (Medium is 2,048 tokens), which cut thinking off at that length; the model's own `reasoning_effort: "medium"` asks it to think less instead, and rarely runs out mid-reasoning.

> [!NOTE]
>
> Large images are downscaled to \~1,024 vision tokens by default on Metal, Vulkan, and CPU to keep latency reasonable; CUDA and ROCm run uncapped. Override with `BONSAI_IMAGE_MAX_TOKENS` (`0` disables the cap, useful for reading small text in screenshots).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.