> ## Documentation Index
> Fetch the complete documentation index at: https://docs.prismml.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Formats & Runtime Support

> GGUF vs MLX, the Q1_0 and Q2_0 quantization types, and which backends have native kernels.

Bonsai ships in two formats. Which you want depends on your hardware and the runtime you plan to use.

<CardGroup cols={2}>
  <Card title="GGUF" icon="box">
    For [llama.cpp](/run/llamacpp) and the ecosystem built on it ([Ollama](/integrations/ollama), [Open WebUI](/run/open-webui) via llama-server). Cross-platform: CPU, CUDA, Metal, Vulkan, ROCm.
  </Card>

  <Card title="MLX" icon="apple">
    Apple's array framework, tuned for Apple Silicon. Best raw performance on M-series Macs and the basis for the iPhone/iPad apps.
  </Card>
</CardGroup>

On a Mac, both work; MLX is usually faster, GGUF is more portable across tools. Everywhere else, use GGUF.

## Quantization types

| Family         | GGUF type | Effective bits/weight | Notes                                                                       |
| -------------- | --------- | --------------------- | --------------------------------------------------------------------------- |
| Bonsai (1-bit) | `Q1_0`    | \~1.13                | Weights in `{−1, +1}`, packed one bit each.                                 |
| Ternary-Bonsai | `Q2_0`    | \~2.13                | Weights in `{−1, 0, +1}` packed in 2 bits, FP16 scale per 128-weight group. |

<Note>
  llama.cpp's stock ternary types (`TQ1_0`, `TQ2_0`) use quantization group size 256. Bonsai's ternary weights are trained with group size **128**, so they don't fit the stock types; that is why ternary ships as `Q2_0` with Bonsai's own kernels for now. An [upstream llama.cpp PR](https://github.com/ggml-org/llama.cpp/pull/24448) for native `Q2_0` support is in progress and will likely standardize on group size 64 — once it lands, expect a group-size-64 repack for out-of-the-box compatibility, alongside a group-size-128 variant on PrismML's fork.
</Note>

## Runtime support matrix

This is the part that matters most for a low-bit model: a runtime **without** native kernels either refuses the file or runs it dequantized, silently losing the speed and memory advantage.

### 1-bit (`Q1_0`): merged upstream

| Backend             | Status                                                                                                                                        |
| ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| CPU (generic)       | ✅ Merged into upstream llama.cpp                                                                                                              |
| CPU (optimized x86) | ✅ Merged                                                                                                                                      |
| Metal               | ✅ Merged                                                                                                                                      |
| CUDA                | ✅ Merged                                                                                                                                      |
| Vulkan              | ✅ Merged                                                                                                                                      |
| MLX                 | ⏳ Pending ([mlx#3161](https://github.com/ml-explore/mlx/pull/3161)); use the [PrismML MLX fork](https://github.com/PrismML-Eng/mlx) meanwhile |

Because `Q1_0` is upstream, any llama.cpp-based tool built from a recent enough version runs 1-bit Bonsai. When in doubt, use the [PrismML llama.cpp binaries](https://github.com/PrismML-Eng/llama.cpp/releases/tag/prism-b9570-0ad1dab), which are known-good.

### Ternary (`Q2_0`): PrismML fork for now

| Backend                       | Status                                                              |
| ----------------------------- | ------------------------------------------------------------------- |
| CPU (NEON / generic)          | `prism` fork; upstream PR planned                                   |
| Metal                         | `prism` fork; upstream PR planned                                   |
| CUDA                          | `prism` fork; upstream PR planned                                   |
| Vulkan / ROCm / x86-optimized | ⏳ Not yet available                                                 |
| MLX (2-bit)                   | ✅ Supported in stock [MLX](https://github.com/ml-explore/mlx) today |

<Warning>
  **Ternary GGUF requires the [PrismML llama.cpp fork](https://github.com/PrismML-Eng/llama.cpp) (`prism` branch) or its pre-built binaries.** Tools that bundle their own stock llama.cpp, including current Ollama releases, cannot run ternary `Q2_0` yet. For ternary on Apple Silicon, MLX works out of the box.
</Warning>

## Pre-built binaries

The [PrismML llama.cpp release](https://github.com/PrismML-Eng/llama.cpp/releases/tag/prism-b9570-0ad1dab) covers 1-bit and ternary on:

| Platform      | Variants                                                       |
| ------------- | -------------------------------------------------------------- |
| macOS         | Apple Silicon (Metal), Apple Silicon (KleidiAI CPU), Intel x64 |
| Linux x64     | CPU, CUDA 12.4, CUDA 12.8, Vulkan, ROCm 7.2                    |
| Linux arm64   | CPU, Vulkan                                                    |
| Windows x64   | CPU, CUDA 12.4, Vulkan, HIP/ROCm                               |
| Windows arm64 | CPU                                                            |
| iOS           | XCFramework                                                    |

`setup.sh` in the demo repo picks the right one automatically. Build-from-source instructions are on the [llama.cpp page](/run/llamacpp#build-from-source).