Skip to main content
Bonsai ships in two formats. Which you want depends on your hardware and the runtime you plan to use.

GGUF

For llama.cpp and the ecosystem built on it (Ollama, Open WebUI via llama-server). Cross-platform: CPU, CUDA, Metal, Vulkan, ROCm.

MLX

Apple’s array framework, tuned for Apple Silicon. Best raw performance on M-series Macs and the basis for the iPhone/iPad apps.
On a Mac, both work; MLX is usually faster, GGUF is more portable across tools. Everywhere else, use GGUF.

​
Quantization types

FamilyGGUF typeEffective bits/weightNotes
Bonsai (1-bit)Q1_0~1.13Weights in {−1, +1}, packed one bit each.
Ternary-BonsaiQ2_0~2.13Weights in {−1, 0, +1} packed in 2 bits, FP16 scale per 128-weight group.
llama.cpp’s stock ternary types (TQ1_0, TQ2_0) use quantization group size 256. Bonsai’s ternary weights are trained with group size 128, so they don’t fit the stock types; that is why ternary ships as Q2_0 with Bonsai’s own kernels for now. An upstream llama.cpp PR for native Q2_0 support is in progress and will likely standardize on group size 64 — once it lands, expect a group-size-64 repack for out-of-the-box compatibility, alongside a group-size-128 variant on PrismML’s fork.

​
Runtime support matrix

This is the part that matters most for a low-bit model: a runtime without native kernels either refuses the file or runs it dequantized, silently losing the speed and memory advantage.

​
1-bit (Q1_0): merged upstream

BackendStatus
CPU (generic)✅ Merged into upstream llama.cpp
CPU (optimized x86)✅ Merged
Metal✅ Merged
CUDA✅ Merged
Vulkan✅ Merged
MLX⏳ Pending (mlx#3161); use the PrismML MLX fork meanwhile
Because Q1_0 is upstream, any llama.cpp-based tool built from a recent enough version runs 1-bit Bonsai. When in doubt, use the PrismML llama.cpp binaries, which are known-good.

​
Ternary (Q2_0): PrismML fork for now

BackendStatus
CPU (NEON / generic)prism fork; upstream PR planned
Metalprism fork; upstream PR planned
CUDAprism fork; upstream PR planned
Vulkan / ROCm / x86-optimized⏳ Not yet available
MLX (2-bit)✅ Supported in stock MLX today
Ternary GGUF requires the PrismML llama.cpp fork (prism branch) or its pre-built binaries. Tools that bundle their own stock llama.cpp, including current Ollama releases, cannot run ternary Q2_0 yet. For ternary on Apple Silicon, MLX works out of the box.

​
Pre-built binaries

The PrismML llama.cpp release covers 1-bit and ternary on:
PlatformVariants
macOSApple Silicon (Metal), Apple Silicon (KleidiAI CPU), Intel x64
Linux x64CPU, CUDA 12.4, CUDA 12.8, Vulkan, ROCm 7.2
Linux arm64CPU, Vulkan
Windows x64CPU, CUDA 12.4, Vulkan, HIP/ROCm
Windows arm64CPU
iOSXCFramework
setup.sh in the demo repo picks the right one automatically. Build-from-source instructions are on the llama.cpp page.