Skip to main content
llama.cpp is the reference runtime for Bonsai GGUF weights. It runs on CPU and every major GPU backend (CUDA, Metal, Vulkan, ROCm).
1-bit (Q1_0) is merged into upstream llama.cpp, so recent upstream builds work. Ternary (Q2_0) needs the PrismML fork (prism branch) or its pre-built binaries; stock builds cannot run it. See Formats & runtime support.

​
Get a binary

The demo repo’s setup.sh downloads a compatible build automatically into bin/. To grab one yourself, use the PrismML llama.cpp release, which covers 1-bit and ternary on macOS (Apple Silicon and Intel), Linux (CPU / CUDA 12.4 / CUDA 12.8 / Vulkan / ROCm), Windows (CPU / CUDA / Vulkan / HIP), and iOS (XCFramework). The full variant list is in Formats & runtime support.

​
Run a chat

./llama-cli -m ./Bonsai-8B-gguf/Bonsai-8B-Q1_0.gguf -c 0 \
  --temp 0.5 --top-p 0.85 --top-k 20 --min-p 0 \
  -p "What is the capital of France?"
  • -c 0 auto-fits the context window to available memory; pass an explicit value (e.g. -c 32768) to pin it and bound memory use. Memory-per-context numbers are on each model card.
  • The sampling flags are Bonsai’s recommended defaults.
  • -ngl 99 offloads all layers to the GPU (the wrapper script sets this automatically when a GPU is present).
Bonsai 27B needs two extra flags: --mmproj <path>.gguf for image input, and --jinja for native OpenAI-style tool_calls (the demo scripts already pass both).
The demo repo wraps all of this: 
./scripts/run_llama.sh -p "What is the capital of France?"

.\scripts\run_llama.ps1 -p "What is the capital of France?"

​
Start a server

./llama-server -m ./Bonsai-8B-gguf/Bonsai-8B-Q1_0.gguf -c 0 --port 8080
This is the OpenAI-compatible endpoint described in Run the server, which also documents the recommended server flags.

​
Build from source

If the pre-built binaries don’t cover your platform or you want a specific backend: 
git clone -b prism https://github.com/PrismML-Eng/llama.cpp
cd llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release   # add a backend flag below
cmake --build build -j
TargetCMake flag
CUDA-DGGML_CUDA=ON
Metal-DGGML_METAL=ON (default on macOS)
Vulkan-DGGML_VULKAN=ON
ROCm / HIP-DGGML_HIP=ON
The prism branch is required for ternary Q2_0. For 1-bit only, upstream llama.cpp works too. The demo repo also has ready-made build scripts: scripts/build_mac.sh, scripts/build_cpu_linux.sh, scripts/build_cuda_linux.sh, and scripts/build_cuda_windows.ps1.