llama.cpp

llama.cpp is the reference runtime for Bonsai GGUF weights. It runs on CPU and every major GPU backend (CUDA, Metal, Vulkan, ROCm).

1-bit (Q1_0) is merged into upstream llama.cpp, so recent upstream builds work. Ternary (Q2_0) needs the PrismML fork (prism branch) or its pre-built binaries; stock builds cannot run it. See Formats & runtime support.

Get a binary

The demo repo’s setup.sh downloads a compatible build automatically into bin/. To grab one yourself, use the PrismML llama.cpp release, which covers 1-bit and ternary on macOS (Apple Silicon and Intel), Linux (CPU / CUDA 12.4 / CUDA 12.8 / Vulkan / ROCm), Windows (CPU / CUDA / Vulkan / HIP), and iOS (XCFramework). The full variant list is in Formats & runtime support.

Run a chat

./llama-cli -m ./Bonsai-8B-gguf/Bonsai-8B-Q1_0.gguf -c 0 \
  --temp 0.5 --top-p 0.85 --top-k 20 --min-p 0 \
  -p "What is the capital of France?"

-c 0 auto-fits the context window to available memory; pass an explicit value (e.g. -c 32768) to pin it and bound memory use. Memory-per-context numbers are on each model card.
The sampling flags are Bonsai’s recommended defaults.
-ngl 99 offloads all layers to the GPU (the wrapper script sets this automatically when a GPU is present).

Bonsai 27B needs two extra flags: --mmproj <path>.gguf for image input, and --jinja for native OpenAI-style tool_calls (the demo scripts already pass both).

The demo repo wraps all of this:

./scripts/run_llama.sh -p "What is the capital of France?"

.\scripts\run_llama.ps1 -p "What is the capital of France?"

Start a server

./llama-server -m ./Bonsai-8B-gguf/Bonsai-8B-Q1_0.gguf -c 0 --port 8080

This is the OpenAI-compatible endpoint described in Run the server, which also documents the recommended server flags.

Build from source

If the pre-built binaries don’t cover your platform or you want a specific backend:

git clone -b prism https://github.com/PrismML-Eng/llama.cpp
cd llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release   # add a backend flag below
cmake --build build -j

Target	CMake flag
CUDA	`-DGGML_CUDA=ON`
Metal	`-DGGML_METAL=ON` (default on macOS)
Vulkan	`-DGGML_VULKAN=ON`
ROCm / HIP	`-DGGML_HIP=ON`

The prism branch is required for ternary Q2_0. For 1-bit only, upstream llama.cpp works too. The demo repo also has ready-made build scripts: scripts/build_mac.sh, scripts/build_cpu_linux.sh, scripts/build_cuda_linux.sh, and scripts/build_cuda_windows.ps1.

Get started

Models

Download

Run Locally

Integrations

Resources

Getting Started

Get a binary

Run a chat

Start a server

Build from source

​Get a binary

​Run a chat

​Start a server

​Build from source

Get a binary

Run a chat

Start a server

Build from source