> ## Documentation Index
> Fetch the complete documentation index at: https://docs.prismml.com/llms.txt
> Use this file to discover all available pages before exploring further.

# llama.cpp

> Run Bonsai GGUF with llama.cpp: pre-built binaries, CLI usage, and building from source.

[llama.cpp](https://github.com/ggml-org/llama.cpp) is the reference runtime for Bonsai **GGUF** weights. It runs on CPU and every major GPU backend (CUDA, Metal, Vulkan, ROCm).

<Warning>
  **1-bit (`Q1_0`)** is merged into upstream llama.cpp, so recent upstream builds work. **Ternary (`Q2_0`)** needs the [PrismML fork](https://github.com/PrismML-Eng/llama.cpp) (`prism` branch) or its pre-built binaries; stock builds cannot run it. See [Formats & runtime support](/download/formats).
</Warning>

## Get a binary

The demo repo's `setup.sh` downloads a compatible build automatically into `bin/`. To grab one yourself, use the [PrismML llama.cpp release](https://github.com/PrismML-Eng/llama.cpp/releases/tag/prism-b9570-0ad1dab), which covers 1-bit and ternary on macOS (Apple Silicon and Intel), Linux (CPU / CUDA 12.4 / CUDA 12.8 / Vulkan / ROCm), Windows (CPU / CUDA / Vulkan / HIP), and iOS (XCFramework). The full variant list is in [Formats & runtime support](/download/formats#pre-built-binaries).

## Run a chat

```bash theme={null}
./llama-cli -m ./Bonsai-8B-gguf/Bonsai-8B-Q1_0.gguf -c 0 \
  --temp 0.5 --top-p 0.85 --top-k 20 --min-p 0 \
  -p "What is the capital of France?"
```

* `-c 0` auto-fits the context window to available memory; pass an explicit value (e.g. `-c 32768`) to pin it and bound memory use. Memory-per-context numbers are on each [model card](/models/bonsai-8b#memory-requirements).
* The sampling flags are Bonsai's recommended defaults.
* `-ngl 99` offloads all layers to the GPU (the wrapper script sets this automatically when a GPU is present).

<Note>
  [Bonsai 27B](/models/bonsai-27b) needs two extra flags: `--mmproj <path>.gguf` for image input, and `--jinja` for native OpenAI-style `tool_calls` (the demo scripts already pass both).
</Note>

The demo repo wraps all of this:

<CodeGroup>
  ```bash macOS / Linux theme={null}
  ./scripts/run_llama.sh -p "What is the capital of France?"
  ```

  ```powershell Windows (PowerShell) theme={null}
  .\scripts\run_llama.ps1 -p "What is the capital of France?"
  ```
</CodeGroup>

## Start a server

```bash theme={null}
./llama-server -m ./Bonsai-8B-gguf/Bonsai-8B-Q1_0.gguf -c 0 --port 8080
```

This is the OpenAI-compatible endpoint described in [Run the server](/run/server), which also documents the recommended server flags.

## Build from source

If the pre-built binaries don't cover your platform or you want a specific backend:

```bash theme={null}
git clone -b prism https://github.com/PrismML-Eng/llama.cpp
cd llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release   # add a backend flag below
cmake --build build -j
```

| Target     | CMake flag                           |
| ---------- | ------------------------------------ |
| CUDA       | `-DGGML_CUDA=ON`                     |
| Metal      | `-DGGML_METAL=ON` (default on macOS) |
| Vulkan     | `-DGGML_VULKAN=ON`                   |
| ROCm / HIP | `-DGGML_HIP=ON`                      |

The `prism` branch is required for ternary `Q2_0`. For 1-bit only, upstream llama.cpp works too. The demo repo also has ready-made build scripts: `scripts/build_mac.sh`, `scripts/build_cpu_linux.sh`, `scripts/build_cuda_linux.sh`, and `scripts/build_cuda_windows.ps1`.