> ## Documentation Index
> Fetch the complete documentation index at: https://docs.prismml.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Troubleshooting

> Fixes for the failure modes people actually hit: kernels, memory, ports, and platform quirks.

## The ternary model won't load (or produces an error about the quantization type)

Ternary GGUF uses the `Q2_0` type with Bonsai's own kernels, which are **not** in upstream llama.cpp yet (an [upstream PR](https://github.com/ggml-org/llama.cpp/pull/24448) is in progress). Stock builds, and tools that bundle them (current Ollama releases), refuse the file.

**Fix:** use the [PrismML llama.cpp fork or its pre-built binaries](/run/llamacpp#get-a-binary), or run ternary through [MLX](/run/mlx) on Apple Silicon, where stock MLX works. For Ollama specifically, use the 1-bit family instead. See [Formats & runtime support](/download/formats).

## Generation is much slower than the published numbers

The published speeds (131 tok/s on M4 Pro for 1-bit 8B, etc.) assume native low-bit kernels and full GPU offload. Check, in order:

1. **Is the runtime dequantizing?** A runtime that recognizes the file but lacks `Q1_0`/`Q2_0` kernels may fall back to higher precision: same output, none of the speed. If memory use during inference is far above the [expected totals](/models/bonsai-8b#memory-requirements) (e.g. multiple GB more than weights + KV cache), that's the signature. Use the [known-good binaries](https://github.com/PrismML-Eng/llama.cpp/releases/tag/prism-b9570-0ad1dab).
2. **Are layers on the GPU?** With raw `llama-cli`/`llama-server`, pass `-ngl 99`. The demo scripts set this automatically. The load log prints how many layers were offloaded.
3. **Right binary for your hardware?** On x86 CPUs, the optimized build is significantly faster than the generic one; on macOS, make sure you're using the Metal (Apple Silicon) build. See the [binary variants](/download/formats#pre-built-binaries).
4. **Long context?** Decode slows as the KV cache grows; benchmark numbers are short-context. Compare against [community benchmarks](https://github.com/PrismML-Eng/Bonsai-demo/tree/main/community-benchmarks) for your hardware.

## Out of memory, or the model won't fit

Total memory = weight file + KV cache, and the KV cache grows linearly with context (144 KiB/token on the 8B; see each [model card](/models/bonsai-8b#memory-requirements)).

* Cap the context explicitly: `-c 8192` instead of the auto-fit `-c 0`.
* Or use llama.cpp's KV-cache quantization (`--cache-type-k q8_0 --cache-type-v q8_0`) to roughly halve the cache term.
* Or step down: ternary → 1-bit, or one size smaller.

## "GGUF model not found" from the scripts

The script's `BONSAI_MODEL`/`BONSAI_FAMILY` don't match what's downloaded. Download the variant you asked for:

```bash theme={null}
BONSAI_FAMILY=ternary BONSAI_MODEL=8B ./scripts/download_models.sh
```

Also note `BONSAI_MODEL=all` / `BONSAI_FAMILY=all` are valid **only** for setup and download scripts, not for run/server scripts, which need exactly one model.

## Port already in use

* The Bonsai server refuses to start if something is on **8080**: stop it with `kill $(lsof -ti TCP:8080)` (macOS/Linux), or start on another port by passing `--port` through the script.
* A standalone `open-webui serve` also defaults to 8080. Use `./scripts/start_openwebui.sh` (which picks 9090+) or pass `--port`. See [Open WebUI](/run/open-webui).

## Windows: "running scripts is disabled on this system"

PowerShell's execution policy blocks `setup.ps1`. Allow it for the current session only:

```powershell theme={null}
Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass
.\setup.ps1
```

## MLX errors on 1-bit weights

1-bit support in stock MLX is pending ([mlx#3161](https://github.com/ml-explore/mlx/pull/3161)). Until it merges, install the fork:

```bash theme={null}
pip install "mlx-lm @ git+https://github.com/PrismML-Eng/mlx"
```

Ternary (2-bit) MLX weights work with stock `mlx-lm`.

## The model emits `<think>` blocks or stalls before answering

Thinking-mode defaults differ by size. [Bonsai 27B](/models/bonsai-27b) reasons **by default**; the 8B/4B/1.7B demo scripts disable it (`--reasoning-budget 0 --reasoning-format none --chat-template-kwargs '{"enable_thinking": false}'`).

* To turn thinking off for 27B: start the server with `BONSAI_THINKING=0`, or cap it with `--reasoning-budget N`.
* If you're using the built-in chat UI, check the **Reasoning effort** picker (lightbulb icon) — it's saved per browser and overrides the server default for every future chat, including new ones. If replies are slower than expected even after setting `BONSAI_THINKING=0`, this is almost always why: set the picker itself to your desired effort, or clear site data for `localhost:8080`.
* If you launched `llama-server` by hand on 8B/4B/1.7B without the disable flags above, add them, or strip reasoning client-side.

## Every new chat is slow to start, even with nothing enabled

If you've turned on an MCP server (Hugging Face Hub, DeepWiki, or a custom one) from the **new-chat** screen rather than inside an existing conversation, that toggle becomes your browser's default for all future chats — so every new chat prefills that server's tool schemas (roughly 2,600 tokens for Hugging Face, 400 for DeepWiki) before the first token. Toggle it off from the new-chat screen's MCP selector, or clear site data for `localhost:8080`; a private/incognito window shows what a fresh browser would see.

## Still stuck?

Ask in the [PrismML Discord](https://discord.gg/prismml) or [open an issue](https://github.com/PrismML-Eng/Bonsai-demo/issues) on the demo repo with your platform, binary variant, and the full load log.