Skip to main content

​
The ternary model won’t load (or produces an error about the quantization type)

Ternary GGUF uses the Q2_0 type with Bonsai’s own kernels, which are not in upstream llama.cpp yet (an upstream PR is in progress). Stock builds, and tools that bundle them (current Ollama releases), refuse the file. Fix: use the PrismML llama.cpp fork or its pre-built binaries, or run ternary through MLX on Apple Silicon, where stock MLX works. For Ollama specifically, use the 1-bit family instead. See Formats & runtime support.

​
Generation is much slower than the published numbers

The published speeds (131 tok/s on M4 Pro for 1-bit 8B, etc.) assume native low-bit kernels and full GPU offload. Check, in order:
  1. Is the runtime dequantizing? A runtime that recognizes the file but lacks Q1_0/Q2_0 kernels may fall back to higher precision: same output, none of the speed. If memory use during inference is far above the expected totals (e.g. multiple GB more than weights + KV cache), that’s the signature. Use the known-good binaries.
  2. Are layers on the GPU? With raw llama-cli/llama-server, pass -ngl 99. The demo scripts set this automatically. The load log prints how many layers were offloaded.
  3. Right binary for your hardware? On x86 CPUs, the optimized build is significantly faster than the generic one; on macOS, make sure you’re using the Metal (Apple Silicon) build. See the binary variants.
  4. Long context? Decode slows as the KV cache grows; benchmark numbers are short-context. Compare against community benchmarks for your hardware.

​
Out of memory, or the model won’t fit

Total memory = weight file + KV cache, and the KV cache grows linearly with context (144 KiB/token on the 8B; see each model card).
  • Cap the context explicitly: -c 8192 instead of the auto-fit -c 0.
  • Or use llama.cpp’s KV-cache quantization (--cache-type-k q8_0 --cache-type-v q8_0) to roughly halve the cache term.
  • Or step down: ternary → 1-bit, or one size smaller.

​
”GGUF model not found” from the scripts

The script’s BONSAI_MODEL/BONSAI_FAMILY don’t match what’s downloaded. Download the variant you asked for: 
BONSAI_FAMILY=ternary BONSAI_MODEL=8B ./scripts/download_models.sh
Also note BONSAI_MODEL=all / BONSAI_FAMILY=all are valid only for setup and download scripts, not for run/server scripts, which need exactly one model.

​
Port already in use

  • The Bonsai server refuses to start if something is on 8080: stop it with kill $(lsof -ti TCP:8080) (macOS/Linux), or start on another port by passing --port through the script.
  • A standalone open-webui serve also defaults to 8080. Use ./scripts/start_openwebui.sh (which picks 9090+) or pass --port. See Open WebUI.

​
Windows: “running scripts is disabled on this system”

PowerShell’s execution policy blocks setup.ps1. Allow it for the current session only: 
Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass
.\setup.ps1

​
MLX errors on 1-bit weights

1-bit support in stock MLX is pending (mlx#3161). Until it merges, install the fork: 
pip install "mlx-lm @ git+https://github.com/PrismML-Eng/mlx"
Ternary (2-bit) MLX weights work with stock mlx-lm.

​
The model emits <think> blocks or stalls before answering

Thinking-mode defaults differ by size. Bonsai 27B reasons by default; the 8B/4B/1.7B demo scripts disable it (--reasoning-budget 0 --reasoning-format none --chat-template-kwargs '{"enable_thinking": false}').
  • To turn thinking off for 27B: start the server with BONSAI_THINKING=0, or cap it with --reasoning-budget N.
  • If you’re using the built-in chat UI, check the Reasoning effort picker (lightbulb icon) — it’s saved per browser and overrides the server default for every future chat, including new ones. If replies are slower than expected even after setting BONSAI_THINKING=0, this is almost always why: set the picker itself to your desired effort, or clear site data for localhost:8080.
  • If you launched llama-server by hand on 8B/4B/1.7B without the disable flags above, add them, or strip reasoning client-side.

​
Every new chat is slow to start, even with nothing enabled

If you’ve turned on an MCP server (Hugging Face Hub, DeepWiki, or a custom one) from the new-chat screen rather than inside an existing conversation, that toggle becomes your browser’s default for all future chats — so every new chat prefills that server’s tool schemas (roughly 2,600 tokens for Hugging Face, 400 for DeepWiki) before the first token. Toggle it off from the new-chat screen’s MCP selector, or clear site data for localhost:8080; a private/incognito window shows what a fresh browser would see.

​
Still stuck?

Ask in the PrismML Discord or open an issue on the demo repo with your platform, binary variant, and the full load log.