Skip to main content
This page gets you from zero to a running Bonsai 27B on macOS, Linux, or Windows. One script installs build tools, downloads the model weights and pre-built inference binaries, and leaves you ready to chat. Prerequisites: git, curl, and enough disk for the model you pick (see sizes). Everything else, including Python tooling and the inference engine, is installed by the setup script.
Examples use BONSAI_MODEL=27B. Set 8B, 4B, or 1.7B for a smaller model; everything else is identical. If you skip step 2 entirely, setup uses its defaults: BONSAI_MODEL=27B, BONSAI_FAMILY=ternary.

​
Install and run

1

Clone the repo

git clone https://github.com/PrismML-Eng/Bonsai-demo.git
cd Bonsai-demo
2

Choose your model (optional)

export BONSAI_MODEL=27B        # 27B (flagship, default), 8B, 4B, or 1.7B
export BONSAI_FAMILY=ternary   # ternary (1.58-bit, default) or bonsai (1-bit)

$env:BONSAI_MODEL = "27B"
$env:BONSAI_FAMILY = "ternary"
Not sure which to pick? Ternary is higher quality; 1-bit is smaller and faster. The model overview has the numbers.
3

Run setup

./setup.sh

Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass
.\setup.ps1
Re-running setup is safe: it skips steps that are already done. To download a different model later, change the environment variables and run ./scripts/download_models.sh (or re-run setup).
4

Chat

./scripts/run_llama.sh -p "What is the capital of France?"

.\scripts\run_llama.ps1 -p "What is the capital of France?"
On Apple Silicon you can also run through MLX, which is typically faster on M-series chips:
source .venv/bin/activate
./scripts/run_mlx.sh -p "What is the capital of France?"

​
Start a server instead

Prefer a browser UI or an API endpoint? The repo ships an OpenAI-compatible server: 
./scripts/start_llama_server.sh    # API + web chat at http://localhost:8080
Most tools in Integrations connects to this endpoint. For a full ChatGPT-style interface, ./scripts/start_openwebui.sh starts the server and Open WebUI together. See Run the server for flags, sampling defaults, and API examples.

​
What setup actually does

Even on a fresh machine, setup.sh / setup.ps1:
  1. Installs system build tools (Xcode Command Line Tools on macOS, build-essential on Linux)
  2. Installs uv and creates a Python virtualenv in .venv/
  3. Downloads the selected model from Hugging Face into models/
  4. Downloads pre-built llama.cpp binaries for your platform into bin/, or builds from source if none match
  5. On Apple Silicon, builds MLX from source for GPU acceleration
Nothing is installed globally except the system build tools; deleting the Bonsai-demo directory removes everything else.

​
Next steps

Run the server

The OpenAI-compatible endpoint, its defaults, and API examples.

Connect your tools

OpenClaw, Hermes, Ollama, and anything OpenAI-compatible.

Pick the right model

Size, family, and format trade-offs with real numbers.

Troubleshooting

Port conflicts, slow generation, models that won’t load.