Run the Server

The demo repo ships an OpenAI-compatible server. Start it once and every tool in these docs (OpenClaw, Open WebUI, Hermes, your own code) connects to the same endpoint.

Start the server

./scripts/start_llama_server.sh    # http://localhost:8080

.\scripts\start_llama_server.ps1   # http://localhost:8080

The server loads whichever model BONSAI_MODEL / BONSAI_FAMILY select — Ternary Bonsai 2 27B unless you set them (see Quickstart) — and also serves a built-in web chat at http://localhost:8080, with image upload, an MCP client, and a per-chat reasoning-effort picker. On Apple Silicon there is a second option, an MLX-backed server on its own port:

./scripts/start_mlx_server.sh      # http://localhost:8081

Both speak the same API. The llama.cpp server is the default; the MLX server is usually faster on M-series Macs.

The endpoint


Base URL	`http://localhost:8080/v1` (llama.cpp) or `http://localhost:8081/v1` (MLX)
API key	Any non-empty string; it isn’t checked
Model ID	Any string works for llama-server (it serves the one loaded model); `GET /v1/models` returns the real name

The scripts bind to 127.0.0.1, so the server is reachable only from the same machine and there is no authentication. Setting BONSAI_HOST to any non-loopback address (for example 0.0.0.0) exposes an unauthenticated server to your network — only do that on a trusted network, or firewall the port.

What the script configures

start_llama_server.sh doesn’t just launch llama-server; it applies Bonsai’s recommended settings:

Setting	Value	Meaning
context	auto (RAM-tiered)	A context size chosen from your machine’s RAM, 8K on small machines up to 131K for the 27B above 71 GB. Override with `BONSAI_CTX` (any value up to `262144`; `0` or unset means this automatic size, not llama.cpp’s `-c 0`)
`-ngl`	auto	Offload all layers to GPU when one is detected (Metal, CUDA, ROCm, Vulkan). Set `BONSAI_NGL=0` for CPU-only, or any layer count for partial offload
`--temp`	0.5	Recommended sampling temperature
`--top-p`	0.85	Recommended nucleus sampling
`--top-k`	20, `--min-p` 0	Recommended truncation settings
thinking	on (27B)	The 27B serves as a reasoning model; clients pick an effort per request, and `--reasoning-budget N` caps the server-wide default
vision projector	loaded	The `mmproj` is loaded automatically for the 27B, so image input works out of the box

Use the same sampling values when you call the API directly or configure a third-party tool; they are what the published benchmarks assume. ‹TODO: confirm benchmark sampling settings match the demo defaults›

Thinking mode defaults differ by size. The 27B is a reasoning model and serves with thinking on by default. In the built-in chat UI, click the lightbulb in the message box and pick a Reasoning effort — Off, Low (512 tokens), Medium (2,048), High (8,192), or Max (unlimited); the pick persists per browser and is sent with every request, no restart needed. Over the API, toggle it per request with thinking_budget_tokens (0 disables it, -1 is unlimited), cap the default for clients that don’t specify one with --reasoning-budget N, or disable it entirely with BONSAI_THINKING=0. The 8B/4B/1.7B demo scripts still disable thinking (enable_thinking: false) by default for faster responses.On slower hardware, thinking is usually the bulk of the wait — pick a lower effort.

Extra arguments pass straight through to llama-server, e.g. ./scripts/start_llama_server.sh --reasoning-budget 2048 to cap the default reasoning budget, or -c 32768 to pin the context size.

Optional extras

Three off-by-default flags for the llama.cpp server:

Variable	Effect
`BONSAI_SPECULATIVE=1`	Speculative decoding with the `dspark` drafter. 1.8–2.4× faster decode for the ternary 27B on an L40S (CUDA), 1.4–1.75× for the 1-bit 27B; on Apple Silicon it only pays off for ternary code/math and is a net slowdown otherwise. Needs the demo’s binaries
`BONSAI_KV4=1`	4-bit KV cache — roughly 3.5× less KV memory for very long contexts, with an optional calibration bias (`./scripts/make_kv_bias.sh`)
`BONSAI_MMPROJ_CPU=1`	Keeps the vision projector in system RAM instead of VRAM, freeing ~0.9 GiB for KV/context on tight cards. Image prompts get slower; text-only chat is unaffected

Call it

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "bonsai",
    "messages": [{"role": "user", "content": "What is the capital of France?"}],
    "temperature": 0.5,
    "top_p": 0.85
  }'

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="not-needed")

response = client.chat.completions.create(
    model="bonsai",
    messages=[{"role": "user", "content": "What is the capital of France?"}],
    temperature=0.5,
    top_p=0.85,
)
print(response.choices[0].message.content)

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "http://localhost:8080/v1",
  apiKey: "not-needed",
});

const response = await client.chat.completions.create({
  model: "bonsai",
  messages: [{ role: "user", content: "What is the capital of France?" }],
  temperature: 0.5,
  top_p: 0.85,
});
console.log(response.choices[0].message.content);

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="not-needed")

stream = client.chat.completions.create(
    model="bonsai",
    messages=[{"role": "user", "content": "Write a haiku about memory bandwidth."}],
    stream=True,
)
for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="", flush=True)

Any client that accepts a custom base URL works the same way.

Images and tool calls

The 27B accepts image_url parts in messages and does native OpenAI-style tool_calls with full round-trips, so vision and tool use work over this endpoint with no special client. The built-in chat UI also ships an MCP client (Hugging Face and DeepWiki preconfigured, per-chat opt-in). Costs, the image-token cap, and adding your own MCP servers: VISION.md and TOOLS.md in the demo repo.

Connect a tool

Open WebUI

A ChatGPT-style interface; one script starts everything.

OpenClaw

Bridge Bonsai into a personal AI agent gateway.

Hermes

Run the Hermes terminal agent on local Bonsai.

All integrations

Everything that speaks OpenAI-compatible.

Get started

Models

Download

Run Locally

Integrations

Resources

Getting Started

Run the Server

Start the server

The endpoint

What the script configures

Optional extras

Call it

Images and tool calls

Connect a tool

Open WebUI

OpenClaw

Hermes

All integrations

​Start the server

​The endpoint

​What the script configures

​Optional extras

​Call it

​Images and tool calls

​Connect a tool

Open WebUI

OpenClaw

Hermes

All integrations

Start the server

The endpoint

What the script configures

Optional extras

Call it

Images and tool calls

Connect a tool