Skip to main content
The demo repo ships an OpenAI-compatible server. Start it once and every tool in these docs (OpenClaw, Open WebUI, Hermes, your own code) connects to the same endpoint.

​
Start the server

./scripts/start_llama_server.sh    # http://localhost:8080

.\scripts\start_llama_server.ps1   # http://localhost:8080
The server loads whichever model BONSAI_MODEL / BONSAI_FAMILY select (see Quickstart) and also serves a built-in web chat at http://localhost:8080. On Apple Silicon there is a second option, an MLX-backed server on its own port: 
./scripts/start_mlx_server.sh      # http://localhost:8081
Both speak the same API. The llama.cpp server is the default; the MLX server is usually faster on M-series Macs.

​
The endpoint

Base URLhttp://localhost:8080/v1 (llama.cpp) or http://localhost:8081/v1 (MLX)
API keyAny non-empty string; it isn’t checked
Model IDAny string works for llama-server (it serves the one loaded model); GET /v1/models returns the real name
The scripts bind to 0.0.0.0, so the server is reachable from other machines on your network, and there is no authentication. On an untrusted network, restrict it (for llama-server, edit the script’s HOST to 127.0.0.1) or firewall the port.

​
What the script configures

start_llama_server.sh doesn’t just launch llama-server; it applies Bonsai’s recommended settings:
SettingValueMeaning
-c 0autoFit the context window to available memory (up to the model max)
-nglautoOffload all layers to GPU when one is available
--temp0.5Recommended sampling temperature
--top-p0.85Recommended nucleus sampling
--top-k20, --min-p 0Recommended truncation settings
--reasoning-budget 0, enable_thinking: falseoffDisables the chat template’s thinking mode
Use the same sampling values when you call the API directly or configure a third-party tool; they are what the published benchmarks assume. ‹TODO: confirm benchmark sampling settings match the demo defaults›
Thinking mode defaults differ by size. Bonsai 27B is a reasoning model and serves with thinking on by default; toggle it per request with thinking_budget_tokens (0 disables it, -1 is unlimited), cap it server-wide with --reasoning-budget N, or disable it entirely with BONSAI_THINKING=0. The 8B/4B/1.7B demo scripts still disable thinking (enable_thinking: false) by default for faster responses.
Extra arguments pass straight through to llama-server, e.g. ./scripts/start_llama_server.sh -c 32768 to pin the context size.

​
Call it

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "bonsai",
    "messages": [{"role": "user", "content": "What is the capital of France?"}],
    "temperature": 0.5,
    "top_p": 0.85
  }'

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="not-needed")

response = client.chat.completions.create(
    model="bonsai",
    messages=[{"role": "user", "content": "What is the capital of France?"}],
    temperature=0.5,
    top_p=0.85,
)
print(response.choices[0].message.content)

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "http://localhost:8080/v1",
  apiKey: "not-needed",
});

const response = await client.chat.completions.create({
  model: "bonsai",
  messages: [{ role: "user", content: "What is the capital of France?" }],
  temperature: 0.5,
  top_p: 0.85,
});
console.log(response.choices[0].message.content);

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="not-needed")

stream = client.chat.completions.create(
    model="bonsai",
    messages=[{"role": "user", "content": "Write a haiku about memory bandwidth."}],
    stream=True,
)
for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="", flush=True)
Any client that accepts a custom base URL works the same way.

​
Connect a tool

Open WebUI

A ChatGPT-style interface; one script starts everything.

OpenClaw

Bridge Bonsai into a personal AI agent gateway.

Hermes

Run the Hermes terminal agent on local Bonsai.

All integrations

Everything that speaks OpenAI-compatible.