> ## Documentation Index
> Fetch the complete documentation index at: https://docs.prismml.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Run the Server
> Start Bonsai's OpenAI-compatible server and call it from curl, Python, or any client.
The demo repo ships an **OpenAI-compatible server**. Start it once and every tool in these docs (OpenClaw, Open WebUI, Hermes, your own code) connects to the same endpoint.
## Start the server
```bash macOS / Linux theme={null}
./scripts/start_llama_server.sh # http://localhost:8080
```
```powershell Windows (PowerShell) theme={null}
.\scripts\start_llama_server.ps1 # http://localhost:8080
```
The server loads whichever model `BONSAI_MODEL` / `BONSAI_FAMILY` select (see [Quickstart](/get-started/quickstart)) and also serves a built-in web chat at `http://localhost:8080`.
On Apple Silicon there is a second option, an MLX-backed server on its own port:
```bash theme={null}
./scripts/start_mlx_server.sh # http://localhost:8081
```
Both speak the same API. The llama.cpp server is the default; the MLX server is usually faster on M-series Macs.
## The endpoint
| | |
| -------- | ---------------------------------------------------------------------------------------------------------- |
| Base URL | `http://localhost:8080/v1` (llama.cpp) or `http://localhost:8081/v1` (MLX) |
| API key | Any non-empty string; it isn't checked |
| Model ID | Any string works for llama-server (it serves the one loaded model); `GET /v1/models` returns the real name |
The scripts bind to `0.0.0.0`, so the server is reachable from other machines on your network, and there is no authentication. On an untrusted network, restrict it (for llama-server, edit the script's `HOST` to `127.0.0.1`) or firewall the port.
## What the script configures
`start_llama_server.sh` doesn't just launch `llama-server`; it applies Bonsai's recommended settings:
| Setting | Value | Meaning |
| ------------------------------------------------ | --------------- | ---------------------------------------------------------------- |
| `-c 0` | auto | Fit the context window to available memory (up to the model max) |
| `-ngl` | auto | Offload all layers to GPU when one is available |
| `--temp` | 0.5 | Recommended sampling temperature |
| `--top-p` | 0.85 | Recommended nucleus sampling |
| `--top-k` | 20, `--min-p` 0 | Recommended truncation settings |
| `--reasoning-budget 0`, `enable_thinking: false` | off | Disables the chat template's thinking mode |
Use the same sampling values when you call the API directly or configure a third-party tool; they are what the published [benchmarks](/models/benchmarks) assume. ‹TODO: confirm benchmark sampling settings match the demo defaults›
**Thinking mode defaults differ by size.** [Bonsai 27B](/models/bonsai-27b) is a reasoning model and serves with thinking **on** by default; toggle it per request with `thinking_budget_tokens` (`0` disables it, `-1` is unlimited), cap it server-wide with `--reasoning-budget N`, or disable it entirely with `BONSAI_THINKING=0`. The 8B/4B/1.7B demo scripts still disable thinking (`enable_thinking: false`) by default for faster responses.
Extra arguments pass straight through to `llama-server`, e.g. `./scripts/start_llama_server.sh -c 32768` to pin the context size.
## Call it
```bash curl theme={null}
curl http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "bonsai",
"messages": [{"role": "user", "content": "What is the capital of France?"}],
"temperature": 0.5,
"top_p": 0.85
}'
```
```python Python (openai SDK) theme={null}
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8080/v1", api_key="not-needed")
response = client.chat.completions.create(
model="bonsai",
messages=[{"role": "user", "content": "What is the capital of France?"}],
temperature=0.5,
top_p=0.85,
)
print(response.choices[0].message.content)
```
```typescript TypeScript (openai SDK) theme={null}
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "http://localhost:8080/v1",
apiKey: "not-needed",
});
const response = await client.chat.completions.create({
model: "bonsai",
messages: [{ role: "user", content: "What is the capital of France?" }],
temperature: 0.5,
top_p: 0.85,
});
console.log(response.choices[0].message.content);
```
```python Streaming (Python) theme={null}
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8080/v1", api_key="not-needed")
stream = client.chat.completions.create(
model="bonsai",
messages=[{"role": "user", "content": "Write a haiku about memory bandwidth."}],
stream=True,
)
for chunk in stream:
print(chunk.choices[0].delta.content or "", end="", flush=True)
```
Any client that accepts a custom base URL works the same way.
## Connect a tool
A ChatGPT-style interface; one script starts everything.
Bridge Bonsai into a personal AI agent gateway.
Run the Hermes terminal agent on local Bonsai.
Everything that speaks OpenAI-compatible.