Local LLM bench lab

Oct 2026 — Local LLMs — In progress

Describe an experiment in one YAML file: the models, the llama.cpp settings and the parameters to vary. The llb command builds llama.cpp, downloads the models, runs every combination while it watches temperature and memory, stores everything in SQLite and tells you which configuration wins, with the llama-server command to deploy it.

It was designed for a Raspberry Pi 5 first, and it works on any Linux PC, including WSL2.

The dashboard's llama-bench chart: two quantizations of Qwen3-0.6B at 4 and 8 threads, with 95% confidence intervalsTokens per second with 95% confidence intervals

What it measures

  • Speed, latency, quality and concurrency in one tool, driving llama-bench, llama-server and, optionally, lm-evaluation-harness.
  • Reproducible: every result records the llama.cpp commit, the model's sha256, the exact flags and the machine. Every flag is checked against the binary before running.
  • Honest numbers: confidence intervals per test, cooldown between runs, and noisy or throttled runs flagged as unstable.
  • Answers, not just tables: pass/fail gates, a winner, HTML and Markdown reports and a web dashboard.

The dashboard

The test tab: an experiment in the form and in YAML, with the validate, plan, run, guide, report and judge buttonsBuilding an experiment

The models tab: the quantizations of Qwen3-8B and the models on diskPicking models and quantizations

The chat tab: Qwen3-0.6B answering in Spanish with its metricsChatting with a model, with its metrics

The suites tab: a writing suite run on Qwen3-0.6B, with each answer's scoreQuality suites, scored answer by answer

Status

Version 0.11: every planned phase is implemented. 1.0 will follow validation on a Raspberry Pi 5.

Code on GitHub