# hexbench

> Source: <https://0xff.nu/hexbench/>
> Published: 2026-08-25 00:00:00+00:00

## What this measures

hex-bench is a personal benchmark for the [hex](https://0xff.nu/hex) project that tests the personal-assistant workflow of this project: every task goes through Inspect AI’s `react`

solver, which may call real tools — Todoist, Obsidian, web search — against a local OpenAI-compatible llama.cpp endpoint. Each of the six categories below has three tasks (18 in the full suite); every task is scored per epoch.

baseline runs are made against bigger cloud models to establish a point of comparison of sorts.

**Tool use**— calls the right real tool (Todoist / Obsidian / web) on live data.** Reasoning**— multi-step analysis over retrieved information.** Verbosity**— answer length: complete but no filler.** Agentic loop**— knowing when to stop acting and answer.** Understanding**— answers that match the actual source material.** Response format**— output conforms to the required format (bullets, JSON, YAML, table, …).

**score** = mean of all scorer values for the task (0–1).

**quality** = mean of the rubric scorers: `judge`

, `loop_eval`

, `clarification_eval`

.

**protocol** = mean of the deterministic compliance scorers: `tool_used`

, `fmt_*`

.

**pass** score ≥ 1.0 · **partial** 0 < score < 1 · **fail** score = 0.

**Timings** — eval t/s, gen t/s, and wall time come from `sample_speed()`

; older runs may lack them (shown as –).

**± / green band** — 95% uncertainty on the pass rate at this sample count (Wilson interval). Overlapping bands mean the models are statistical ties, not a real ranking.