What this measures #
hex-bench is a personal benchmark for the hex project that tests the personal-assistant workflow of this project: every task goes through Inspect AI’s react
solver, which may call real tools — Todoist, Obsidian, web search — against a local OpenAI-compatible llama.cpp endpoint. Each of the six categories below has three tasks (18 in the full suite); every task is scored per epoch.
baseline runs are made against bigger cloud models to establish a point of comparison of sorts.
Tool use— calls the right real tool (Todoist / Obsidian / web) on live data.** Reasoning**— multi-step analysis over retrieved information.** Verbosity**— answer length: complete but no filler.** Agentic loop**— knowing when to stop acting and answer.** Understanding**— answers that match the actual source material.** Response format**— output conforms to the required format (bullets, JSON, YAML, table, …).
score = mean of all scorer values for the task (0–1).
quality = mean of the rubric scorers: judge
, loop_eval
, clarification_eval
.
protocol = mean of the deterministic compliance scorers: tool_used
, fmt_*
.
pass score ≥ 1.0 · partial 0 < score < 1 · fail score = 0.
Timings — eval t/s, gen t/s, and wall time come from sample_speed()
; older runs may lack them (shown as –). ± / green band — 95% uncertainty on the pass rate at this sample count (Wilson interval). Overlapping bands mean the models are statistical ties, not a real ranking.