hexbench

The hex-bench benchmark, a personal evaluation suite for the hex project, tests personal-assistant workflows using Inspect AI's react solver against a local OpenAI-compatible llama.cpp endpoint, with 18 tasks across six categories scored per epoch. Baseline runs use larger cloud models for comparison, and scores are reported with 95% Wilson confidence intervals, where overlapping bands indicate statistical ties rather than definitive rankings.

What this measures hex-bench is a personal benchmark for the hex https://0xff.nu/hex project that tests the personal-assistant workflow of this project: every task goes through Inspect AI’s react solver, which may call real tools — Todoist, Obsidian, web search — against a local OpenAI-compatible llama.cpp endpoint. Each of the six categories below has three tasks 18 in the full suite ; every task is scored per epoch. baseline runs are made against bigger cloud models to establish a point of comparison of sorts. Tool use — calls the right real tool Todoist / Obsidian / web on live data. Reasoning — multi-step analysis over retrieved information. Verbosity — answer length: complete but no filler. Agentic loop — knowing when to stop acting and answer. Understanding — answers that match the actual source material. Response format — output conforms to the required format bullets, JSON, YAML, table, … . score = mean of all scorer values for the task 0–1 . quality = mean of the rubric scorers: judge , loop eval , clarification eval . protocol = mean of the deterministic compliance scorers: tool used , fmt . pass score ≥ 1.0 · partial 0 < score < 1 · fail score = 0. Timings — eval t/s, gen t/s, and wall time come from sample speed ; older runs may lack them shown as – . ± / green band — 95% uncertainty on the pass rate at this sample count Wilson interval . Overlapping bands mean the models are statistical ties, not a real ranking.