Werkbank: 40 tasks for testing local LLMs with tools A hobbyist developer released Werkbank, a set of 40 fixed, synthetic tasks for testing local large language models on multi-step tool use, with deterministic checks of final state, code or answer instead of an LLM judge. Werkbank reports time and token costs per solved task and aims to sit between ad hoc prompts and large benchmark suites, with initial tests run on GB10 and RTX 6000 hardware. The developer is asking others to run the tasks on their own setups and report results. Hi everyone I built “Werkbank” as a hobby project, mostly for fun and the experience and partly because I wanted something between ad hoc prompts and a large benchmark suite for my local setup. A lot of benchmarks already cover multi-step tool use. What I wanted was a small set of controlled tasks with deterministic checks of the final state, code or answer, rather than an LLM judge, plus time and token costs per solved task without having to spend 3 days to run a single seed. Werkbank has 40 fixed, synthetic tasks, leaning toward the harder end, especially in the Full set. My initial tests include GB10 nad RTX 6000 hardware. Results try to describe the whole setup, including runtime and quantization, not just the model. If you have some time and inference to spare, I’d be happy if you gave it a try on your own setup. Would love to hear how it goes and what you think of the tasks. reposted here from Nvidia Forum