SkillBuffBench: Measuring What Agent Skills Contribute across Harnesses, Models, Skill Access Modes and Memory A new arXiv paper (2611.22017v1) introduces SkillBuffBench, a benchmark and paired evaluation protocol that measures how much Agent Skills — file-based packages of procedural knowledge a coding agent loads on demand — actually contribute to task performance. SkillBuffBench treats Skill access as a three-level factor (no Skill, a curated Skill, and automatic invocation from an installed library), crosses it with memory on and off, four agent harnesses and multiple models from Anthropic, Google, OpenAI and Chinese open-weight providers, and scores runs on expert-contributed research tasks in materials science and biology with a deterministic verifier and paired, task-level bootstrap uncertainty. The paper describes only the benchmark design, protocol and current tasks and Skills; the authors state the evaluation is in progress and results will be added once runs are complete, with the suite updated on a rolling basis. arXiv:2611.22017v1 Announce Type: new Abstract: Agent Skills, file-based packages of procedural knowledge that a coding agent can discover and load on demand, are now shipped by every major agent harness, yet the evidence for how much they help comes almost entirely from settings in which the right Skill is hand-picked for the task and the agent starts from a blank slate. Neither holds in deployment, where an agent chooses from whatever library happens to be installed and frequently carries memory from earlier work. We present SkillBuffBench, a benchmark and paired evaluation protocol that treats Skill access as a three-level factor no Skill, a curated Skill, and automatic invocation from an installed library , crosses it with memory off and on , with four agent harnesses and with multiple models the state-of-the-art proprietary models from Anthropic, Google and OpenAI, and the state-of-the-art open-weight models from Chinese providers , every model being run under every harness, and evaluates the result on research tasks contributed by domain experts together with the Skills they use, currently in materials science and biology, with the suite extended on a rolling basis. Every run is marked by a deterministic verifier; uncertainty is reported with a paired, task-level bootstrap; and cost, agent rounds and wall-clock time are reported alongside the score rather than folded into it. This version of the paper describes the design of the benchmark, its evaluation protocol and its current tasks and Skills. The evaluation is in progress: results will be added once the runs are complete, and the task suite, results and analysis will continue to be updated on a rolling basis.