Show HN: Do Codex skills save tokens? A six-run task-size benchmark

A six-run benchmark by an independent developer found that Codex skills saved tokens on a medium 2048 game build but lost on a small fix, with GPT-5.6-sol runs showing the same engineering-loop skill produced mixed results. The study held model, reasoning effort, starting commit, task contract, sandbox, and acceptance criteria constant, varying only repository-skill routing, and measured Codex CLI input plus output tokens with cached input included once. The author urges replication with forked fixtures and publication of negative findings.

Medium implementation Dependency-free 2048 Four browser-game files, ten engine tests, syntax checks, and a post-run evaluator. Six controlled GPT-5.6-sol runs The same engineering-loop skill lost on a small fix and won on a medium build. Explore the result, inspect the evidence, then run your own replication. Task-size boundary Medium implementation Four browser-game files, ten engine tests, syntax checks, and a post-run evaluator. What was held constant Same model, reasoning effort, starting commit, task contract, sandbox, and acceptance criteria. Only repository-skill routing changed. Acceptance, required checks, and evidence completeness were primary. A cheaper failed run would not win. No repository skill, the original v0.2.0 loop, and the lean v0.4.0 loop started from equivalent fresh copies. Token totals are Codex CLI input plus output tokens. Cached input is already included and was not counted twice. Read this before sharing Make the evidence better Fork the fixture, hold the environment constant, report every result, and publish negative findings too.