{"slug": "show-hn-i-forked-an-agent-stack-and-measured-myself-against-it-losses-included", "title": "Show HN: I forked an agent stack and measured myself against it, losses included", "summary": "Toolbay Stack v0.2.0, a free MIT-licensed agent stack forked from an unnamed upstream, claims to reduce context cost by 83% across 53 paired skills and includes 145 engines each with its own test, with a backtest that reproduces all measurements. The stack's 20 free skills are available via npx, and it is designed to fail loudly rather than silently, as demonstrated by a seeded failure where the upstream's guard exits 0 when missing while Toolbay exits 1.", "body_md": "Toolbay Stack · v0.2.0\n\n# An agent stack that tells you when it failed.\n\n60 skills for Claude Code. 145 engines, and every single one carries its own test — an engine without one fails the suite rather than shipping. Free, MIT, no dependencies.\n\nMost agent tooling fails silently: a guard that allows everything when it is missing, a check that exits 0 having done nothing, a save that writes somewhere you will never look. This is built around refusing that one behaviour, and the refusals are measured rather than claimed — `npm run backtest`\n\nreproduces every number on this page.\n\nGet it\n\nToolbay Stack is public and MIT licensed. Clone it and run it — there is no account, no telemetry, and no dependency to install.\n\n```\ngit clone https://github.com/sriptcollector/toolbay-stack.git\n```\n\nIt is not on npm yet, so there is no npx one-liner. Clone is the install path. Verified 2026-08-18:\n\n`curl -s -o /dev/null -w '%{http_code}' https://github.com/sriptcollector/toolbay-stack`\n\n→ 200 — public`gh api repos/sriptcollector/toolbay-stack --jq .license.spdx_id`\n\n→ MIT`npm view toolbay-stack`\n\n→ E404 — not on npm yet\n\nWhat does run today, on the same machine, with no account and no telemetry, is the 20 free MIT-licensed skills on the marketplace. That command is verified end to end, not asserted.\n\n20 free skills, no account\n\nnpx toolbay add pr-reviewerInstalls a real Claude Code skill that reviews your diff before you push. [See all 20](/free).\n\nMeasurement 1\n\n## Context cost\n\nA `SKILL.md`\n\nis injected into the conversation every time its skill is invoked, so its size is a cost you pay per invocation. The backtest reads both stacks off disk and pairs them by skill name.\n\n| Skill | Toolbay | gstack | Smaller |\n|---|---|---|---|\n| /spec | 9.2 KB | 124 KB | 93% |\n| /review | 8.8 KB | 103.3 KB | 91% |\n| /qa | 8.1 KB | 81.2 KB | 90% |\n| /ship | 8.4 KB | 79.2 KB | 89% |\n| /retro | 10.5 KB | 90 KB | 88% |\n| /design-review | 14.3 KB | 102.5 KB | 86% |\n| 53 paired | 557.4 KB | 3193.2 KB | 83% |\n\nOne session\n\ninvestigate → review → ship → context-restore\n\nOf gstack’s 290.4 KB in that four-skill session, 134KB is lines an earlier skill in the same session already injected — an auto-generated preamble stamped into every skill and paid for again on every invocation. Toolbay Stack’s repeated share is 1.9 KB.\n\nBytes are measured; tokens are an estimate at 4 bytes/token. Files a skill reads while it runs are not counted, on either side.\n\nThree skills are bigger here\n\nThe tool prints this and so do we. These are the guard skills, and the extra bytes are the documented failure modes and the list of what the guard does *not* cover. Trimming them to win a byte count would be gaming the number the measurement exists to report.\n\n- /careful — 6.1 KB here vs 2.5 KB upstream\n- /freeze — 5.7 KB here vs 3.1 KB upstream\n- /unfreeze — 1.7 KB here vs 1.5 KB upstream\n\nMeasurement 2\n\n## Correctness on seeded failures\n\nThe backtest builds a fixture with a known defect, runs the equivalent command from both stacks against it, and records CAUGHT or MISSED per side. It fails closed: a scenario whose fixture did not build, whose upstream is not installed, or which timed out is scored for nobody.\n\n### The defects, with the file and the line\n\nOpen them yourself. Paths are inside a gstack install at `~/.claude/skills/`\n\n.\n\n- 01\n#### A safety guard that allows everything when it is missing\n\ngstack/investigate/SKILL.md:26\n\n```\n[ -x \"$S\" ] && bash \"$S\" || exit 0\n```\n\nThis is the PreToolUse hook that is supposed to hold the edit boundary during a debugging session. If the guard script is not on the machine, the `|| exit 0` fires — and exit 0 with no output is how a PreToolUse hook spells ALLOW. The protection does not fail loudly, it evaporates silently.\n\nRead directly from the file, and reproduced: backtest scenario guard-not-installed, where gstack exits 0 with no output at all and Toolbay Stack exits 1 so the failure is visible.\n\n- 02\n#### A failing test suite that reports success\n\ngstack/ship/sections/tests.md:170\n\n```\nbin/test-lane 2>&1 | tee /tmp/ship_tests.txt &\n```\n\nThe exit status of a shell pipeline is the status of its LAST command, and the last command here is tee, which essentially always succeeds. The test suite's own exit code is discarded before anything can read it. This is the step that decides whether a change is safe to land.\n\nRead directly from the file in gstack 1.60.1.0.\n\n- 03\n#### An expiry that cannot be parsed means never expires\n\ngstack/ios-qa/daemon/src/allowlist.ts:56\n\n```\nif (Number.isFinite(exp) && exp < now) continue;\n```\n\n`exp` comes from Date.parse on the entry's expires_at. A malformed date gives NaN, Number.isFinite(NaN) is false, so the `continue` never runs and the entry is returned as valid. A device grant with a garbled expiry is not rejected and is not expired — it is permanent.\n\nRead directly from the file in gstack 1.60.1.0.\n\n- 04\n#### A destructive command waved straight through\n\nbacktest scenario careful-block-device\n\n```\ndd if=/dev/zero of=/dev/sda\n```\n\nA command that destroys a filesystem. gstack's guard allowed it silently — exit 0, output \"{}\". Its sibling scenario careful-quoted-sql does the same with psql -c \"DROP TABLE users;\": the pattern match stops at the first escaped quote, so the guard never sees the DROP.\n\nnpm run backtest --only careful-block-device (raw log: .toolbay/backtest/raw/careful-block-device.gstack.log)\n\n- 05\n#### A corrupt row that silently vanishes from your learnings\n\nbacktest scenario learn-corrupt-row\n\n```\nLEARNINGS: 3 loaded   # over a store with 4 rows on disk\n```\n\nGiven a learnings store with four rows, one of them unparseable, gstack exits 0 and reports three. The fourth is gone and nothing says so. Toolbay Stack names the damaged row and exits 3.\n\nnpm run backtest --only learn-corrupt-row (raw log: .toolbay/backtest/raw/learn-corrupt-row.gstack.log)\n\n- 06\n#### Saved context that reports NO_CHECKPOINTS over files that exist\n\nbacktest scenario restore-slug-unresolvable\n\n```\nresolved CHECKPOINT_DIR=[/projects//checkpoints]\n```\n\nOn a machine where the slug helper is not at the path the skill hard-codes, the directory resolves with an empty middle segment, and the skill prints NO_CHECKPOINTS at exit 0 over saved context that is sitting on disk. An empty answer that looks like a real answer.\n\nnpm run backtest --only restore-slug-unresolvable (raw log: .toolbay/backtest/raw/restore-slug-unresolvable.gstack.log)\n\nThe other direction\n\n## What gstack does better\n\nThis section is the reason to believe the rest of the page. If you want the original, use the original — it is upstream, it is very good, and it is where most of these ideas came from.\n\n### Four scenarios where upstream is correct too\n\nScored as ties. These are not wins for us.\n\n- freeze-boundary-posix\n\nfreeze boundary enforced with POSIX-style absolute paths — both allowed the edit inside and denied the one outside - careful-rm-rf\n\nrm -rf on a real directory — both warned first - careful-safe-artifact\n\nrm -rf node_modules — both correctly stayed silent - freeze-traversal-escape\n\nan edit that leaves the boundary through .. — both denied it\n\n### One command both stacks wave straight through\n\n```\naws s3 rm s3://bucket --recursive\n```\n\nNeither stack's pattern list covers it. Both allowed it silently (exit 0, output \"{}\"). This is a real hole in Toolbay Stack, not just in upstream.\n\n### A capability upstream has that this fork never ported\n\nlearn-cross-project-trust-gate\n\nCross-project learnings with a trust allowlist. A learning written by another project, without trusted: true, is correctly refused by gstack. Toolbay Stack has no cross-project learnings reader at all, so there is nothing to gate.\n\n### Skills that exist only upstream: none, as of 2026-08-17\n\nThis section used to name six. The last four — benchmark-models, landing-report, setup-gbrain, sync-gbrain — landed on 2026-08-17, so every one of upstream's 53 skills now has a counterpart here. The heading is generated from the list rather than written, so it will say a number again the moment upstream ships something this fork has not matched.\n\nSeven exist only here (backtest, memory, modes, router, suggest, toolbay, video) and are not compared, because only one stack has them. Counts: 60 skills here, 53 upstream, 53 paired. Parity in the skill list is not parity in behaviour — the capability named above is still missing.\n\n### Two results you should not quote at us\n\nOne scenario (freeze-boundary-windows-paths) is Windows-only: gstack denied an edit that was inside the boundary because it reads a drive-letter path as relative. It does NOT reproduce on macOS or Linux, where Claude Code sends POSIX paths and upstream's guard is correct — see freeze-boundary-posix, which upstream passes. The backtest prints this caveat itself.\n\nA third scenario (ios-qa-expiry-unparseable) is INCONCLUSIVE: the upstream half only runs on macOS and this was measured on win32. It is scored for nobody.\n\nMeasurement 3\n\n## The hook that runs before every tool call\n\nThe `careful`\n\nguard is a PreToolUse hook, so its latency is added to every single tool call in a session. Median over 12 runs:\n\nMost of each number is interpreter startup: 59ms of the Toolbay hook is bare node, 37ms of the gstack hook is bare bash. The gap is real but it is not 7x of our own code.\n\nOn top of the fork\n\n## What is actually ours\n\n### Modes\n\nA stance layer. Set SHIP, CRAFT, WILD or LOCK per project and every other skill changes what it does — /review in SHIP flags blockers only, the same /review in LOCK flags everything and rejects any claim without a citation. The four modes live in one hand-edited modes.json and the code knows none of their names, so a stance is a thing you edit rather than argue with. Everything fails closed: no mode, an unknown name, a corrupt or empty config all land on a restrained built-in stance and print the reason.\n\n### Memory\n\nClaude's own memory keeps facts. This keeps corrections — it writes at the moment you say \"no, not like that\", and refuses to store one without the reason, because the reason is the only part that transfers. A newer entry never silently wins: a contradiction is surfaced with both sides and neither is applied until you say which is true. Every rule knows its scope, and when that is unclear it goes narrow. Off until you run enable twice.\n\n### Suggest\n\nReads the project's actual state and says what is worth doing next, with the evidence. Names the command, never runs it, and says \"nothing worth suggesting\" when that is true. The active mode changes what gets offered, and every suppressed suggestion is printed with the weight that dropped it — but no mode at any weight can suppress a credential about to be committed.\n\n### Backtest\n\nThe reason you are reading real numbers instead of adjectives. It seeds fixtures with a known defect, runs the equivalent command from BOTH stacks, and records CAUGHT or MISSED per side. It fails closed: a scenario whose fixture did not build, whose upstream is missing, or which timed out is scored for nobody. You can point it at your own setup and prove your own hooks fire.\n\n### Cursor support\n\nThe guards port to Cursor, and the installer refuses to install a skill whose guard Cursor cannot actually enforce, naming the missing capability. What is not covered is printed at the top of every installed copy rather than left for you to find: Tab autocomplete does not go through the freeze boundary, and your own typing is never guarded.\n\nWhy this lives on Toolbay\n\n## The skill that reaches the marketplace\n\nEvery agent stack dead-ends the same way. gstack’s routing rule ends with *“If nothing matches, answer directly.”* (its `SKILL.md`\n\n, line 531 — quoted exactly, because a page like this one does not get to paraphrase inside quotation marks). The `toolbay`\n\nskill is the one thing a fork can add that upstream structurally cannot: somewhere to go. Blocked on a capability that does not exist locally, it searches the live catalogue on this site. Finished something reusable, it can evaluate and list it without you opening a form.\n\nThe rails on it are not optional. Search results are paid listings you would be buying, and are always shown with the price. An empty result stays empty rather than being padded with something loosely related. Revenue figures are ranges with a stated basis, never a single number and never a forecast. It is offered once, and declined means dropped. And the mode can switch it off outright — LOCK refuses `find`\n\nand `call`\n\nbefore a token is read or a packet is sent. There is no bypass flag.\n\nCheck it yourself\n\n## Reproduce every number on this page\n\nThese are the commands the figures above came from. They read the upstream install, never write inside it, and log every process they start — argv, cwd, env overrides, exit status, stdout, stderr, duration — under `.toolbay/backtest/raw/`\n\n.\n\n```\nnpm run backtest                    # every scenario, both stacks\nnode tools/backtest.mjs --list      # what the scenarios are\nnode tools/backtest.mjs --family context\nnode tools/backtest.mjs --gstack <dir>   # point it at your own install\nnpm test                            # 145 engine selftests\n```\n\nAll 145 engines passed their own selftest, and every engine on disk has one. An engine with no selftest fails the suite, because the list is read off the filesystem instead of maintained by hand — which is how the old hand-written list came to name seven engines while fifteen shipped.\n\nMeasured 2026-08-17 on node v24.14.1 / win32 against gstack 1.60.1.0. Your numbers will differ with your install; that is the point of shipping the command rather than the screenshot.\n\nToolbay Stack is an MIT fork of [gstack](https://github.com/garrytan/gstack) by [Garry Tan](https://github.com/garrytan), which is excellent software and the origin of most of the workflow skills here. Every derived skill carries an attribution header, the MIT notice is preserved, and this project is not affiliated with or endorsed by him. If you want the original, use the original.", "url": "https://wpnews.pro/news/show-hn-i-forked-an-agent-stack-and-measured-myself-against-it-losses-included", "canonical_source": "https://toolbay.ai/stack", "published_at": "2026-08-19 20:59:35+00:00", "updated_at": "2026-08-19 21:16:43.180347+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "ai-tools", "ai-safety"], "entities": ["Toolbay Stack", "Claude Code", "gstack"], "alternates": {"html": "https://wpnews.pro/news/show-hn-i-forked-an-agent-stack-and-measured-myself-against-it-losses-included", "markdown": "https://wpnews.pro/news/show-hn-i-forked-an-agent-stack-and-measured-myself-against-it-losses-included.md", "text": "https://wpnews.pro/news/show-hn-i-forked-an-agent-stack-and-measured-myself-against-it-losses-included.txt", "jsonld": "https://wpnews.pro/news/show-hn-i-forked-an-agent-stack-and-measured-myself-against-it-losses-included.jsonld"}}