This release folds in two PRs: #2956 (the main MetaHarness work) and #2957 (version bump + a small test fix). This write-up walks through what changed, why it mattered, and — the part worth reading closely — three real bugs a pre-merge code review found and got fixed before they shipped.
MetaHarness is a separate, optional toolkit (@metaharness/darwin
, @metaharness/flywheel
, @metaharness/radio
) that ruflo can use for extra capabilities — things like scoring how "ready" a codebase is for AI agents, or self-optimizing its own retrieval settings over time. Critically, ruflo is designed to work completely fine without any of it installed — MetaHarness is an augmentation, never a requirement.
The bug: in package.json
, these packages were declared as optional peer dependencies. That sounds like it should mean "install this if you can, but don't fail if you can't" — which is what everyone assumed. In practice, npm has a quirk: an optional peer dependency is never automatically installed, by anyone, ever. It only gets installed if some other, unrelated package in your project happens to also list it as a real dependency. So on a completely normal, clean npm install
of ruflo, all three MetaHarness packages were silently absent — every time, for every user — even though the intent was "install these optionally."
Worse: the automated tool that was supposed to catch exactly this kind of drift (check-metaharness-pins.mjs
) only looked at two dependency categories (dependencies
and optionalDependencies
). Since these packages lived in a third category (peerDependencies
), the checker reported them as undeclared
— and undeclared
wasn't treated as a failure. So the bug passed every automated check, silently, for a while.
The fix:
- Moved all three packages into the category that npm actually auto-installs from: real
optionalDependencies
, with tilde version pins (~0.8.3
style — absorbs bugfixes, never jumps to a new minor version behind your back). - Rewrote the pin-checker to search all three dependency categories, and to treat "declared as a peer only" as a real failure, not a shrug.
- Added a new CI job that does a genuinely clean install (empty directory, no cache) and asserts every advertised feature of each MetaHarness package is actually there and working — so this specific failure mode can never silently reappear.
- Made
ruflo doctor
fail(not just warn) if a MetaHarness package is declared but doesn't actually resolve at runtime.
The combined packages are small (~2.3 MB, install in under a second) and have zero further dependencies of their own, so this fix doesn't meaningfully change install size or time for anyone — it just makes the "optional" part actually optional-but-present, instead of optional-and-silently-absent.
Ruflo has a self-optimizing "flywheel": it periodically tries small tweaks to its own retrieval settings, measures whether the tweak actually helped on held-back test data, and — if it did — promotes the tweak to become the new default. This is powerful, but it has a classic statistics trap baked in.
The trap, in plain terms: if you keep trying new candidate settings and only report the ones that "worked," you will eventually get a candidate that looks like it worked purely by chance — even if none of your ideas were actually any good. This is the same reason "we tried 20 different vitamins and one of them showed a statistically significant effect" headlines are usually nonsense: test enough things and pure luck will hand you a false positive. Real published measurements of this exact "keep trying, promote if it looks good" pattern put the false-promotion rate at 30–42% — way too high to trust blindly.
The fix (ADR-381) puts a hard mathematical ceiling on this: across the entire stream of candidates the flywheel will ever try, the probability that any single one of them gets promoted by pure luck is capped at 5%, total — not 5% per candidate, 5% across the whole ongoing history. It does this by:
- Giving each candidate in the sequence a shrinking "budget" of statistical confidence it needs to clear to get promoted — the first candidate needs the least evidence, the 100th candidate needs (proportionally) much more, because by then you've had 99 other chances to get lucky.
- Recording, permanently, which "slot" in that sequence each candidate's evidence was actually judged against — so nobody can quietly re-submit the same evidence hoping for an easier slot later ("index shopping").
- Making the whole thing strict by default: if a candidate's evidence can't be checked against this scheme (e.g., it's from before this upgrade), it's refused, not silently waved through with a weaker check.
There's also a formal "reset" mechanism: sometimes you genuinely want to start a fresh evaluation campaign (say, after rolling back to a different baseline). flywheel evidence-reset
lets you do that explicitly, but it:
archives the old campaign's spend into a permanent audit trail (nothing is ever silently deleted),expires every evidence result that was still "in flight" when the reset happened, so nobody can reopen an old, cheaper budget and slot new results into it,- requires a human-readable reason and explicit confirmation — it's a deliberate act, not something that happens by accident.
Before this went out, a review pass specifically targeted the new statistical-guarantee code, on the theory that "we just built a system that promises a mathematical ceiling on false promotions — let's make sure nothing quietly breaks that promise." It found three real bugs. All three share a theme: the code was correct for one thing happening at a time, but broke down if two things happened close together, or out of the expected order.
The reset mechanism (above) is supposed to guarantee: "a fresh campaign can only be won using evidence collected after the reset — you can't reopen the cheap new budget and slide old evidence into it." But the code only checked when a result was recorded into the system (registration time), not when the underlying evidence was actually measured (evaluation time).
Concrete failure: imagine an evaluation result was computed just before a reset happened, but — for whatever reason (a queued job, a cached result being replayed later, a future code path that separates "measure" from "record") — it doesn't get registered into the system until after the reset. Under the old code, that old evidence would be treated as brand-new, get slotted into the first, cheapest position of the fresh budget, and could ride through on old evidence that was never supposed to be eligible for the new campaign. That's exactly the "index shopping" loophole the reset mechanism exists to prevent — just via a side door.
The fix: the system now remembers the exact moment each reset happened, and compares that against the evidence's own recorded measurement time (not its registration time) before allowing it into the new campaign. Evidence measured before the boundary is refused, no matter when it happens to get registered.
Every candidate the flywheel evaluates gets assigned a position in the sequence (1st, 2nd, 3rd, ...) — that position determines how much statistical evidence it needs, and each position is only supposed to ever be used once. Assigning that position was done by counting "how many evaluations exist so far, plus one" — a plain read of a file, with no protection against two evaluations doing that read at nearly the same instant.
Concrete failure: imagine the automated background process is in the middle of one evaluation cycle, and someone also manually kicks off a run at the same time (or a retry overlaps a slow run). Both processes read "9 evaluations exist so far" before either one writes its own result, so both compute "I'm evaluation #10." Now two different candidates are both judged against slot #10's statistical budget — which means that budget slot has effectively been spent twice, silently pushing the true false-promotion rate above the promised 5% ceiling, with no error or warning anywhere.
The fix: the "count existing evaluations, then add mine" step now happens inside a lock — the same file-locking mechanism the rest of the system already used elsewhere, just extended to cover this specific counting step too. Two overlapping runs now safely queue instead of racing.
Before a candidate is submitted for promotion, the system shows the operator a quick preview: "based on what I can see right now, this looks promotable." That preview was computed before the (potentially slow) work of actually scoring the candidate happened — so by the time the operator acted on it and actually tried to promote, the real, authoritative check (which happens later, under a lock, at promotion time) could have moved on, because someone else's candidate got promoted in the meantime and took the "slot" this one was counting on.
Concrete failure: an operator runs an evaluation, gets told "this is ready to promote," and — before they act on it — a second, unrelated evaluation gets promoted first, consuming the next available slot. When the operator then tries to promote their (still perfectly good) candidate, it's now judged against a harder slot than the preview promised, and gets refused — which looks like a bug or a broken promise to the operator, even though nothing was actually wrong with their candidate.
The fix: the preview is now computed as late as possible — right after all the real evaluation work is done, instead of before it — which shrinks the window where this mismatch can happen dramatically (it used to include the entire scoring computation; now it's essentially just the moment before the result is returned). It can't be made perfectly race-proof without holding a lock across the operator's entire decision-making process, which the design deliberately avoids (the system is built to never block an evaluation just because a promotion might be in flight elsewhere). So the fix also makes explicit, in the code and its documentation, that this preview is advisory — "this looked ready as of a moment ago," not an ironclad promise — so nobody downstream mistakes it for a guarantee it was never meant to make.
The whole point of PR #2956's headline feature is a specific mathematical promise: the probability of the flywheel ever promoting a bad candidate, across its entire operating lifetime, stays under 5%. Bugs 1 and 2 above are exactly the kind of thing that quietly breaks that promise — not through some obvious crash or wrong number, but through a timing edge case that only shows up when two things happen close together, which is precisely the scenario the guarantee is supposed to hold up under. Shipping a feature whose entire purpose is "you can trust this number" while a review had already found ways to make that number wrong wouldn't have been acceptable, so those two were fixed and covered with dedicated regression tests before anything went out the door. Bug 3 is more of an operator-experience issue than a statistical-soundness one, but it was fixed alongside the others since it touches the same code paths.
All three fixes, plus regression tests proving the concurrency race and the epoch-boundary refusal actually work, were pushed to the PR branch, re-validated against the full CI suite (122 checks, all green), and only then merged.
@claude-flow/cli
,claude-flow
, andruflo
are all now at3.35.0 on npm,latest
/alpha
/v3alpha
tags aligned.- No breaking changes — this is a minor version bump (new optional dependency, new doctor checks, new CLI/MCP surface for the evidence-reset command).
- Verified live:
`npx ruflo@latest --version`
reports3.35.0
from a fresh install.
Full technical PRs: [#2956](https://github.com/ruvnet/ruflo/pull/2956) · [#2957](https://github.com/ruvnet/ruflo/pull/2957)
This section covers a same-day follow-up release. The upstream MetaHarness project (a sibling toolkit ruflo can optionally use) shipped a new capability in PR #176, and this release brings ruflo's side of that integration up to date.
Imagine an AI agent takes 30 actions to complete a task, and at the end you're told only "it succeeded" or "it failed." That single pass/fail verdict says nothing about which of those 30 actions actually mattered — some were probably decisive, most were probably neutral, and a few might even have hurt. If you want to learn from that trajectory (reward the good decisions, deprioritize the bad ones), a single terminal score can't tell you where to look.
turn-credit
solves this with an idea borrowed from a recent research paper (AgentOPSD): after every action, it tracks how the agent's "belief that this will succeed" shifts up or down. A big upward jump after a particular action is a strong signal that action was pivotal; the trajectory ends up with a per-action credit score instead of one flat number for the whole run. Crucially, it's built so that this per-action re-weighting can only ever adjust emphasis, never flip the final verdict — if the run failed overall, no amount of "this action looked promising" can turn it into a success in the record. That guarantee is mathematically enforced (verified in code review by checking the actual bounds on the math, not just trusting the docstring), not just asserted.
This is purely a post-processing / analysis tool — it doesn't touch model weights and doesn't change how the agent behaves live. It produces signals that other systems (routing decisions, retry policy, which memories to reinforce) can optionally use later.
When turn-credit
shipped, the upstream project also bumped two packages ruflo already depends on: @metaharness/darwin
(adds an opt-in way to feed it better-quality trace data) and @metaharness/router
(adds a new tool for auditing whether the router's cost-saving predictions are actually well-calibrated against reality). Neither of ruflo's version pins had been updated to match:
@metaharness/darwin
was pinned~0.8.3
, which — because of how "tilde" version ranges work — silently stopped covering the new0.9.0
release. Anyone installing ruflo would have kept getting theolddarwin version even though a newer one existed.@metaharness/router
's pin had the same problem one version-range type up (^0.3.2
didn't reach0.4.0
).
Both are now fixed to track the versions that are actually published.
The previous release (v3.35.0) had raised a CI budget check from 8 to exactly 10 — matching the count of dependencies at that moment, with zero room to spare. That's a subtle trap: the very next time anyone adds one more optional dependency for a completely unrelated reason, that specific CI check fails — not because anything is actually wrong, but because the budget was set with no headroom. This release's guard bump (10 → 13) deliberately leaves real slack instead of repeating that mistake — a small but concrete example of "fix it so it doesn't just work today, fix it so it doesn't quietly become tomorrow's problem."
While preparing this release, editing the dependency file directly (rather than through the package manager) broke roughly twenty unrelated-looking CI checks all at once — Windows tests, plugin smoke tests, type checks, and more. The real cause was one thing: this project uses two separate package managers for two separate parts of the repo, and editing one dependency file by hand left its companion lockfile out of sync. CI enforces that the lockfile and the dependency file agree exactly, so every job that needed to install anything failed at the very first step — which is why the failure list looked so broad and scary even though the actual bug was narrow and simple. Regenerating the lockfile properly fixed all of it in one shot. The lesson: when a change causes a wide, seemingly-unrelated wave of CI failures, look for one shared setup step breaking for everyone, rather than assuming many separate things broke at once.
@claude-flow/cli
,claude-flow
, andruflo
are now at3.36.0 on npm,latest
/alpha
/v3alpha
tags aligned.- No breaking changes — backward-compatible minor release.
- Verified live:
`npx ruflo@latest --version`
reports3.36.0
from a fresh install.
Full technical PR: [ruflo#2958](https://github.com/ruvnet/ruflo/pull/2958) · upstream: [metaharness#176](https://github.com/ruvnet/metaharness/pull/176)