# I didn't fix the bug: contributing to a 20k-star ML repo by measuring it

> Source: <https://dev.to/genelab_999/i-didnt-fix-the-bug-contributing-to-a-20k-star-ml-repo-by-measuring-it-1lg7>
> Published: 2026-09-24 04:41:45+00:00

Last week I opened an issue on [laya](https://github.com/NandhaKishorM/laya), an open-source decision model with a little over 20k stars. The issue reported that one of its checkpoints almost never picks the first-listed option on ordinal questions. About 36 hours later, three things had happened. The maintainer had documented the limitation in a release. He had asked me to run the before/after on the retrained checkpoint. And he had merged a regression check I wrote. That was four days after my first measurement.

Nothing is fixed. The fix is a retrain, and the retrain hasn't happened yet. What I contributed is narrower. I think it is also the kind of thing an outsider is best placed to contribute: a measurement nobody could wave away, and a tool that says whether the fix worked.

The fastest way I've found to contribute to an ML repo you don't maintain: measure it, make the measurement impossible to explain away, and then hand the maintainer a tool.

The timeline, in UTC:

laya is a non-autoregressive "System 1" decision model. You give it some text and a set of typed questions: `choice`, `score` for ordinal levels, and `noul` for yes/no. It answers all of them in one forward pass, with probabilities and no text generation. It ships several checkpoints, including an English `laya` and a `laya-multilingual`.

I'm building a Japanese model in the same space ([sokudan](https://github.com/hiroki-abe-58/sokudan)). Before writing any training code, I wanted a baseline: how well does the existing multilingual checkpoint handle Japanese?

I built 300 label-conditioned Japanese business emails. The labels were fixed first. Then a local LLM wrote an email to match them, and any email containing the label words themselves was thrown out and regenerated. Each email got three questions: department (4-way `choice`), urgency (3-level `score`) and whether the customer is hinting at cancelling (` bool`).

The results split by question type. `choice` worked: 0.747 accuracy against a 0.380 majority-class baseline. `score` lost to the majority baseline (RPS 0.232 vs 0.197, lower is better). `bool` also lost to the majority baseline (0.543 vs 0.703), with an AUROC of 0.523. That is barely better than ranking at random.

One comparison I did not run is TypeSafe's Jev, the hosted API laya benchmarks itself against. Its customer agreement prohibits using the service or its outputs to develop similar products, and my project is a similar product.

"Worse than majority" is a finding, not an explanation. The confusion matrix for `score` was more interesting. The lowest level, "not urgent", was never predicted in all 300 emails, even though 77 emails had it as the gold label.

At least three explanations fit that. The ordinal head could be degenerate. The model could be reading position rather than content. Or this particular Japanese phrase could be the problem.

To separate them, I re-ran the same 300 emails under five option schemas: original order, reversed, reworded (low / medium / high), reworded and reversed, and four levels. In all five, the first-listed option was chosen 0 or 1 time out of 300.

The cleanest pair was the original order against the reversed one. "Not urgent" was chosen 0 times when listed first and 250 times when listed last. The emails were the same, the words were the same, and only the slot had changed.

Before reporting, I tried to list the replies a busy maintainer could reasonably give, and to answer each one with data before he had to ask.

**"Your harness is wrong."** I re-ran the benchmark using only the two lines from the README, `laya.load()` and `agent.predict()`, with no wrapper of mine in between. Then I compared the results to my saved probabilities. The maximum difference was 0.0 after the probability floor I apply, and 2.2e-16 on the raw values.

**"It's a Japanese problem."** I built 290 English emails with the same label weights, the same generator and the same rejection rules, and ran the same five conditions. The multilingual checkpoint went 0 for 290 in every one of them. English is one of its training languages, so "it can't read the input" doesn't explain this.

First-listed option chosen, per condition (original / reversed / reworded / reworded reversed / four levels):

| checkpoint | Japanese (n=300) | English (n=290) | 
|---|---|---|
| `laya-multilingual` | 0, 0, 1, 1, 0 | 0, 0, 0, 0, 0 | 
| `laya` (English) | 13, 56, 8, 1, 110 | 65, 74, 0, 5, 4 | 

The English checkpoint picks the first slot in most conditions, on the same data through the same harness. So the task isn't the cause, and neither is the harness. It comes down to one checkpoint. I opened [#131](https://github.com/NandhaKishorM/laya/issues/131) with a title that said exactly that.

(A correction to myself: in the issue I summarised the English checkpoint as picking the first option "22–26% of the time". That holds for the two original orderings. It is 0 in one condition and under 2% in two others. The table above is the honest version.)

**"Is it the position or the word?"** Within any single fixed ordering, "the model rejects slot 1" and "the model rejects whichever word sits in slot 1" predict the same table. Someone on X asked whether the effect survives shuffling. Twenty-two minutes later I had condition F: the option order is shuffled per item, with a fixed seed. The results:

So every label lost exactly the picks it had while it was in slot 1. Slots 2 and 3 split evenly, so this isn't a preference for the last slot either. One run was enough to separate position from label.

What happened next was the best part, and it wasn't my work. [AlKor13](https://github.com/AlKor13), who is not a maintainer, read the raw marker logits straight off the forward pass, before temperature and softmax.

On the identical code path, swapping only the checkpoint made the hole appear or disappear. Then he ran the control I wish I'd thought of: three identical options, so the inputs differ only by position. The English checkpoint came out essentially flat. The multilingual one came out sharply position-dependent. That settled it: the cause is in the weights, not the code.

He went one step further. Score options are rendered with a `level N:` prefix, and dropping that prefix made the slot-0 suppression vanish from the raw logits. But he stated the limit of his own result. Without the prefix the input is off-distribution, so the recovery might just be the learned prior getting scrambled. Settling that would take labelled data.

I had the labelled data, so I ran three renderings on both benchmarks: A is the shipped `level N:` rendering, C drops the prefix, and D uses word ordinals. My harness reproduced `agent.predict()` to within 4.6e-5 under condition A. I compared the renderings with paired McNemar tests on identical items:

| bench | change | score accuracy | items that flipped correctness | p | 
|---|---|---|---|---|
| ja | A → C | 0.447 → 0.513 | 56.7% | 0.145 | 
| ja | A → D | 0.447 → 0.570 | 38.3% | 0.0007 | 
| en | A → C | 0.266 → 0.428 | 56.9% | 0.0003 | 
| en | A → D | 0.266 → 0.293 | 19.3% | 0.350 | 

Two of the four comparisons are significant, and they are different conditions in different languages. D wins in Japanese and C wins in English, and each is null in the other language. The claim this supports is "the shipped rendering isn't the best one for this checkpoint". It does not support "drop the prefix and it's fixed".

I'll admit the accuracy column alone had me halfway to writing the second claim. The paired test is what stopped me, and I said so in the thread.

The column that matters is the flip rate. Under C, 57% of items change correctness, while headline accuracy moves by 6.7 to 16.2 points. A cosmetic change to the option string nearly re-rolls the prediction. That is an instability, not a mitigation. As a control, removing the prefix makes the English checkpoint *worse* on the English set (0.583 → 0.500). So `level N:` isn't harmful in general. The multilingual checkpoint has learned something wrong about that specific pattern.

About fifteen hours after I opened the issue, the maintainer, [NandhaKishorM](https://github.com/NandhaKishorM), replied to both of us. He called the thread "a model of how to run a bug down" and made three decisions:

`choice` options already get.
The next morning, v0.3.7 shipped with a line in the README's limits section. It says `laya-multilingual` has a position bias on `score` questions, that English score questions should go to the English checkpoint, and that users should validate score outputs on their own data for other languages. The issue stays open until a retrained checkpoint lands.

I offered to run the same A–F and A/C/D before/after on that checkpoint. He said he'd ping me when it's ready.

That offer had a weak point: it depended on me. My benchmarks are CC BY 4.0, so anyone can rerun them. But the question the retrain has to answer doesn't need labels at all: did the slot prior go away? So I wrote PR [#259](https://github.com/NandhaKishorM/laya/pull/259):

```
python research/eval/presentation_checks.py --model <retrained checkpoint> [--subfolder multilingual]
```

It uses ten short English support messages that are fixed in the file, with no dataset and no labels. It runs two checks:

`score_slot0_identical` is AlKor13's identical-option control turned into a gate. It measures slot 0's raw logit minus the slot mean, over K = 3, 4 and 5 identical levels, and requires the result to be at least −0.20.`score_first_slot_permuted` puts the three real levels in all six orders for each message, so every level sits in every slot exactly twice. A model whose answer doesn't depend on order picks the first slot in exactly 1/3 of decisions. The gate is at least 0.15.
Results on the checkpoints that ship today:

| checkpoint | slot-0 logit (gate ≥ −0.20) | first-slot rate (gate ≥ 0.15) | verdict | 
|---|---|---|---|
| `laya` (English) | +0.664 | 0.217 | PASS | 
| `laya-multilingual` | −0.492 | 0.017 | FAIL | 

**I fixed the threshold conditions before the real 10-message run, and I didn't move them afterwards.** There were two conditions. First, in every leave-one-out subset, multilingual must fail and English must pass. Second, each checkpoint's worst leave-one-out value must clear its gate by at least 0.10 logit (slot 0) or 0.05 (first-slot rate). The tightest margin turned out to be the English first-slot rate, at 0.054.

**The gate is one-sided on purpose.** With identical options, the English checkpoint isn't flat either. It leans toward early slots, and the lean grows with K. A two-sided "no position effect" gate would fail the checkpoint everyone agrees is fine. So the gate asks only the narrower question #131 is about: is slot 0 suppressed?

**The harness checks itself before it gives a verdict.** It first compares its logit path against `Agent.system_one` on every message. On the shipped checkpoints, the maximum |Δp| was 4.98e-5, which is just the API's 4-decimal rounding. If the difference exceeds 1e-3, the script exits with code 2 and reports no verdict. Exit 0 means pass, 1 means a check failed, and 2 means the harness disagrees with the package. There are 69 offline tests using scripted logits. For example, a scripted slot-0 hole fails both checks, and an order-invariant model scores exactly 1/3.

The PR also lists its own limits. Passing is not accuracy. The check covers English only, `score` only and ten short messages. And the thresholds were set on CPU with fp32.

I spent more time reading the repo than writing the script. Tests there are plain scripts, not pytest. CI doesn't load model checkpoints. Research code lives under `research/`. A new test has to be registered in both `ci.yml` and `release.yml`, and an open PR, [#184](https://github.com/NandhaKishorM/laya/pull/184), hadn't yet settled how research tests get wired in. So I didn't wire mine in. I said so in the PR and offered to do it once #184 lands. Nothing under `laya/` changed, and I added no new dependencies.

The same afternoon, another contributor, Rukafuu, had proposed a general metamorphic robustness framework in [#244](https://github.com/NandhaKishorM/laya/issues/244), and the maintainer had invited them to start with option permutation. That was their work to do. I kept my check specific to #131 and noted that it could move into their framework once it exists. Their PR, #269, has since been merged.

I used Claude Code throughout: for the harness, the offline tests and drafting the PR text. The commit carries a `Co-Authored-By` line. The decisions about what to measure, what to claim, what to take back and where to stop were mine, and so are any mistakes.

`presentation_checks.py` gives the first answer, and I'll run the labelled A–F and A/C/D before/after on top of it.
What I posted the day it merged:

// Detect dark theme var iframe = document.getElementById('tweet-2102969840386687112-803'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2102969840386687112&theme=dark" }