I didn't fix the bug: contributing to a 20k-star ML repo by measuring it A developer building a Japanese decision model measured the multilingual checkpoint of the 20k-star open-source model laya and found its ordinal head effectively ignores content, always selecting the first-listed option: "not urgent" was chosen 0 times when listed first and 250 times when listed last across the same 300 emails. The maintainer documented the limitation in a release, requested a before/after run on a retrained checkpoint, and merged the developer's regression check, though the retrain itself has not yet happened. Last week I opened an issue on laya https://github.com/NandhaKishorM/laya , an open-source decision model with a little over 20k stars. The issue reported that one of its checkpoints almost never picks the first-listed option on ordinal questions. About 36 hours later, three things had happened. The maintainer had documented the limitation in a release. He had asked me to run the before/after on the retrained checkpoint. And he had merged a regression check I wrote. That was four days after my first measurement. Nothing is fixed. The fix is a retrain, and the retrain hasn't happened yet. What I contributed is narrower. I think it is also the kind of thing an outsider is best placed to contribute: a measurement nobody could wave away, and a tool that says whether the fix worked. The fastest way I've found to contribute to an ML repo you don't maintain: measure it, make the measurement impossible to explain away, and then hand the maintainer a tool. The timeline, in UTC: laya is a non-autoregressive "System 1" decision model. You give it some text and a set of typed questions: choice , score for ordinal levels, and noul for yes/no. It answers all of them in one forward pass, with probabilities and no text generation. It ships several checkpoints, including an English laya and a laya-multilingual . I'm building a Japanese model in the same space sokudan https://github.com/hiroki-abe-58/sokudan . Before writing any training code, I wanted a baseline: how well does the existing multilingual checkpoint handle Japanese? I built 300 label-conditioned Japanese business emails. The labels were fixed first. Then a local LLM wrote an email to match them, and any email containing the label words themselves was thrown out and regenerated. Each email got three questions: department 4-way choice , urgency 3-level score and whether the customer is hinting at cancelling bool . The results split by question type. choice worked: 0.747 accuracy against a 0.380 majority-class baseline. score lost to the majority baseline RPS 0.232 vs 0.197, lower is better . bool also lost to the majority baseline 0.543 vs 0.703 , with an AUROC of 0.523. That is barely better than ranking at random. One comparison I did not run is TypeSafe's Jev, the hosted API laya benchmarks itself against. Its customer agreement prohibits using the service or its outputs to develop similar products, and my project is a similar product. "Worse than majority" is a finding, not an explanation. The confusion matrix for score was more interesting. The lowest level, "not urgent", was never predicted in all 300 emails, even though 77 emails had it as the gold label. At least three explanations fit that. The ordinal head could be degenerate. The model could be reading position rather than content. Or this particular Japanese phrase could be the problem. To separate them, I re-ran the same 300 emails under five option schemas: original order, reversed, reworded low / medium / high , reworded and reversed, and four levels. In all five, the first-listed option was chosen 0 or 1 time out of 300. The cleanest pair was the original order against the reversed one. "Not urgent" was chosen 0 times when listed first and 250 times when listed last. The emails were the same, the words were the same, and only the slot had changed. Before reporting, I tried to list the replies a busy maintainer could reasonably give, and to answer each one with data before he had to ask. "Your harness is wrong." I re-ran the benchmark using only the two lines from the README, laya.load and agent.predict , with no wrapper of mine in between. Then I compared the results to my saved probabilities. The maximum difference was 0.0 after the probability floor I apply, and 2.2e-16 on the raw values. "It's a Japanese problem." I built 290 English emails with the same label weights, the same generator and the same rejection rules, and ran the same five conditions. The multilingual checkpoint went 0 for 290 in every one of them. English is one of its training languages, so "it can't read the input" doesn't explain this. First-listed option chosen, per condition original / reversed / reworded / reworded reversed / four levels : | checkpoint | Japanese n=300 | English n=290 | |---|---|---| | laya-multilingual | 0, 0, 1, 1, 0 | 0, 0, 0, 0, 0 | | laya English | 13, 56, 8, 1, 110 | 65, 74, 0, 5, 4 | The English checkpoint picks the first slot in most conditions, on the same data through the same harness. So the task isn't the cause, and neither is the harness. It comes down to one checkpoint. I opened 131 https://github.com/NandhaKishorM/laya/issues/131 with a title that said exactly that. A correction to myself: in the issue I summarised the English checkpoint as picking the first option "22–26% of the time". That holds for the two original orderings. It is 0 in one condition and under 2% in two others. The table above is the honest version. "Is it the position or the word?" Within any single fixed ordering, "the model rejects slot 1" and "the model rejects whichever word sits in slot 1" predict the same table. Someone on X asked whether the effect survives shuffling. Twenty-two minutes later I had condition F: the option order is shuffled per item, with a fixed seed. The results: So every label lost exactly the picks it had while it was in slot 1. Slots 2 and 3 split evenly, so this isn't a preference for the last slot either. One run was enough to separate position from label. What happened next was the best part, and it wasn't my work. AlKor13 https://github.com/AlKor13 , who is not a maintainer, read the raw marker logits straight off the forward pass, before temperature and softmax. On the identical code path, swapping only the checkpoint made the hole appear or disappear. Then he ran the control I wish I'd thought of: three identical options, so the inputs differ only by position. The English checkpoint came out essentially flat. The multilingual one came out sharply position-dependent. That settled it: the cause is in the weights, not the code. He went one step further. Score options are rendered with a level N: prefix, and dropping that prefix made the slot-0 suppression vanish from the raw logits. But he stated the limit of his own result. Without the prefix the input is off-distribution, so the recovery might just be the learned prior getting scrambled. Settling that would take labelled data. I had the labelled data, so I ran three renderings on both benchmarks: A is the shipped level N: rendering, C drops the prefix, and D uses word ordinals. My harness reproduced agent.predict to within 4.6e-5 under condition A. I compared the renderings with paired McNemar tests on identical items: | bench | change | score accuracy | items that flipped correctness | p | |---|---|---|---|---| | ja | A → C | 0.447 → 0.513 | 56.7% | 0.145 | | ja | A → D | 0.447 → 0.570 | 38.3% | 0.0007 | | en | A → C | 0.266 → 0.428 | 56.9% | 0.0003 | | en | A → D | 0.266 → 0.293 | 19.3% | 0.350 | Two of the four comparisons are significant, and they are different conditions in different languages. D wins in Japanese and C wins in English, and each is null in the other language. The claim this supports is "the shipped rendering isn't the best one for this checkpoint". It does not support "drop the prefix and it's fixed". I'll admit the accuracy column alone had me halfway to writing the second claim. The paired test is what stopped me, and I said so in the thread. The column that matters is the flip rate. Under C, 57% of items change correctness, while headline accuracy moves by 6.7 to 16.2 points. A cosmetic change to the option string nearly re-rolls the prediction. That is an instability, not a mitigation. As a control, removing the prefix makes the English checkpoint worse on the English set 0.583 → 0.500 . So level N: isn't harmful in general. The multilingual checkpoint has learned something wrong about that specific pattern. About fifteen hours after I opened the issue, the maintainer, NandhaKishorM https://github.com/NandhaKishorM , replied to both of us. He called the thread "a model of how to run a bug down" and made three decisions: choice options already get. The next morning, v0.3.7 shipped with a line in the README's limits section. It says laya-multilingual has a position bias on score questions, that English score questions should go to the English checkpoint, and that users should validate score outputs on their own data for other languages. The issue stays open until a retrained checkpoint lands. I offered to run the same A–F and A/C/D before/after on that checkpoint. He said he'd ping me when it's ready. That offer had a weak point: it depended on me. My benchmarks are CC BY 4.0, so anyone can rerun them. But the question the retrain has to answer doesn't need labels at all: did the slot prior go away? So I wrote PR 259 https://github.com/NandhaKishorM/laya/pull/259 : python research/eval/presentation checks.py --model