A Ruby type checker has found 15 bugs in Discourse, Rails, Zammad and Ruby-lsp

Itaruby, a Ruby type checker built in Rust, has found 15 bugs across Discourse, Rails, Zammad, and ruby-lsp, with 7 accepted, according to its developer. The checker, which is not yet released, uses inference and has scanned 20,615 files, producing 13,208 raw diagnostics, with submissions verified by execution. It runs as a language server and uses the salsa incremental recomputation engine.

itaruby, not released yet A Ruby type checker that has already found 15 bugs in Discourse, Rails, Zammad and ruby-lsp itaruby is a Ruby type checker I am building. Instead of announcing what it promises to do, this page shows what it has already done: 14 submissions, 3 in discourse/discourse, 5 in rails/rails, 5 in zammad/zammad and 1 in Shopify/ruby-lsp, each bug verified by running it against the real file in the project before it was submitted. Discourse is a forum, Rails is a framework, Zammad is a helpdesk and ruby-lsp is a language server: different domains, no special-casing for any of them. Of the total, 7 have been accepted. What it is From the initial commit's own description: "inference-first Ruby type checker in Rust". It infers instead of requiring you to declare. 4 crates: itaruby the CLI , itaruby syntax, itaruby semantic and itaruby server the language server . 98 .rs files, 15,075 lines of production Rust and 13,793 of test code, close to a test line for every line of production code. Incremental recomputation engine: salsa, the same one rust-analyzer uses. Beyond the command-line check, it runs as a language server: definition and hover at file:line:column . The initial commit is from August 25, 2026, 18:18 UTC-3. The most recent one measured here is from August 26, 2026, 08:30 UTC-3: 26 commits over two days 8 on the first, 18 on the second , with no rewritten history. By the time the 14 submissions on this page went out, the checker was about 14 hours old. It advances in waves, each tested against a mutation harness that deliberately breaks the code to prove the tests would catch it: wave 7 caught 16 mutants. The funnel 13208 raw diagnostics, 14 submissions, across four triages from different moments, not one snapshot: rails, discourse and zammad in a round before wave 6; ruby-lsp in wave 6, during the Sorbet comparison. Most of the 13208 is noise: the same unresolved constant, flagged again on every reference in the code. The funnel from one number to the other is manual. I read every diagnostic, group the ones that share a cause, prove by execution whatever is left, and only then does it become a submission. None of the 13208 became a submission on its own. What this number deliberately leaves out: twelve pull requests I opened on software-factory, a project that was part of this same run and the thing that got me unblocked enough to open the rest. They are excluded because the checker did not find them. Adding the two together would suggest itaruby surfaced twenty-six things, and "every submission proved by execution" would stop being true of half of them. I would rather publish the smaller number that is true all the way through. | Repository | Files | Diagnostics | Round | Submissions | | rails/rails | 3457 | 2364 | before wave 6 | 5 | | discourse/discourse | 11201 | 7959 | before wave 6 | 3 | | zammad/zammad | 5586 | 2834 | before wave 6 | 5 | | Shopify/ruby-lsp | 371 | 51 | wave 6 7d3a781 | 1 | | Total | 20615 | 13208 | | 14 | The rails, discourse and zammad counts are from the moment of triage, a round before wave 6 whose binary sha was never recorded. The ruby-lsp count is from wave 6, commit 7d3a781 . None of these numbers change afterward: today's raw count, which is lower, is in the "Wave 8" section further down, sha 5c5c8c3 . File counts were reconfirmed stable across rounds; scan time is not in this table because it would pair one round's time with another round's count, the same mistake this page calls out elsewhere. Time belongs in the "Throughput" section, which uses time and count from the same round. Throughput 11201 Discourse files scanned in 8,02s. No configuration, no annotation, no setup step: the requirement to run is zero. The submissions FIX: two crashes in the Lithium import script Two crashes in the Lithium import script. A file.unlind that does not exist on Tempfile , swallowed by the surrounding rescue , leaking a temp file per imported avatar. And a link post uploads called on CookedPostProcessor , which has no such method: that one is not swallowed and takes down the entire attachment post-processing step. script/import scripts/lithium.rb FIX: forward force as a keyword argument in the TopicAssigner shim The deprecated shim received force: as a kwarg and forwarded it positionally to a method that only accepts a positional argument. Any call blew up, including the simplest one, without force . Ships with the regression spec the class was missing. plugins/discourse-assign/lib/topic assigner.rb, plugins/discourse-assign/spec/lib/topic assigner spec.rb FIX: pass all five arguments to replace internal link in Discuz import The private-message-link branch called replace internal link with 2 arguments, and the method requires 5. Every post whose body matched the regexp would crash the import with an ArgumentError. script/import scripts/discuz x.rb Remove unreachable default for DerivedSecretKeyProvider derive key from The using: key generator default points to a method that does not exist on the instance, so falling into it raises NameError. Unreachable: the method is private and the only caller passes the argument explicitly. activerecord/lib/active record/encryption/derived secret key provider.rb Remove unused Active Record internals Two dead code paths. An else branch that falls into expr , a method that does not exist anywhere in Arel, and a columns hash that calls the connection without the argument it requires. activerecord/lib/arel/nodes/homogeneous in.rb, activerecord/test/cases/arel/support/fake record.rb Remove unused request helper from RouteSetTest A helper that memoizes TestRequest.new with no arguments, while the constructor requires three. Any call would raise ArgumentError. Nobody calls it. actionpack/test/controller/routing test.rb 58566 https://github.com/rails/rails/pull/58566 applied by commit +13 / -1 Fix QueryCacheRegistry clear calling an undefined method The method called @map.synchronize . The class's mutex is @mutex ; @map is a Hash and does not respond to synchronize, so clearing the cache would blow up. activerecord/lib/active record/connection adapters/abstract/query cache.rb Finding applied in commit 22a02ee12e https://github.com/rails/rails/commit/22a02ee12e , not through this PR's branch. Fix RouteSetTest::SimpleApp call referencing an undefined local variable call returned response , a local variable that does not exist. The value is in @response . Any request routed to this test app raised NameError. actionpack/test/dispatch/routing/route set test.rb Fix: Wrong number of arguments in create knowledge base categories call A call passed 4 positional arguments to a method that takes a hash, raising ArgumentError: wrong number of arguments given 4, expected 1 . The right shape is documented in the file's own docblock. lib/fill db.rb Fix: Compare limit against @size, not undefined size The comparison used size , which does not exist in scope; only @size exists, used on four other lines. Raises NameError the first time anyone chains .limit before .as batches . config/initializers/active record as batches.rb, spec/lib/core ext/relation/as batches spec.rb Remove unused OnlineNotificationPolicy relation The method called a name that does not exist and nothing reaches it. Proven from pundit's own source, whose only dynamic dispatch is the action name plus a question mark. app/policies/online notification policy.rb Fix: Auth::TwoFactor initiate authentication calls an undefined method The method called method object instead of authentication method object , which exists and is used correctly on two other lines in the same class. lib/auth/two factor.rb, spec/lib/auth/two factor spec.rb Fix: Cleanup action job name misses job prefix on payload args One line forgot the job. receiver, so the bare identifier raises NameError ; the neighboring lines in the same method already used the right receiver. lib/background services/service/base delayed jobs/cleanup action.rb, spec/lib/background services/service/base delayed jobs/cleanup action spec.rb Fix NameError raised by RubyIndexer's abstract method guard A raise AbstractMethodInvokedError inside RubyIndexer::Entry::Member points to a constant that only exists as RubyLsp::AbstractMethodInvokedError , a sibling module, not an ancestor: the guard raises NameError instead of the intended error. It is the only one of the 17 sites of this raise in the repository outside lib/ruby lsp/ ; the other 16 resolve correctly. A latent defensive guard, not reachable today: both concrete subclasses of Member already implement signatures . lib/ruby indexer/lib/ruby indexer/entry.rb The gates Once the PR is open, what decides whether it moves forward is not the patch. Three gates, measured, not about the tool. Shopify's CLA wants a human signature plus a comment that matches the workflow's regex. My first comment, "CLA signed.", passed the workflow's if , the job concluded success , and the log said Comment does not match CLA pattern : a green run does not mean the gate is cleared. What worked was "I have signed the CLA." action required is first-contribution workflow approval, per repository: it blocks zammad 5 PRs and ruby-lsp 1 . Rails and Discourse do not have this gate. On GitLab, quota and membership block even the push : that is why 6 ready MRs remain unopened. None of the three gates have anything to do with patch quality, and none of them show up before the PR is open. How each finding was verified A checker's finding is a hypothesis, not a bug. Every item on this list went through two-sided proof against the project's real file: run and fail before the fix, run and pass after. Reproducing it in a separate file does not count, because that tests my understanding of the code, not the project's behavior. For dead-code removals the bar is different: prove that nothing reaches that code. That means scanning the whole repository, including dynamic dispatch send , public send , respond to? , method missing , delegate and the class's subclasses, not just direct callers. That last part almost slipped past me on 58569 https://github.com/rails/rails/pull/58569 in Rails: the class has two subclasses, and if either one had called the method the story would have flipped from cleanup to real bug. The shape of the PRs did not come from personal taste either. It came from measuring what the project already accepts. I first counted Rails's last hundred merges, found no dead-code removal and concluded there was no precedent. A hundred Rails merges cover two weeks: searching the full history turned up a twelve-PR removal campaign merged a month earlier. A short sample does not measure a rare category. Acceptance is not always the same thing either. At Zammad, real development happens on an internal GitLab; GitHub is a mirror, and native merging there has been stalled since 2021. When a maintainer accepts a fix, they apply the patch as a manual commit and close the PR citing the hash. That is why the list distinguishes merged from applied by commit : two ways of saying yes, without hiding which one it was. Proof, not adjective Rails 58566 https://github.com/rails/rails/pull/58566 is the most complete case in this section: closed without a merge, with a single comment, from byroot Jean Boussier, Rails core : "If it's never called, let's just remove the method entirely." That was not a rejection of the finding, it was a rejection of the form: I sent a fix for a method that was dead, when removal was the right call. I had classified the method as reachable without proving it. The proof, done afterward: @thread query caches appears in exactly two places in the repository, the instantiation and one call to compute if absent ; .clear is never called. And the method's body called synchronize on an object that does not have that method, so it would raise an error if anyone called it: dead and broken, and I sent a fix instead of a removal. The outcome is the applied by commit state: byroot removed the method himself in commit 22a02ee12e https://github.com/rails/rails/commit/22a02ee12e , literal message: Get rid of QueryCacheRegistry clear The method is dead code. With a co-author credit for the finding. It went in, just not through my branch. On Rails 58568 https://github.com/rails/rails/pull/58568 , the method itaruby flagged had zero test coverage in the suite. The first proof that the code was dead came from a grep 'def expr' against the repository, which would never find an attr accessor :expr . The right instrument was following the inheritance chain, not grepping for the method name. The mistake was caught and fixed before the PR shipped, and the missing test went in with it. Discourse 42898 https://github.com/discourse/discourse/pull/42898 shows the same kind of failure with opposite consequences, in the same Lithium importer file: Two crashes in the Lithium import script. A file.unlind that does not exist on Tempfile , swallowed by the surrounding rescue , leaking a temp file per imported avatar. And a link post uploads called on CookedPostProcessor , which has no such method: that one is not swallowed and takes down the entire attachment post-processing step. Three Zammad findings 6329 https://github.com/zammad/zammad/pull/6329 , 6331 https://github.com/zammad/zammad/pull/6331 , 6332 https://github.com/zammad/zammad/pull/6332 shipped with a regression spec the suite did not have. In all three, the production fix is one line; the rest of the diff, thirteen to fourteen lines, is test. Where Sorbet wins What is proven: itaruby does not require anything, and it reads inline RBS if you have it. No configuration, no adoption step. The 14 bugs on this page came out of 3 repositories with zero sorbet/ , zero .rbi and zero extend T::Sig : 13 submissions, 7 accepted. The remaining repository in the corpus, ruby-lsp, runs Sorbet and has its own bug, in the next section. Scope note: from a Sorbet sig block, itaruby picks up the return type; it does not pick up the parameter type. The sig already speaks return, it still has to speak parameter. This matters less than it looks, because the corpus barely uses sig at all. I tested it: a class with extend T::Sig and sig { params n: Integer .returns String } , called passing a String where the signature wants an Integer parameter, produces no diagnostic at all: that is exactly the side that is still missing. But rails, discourse and zammad have zero sig . ruby-lsp, the most typed repository in the corpus, has zero real production sig : the leftover sig text in the repository is documentation comments and spec fixtures, not annotation. tapioca has one, in lib/ . There is a sharper reason behind the zero above, and it is Sorbet's own documented behaviour rather than an opinion about it. A file with no sigil defaults to typed: false , and at that level Sorbet reports only syntax, constant resolution and sig correctness. Its manual is explicit that the level "only prevents that method's body from being typechecked". So on a repository with no annotations, Sorbet is not failing to find these bugs: it is never asked to look. That is the whole gap itaruby aims at, and it is also why the comparison is about reach, not about speed. sorbet.org/docs/static https://sorbet.org/docs/static What changes the read: itaruby reads inline RBS. It is the annotation the tapioca team itself started using in ruby-lsp's production code, instead of sig : a one-line comment above the method, in the : Type - Type format. I tested the same method in the three annotation styles: python no annotation def format n n.to s end format "7" itaruby: silence correct, nothing to go on with inline RBS : Integer - String def format n n.to s end format "7" itaruby: p1.rb:7:18: error E0103 : argument 1 of 'format' expects Integer, got String with Sorbet sig extend T::Sig sig { params n: Integer .returns String } def format n n.to s end format "7" itaruby: silence The cost contrast is the strongest data point in this section. For Sorbet to see the dependencies, ruby-lsp and tapioca keep 754,669 lines of vendored .rbi 246,830 + 507,839 , plus a runtime gem, plus a sigil per file, plus a generation step. Inline RBS asks for none of that: it is a comment above the method, zero vendored files, zero runtime gem, zero generation. RBS for core/stdlib, in a separate file, not inline: not consumed. RBS for a gem via rbs collection , also in a separate file: not consumed. RBI is not missing coverage, it is a lookup problem: a declaration inside .rbi written as a nested block indexes correctly, the full path exists in the index; the query is what fails, looking up the qualified path in a map where that name was registered unqualified, and giving up before applying the refilter that exists exactly for this. Only the compact module X::Y form avoids the problem. It is the same bug that explains why the alias fix cured ruby-lsp and not tapioca, in the "The cycle" section further down. This table changed since this page's original publication: ruby-lsp went from worst to best in the corpus after a fix in itaruby itself, not in the scanned code. The mechanism and the verified prediction are in "The cycle" section, right after "Head-to-head with Sorbet". | Repository | Diagnostics / 1,000 files | | ruby-lsp | 137 | | tapioca | 275 | | zammad/zammad | 505 | | rails/rails | 670 | | discourse/discourse | 704 | What holds up, before and after the fix: neither "worst" nor "best" measured typing. They measured how much repository-specific parsing bug itaruby still had. The contract that survives is not "works better with more types": it is no requirement at all, and it reads inline RBS when the code already has it. Head-to-head with Sorbet The hardest test I can run: pit itaruby against Sorbet on the only two repositories in the exploratory corpus that have genuinely adopted Sorbet. Global srb tc , no bundle install , on both: | Tool | Repository | Files | Time | Diagnostics | | srb tc | ruby-lsp | 235 | 0,172s | 0 | | ita check | ruby-lsp | 371 | 0,496s | 51 | | srb tc | tapioca | 370 | 0,242s | 0 | | ita check | tapioca | 218 | 0,664s | 60 | All four time measurements are from the same session, same machine, median of three runs each. Sorbet is 2,9x faster on ruby-lsp and 2,7x on tapioca. Wall-clock time is not comparable across different sessions on this page: in an earlier remeasurement, ita check on ruby-lsp took 445ms; in this one, 496ms, with no binary change between the two. Session-to-session variance of 10% to 20% on this machine, which rules out any reading of "the fix cost time" without a before/after measurement taken in the same round. File counts differ because srb applies the project's sorbet/config --ignore list fixtures, vendor and counts .rbi files alongside them; itaruby counts raw .rb files and has no equivalent ignore list. Sorbet is green on both: literal output No errors Great job. . itaruby, on the same code, produces 111 raw diagnostics. I did not find 111 bugs. I classified both full sets, item by item, not a sample: | Repository | Diagnostics | Real bug | False positive | Genuine noise | | ruby-lsp | 51 | 1 | 16 31% | 34 67% | | tapioca | 60 | 0 | 34 57% | 26 43% | srb tc remains at zero errors on both. One number improved and the other got worse, and both deserve equal billing: in the earlier sample of 10 for ruby-lsp above, 7 of 10 were false positives 70% ; in the exhaustive classification of 51 , that fell to 31%. tapioca went the other way: it became the repository with the highest false-positive share in the corpus, 57%. It was not a terminal state: the cause had a name, an address in the code, and the fix went in. The result is in the "Wave 8" section further down: tapioca dropped from 17 to 1, a documented ceiling, not a reclassification to zero. I rechecked the original 10 ruby-lsp items against the exhaustive classification: 2 disappeared they were false positives, resolved by the alias fix , 8 remain, and 1 of those changed classification because the earlier evidence had checked the wrong directory. A measurement mistake on our side, not a tool behavior. False positives have a cause. The first time I measured that cause, the number came out wrong; the corrected version is below. I reclassified the 41 constant warnings left in ruby-lsp after the alias fix, by mechanism: | Mechanism | Warnings | What it is | | Declared as T.let in already-committed RBI | 11 | Test::Unit:: 7 , LanguageServer::Protocol::Constant:: 3 , RuboCop::Version::STRING 1 | | Exists in on-disk core/stdlib RBS | 9 | Minitest::Spec 7 , OptionParser and one from Gem:: | | Deliberate fixture in the repository itself | 11 | fake constants ruby-lsp's own tests define on purpose | | Rest | 10 | includes the entry.rb bug that became PR 4197 | The two type-info levers, already-committed RBI and RBS from core/stdlib on disk, are close to the same size: 11 against 9. Neither one alone resolves half of the 41. There is no single saving type source here: it is always both together, and even both together do not cover everything. 11 of the 41 are fixture: fake constants that ruby-lsp itself defines on purpose to test its own syntax diagnostic detector. The project's sorbet/config excludes that folder via --ignore ; itaruby does not honor that --ignore . That is 27% of what is left being itaruby reporting on a file the project's own maintainers already declared out of scope, a concrete and verifiable limitation. About the method, because it is the foundation of every number on this page: the first measurement of this mechanism had two mistakes. The original denominator, 160 unique unresolved constants, dropped to 29 as soon as the alias-caused ones left the list with the fix in commit 7d3a781 . And the original numerator, 78, was also wrong: the regex matched a constant by name suffix, not full path, so names like A and C counted because they happened to end the same way as a real T.let declaration somewhere else in the file. Both mistakes only surfaced when I swapped grep for a direct query against the RBS API, loading the 61 stdlib libraries and asking by type name instead of by text. The correct number, after that, is the one in the table above: 11 of 41, 27%. Swapping grep for the API fixed the suffix bug, but was not enough on its own: the query accepted a shortcut, a top-level name existing already counted as a match, so Gem existing was enough for Gem::DefaultUserInteraction to pass, when that constant's only real occurrence in the entire RBS is a comment. The full lesson is not "API instead of text": it is that the question has to be about the exact path, because accepting a prefix is suffix grep dressed up as an API. The 7 ruby-lsp diagnostics that are not constant warnings have a complete partition, verified against the source, not a sample: | Mechanism | Diagnostics | Where | | Flow narrowing | 3 | declaration listener.rb:512 and :515 is a? in a ternary , code action resolve.rb:435 case without else | | Generic T in the signature | 3 | erb document test.rb:161, ruby document test.rb:906 and :1010 all in cache set | | Cast : as untyped ignored | 1 | addon test.rb:149 | The code action resolve.rb:435 case deserves its own line: the : as nil cast stuck to the end on line 417 is read correctly, that is confirmed. What is missing is narrowing the type before the cast is reached: lines 410 to 417 are a case @code action :title with three when branches and no else , and without that branch the type left over is String | nil . The cast is not ignored: it is applied to a type that already arrived wrong. Not every row in the table is a bug. Narrowing on a case without else is a family declared out of scope by decision, a chosen boundary, not a flaw waiting to be found. The generic T treated as a fixed type is a real bug, and it is the third time this mechanism has shown up in this investigation: it already has a prioritized fix. The : as untyped cast being ignored remains unfixed. The framing that closes this section: the value is in untyped code. Across the 3 untyped repositories, the funnel produced 14 real bugs, 13 submissions, 7 accepted. Across the two repositories with Sorbet, Sorbet is already green and itaruby produces mostly false positives. That is scope, not defeat: where Sorbet is already configured and green, itaruby today does not add, it gets in the way. The one real bug itaruby found across the two repositories with Sorbet, out of 111 raw diagnostics in those two repositories. Not a win, it is the exception that proves the scope. The bug: a raise AbstractMethodInvokedError inside module RubyIndexer when the constant only exists as RubyLsp::AbstractMethodInvokedError , a sibling module, not an ancestor. The raise never resolves; I reproduced the structure separately and confirmed it: it raises NameError . Of the 17 sites of this same raise in the repository, 16 are in lib/ruby lsp/ and resolve correctly; only this one is outside the right namespace. It is a latent defensive guard, not a production crash, because both concrete subclasses already implement the abstract method. The PR is open, 4197 https://github.com/Shopify/ruby-lsp/pull/4197 , on ruby-lsp: 2 lines added, none removed. Contributions from outside Shopify go through two gates before shipping: the company's CLA and first-contribution workflow approval, both pending now. The most interesting data point in this section: this bug was in a repository that runs Sorbet in CI, green, and Sorbet does not catch it because the project's own sorbet/config --enable-experimental-rbs-comments flag silences this kind of guard. It is not "itaruby is stronger than Sorbet": it is a configuration flag that created a blind spot, and a tool that does not read the project's configuration went right through it without knowing it existed. That is the real argument for running two different tools instead of one. Overlap between the two real-error sets: zero. That is not a finding, it is an absence of data: Sorbet's set in the two repositories is empty, so there is no intersection to measure. Not a win, it is zero degrees of freedom in the comparison. The cycle The rate table above showed ruby-lsp at 1240 diagnostics per thousand files, the worst in the corpus, while being the most typed repository. Counterintuitive enough to be worth investigating the cause instead of taking the number at face value. The cause has a name and a line, in lib/ruby lsp/utils.rb:6-7 : Interface = LanguageServer::Protocol::Interface Constant = LanguageServer::Protocol::Constant itaruby did not follow this kind of constant alias; every reference to Interface::X or Constant::X in the rest of the file turned into an unresolved-constant diagnostic. That accounted for 405 of the 449 constant warnings in the repository. Prediction made before touching the code: about 405 warnings should disappear with the fix. After the fix, commit 7d3a781 : 408 disappeared. Prediction error of 0,7%. That conclusion came from comparing totals per repository, before and after, and comparing totals does not establish a claim about a set: I made this exact mistake myself later, in wave 7, where rails's total came out identical between two waves, 2316 against 2316, hiding 5 diagnostics leaving and 5 entering. The set-level diff for this alias fix was never run, and I no longer have that wave's binary to run it now: I cannot say whether any new diagnostic showed up anywhere in the corpus. | Repository | Diagnostics before | Diagnostics after | Diag/1k before | Diag/1k after | | ruby-lsp | 456 | 51 | 1240 | 137 | | tapioca | 60 | 60 | 289 | 275 | | rails/rails | 2364 | 2316 | 684 | 670 | | discourse/discourse | 7959 | 7884 | 711 | 704 | | zammad/zammad | 2834 | 2822 | 507 | 505 | Measured at commit 7d3a781 , same file count as before. What survives this comparison is magnitude, not the set: the untyped repositories barely moved, net deltas of rails -48, discourse -75, zammad -12, while the typed repository dropped 405 also a net delta . If the gain had come spread out evenly, that would be cause for suspicion: a generic fix that reduces noise everywhere equally proves no mechanism at all. Here the gain landed exactly where the mechanism predicted it would. Why did the fix cure ruby-lsp and leave tapioca untouched, when both have the same kind of alias? I isolated it with a 9-line test, three variations of the same alias, changing only where the target is declared: | Case | Target declared in | Form | Result | | A | .rb file | nested block | resolves | | B | .rbi file | nested block | fails | | C | .rbi file | compact module X::Y | resolves | In a .rbi file, a declaration written as a nested block indexes correctly: the full path exists in the index. The query is what fails, looking up the qualified path in a map where that name was registered unqualified, and giving up before applying the refilter that exists exactly for this. Only the compact form avoids the problem. In .rb both work. The two real lines, side by side: ruby-lsp, lib/ruby lsp/utils.rb:7 alias in .rb, cured Constant = LanguageServer::Protocol::Constant tapioca, sorbet/rbi/gems/ruby-lsp@0.26.10.rbi:2380 alias in .rbi, not cured RubyLsp::Constant = LanguageServer::Protocol::Constant The difference between the two repositories is not a code profile, it is a lookup bug with an address. And the consequence that matters to a reader: fixing this is not increasing coverage, it is knocking down false positives in the repository where the tool looks worst. The reclassification closed, exhaustive, item by item, not a sample of 20. This replaces the provisional "undecided" reading I had left here: on ruby-lsp, false positives dropped from 70% 7 of 10, old sample to 31% 16 of 51, full count ; on tapioca it went the other way and it became the repository with the highest false-positive share in the corpus, 57% 34 of 60 . srb tc remains at zero errors on both. The sentence from two hours earlier, "where Sorbet is already configured and green, itaruby today does not add, it gets in the way", no longer holds as a single sentence for both repositories: on ruby-lsp it got weaker, false positives are already a minority 31% ; on tapioca it got stronger, false positives are a clear majority 57% , and it was not a terminal state: the cause was the .rbi lookup bug above, it had an address in the code, and the fix went in. The result is in the "Wave 8" section: tapioca dropped from 17 to 1, a documented ceiling. Wave 8 Wave 8 replaces wave 7 as the current snapshot: binary compiled from commit 5c5c8c3 , 2026-08-26, same trees pinned by sha as before. Composition by diagnostic code, all six repositories in the corpus: | Repository | Total | Composition | | ruby-lsp | 30 | 24 E0104, 3 E0103, 3 E0001 | | tapioca | 1 | 1 E0104 | | rails/rails | 2302 | 2079 E0104, 222 E0101, 1 E0102 | | discourse/discourse | 6217 | 6200 E0104, 12 E0101, 5 E0102 | | zammad/zammad | 2821 | 2611 E0104, 207 E0101, 3 E0102 | | chatwoot/chatwoot | 2915 | 2915 E0104 | chatwoot had its first measurement in wave 7. This is the first comparison that exists for it: the set diff silenced 2 diagnostics, zero new ones. Of ruby-lsp's 30, 3 are E0001 syntax error in test/fixtures/syntax diagnostics.rb and test/fixtures/multiple invocations.rb , files that are invalid Ruby on purpose, written for ruby-lsp itself to test its own syntax diagnostic. The checker is right there: it is not a hit against a real bug, it is recognizing a deliberate fixture for what it is. tapioca, which this page classified as 60 diagnostics, 0 real bug and 34 false positives 57%, the worst share in the corpus , dropped to 1. tapioca is not clean: 1 remains, and it is a documented ceiling. The author says this remaining case is a case where INSTANCE is a value, a known limitation, not an open defect. Trajectory of the two typed repositories, four measurement rounds: ruby-lsp 456 - 51 - 31 - 30; tapioca 60 - 60 - 17 - 1. Set diff between wave 7 and wave 8, keyed by path, line, column and code: | Repository | Silenced | New | | ruby-lsp | 1 | 0 | | tapioca | 16 | 0 | | rails/rails | 14 | 0 | | discourse/discourse | 1658 | 0 | | zammad/zammad | 0 | 0 | | chatwoot/chatwoot | 2 | 0 | Zero new diagnostics across the six repositories in the corpus. This time the claim is established by a set diff, not by a totals comparison, which is exactly the distinction the "Comparing totals is not comparing sets" section calls out. Where this page already said that claim had not been established for a given wave, it still holds for that wave: this is where, in wave 8, it became established. The diff key gained a column, and this is a correction to this page's own published rule. The "Comparing totals is not comparing sets" section, further down, says the diff must be keyed by file, line and code, never by message. That rule was incomplete: file:line:code does not distinguish two diagnostics with the same code on the same line, and there are real pairs like that in discourse. Keying without a column makes those pairs look silenced when both are still correctly flagged. The correct key includes the column: file:line:column:code . Most of discourse's drop, 1658 , is the single root cause already covered in the section below: 1642 of those are DB written with cbase. The rest, 16 , is the defined? guard. Comparing totals is not comparing sets Wave 7 was declared drift-free by the checker's author. I remeasured it, 5c5c8c3 , and what looked like 4 isolated false positives were two root causes with 1675 sites total, not four separate cases, one of them an order of magnitude bigger than the first read suggested. Root 1, top-level constant write with cbase: 1645 sites across 3 repositories. | Repository | Sites | Detail | | discourse/discourse | 1642 | all the same name, DB, written as ::DB = MiniSqlMultisiteConnection.instance in an initializer | | rails/rails | 2 | ::DEFAULT APP FILES and ::DEFAULT PLUGIN FILES, each defined at the top of the very test file that reads it | | Shopify/ruby-lsp | 1 | ::C2, ruby-lsp's own const.rb fixture, which writes ::C2 = 42 on line 2 | When code writes a top-level constant using cbase form, ::X = value, that constant never made it into the top-level constant table. The other side of the same defect: the existence check keyed the cbase fallback off the untrimmed name. Root 2, defined? guard: 30 sites across 3 repositories. | Repository | Sites | | rails/rails | 12 | | discourse/discourse | 16 | | chatwoot/chatwoot | 2 | Flagging defined?'s argument was never the problem: the checker already got that right. The real false positive was reading that same constant in the if or ternary's then branch, the branch that only runs when it exists. The rule that fixed it controls both sides: a read in the else branch is still flagged, because there it is genuinely unsafe, and defined? X guarding a read of Y still flags Y. The single biggest name in the whole corpus was not a gem constant missing a declaration: it was a lookup bug on our side. One line of discourse code produced 1642 false diagnostics. Both roots were fixed at the source in wave 8, binary compiled from commit 5c5c8c3 , and I verified the fix myself: | Constant | Before | After | | DB discourse/discourse | 1642 | 0 | | DEFAULT APP FILES rails/rails | 1 | 0 | | AppBuilder rails/rails | 1 | 0 | The proof for the other side, the one that separates fixing from suppressing: constants that are genuinely gem constants keep getting flagged. ::Trilogy stays at 3 in rails/rails, unchanged. On top of that, zero new diagnostics across the six repositories, and this time that claim is established by a set diff, not a totals comparison. Worth publishing as verification, not praise: the attribution the author sent cbase versus the defined? guard sums up exactly to the totals I measured independently, repository by repository: | Repository | Independent total | cbase | defined? | Author's sum | | rails/rails | 14 | 2 | 12 | 14 | | discourse/discourse | 1658 | 1642 | 16 | 1658 | | chatwoot/chatwoot | 2 | 0 | 2 | 2 | | Shopify/ruby-lsp | 1 | 1 | 0 | 1 | Two counts taken through different paths landed on the same number, repository by repository. It is the opposite of the wave 7 case, where the composition did not add up to the total and 3 were left unaccounted for. The method finding that triggered this check: comparing totals is not comparing sets. On rails, the total came out identical between the two waves, 2316 and 2316, and underneath, 5 diagnostics left and 5 entered: a net-zero delta was hiding 10 items moving. Worse: if the diff had been keyed by message instead of by file:line:code , rails would have flagged 187 changes, because wave 7 started printing the constant name in absolute form ::WebSocket::Driver instead of WebSocket::Driver and every diagnostic's text changed without its site changing. A large false number is more dangerous than a small correct one. The value of measuring things yourself is not distrusting the other side: it is that a net-zero delta and a cosmetic rewrite hide things in opposite directions. This page itself made this exact mistake in the alias-fix section, further up: the passage was corrected, not deleted. Follow the launch itaruby does not have a public repository yet. Leave your email to hear when it opens. Message sent. I'll get back to you soon. Could not send. Please try again. Each submission's state checked via the host CLI: gh pr view <n --repo <owner/repo on GitHub, glab mr view <n --repo <owner/repo on GitLab.