We benchmarked a security detector against 500k lines of Go. It found a real bug.

The deslop project benchmarked a security detector against roughly 503,000 lines of Go across eight open-source projects, producing exactly one real finding. The detector, which flags non-constant-time comparisons of secrets, identified a timing side-channel vulnerability in goreleaser's GitLab client, where a plain string equality check compared a user-configured API token with an environment variable. The issue was fixed by using crypto/subtle.ConstantTimeCompare, and the maintainers accepted it as a reasonable correctness improvement.

We're building deslop — rule packs that stop coding agents from re-introducing bugs that were already fixed. One half of that is detection: if a rule claims agents keep writing a certain kind of bug, there should be a detector that catches it, and that detector should be measured before it's allowed to fail anyone's build. So we benchmarked one of our security detectors against roughly 503,000 lines of Go across eight well-known open-source projects. It produced exactly one finding. The finding was real. The rule is simple to state: secrets and tokens must be compared in constant time. A plain string equality check == against a secret leaks information through timing — on shared infrastructure, an attacker who can measure how long a comparison takes can, in principle, recover the secret piece by piece. The standard fix is crypto/subtle.ConstantTimeCompare . Writing a detector that fires on every == next to a variable named token is easy and useless. Real code compares all kinds of things to all kinds of things, and most of them are fine. Our detector only fires when an operand has a traceable credential source — a value that came from r.Header.Get ... , os.Getenv ... , or an assignment chained from one of those. It skips test files and vendored code. It resolves named helper functions one level deep, because CheckOrigin: isValidOrigin and CheckOrigin: func ... { return true } are the same bug wearing different clothes. Eight repositories, pinned commits, ~503k lines of Go counted as non-blank lines, tests and vendor excluded : | Repo | What it is | LOC | Findings | |---|---|---|---| | caddy | web server / reverse proxy | 104,660 | 0 | | lazygit | TUI git client | 141,754 | 0 | | restic | backup / storage infra | 88,525 | 0 | | goreleaser | release automation | 92,357 | 1 | | fzf | CLI fuzzy finder | 33,306 | 0 | | task | task runner | 23,162 | 0 | | chi | HTTP router | 12,082 | 0 | | viper | configuration | 7,194 | 0 | Seven clean repos matter as much as the hit. A detector that cries wolf on if key == "border-native" fzf compares lexer tokens, not credentials or if setting.Key == "vcs.modified" task dispatches on config keys is a detector nobody will leave enabled. Our first version produced exactly those false positives — twenty-two of them. We hardened the operand analysis require the credential source, not the name and re-ran. One finding. goreleaser , internal/client/gitlab.go , in checkUseJobToken : ciToken := os.Getenv "CI JOB TOKEN" if ciToken == "" { return false } // ... if ctx.Config.GitLabURLs.UseJobToken { return token == ciToken } token is the user-configured GitLab API token. ciToken is the CI job token from the environment. The comparison decides which API client to use — and it's done with plain string equality. To be clear about severity: this is hardening, not a zero-day . Exploiting a timing side channel over a network on a comparison like this is largely theoretical, and ConstantTimeCompare still leaks length though CI JOB TOKEN lengths are fixed per environment . We said exactly that in the PR. The maintainers' response treated it as a reasonable correctness improvement — which is the right way to think about it. import "crypto/subtle" // ... return subtle.ConstantTimeCompare byte token , byte ciToken == 1 Two hunks, one line changed. The PR is here: goreleaser 6813 https://github.com/goreleaser/goreleaser/pull/6813 . token == "giteatoken" in goreleaser a placeholder comparison looks identical to a credential check if you match on identifiers. Trace where the value came from instead.deslop is open source: github.com/Amaresh/deslop https://github.com/Amaresh/deslop . The detector that found this, and the benchmark that measured it, are part of the repo. Found a false positive in your own Go codebase? That's data — the issue tracker is open.