We're building deslop — rule packs that stop coding agents from
re-introducing bugs that were already fixed. One half of that is
detection: if a rule claims agents keep writing a certain kind of bug,
there should be a detector that catches it, and that detector should be
measured before it's allowed to fail anyone's build.
So we benchmarked one of our security detectors against roughly 503,000
lines of Go across eight well-known open-source projects.
It produced exactly one finding. The finding was real.
The rule is simple to state: secrets and tokens must be compared in
constant time. A plain string equality check (==
) against a secret leaks
information through timing — on shared infrastructure, an attacker who can
measure how long a comparison takes can, in principle, recover the secret
piece by piece. The standard fix is crypto/subtle.ConstantTimeCompare
.
Writing a detector that fires on every ==
next to a variable named
token
is easy and useless. Real code compares all kinds of things to all
kinds of things, and most of them are fine. Our detector only fires when
an operand has a traceable credential source — a value that came from
r.Header.Get(...)
, os.Getenv(...)
, or an assignment chained from one
of those. It skips test files and vendored code. It resolves named helper
functions one level deep, because CheckOrigin: isValidOrigin
and
CheckOrigin: func(...) { return true }
are the same bug wearing
different clothes.
Eight repositories, pinned commits, ~503k lines of Go (counted as
non-blank lines, tests and vendor excluded):
| Repo | What it is | LOC | Findings |
|---|---|---|---|
| caddy | web server / reverse proxy | 104,660 | 0 |
| lazygit | TUI git client | 141,754 | 0 |
| restic | backup / storage infra | 88,525 | 0 |
| goreleaser | release automation | 92,357 | 1 |
| fzf | CLI fuzzy finder | 33,306 | 0 |
| task | task runner | 23,162 | 0 |
| chi | HTTP router | 12,082 | 0 |
| viper | configuration | 7,194 | 0 |
Seven clean repos matter as much as the hit. A detector that cries wolf on
if key == "border-native"
(fzf compares lexer tokens, not credentials)
or if setting.Key == "vcs.modified"
(task dispatches on config keys)
is a detector nobody will leave enabled. Our first version produced
exactly those false positives — twenty-two of them. We hardened the
operand analysis (require the credential source, not the name) and
re-ran. One finding.
goreleaser
, internal/client/gitlab.go
, in checkUseJobToken
:
ciToken := os.Getenv("CI_JOB_TOKEN")
if ciToken == "" {
return false
}
// ...
if ctx.Config.GitLabURLs.UseJobToken {
return token == ciToken
}
token
is the user-configured GitLab API token. ciToken
is the CI job
token from the environment. The comparison decides which API client to
use — and it's done with plain string equality.
To be clear about severity: this is hardening, not a zero-day.
Exploiting a timing side channel over a network on a comparison like this
is largely theoretical, and ConstantTimeCompare
still leaks length
(though CI_JOB_TOKEN
lengths are fixed per environment). We said exactly
that in the PR. The maintainers' response treated it as a reasonable
correctness improvement — which is the right way to think about it.
import "crypto/subtle"
// ...
return subtle.ConstantTimeCompare([]byte(token), []byte(ciToken)) == 1
Two hunks, one line changed. The PR is here:
token == "giteatoken"
in goreleaser (a placeholder comparison) looks identical to a credential check if you match on identifiers. Trace where the value came from instead.deslop is open source: github.com/Amaresh/deslop.
The detector that found this, and the benchmark that measured it, are part
of the repo.
Found a false positive in your own Go codebase? That's data — the issue tracker is open.