{"slug": "we-benchmarked-a-security-detector-against-500k-lines-of-go-it-found-a-real-bug", "title": "We benchmarked a security detector against 500k lines of Go. It found a real bug.", "summary": "The deslop project benchmarked a security detector against roughly 503,000 lines of Go across eight open-source projects, producing exactly one real finding. The detector, which flags non-constant-time comparisons of secrets, identified a timing side-channel vulnerability in goreleaser's GitLab client, where a plain string equality check compared a user-configured API token with an environment variable. The issue was fixed by using crypto/subtle.ConstantTimeCompare, and the maintainers accepted it as a reasonable correctness improvement.", "body_md": "We're building deslop — rule packs that stop coding agents from\n\nre-introducing bugs that were already fixed. One half of that is\n\ndetection: if a rule claims agents keep writing a certain kind of bug,\n\nthere should be a detector that catches it, and that detector should be\n\nmeasured before it's allowed to fail anyone's build.\n\nSo we benchmarked one of our security detectors against roughly 503,000\n\nlines of Go across eight well-known open-source projects.\n\nIt produced exactly one finding. The finding was real.\n\nThe rule is simple to state: secrets and tokens must be compared in\n\nconstant time. A plain string equality check (`==`\n\n) against a secret leaks\n\ninformation through timing — on shared infrastructure, an attacker who can\n\nmeasure how long a comparison takes can, in principle, recover the secret\n\npiece by piece. The standard fix is `crypto/subtle.ConstantTimeCompare`\n\n.\n\nWriting a detector that fires on every `==`\n\nnext to a variable named\n\n`token`\n\nis easy and useless. Real code compares all kinds of things to all\n\nkinds of things, and most of them are fine. Our detector only fires when\n\nan operand has a **traceable credential source** — a value that came from\n\n`r.Header.Get(...)`\n\n, `os.Getenv(...)`\n\n, or an assignment chained from one\n\nof those. It skips test files and vendored code. It resolves named helper\n\nfunctions one level deep, because `CheckOrigin: isValidOrigin`\n\nand\n\n`CheckOrigin: func(...) { return true }`\n\nare the same bug wearing\n\ndifferent clothes.\n\nEight repositories, pinned commits, ~503k lines of Go (counted as\n\nnon-blank lines, tests and vendor excluded):\n\n| Repo | What it is | LOC | Findings |\n|---|---|---|---|\n| caddy | web server / reverse proxy | 104,660 | 0 |\n| lazygit | TUI git client | 141,754 | 0 |\n| restic | backup / storage infra | 88,525 | 0 |\n| goreleaser | release automation | 92,357 | 1 |\n| fzf | CLI fuzzy finder | 33,306 | 0 |\n| task | task runner | 23,162 | 0 |\n| chi | HTTP router | 12,082 | 0 |\n| viper | configuration | 7,194 | 0 |\n\nSeven clean repos matter as much as the hit. A detector that cries wolf on\n\n`if key == \"border-native\"`\n\n(fzf compares lexer tokens, not credentials)\n\nor `if setting.Key == \"vcs.modified\"`\n\n(task dispatches on config keys)\n\nis a detector nobody will leave enabled. Our first version produced\n\nexactly those false positives — twenty-two of them. We hardened the\n\noperand analysis (require the credential source, not the name) and\n\nre-ran. One finding.\n\n`goreleaser`\n\n, `internal/client/gitlab.go`\n\n, in `checkUseJobToken`\n\n:\n\n```\nciToken := os.Getenv(\"CI_JOB_TOKEN\")\nif ciToken == \"\" {\n    return false\n}\n// ...\nif ctx.Config.GitLabURLs.UseJobToken {\n    return token == ciToken\n}\n```\n\n`token`\n\nis the user-configured GitLab API token. `ciToken`\n\nis the CI job\n\ntoken from the environment. The comparison decides which API client to\n\nuse — and it's done with plain string equality.\n\nTo be clear about severity: this is **hardening, not a zero-day**.\n\nExploiting a timing side channel over a network on a comparison like this\n\nis largely theoretical, and `ConstantTimeCompare`\n\nstill leaks length\n\n(though `CI_JOB_TOKEN`\n\nlengths are fixed per environment). We said exactly\n\nthat in the PR. The maintainers' response treated it as a reasonable\n\ncorrectness improvement — which is the right way to think about it.\n\n```\nimport \"crypto/subtle\"\n// ...\nreturn subtle.ConstantTimeCompare([]byte(token), []byte(ciToken)) == 1\n```\n\nTwo hunks, one line changed. The PR is here:\n\n[goreleaser#6813](https://github.com/goreleaser/goreleaser/pull/6813).\n\n`token == \"giteatoken\"`\n\nin\ngoreleaser (a placeholder comparison) looks identical to a credential\ncheck if you match on identifiers. Trace where the value came from\ninstead.deslop is open source: [github.com/Amaresh/deslop](https://github.com/Amaresh/deslop).\n\nThe detector that found this, and the benchmark that measured it, are part\n\nof the repo.\n\n*Found a false positive in your own Go codebase? That's data — the issue\ntracker is open.*", "url": "https://wpnews.pro/news/we-benchmarked-a-security-detector-against-500k-lines-of-go-it-found-a-real-bug", "canonical_source": "https://dev.to/stopthatslop/we-benchmarked-a-security-detector-against-500k-lines-of-go-it-found-a-real-bug-3pj2", "published_at": "2026-08-24 19:05:48+00:00", "updated_at": "2026-08-24 19:14:13.281770+00:00", "lang": "en", "topics": ["developer-tools", "ai-safety"], "entities": ["deslop", "goreleaser", "caddy", "lazygit", "restic", "fzf", "task", "chi"], "alternates": {"html": "https://wpnews.pro/news/we-benchmarked-a-security-detector-against-500k-lines-of-go-it-found-a-real-bug", "markdown": "https://wpnews.pro/news/we-benchmarked-a-security-detector-against-500k-lines-of-go-it-found-a-real-bug.md", "text": "https://wpnews.pro/news/we-benchmarked-a-security-detector-against-500k-lines-of-go-it-found-a-real-bug.txt", "jsonld": "https://wpnews.pro/news/we-benchmarked-a-security-detector-against-500k-lines-of-go-it-found-a-real-bug.jsonld"}}