{"slug": "i-invented-a-cve-number-to-test-my-tool-it-was-real", "title": "I invented a CVE number to test my tool. It was real", "summary": "A developer building a tool to verify claims in bug reports fabricated a security report against one of their own projects, inventing a CVE number that turned out to be a real Linux kernel vulnerability. The tool correctly flagged all other fake claims but failed to catch the CVE, highlighting the challenge of validating identifiers in AI-generated reports. The developer tested the tool against curl's advisory corpus, finding that 9.3% of claims at the affected release were not found, and noted issues with tree-sitter's error recovery in parsing macro-wrapped enumerators.", "body_md": "I've been building a thing that checks whether the claims in a bug report\n\ncorrespond to anything that exists. Not whether the report is AI-generated —\n\nwhether the file it names is in the repository, whether the function is\n\ndeclared, whether the version was ever tagged.\n\nTo test it end to end I wrote a deliberately fabricated security report against\n\none of my own projects. Fake file, fake function, fake commit hash, and a CVE\n\nnumber I made up on the spot: `CVE-2026-45871`\n\n.\n\nThe tool came back and told me the CVE was real. It's a Linux kernel TPM bug —\n\n\"st33zp24: Fix missing cleanup on get_burstcount() error.\" I had invented a\n\nnumber and hit a live one.\n\nEvery other fabricated claim in that report was correctly flagged. The CVE, the\n\none claim a triager would most want challenged, came back green and read like\n\ncorroboration. My tool was asking \"does this identifier exist\" when the question\n\nthat mattered was \"does it have anything to do with this project.\"\n\nThat's now the only open issue on the project, and it's a better bug than\n\nanything I found by staring at the code.\n\nMaintainers are getting buried in bug reports and security disclosures that read\n\nfluently and reference code that doesn't exist. The reports are cheap to\n\ngenerate and expensive to triage, and that asymmetry is the whole problem: a\n\nmaintainer has to read carefully to find out there was nothing there.\n\nThe useful property of these reports is that they hallucinate *plausible\nidentifiers*.\n\n`Curl_hpack_decode()`\n\ninstead of `Curl_hpack_decode_header()`\n\n.`lib/vtls/openssl_helper.c`\n\n, which sounds exactly like a file curl would have.So: extract every checkable claim from the report, resolve each one against the\n\nrepo at the revision the report says it affects, and report what didn't resolve.\n\nNo judgement about who wrote it, no score, no auto-close.\n\nDocumentation is a convenient corpus and a bad one — it drifts, it's full of\n\nillustrative examples, it references other projects. Security advisories are the\n\nactual workload.\n\ncurl publishes all 206 of its advisories in OSV format, each with the prose\n\nwriteup and the exact affected version. Every one is human-written, every one was\n\naccepted as valid, and every claim in them was true of the release it describes.\n\nSo any claim that fails to resolve is a false positive — with one controllable\n\nexception, which is checking an advisory against the wrong revision. The harness\n\nruns each advisory twice, once against HEAD and once against the release it\n\nactually names.\n\n| Corpus | Claims | Not found | Unexplained |\n|---|---|---|---|\n| curl advisories, at the affected release | 129 | 9.3% | 3.1% |\n| curl advisories, at HEAD | 124 | 12.1% | 5.6% |\ncurl `docs/` (4,449 files) |\n1,165 | 40.7% | 32.0% |\n\n\"Unexplained\" means a miss carrying no hint. A miss that says *\"no file by that\nname, but lib/hpack.c exists\"* is useful to everyone. A bare miss on an honest\n\nThe gap between the two advisory rows is the cost of not telling it which\n\nrelease you mean. Check a 2019 advisory against today's HEAD and it will\n\ncorrectly tell you the function is gone, which is true and useless.\n\n**Tree-sitter's error recovery is arbitrary, and I was reading declarations off\nit.** curl declares every option through a macro:\n\n```\nCURLOPT(CURLOPT_AUTOREFERER, CURLOPTTYPE_LONG, 58),\nCURLOPT(CURLOPT_SSL_VERIFYPEER, CURLOPTTYPE_LONG, 64),\n```\n\nThe C grammar has no rule for a macro-wrapped enumerator, so the parser\n\nerror-recovers. Where the recovered ERROR node *ends* is arbitrary: for one\n\nentry it stopped after `CURLOPT(`\n\nand the name survived as a real enumerator,\n\nfor the other it swallowed `CURLOPT(CURLOPT_AUTOREFERER`\n\nwhole and the name was\n\nnever seen. Identical syntax, 22 lines apart in one enum, opposite answers.\n\nTo a maintainer that doesn't read as \"the claim was wrong.\" It reads as \"this\n\ntool is broken.\" I now read enum bodies from their own text instead of trusting\n\nthe recovered tree.\n\n**Some names don't exist as text anywhere.** Through curl 7.62, every option was\n\nwritten like this:\n\n```\n#define CINIT(na, t, nu) CURLOPT_ ## na = t + nu\nCINIT(SSL_VERIFYPEER, LONG, 64),\n```\n\n`CURLOPT_SSL_VERIFYPEER`\n\nappears **zero times** in the header that declares it.\n\nThe preprocessor builds it. And my candidate files were chosen by grepping for\n\nthe name — so grepping found the docs that mention it and never the header that\n\ndeclares it. Fixing the parser did nothing until I fixed that too.\n\nFixing both took the pinned advisory row from 24.0% unexplained to 3.1%. The\n\nfour remaining misses are internal functions curl has since renamed or removed.\n\nIt catches *lazy* fabrication. A report that only names real symbols passes\n\nclean, and by design I have no way to distinguish a well-grounded fabrication\n\nfrom an honest report — the tool refuses to judge authorship, which is the whole\n\nreason a maintainer can run it without poisoning their contributor\n\nrelationships.\n\nThe 32% on curl's docs is the number I'd push back on if I were reading this.\n\nDocumentation is a harsher corpus than reports — it's full of build variables,\n\nother projects' APIs, and illustrative examples — and some of those \"misses\" are\n\ncorrect: `CURLOPT_CONNECTIMEOUT`\n\nis a typo in curl's own prose for\n\n`CURLOPT_CONNECTTIMEOUT`\n\n, and reporting that it resolves to nothing is right. But\n\nI won't pretend that rate would be pleasant on a live issue tracker.\n\nWhich is the honest limitation: every number above is a replay. No maintainer has\n\nrun this against real inbound traffic, and \"does it stay quiet enough on honest\n\nreports that you leave it enabled\" is a question I can't answer by myself.\n\nIf you maintain something that gets this kind of report, I'd genuinely like to\n\nknow what it does on yours.\n\n```\n- uses: Dgotlieb/substantiate@86b171f7b7afadbfd0cf95dd62a12b579f9078c5  # v0.1.4\n  with:\n    report: ${{ github.event.issue.body }}\n```\n\nApache-2.0. Zero dependencies for the default path.", "url": "https://wpnews.pro/news/i-invented-a-cve-number-to-test-my-tool-it-was-real", "canonical_source": "https://dev.to/dgotlieb/i-invented-a-cve-number-to-test-my-tool-it-was-real-3di1", "published_at": "2026-08-27 01:26:24+00:00", "updated_at": "2026-08-27 01:48:33.370569+00:00", "lang": "en", "topics": ["developer-tools", "ai-safety"], "entities": ["curl", "Linux kernel", "tree-sitter", "OSV"], "alternates": {"html": "https://wpnews.pro/news/i-invented-a-cve-number-to-test-my-tool-it-was-real", "markdown": "https://wpnews.pro/news/i-invented-a-cve-number-to-test-my-tool-it-was-real.md", "text": "https://wpnews.pro/news/i-invented-a-cve-number-to-test-my-tool-it-was-real.txt", "jsonld": "https://wpnews.pro/news/i-invented-a-cve-number-to-test-my-tool-it-was-real.jsonld"}}