{"slug": "detection-rate-alone-ranks-the-worst-llm-code-reviewers-first", "title": "Detection rate alone ranks the worst LLM code reviewers first", "summary": "Sunny Chokshi, a PhD student at the University of the Cumberlands, benchmarked 16 open-weight LLM code reviewers on 180 Python snippets across nine cryptographic misuse classes, running 8,640 scored calls with paired vulnerable and secure twins. Ranking by true positive rate alone put codellama 7B, devstral 24B and qwen2.5-coder 0.5B near the top, but those models flagged most or all secure twins, while gpt-oss 20B had the highest balanced accuracy at 0.952 with a 93.7% TPR and 3.3% FPR. Chokshi concludes that detection rate should be reported alongside false positive rate using matched secure twins, and notes the preprint is not peer reviewed and covers only open models and Python.", "body_md": "When I tested 16 open LLM code reviewers, the first table in my paper showed which of them separate secure code from vulnerable code at all. It is worth reading on its own.\n\nI used 180 Python snippets across nine cryptographic misuse classes, ten pairs per class. Each pair has a vulnerable snippet and a secure twin that does the same task correctly. I ran 16 open models, from 0.5B to 120B parameters, with three repeats per snippet, which gives 8,640 scored calls. For each model I report the true positive rate (TPR, vulnerable snippets flagged), the false positive rate (FPR, secure twins flagged), and balanced accuracy, the average of TPR and one minus FPR. A model that flags everything scores 0.5.\n\nJudged on TPR alone, codellama 7B, devstral 24B, and qwen2.5-coder 0.5B would rank near the top, each at or near 100%. Their FPRs tell a different story. codellama 7B and qwen2.5-coder 0.5B flagged 539 and 540 of 540 scored calls. devstral 24B flagged 82.6% of the secure twins. Three more models (codellama 34B, qwen2.5-coder 1.5B and 3B) also flagged more than 70% of secure code. The 0.5B model also left out the requested verdict line in all but 37 of its 540 replies.\n\nWith the secure twins in the table, the order changes. Of the 16 models, gpt-oss 20B had the highest balanced accuracy, 0.952, with a TPR of 93.7% and an FPR of 3.3%. Ranked by TPR alone, these models would sit at or near the top.\n\nSize and score go together to some degree, but size does not settle the ranking. Over all 16 models, parameter count and balanced accuracy are related (Spearman 0.76). Across families it did not order the results: codellama 34B (0.609) and devstral 24B (0.587) were among the weakest of the larger models. Even within gpt-oss, the 120B model (0.931) scored below the 20B (0.952).\n\nWhat I take from this test: a detection rate on its own rewarded models that said \"vulnerable\" almost every time. I would report the false positive rate next to it and use secure twins that match the vulnerable code.\n\nThe snippets are short Python functions, reviewed one at a time, so the result speaks to a single-snippet review pass. The models are open models run with temperature 0. I tested only open-weight models and only Python, so I make no claim about closed models or other languages. TPR counts a vulnerable verdict given for any reason, and I did not check whether each detection named the right misuse. This is a preprint, it is not peer reviewed, and it carries the notice that it has been submitted to the IEEE for possible publication.\n\nThe same paper goes on to test what happens when the same misuse is written indirectly. I covered one result from that part in my [earlier post on a hex key size](https://sunny-chokshi.github.io/blog/hex-key-size-false-alarms.html).\n\n[Preprint on Zenodo (doi:10.5281/zenodo.23351769)](https://doi.org/10.5281/zenodo.23351769) · [Paper page and PDF](https://sunny-chokshi.github.io/papers/does-indirection-hurt.html)\n\nRelated work in the same line, an earlier benchmark: [CryptoBench (doi:10.5281/zenodo.23067051)](https://doi.org/10.5281/zenodo.23067051).\n\nCite as: Chokshi, S. (2026). Does Indirection Hurt? Paired Evidence on LLM Review of Indirectly Written Cryptographic API Misuse [Preprint]. Zenodo. [https://doi.org/10.5281/zenodo.23351769](https://doi.org/10.5281/zenodo.23351769)\n\nDataset: Chokshi, S. (2026). CryptoBench: A benchmark of cryptographic API misuse for evaluating LLM code reviewers [Dataset]. Zenodo. [https://doi.org/10.5281/zenodo.23067051](https://doi.org/10.5281/zenodo.23067051)\n\nSunny Chokshi, PhD student, Information Technology, University of the Cumberlands. Research: adversarial robustness of LLMs in cybersecurity. [ORCID](https://orcid.org/0009-0003-4738-7759) · [GitHub](https://github.com/sunny-chokshi)\n\nOriginally published at [https://sunny-chokshi.github.io/blog/reviewer-that-flags-everything.html](https://sunny-chokshi.github.io/blog/reviewer-that-flags-everything.html)", "url": "https://wpnews.pro/news/detection-rate-alone-ranks-the-worst-llm-code-reviewers-first", "canonical_source": "https://dev.to/sunnychokshi/detection-rate-alone-ranks-the-worst-llm-code-reviewers-first-5825", "published_at": "2026-10-28 19:31:01+00:00", "updated_at": "2026-10-28 19:41:38.981488+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-research", "ai-tools", "machine-learning"], "entities": ["Sunny Chokshi", "University of the Cumberlands", "codellama", "devstral", "qwen2.5-coder", "gpt-oss", "CryptoBench", "Zenodo"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/detection-rate-alone-ranks-the-worst-llm-code-reviewers-first", "markdown": "https://wpnews.pro/news/detection-rate-alone-ranks-the-worst-llm-code-reviewers-first.md", "text": "https://wpnews.pro/news/detection-rate-alone-ranks-the-worst-llm-code-reviewers-first.txt", "jsonld": "https://wpnews.pro/news/detection-rate-alone-ranks-the-worst-llm-code-reviewers-first.jsonld"}}