{"slug": "i-built-an-ai-code-reviewer-then-owasp-broke-it", "title": "I Built an AI Code Reviewer. Then OWASP Broke It.", "summary": "An engineer built EdgeGuard, an open-source VS Code extension that uses AI to find security vulnerabilities in code, and discovered that LLMs tend to make optimistic assumptions about unknown functions, leading to false negatives. By implementing a local static risk screening stage and evidence-based reasoning, EdgeGuard reduced API costs and improved detection, reporting 2,145 potential defects on the OWASP Benchmark for Java.", "body_md": "Most AI coding tools are very good at helping developers **write code**.\n\nBut I wanted to explore a different question:\n\nCan AI help developers find the edge cases and security problems they didn't think about?\n\nThat question led me to build **EdgeGuard**, an open-source VS Code extension designed to investigate code from an adversarial perspective.\n\nThe idea is simple:\n\nTry to break the code before production does.\n\nBut when I first tested the idea against real security benchmarks, I discovered a problem.\n\n**The AI was too optimistic.**\n\nAnd OWASP exposed it very quickly.\n\nI started testing EdgeGuard against the **OWASP Benchmark for Java**.\n\nOne of the first things I noticed was surprisingly simple.\n\nGiven code like this:\n\n```\nString param = request.getParameter(\"id\");\nString bar = DatabaseHelper.doSomething(param);\nString sql = \"SELECT * FROM USERS WHERE ID='\" + bar + \"'\";\n```\n\nThe LLM could see the untrusted HTTP input.\n\nIt could see the SQL construction.\n\nBut it couldn't see what `doSomething()`\n\nactually did.\n\nSo it made an optimistic assumption:\n\n\"This is probably an internal helper that sanitizes the input.\"\n\nAnd the result could be:\n\n**SAFE**\n\neven though the data was still flowing into a SQL sink.\n\nThis was exactly the kind of false negative I wanted EdgeGuard to find.\n\nThe problem wasn't that the model couldn't understand SQL injection.\n\nThe problem was that **the model was filling in missing information with an optimistic assumption**.\n\nI changed the investigation strategy.\n\nInstead of sending raw code and asking:\n\n\"Is this vulnerable?\"\n\nEdgeGuard now provides explicit security assumptions and asks the model to preserve taint unless there is evidence that it has been sanitized.\n\nOne important rule became:\n\nIf tainted data enters an unknown function, treat the data as still tainted unless there is evidence that the function sanitizes it.\n\nThe investigation also became evidence-oriented instead of relying on a simple `SAFE`\n\nor `VULNERABLE`\n\nclassification.\n\nFor example:\n\nThat small change made a big difference.\n\nThe AI was no longer being asked to guess what an unknown function *probably* did.\n\nIt had to reason from the available evidence and follow explicit security assumptions.\n\nFinding a potential vulnerability in one function is one thing.\n\nDoing it across thousands of functions is another.\n\nLarge projects contain enormous numbers of methods that simply aren't interesting from a security perspective:\n\n```\nget()\nset()\nToString()\nEquals()\nsimple CRUD methods\ndata mapping\nutility functions\n```\n\nSending all of them to an LLM would be wasteful.\n\nIt would also create two immediate problems:\n\n**API rate limits and API cost.**\n\nMy philosophy has always been:\n\nStart with the simplest solution, and only add complexity when the simple thing breaks.\n\nSo I didn't want the LLM to analyze everything.\n\nI added a local **Static Risk Screening** stage that runs directly inside VS Code.\n\nBefore making an API call, EdgeGuard parses the code locally and looks for characteristics that make a function worth investigating.\n\nFor example:\n\nThe local stage acts as a filter.\n\nInstead of:\n\n```\n7,000 functions\n        ↓\n     LLM\n        ↓\n  $$$$$$$$$\n```\n\nthe architecture becomes:\n\n```\n7,000 functions\n        ↓\nLocal Static Screening\n        ↓\nHigh / Medium Risk\n        ↓\nLLM Investigation\n        ↓\nEvidence + Verification\n```\n\nThis makes the LLM a **reasoning engine**, rather than the first line of analysis.\n\nI wanted to know whether this architecture would actually work at project scale.\n\nSo I loaded the entire OWASP Benchmark Java project into VS Code and triggered a workspace scan.\n\nThe scan discovered:\n\nThe local screening stage first filtered the code and identified functions that deserved deeper investigation.\n\nThe LLM then focused on the higher-risk candidates instead of blindly processing every function.\n\nIn this test, EdgeGuard reported:\n\n**2,145 potential defects**\n\nThe findings included security issues such as:\n\nThe important part for me wasn't simply the number of findings.\n\nIt was that the architecture could move from:\n\none function → one LLM request\n\nto:\n\nwhole workspace → local screening → targeted AI investigation\n\nwithout turning every method into an API call.\n\nI also wanted EdgeGuard to work beyond Java.\n\nSo I tested the architecture against projects in three different ecosystems:\n\n**OWASP Benchmark**\n\n**OWASP Juice Shop**\n\n**Microsoft eShopOnWeb**\n\nThis introduced another problem.\n\nJava, C#, and TypeScript have very different:\n\nI didn't want language-specific parsing logic leaking into the investigation engine.\n\nSo I separated the language context layers.\n\nThe architecture became roughly:\n\n```\n                EdgeGuard\n                    │\n          Investigation Engine\n                    │\n        ┌───────────┼───────────┐\n        │           │           │\n      Java         C#       TypeScript\n     Context      Context      Context\n        │           │           │\n       AST         AST         AST\n```\n\nThe investigation logic can therefore remain shared while the language-specific context stays isolated.\n\nThis became an important design principle:\n\nShare the investigation logic. Isolate the language-specific context.\n\nAt this point, I realized that simply generating a vulnerability report wasn't enough.\n\nAn AI can say:\n\n\"This might be vulnerable.\"\n\nBut what I really want is:\n\n\"Here is how you can reproduce it.\"\n\nSo EdgeGuard is designed to go beyond detection.\n\nFor potential vulnerabilities, the agent can attempt to construct counterexample inputs and generate runnable verification tests.\n\nDepending on the language, that can mean:\n\nThe goal is to move from:\n\n```\nAI says:\n\"This might be vulnerable.\"\n```\n\nto:\n\n```\nAI hypothesis\n      ↓\nCounterexample\n      ↓\nGenerated test\n      ↓\nExecution\n      ↓\nEvidence\n```\n\nThe principle is:\n\nEvidence over assumptions.\n\nBuilding EdgeGuard changed how I think about AI-assisted code analysis.\n\nThe difficult part isn't simply getting an LLM to understand code.\n\nThe difficult part is controlling what the model is allowed to **assume**.\n\nIf an unknown helper is assumed safe, a vulnerability can disappear.\n\nIf every function is sent to an LLM, the system becomes expensive and difficult to scale.\n\nIf language-specific context is mixed together, supporting multiple languages becomes increasingly fragile.\n\nSo the architecture ended up combining three different approaches:\n\n**Static analysis** for fast deterministic screening.\n\n**LLM investigation** for reasoning about complex code paths.\n\n**Verification** for turning hypotheses into evidence.\n\nNone of these approaches is perfect on its own.\n\nTogether, they are much more interesting.\n\nEdgeGuard is an open-source VS Code extension, and it is still evolving.\n\nThe project is available here:\n\n**GitHub:** [https://github.com/phucphungbk/edgeguard](https://github.com/phucphungbk/edgeguard)\n\nThe idea behind EdgeGuard is deliberately simple:\n\nDon't just ask AI to write your code. Ask it how your code could fail.\n\nIf you work with security-sensitive applications, large codebases, or legacy systems, I'd love to hear how you approach this problem.\n\nHow do you find edge cases?\n\nHow do you deal with false negatives in automated code analysis?\n\nAnd most importantly:\n\n**How do you prove that an AI-generated security finding is actually real?**\n\nI'm building EdgeGuard to explore those questions.\n\nFeedback, bug reports, benchmark results, and architectural criticism are all welcome.", "url": "https://wpnews.pro/news/i-built-an-ai-code-reviewer-then-owasp-broke-it", "canonical_source": "https://dev.to/phucphungbk/i-built-an-ai-code-reviewer-then-owasp-broke-it-2ika", "published_at": "2026-08-19 22:57:14+00:00", "updated_at": "2026-08-19 23:44:08.263412+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-tools", "ai-safety", "developer-tools"], "entities": ["EdgeGuard", "OWASP Benchmark", "VS Code", "Java"], "alternates": {"html": "https://wpnews.pro/news/i-built-an-ai-code-reviewer-then-owasp-broke-it", "markdown": "https://wpnews.pro/news/i-built-an-ai-code-reviewer-then-owasp-broke-it.md", "text": "https://wpnews.pro/news/i-built-an-ai-code-reviewer-then-owasp-broke-it.txt", "jsonld": "https://wpnews.pro/news/i-built-an-ai-code-reviewer-then-owasp-broke-it.jsonld"}}