{"slug": "give-your-coding-agent-a-deterministic-vulnerability-oracle", "title": "Give Your Coding Agent a Deterministic Vulnerability Oracle", "summary": "A developer created VulnGraph, a deterministic vulnerability oracle for AI coding agents that turns continuously changing vulnerability intelligence into verified, content-addressed snapshots. The tool provides typed evidence and explicit freshness checks, ensuring agents never confuse \"unknown\" with \"safe\" and can make informed security decisions.", "body_md": "AI agents can write code, run tests, inspect dependencies, and open pull requests. But when they encounter a vulnerable package, their security reasoning often collapses into a web search, an opaque API score, or whatever the model remembers from training.\n\nThe problem is not simply missing vulnerability data. It is the contract between the agent and its harness—the automation layer that supplies tools, executes commands, and interprets results. If “no match” silently becomes “safe,” stale intelligence looks current, or a network failure resembles an empty result, the agent can produce a confident answer without trustworthy evidence.\n\nI designed VulnGraph around a different premise: turn continuously changing vulnerability intelligence into a deterministic local primitive. [ vulngraph-data](https://github.com/copyleftdev/vulngraph-data) compiles upstream security sources into verified, content-addressed snapshots.\n\n`vulngraph-cli`\n\nThe result is a security tool shaped for both humans and machines: an accepted snapshot and target produce the same verdict; every verdict carries typed evidence; stale data fails explicitly; and `unknown`\n\nis never misrepresented as clean.\n\nThis article explains the design decisions behind that contract and shows how to embed VulnGraph into agent and harness workflows.\n\nSuppose an agent is reviewing a dependency update. It needs to answer a seemingly simple question: *Is this version safe to ship?* A useful answer depends on distinctions that are easy for a harness to erase.\n\n| What happened | A careless harness concludes | What the contract must preserve |\n|---|---|---|\n| The package exists and the version is outside every affected range | No vulnerabilities found | `not-affected` |\n| The package is absent from the dataset | No vulnerabilities found | `unknown` |\n| The local snapshot is too old | The last result is probably fine | A freshness failure |\n| The tool cannot open or verify its data | Empty result | An operational or integrity error |\n\nThese are not cosmetic labels. They lead to different agent actions: proceed, investigate, refresh the data, or stop the workflow. Once a harness flattens them into a Boolean, the model cannot recover the missing meaning through better prompting.\n\nHere is the distinction in practice. One command checks an exploited CVE, a real package version, and a package the snapshot has never observed:\n\n``` bash\n$ vulngraph check CVE-2024-4577 npm:lodash@4.17.15 npm:no-such-package@1.0.0\n\n  CVE-2024-4577\n  verdict: ACTIVELY EXPLOITED  (confidence 0.99)\n  action:  patch now\n  reasons: CRITICAL_SEVERITY, HIGH_EXPLOIT_PROBABILITY,\n           KNOWN_EXPLOITED, PUBLIC_EXPLOIT, ...\n  cvss:    9.8\n  epss:    100.0%\n  kev:     listed\n\n  npm:lodash@4.17.15\n  verdict: PROOF OF CONCEPT  (confidence 0.96)\n  action:  prioritize\n  package: npm:lodash — 6 CVE(s) affect this version\n\n  npm:no-such-package@1.0.0\n  verdict: UNKNOWN  (confidence 0.00)\n  action:  investigate\n  reasons: NOT_OBSERVED\n\n  snapshot: sha256:41ee5fbbf58f0a46b99234af89c5388e5f27e0dcd3dbd52298bcd2c2d87ca90e\n```\n\nThe output does not ask the agent to infer policy from prose. It supplies a disposition, an action, stable reason codes, confidence, and the exact snapshot that produced the result. The human-readable rendering is useful in a terminal; `--json`\n\nexposes the same distinctions through a versioned machine envelope.\n\nVulnerability intelligence changes every day. Agent execution still needs to be reproducible. I separated those concerns instead of allowing every check to fetch and reconcile live data.\n\n```\nCVE List V5 · EPSS · CISA KEV · exploit sources · ATT&CK · OSV · …\n                              │\n                              ▼\n                    vulngraph-data\n              fetch → normalize → build → verify\n                              │\n                              ▼\n            immutable data-YYYYMMDD release\n       manifest + checksums + content snapshot_id\n                              │\n                  vulngraph update\n                              ▼\n             verified local snapshot\n                              │\n          vulngraph check --offline --json\n                              ▼\n                  agent or harness\n```\n\nThe [ vulngraph-data](https://github.com/copyleftdev/vulngraph-data) repository owns the nondeterministic edge: retrieving bulk publications from upstream sources and reconciling them into one graph. Its output is deterministic. Identical inputs produce byte-identical semantic files, and their hashes produce a\n\n`snapshot_id`\n\nthat names the graph by content. If a daily build has no semantic change, there is no new release.The release artifact is the distribution boundary. A consumer does not need the raw source collection or the machinery that built it. It receives a fixed database archive and a manifest recording the snapshot identity, file hashes, source freshness, graph format, and engine revision.\n\nThe [ vulngraph-cli](https://github.com/copyleftdev/vulngraph-cli) owns the deterministic side.\n\n`vulngraph update`\n\ndownloads a release into a staging area, verifies the archive checksum and every file hash, recomputes the snapshot identity, checks format compatibility, sanity-opens the graph, and only then activates it. Failed updates cannot replace the last verified snapshot.After activation, checks do not touch the network. The snapshot becomes an explicit input to the decision:\n\n```\nverdict = policy(snapshot_id, target)\n```\n\nThere is no wall clock, random value, model call, or remote response inside that function. Once a snapshot passes the freshness and integrity preflight, the same snapshot and target give a developer laptop, a CI job, and an agent sandbox the same verdict.\n\nA reliable integration has three phases. Only the update phase needs network access; discovery and checks are stable interfaces that the harness can validate.\n\nDo not make the agent reverse-engineer help text. Let the harness inspect the machine-readable capability document and the bundled JSON Schemas:\n\n```\nvulngraph --json capabilities\nvulngraph schema command\nvulngraph schema observation\nvulngraph schema status\n```\n\n`capabilities`\n\ndeclares the commands, output schema versions, exit codes, and two behavioral guarantees: checks are offline and deterministic. A harness can validate support during startup instead of discovering an incompatibility halfway through a run.\n\nInstall or refresh the snapshot in a setup job that is allowed to use the network:\n\n```\nvulngraph update\nvulngraph --json status\n```\n\nThen mount or cache the resulting VulnGraph home directory inside the agent environment. This makes network access a controlled data-provisioning concern instead of an ambient capability available during every security decision.\n\n`status`\n\nreports whether a snapshot is installed, its identity, its integrity, and its freshness. A dataset older than fourteen days is not merely accompanied by a warning: `check`\n\nrefuses to answer and exits with code `4`\n\n.\n\nThe `check`\n\ncommand accepts individual CVEs, package coordinates, and dependency files. Pointing it at a lockfile expands every dependency and applies the same version-range policy:\n\n```\nvulngraph --offline --json check package-lock.json > vulngraph.json\n```\n\nThe CLI recognizes lockfiles from npm, Yarn, pnpm, Cargo, Bundler, Poetry, pip, Go, Composer, Maven, Gradle, and Pipenv. The abridged JSON response below shows the stable envelope:\n\n```\n{\n  \"schema\": \"vulngraph.command.v1\",\n  \"command\": \"check\",\n  \"ok\": true,\n  \"partial\": false,\n  \"snapshot_id\": \"sha256:41ee5fbb…\",\n  \"data\": [\n    {\n      \"schema\": \"vulngraph.observation.v1\",\n      \"target\": { \"type\": \"package\", \"value\": \"npm:lodash@4.17.15\" },\n      \"verdict\": {\n        \"disposition\": \"proof-of-concept\",\n        \"confidence\": 0.96,\n        \"action\": \"prioritize\",\n        \"reason_codes\": [\n          \"ATTACK_MAPPED\",\n          \"HIGH_SEVERITY\",\n          \"MULTISOURCE_CORROBORATION\",\n          \"PUBLIC_EXPLOIT\",\n          \"SEVERITY_SCORED\",\n          \"VERSION_IN_AFFECTED_RANGE\",\n          \"WEAKNESS_CLASSIFIED\"\n        ]\n      }\n    }\n  ],\n  \"warnings\": [],\n  \"errors\": [],\n  \"metrics\": { \"elapsed_us\": 781250 }\n}\n```\n\nThe important separation is that VulnGraph **observes; it does not gate**. A successful check exits with `0`\n\neven when it finds an actively exploited vulnerability. Exit codes describe whether the tool completed reliably; the harness applies organizational policy to the returned dispositions.\n\n| Disposition | Evidence | Sensible default harness action |\n|---|---|---|\n`actively-exploited` |\nListed in CISA KEV | Block or require an explicit emergency exception |\n`weaponized` |\nPublic exploit and very high EPSS | Block or require security approval |\n`proof-of-concept` |\nAt least one public exploit | Prioritize remediation and require review |\n`scored` |\nCVSS or EPSS evidence, no public exploit | Apply the repository's severity policy |\n`recorded` |\nKnown record without stronger risk signals | Monitor and retain the evidence |\n`not-affected` |\nKnown package outside all affected ranges | Proceed for this snapshot |\n`unknown` |\nTarget absent from the snapshot | Investigate; never translate to clean |\n\nThis split keeps probabilistic reasoning in the right place. The harness deterministically decides which states require a stop, review, or escalation. The agent then uses the evidence to explain the finding, inspect reachability, propose the smallest safe upgrade, and communicate the trade-off to a human.\n\nThe following wrapper turns the versioned VulnGraph envelope into one of five local workflow decisions. It does not ask the model whether an operational failure is acceptable, and it never treats an unrecognized disposition as approval.\n\n``` js\nimport { spawnSync } from \"node:child_process\";\n\nconst binary = process.env.VULNGRAPH_BIN ?? \"vulngraph\";\nconst target = process.argv[2] ?? \"package-lock.json\";\nconst run = spawnSync(\n  binary,\n  [\"--offline\", \"--json\", \"check\", target],\n  { encoding: \"utf8\" }\n);\n\nlet envelope;\ntry {\n  envelope = JSON.parse(run.stdout);\n} catch {\n  throw new Error(`vulngraph returned invalid JSON: ${run.stderr}`);\n}\n\n// Exit codes describe tool reliability, not vulnerability severity.\nif (run.status === 4) {\n  throw new Error(\"VulnGraph snapshot is stale; refresh before continuing\");\n}\nif (run.status !== 0 || !envelope.ok) {\n  throw new Error(`VulnGraph failed: ${JSON.stringify(envelope.errors)}`);\n}\nif (envelope.schema !== \"vulngraph.command.v1\") {\n  throw new Error(`Unsupported schema: ${envelope.schema}`);\n}\n\nconst policy = new Map([\n  [\"actively-exploited\", \"block\"],\n  [\"weaponized\", \"block\"],\n  [\"proof-of-concept\", \"review\"],\n  [\"scored\", \"review\"],\n  [\"recorded\", \"monitor\"],\n  [\"not-affected\", \"proceed\"],\n  [\"unknown\", \"investigate\"]\n]);\n\nconst decisions = envelope.data.map(({ target, verdict }) => ({\n  target: target.value,\n  disposition: verdict.disposition,\n  decision: policy.get(verdict.disposition) ?? \"investigate\",\n  action: verdict.action,\n  confidence: verdict.confidence,\n  reasons: verdict.reason_codes\n}));\n\nconst evidence = {\n  snapshotId: envelope.snapshot_id,\n  decisions\n};\n\nconsole.log(JSON.stringify(evidence, null, 2));\n\nif (decisions.some(({ decision }) => decision === \"block\")) {\n  process.exitCode = 10; // This is our harness policy, not a VulnGraph exit code.\n}\n```\n\nThe `evidence`\n\nobject is what I would place in the agent's context. The prompt can remain narrow:\n\n```\nReview the attached vulnerability evidence for this dependency change.\n\n- Never describe `unknown` as safe or clean.\n- Preserve the snapshot ID in your report.\n- Explain the reason codes in plain language.\n- Check whether each affected dependency is reachable from the changed code.\n- Propose the smallest compatible upgrade, but do not edit files until approved.\n```\n\nThe deterministic layer establishes what was observed and which policy branch applies. The agent contributes the work it is good at: repository-specific investigation, explanation, remediation planning, and communication.\n\nBuilding the graph was only half the problem. Making it safe for automation required treating the interface as part of the security model.\n\n`unknown`\n\nmust survive every layer from data lookup to terminal output to agent context.An agent does not become trustworthy because it receives more tokens or a longer system prompt. It becomes more trustworthy when its environment preserves distinctions, constrains side effects, and supplies evidence through contracts that cannot silently change meaning.\n\nThat is the role I designed VulnGraph to play: not an autonomous security authority, but a fast, local, auditable source of vulnerability evidence that gives agents and their harnesses something solid to reason from.\n\n`copyleftdev/vulngraph-data`\n\n`copyleftdev/vulngraph-cli`\n\nDisclosure:I designed VulnGraph. This article and its cover were created with AI assistance. I reviewed the technical claims against the linked repositories and live CLI output.\n\nIf this open-source work is useful to you, you can support its continued development:", "url": "https://wpnews.pro/news/give-your-coding-agent-a-deterministic-vulnerability-oracle", "canonical_source": "https://dev.to/copyleftdev/give-your-coding-agent-a-deterministic-vulnerability-oracle-4ngc", "published_at": "2026-07-21 15:04:00+00:00", "updated_at": "2026-07-21 15:23:05.135280+00:00", "lang": "en", "topics": ["ai-agents", "ai-safety", "developer-tools", "artificial-intelligence"], "entities": ["VulnGraph", "CVE-2024-4577", "npm", "lodash", "CISA KEV", "EPSS", "ATT&CK", "OSV"], "alternates": {"html": "https://wpnews.pro/news/give-your-coding-agent-a-deterministic-vulnerability-oracle", "markdown": "https://wpnews.pro/news/give-your-coding-agent-a-deterministic-vulnerability-oracle.md", "text": "https://wpnews.pro/news/give-your-coding-agent-a-deterministic-vulnerability-oracle.txt", "jsonld": "https://wpnews.pro/news/give-your-coding-agent-a-deterministic-vulnerability-oracle.jsonld"}}