{"slug": "where-the-llm-stops-deterministic-scoring-in-an-ai-assisted-vapt-pipeline", "title": "Where the LLM Stops: Deterministic Scoring in an AI-Assisted VAPT Pipeline", "summary": "ONUS, an open-source, self-hosted DAST platform developed at IIT Kanpur, ensures that all numeric scores in its vulnerability reports come from deterministic formulas rather than its local language model. The project's architecture, validated by 690 tests, guarantees that re-running scans produces byte-identical numbers even when the LLM is slow or unreachable, degrading only the prose. This design choice prioritizes reproducibility and availability over the variability of model outputs.", "body_md": "Every VAPT report ends the same way: a handful of numbers. A CVSS score. A severity label. A priority rank. Sometimes an aggregate risk score. Those are the numbers a remediation team actually acts on: what gets patched this sprint, and what waits.\n\nOnce a large language model enters that pipeline (writing summaries, explaining findings, drafting remediation steps), a quieter architectural question follows it in: is the model narrating scores that already exist, or is it, somewhere along the way, actually shaping them?\n\nThis is a technical write-up of how ONUS, an open-source, self-hosted DAST (dynamic application security testing) platform, answers that question by construction rather than by policy. It's drawn from ONUS's internal project report, its architecture, its test suite, and scan data collected during development and validation, with the test count independently verified against the live repository (more on that below), not from GitHub stars, scan counts, or its listing in OWASP's Vulnerability Scanning Tools directory. None of those are evidence that an architecture is sound, and this article deliberately sets them aside in favor of what was actually built, tested, and observed. Links to the repository, project site, and that listing are at the end, as references, not as proof of anything.\n\nONUS's own validation confirms the design held in practice: every scored field traces to a single function, so re-running a scan against an unchanged target reproduces identical numbers. When the local language model is slow or unreachable, only the report's prose degrades. The severity, CVSS, and priority numbers never move.\n\nI originally built and validated ONUS as a supervised project for the IIT Kanpur Computer Centre, under Navpreet Singh, before open-sourcing it.\n\nFor readers meeting it for the first time: ONUS is a self-hosted DAST platform. An operator submits an authorized domain; eight scanning modules run against it in parallel; results are deduplicated, passively re-verified, scored against the official CVSS v3.1 formula (the industry-standard 0–10 vulnerability severity scale), described in plain English by a local language model, and delivered as both a downloadable PDF and a live dashboard.\n\nThree commitments run through the whole system:\n\nThat architecture isn't really this article's subject, though. What is: the line ONUS draws around what its LLM is allowed to touch, and what happens on both sides of that line when things go wrong.\n\nONUS's project report states this as one of its core design objectives, in plain terms: every numeric score in the final report (CVSS score, CVSS vector, severity, priority, aggregate risk score) has to come from a deterministic formula, never from the language model, so that running the same scan twice produces byte-identical numbers.\n\nThat's a strong claim. It's worth asking why it's the right one, not just noting that ONUS makes it.\n\n**Reproducibility is testable; \"usually right\" isn't.** A pure function (`cvss_scorer.py::score_finding()`\n\n) can be checked against known CVSS vectors in a unit test and asserted to never drift. ONUS's test suite (690 tests, as of this writing) does exactly that: the CVSS formula is validated against known vectors as one specific, ongoing test category. There's no equivalent test you can write for \"the model rates this finding as Medium,\" because a model's output can vary across prompts, model versions, and (for a self-hosted, comparatively small model like the 7B one ONUS runs to stay air-gapped) whether it's even reachable that day.\n\n**Availability and correctness shouldn't share a failure mode.** If the LLM produced the severity number, LLM downtime would force a choice: block the report entirely, or quietly fall back to some second, less-tested scoring path: exactly where a less-tested path is most dangerous. ONUS's own discussion of results confirms this decoupling held in practice: a slow or unreachable Ollama instance degrades the report's prose quality, but never its severity, CVSS, or priority numbers. When Ollama can't be reached after retries, a rule-based fallback description is substituted and the report is explicitly flagged (`ai_unavailable`\n\n), never silently left blank, and never routed through a different scoring path.\n\n**The model never sees evidence before it's scored.** This is structural, not just policy: aggregation, passive re-verification, and CVSS scoring all happen, and all complete, before the LLM is invoked at all. By the time Qwen 2.5 7B sees a finding, it's already deduplicated, already tiered by confidence, and already carries its final CVSS score and priority. Its only job from there is to write about it.\n\nThat last point has a second, more general benefit worth naming, even though it isn't a claim the report itself makes or set out to test: a scanner's raw evidence often includes text pulled directly from the target: page titles, error strings, response bodies. That's attacker-adjacent content, and occasionally attacker-controlled. A model that reads raw target content *and* assigns the severity number is a model whose score a sufficiently motivated target could, in principle, try to influence. A model that only writes prose about a number it has no ability to change has a much smaller blast radius if something in that evidence turns out to be adversarial. Scoring first and describing second closes that door structurally, whether or not it was the original motivation for the design.\n\nONUS is a six-layer pipeline. Each layer has one job and talks to its neighbors.\n\n| Layer | Component | Responsibility |\n|---|---|---|\n| 1: Input | Next.js frontend | Domain entry form, authorization checkbox, live scan status |\n| 2: Backend | FastAPI + PostgreSQL | Request validation, job creation, status API, report delivery |\n| 3: Queue | Celery + Redis | Async dispatch, parallel worker orchestration, task state |\n| 4: Scanning | 8 Python modules | Execute external tools, normalize output to a shared JSON schema |\n| 5: Intelligence | Ollama + Qwen 2.5 7B | CVSS scoring, risk ranking, remediation prose |\n| 6: Output | WeasyPrint + Next.js | PDF report, interactive vulnerability dashboard |\n\nThat Layer 5 label is a simplification: the architecture diagram it's drawn from only has room for one box. Layer 5 is actually four sequential stages, and only the last one touches Ollama at all. That's the subject of \"Inside the Analysis Pipeline,\" below.\n\nA scan starts when an operator submits a domain with authorization confirmed. FastAPI validates the request, rejecting private IP ranges (RFC 1918) and localhost outright, checks for duplicate or concurrent scans against the same domain, creates a `Scan`\n\nrow, and pushes a job to Redis. Celery dispatches a group of eight parallel scanning subtasks; a chord callback (a Celery primitive that fires once every task in a group has completed) triggers once all eight report back, and only then does aggregation, verification, scoring, description, and PDF rendering begin.\n\nTwo schema-level choices are worth calling out, because they show the layering is enforced in the database, not just the diagram. The generated PDF is stored in a separate `reports`\n\ntable (as `BYTEA`\n\n), deliberately kept apart from the `scans`\n\nrow, so that polling a scan's status never has to read or write binary PDF data. And updates to a scan's per-module status map use a raw, atomic `jsonb_set`\n\nSQL statement rather than a read-modify-write ORM update, specifically to avoid a race condition when multiple parallel Celery workers try to update the same scan's status at once. (WeasyPrint, notably, is also what rendered ONUS's own underlying project report: same renderer, both jobs.)\n\n| # | Module | Tools | Finds |\n|---|---|---|---|\n| 1 | Recon | nmap, subfinder, Amass, httpx, Naabu, WHOIS, dnspython | Ports/services, subdomains, live-host tech, WHOIS, DNS/SPF/DMARC/DKIM |\n| 2 | Web Scan | OWASP ZAP, Nikto, Katana | XSS/SQLi/CSRF/broken auth, misconfigurations, JS-aware endpoints |\n| 3 | SSL/TLS | testssl.sh, sslscan | Protocol/cipher issues, certificate validity, HSTS |\n| 4 | Headers | pure `requests`\n|\nCSP/HSTS/X-Frame-Options/CORS/cookie flags |\n| 5 | OWASP Top 10 |\n`requests` , 6 test functions |\nSQLi, XSS, IDOR, path traversal, open redirect, error disclosure |\n| 6 | Tech Fingerprint | WhatWeb, WAFW00F | CMS/framework/server detection, WAF (Web Application Firewall) presence |\n| 7 | Nuclei CVE | Nuclei | Known CVEs, misconfigurations, exposed panels |\n| 8 | Dir Enum | FFUF | Exposed files, admin panels, auth-gated paths |\n\nNone of these tools is novel: nmap, ZAP, Nikto, testssl.sh, and Nuclei already do their individual jobs well on their own. What none of them do alone, and what the project report frames as the actual contribution over just running them separately, is operate in one coordinated pipeline against a single target, deduplicate and cross-reference their overlapping findings, apply one consistent formula-derived score across all of them, and produce a single narrative a non-technical reader can act on.\n\nEvery module wraps its external tool via `subprocess`\n\nunder a controlled timeout, and every module has to normalize its output to one shared finding schema: module, tool, type, title, evidence, target, `found_by`\n\n, and a confidence/verifiable flag. That schema is treated as non-negotiable throughout the codebase, for a specific reason: a module that emits a malformed finding causes silent data loss at the aggregation stage. Hold onto that: it comes back later, once as a design decision and once as a real bug.\n\nFor targets that sit behind authentication, an operator can supply login credentials on submission. ONUS stores them in Redis, keyed by scan ID, never as a Celery task argument, and never written to the `scans`\n\ntable. A scanning module retrieves the credentials, auto-detects whether the login is an HTML form or a JSON API, logs in, and crawls and tests only the authenticated surface. Logout-shaped links are explicitly excluded from the crawl: a rule that exists because of a real bug, covered below. Credentials are deleted from Redis once the scan finalizes.\n\nOnce all eight modules report back, ONUS runs a fixed four-stage pipeline before anything reaches a human, or an AI model acting as narrator. This pipeline (and the fact that its stages run in a fixed order, each depending on the last one having already finished) is the actual trust boundary this article is about.\n\n``` php\nflowchart TD\n    A[\"8 module result envelopes\"] --> B[\"Aggregator<br/>dedupe + fingerprint collapse\"]\n    B --> C[\"Confidence Verifier<br/>passive re-observation only\"]\n    C --> D[\"Deterministic CVSS Scorer<br/>CVSS v3.1 formula\"]\n    D --> E{\"Ollama reachable?\"}\n    E -->|Yes| F[\"Ollama (Qwen 2.5 7B)<br/>description + remediation prose\"]\n    E -->|No, after retries| G[\"Rule-based fallback<br/>ai_unavailable = true\"]\n    F --> H[\"Merge: scores from D, prose from F/G\"]\n    G --> H\n    H --> I[\"Scored + described findings\"]\n```\n\nThe aggregator deduplicates findings reported by more than one module into a single entry, and collapses large groups of identically-shaped responses into one summarized finding. That second part matters more than it sounds: it's the defense against a wordlist-based directory brute-force returning thousands of near-identical \"findings\" and drowning out everything else in the report.\n\nThis is the stage most worth slowing down for, because it's the one most easily skipped in a simpler design, and ONUS's report is explicit that it wasn't: confidence verification is a distinct, standalone stage, in its own module (`backend/analysis/verifier.py`\n\n), strictly between aggregation and scoring.\n\nIts rule is absolute: every verifier re-issues the *exact same* non-destructive request a scanning module already sent, and checks whether the same evidence still reproduces. It is never given a new exploitation technique or payload, even where adding one would be trivial. That constraint is what keeps \"verification\" from quietly turning into a second, less-authorized exploitation pass. For reflected XSS specifically, that re-observation happens in an actual browser (Playwright-driven headless Chromium) rather than a raw HTTP replay, since confirming a reflected payload actually executes needs a real rendering context, not just a string match in a response body.\n\nEvery finding lands in one of three tiers:\n\n| Tier | What it means | How a finding gets there |\n|---|---|---|\nConfirmed |\nRe-verified proof, or a signal that already needed no further check (e.g., a database error string returned directly in a response) | The verifier re-issues the same request and the evidence still reproduces |\nProbable |\nThe default | Not yet re-checked, or not currently verifiable |\nUnverified |\nCould not be re-proven | A verifier ran and the original evidence didn't reproduce |\n\nA finding that fails to reproduce is **never dropped**: it's demoted to `unverified`\n\n, with a recorded reason. The report is explicit about why: silently dropping it would reintroduce the exact class of data-loss bug this stage exists to prevent.\n\nThe tier isn't just a label; it deterministically shifts two things: a finding's priority (a confirmed finding moves one step more urgent, an unverified one moves one step less) and its contribution to the overall risk score. The final PDF also groups its findings catalogue by tier (Confirmed, then Probable, then Unverified) rather than mixing them, specifically so a reader can immediately tell a re-proven vulnerability apart from one that still needs manual review.\n\nEvery finding (now carrying both a deduplicated identity and a confidence tier) runs through a CVSS v3.1 scorer with an explicit rule for 73 distinct finding types. This is the single function the reproducibility claim at the top of this article rests on.\n\nOnly after scoring is complete does Ollama see the findings, and only to produce descriptive prose and remediation text. If it's unreachable or times out after retries, a rule-based fallback description is substituted and the report is flagged accordingly: never left blank, and critically, never routed back through a different scoring path. A final merge step combines the scores from stage three with the prose from whichever of the two description sources ran.\n\nArchitecture diagrams are easy to nod along to and hard to actually picture. Here's what the pipeline above produced against `testphp.vulnweb.com`\n\n, an intentionally-vulnerable public test site, as documented in the report's own dashboard screenshots.\n\n| Finding | Severity | CVSS | OWASP Category | Module | Priority |\n|---|---|---|---|---|---|\n| Missing SPF record | Medium | 4.3 | A05:2021 – Security Misconfiguration | RECON | 3 |\n| Missing DMARC record | Medium | 4.3 | A05:2021 – Security Misconfiguration | RECON | 3 |\n| DKIM record not found (common selectors) | Medium | 4.3 | A05:2021 – Security Misconfiguration | RECON | 3 |\n| nmap found no open ports (or scan timed out) | Informational | 0.0 | N/A | RECON | 5 |\n| A record found | Informational | 0.0 | N/A | RECON | 5 |\n| TXT record found | Informational | 0.0 | N/A | RECON | 5 |\n| No HTTPS service detected on port 443 | Informational | 0.0 | N/A | SSL_TLS | 5 |\n| Target unreachable for header analysis | Informational | 0.0 | N/A | HEADERS | 5 |\n| No WAF detected | Informational | 0.0 | N/A | TECH_FINGERPRINT | 5 |\n\nOverall: **4/100, Low Risk** (0 Critical, 0 High, 3 Medium, 0 Low, 6 Informational).\n\nLayered on top of that table, here's what the LLM contributed for the same scan:\n\nThe security scan of testphp.vulnweb.com revealed several issues related to domain security configurations and network accessibility. The site lacks essential email authentication records (SPF, DMARC, DKIM) which can lead to phishing attacks and undetected spam. Additionally, the absence of an HTTPS service on port 443 and no web application firewall suggests potential vulnerabilities in data protection and traffic filtering. Overall, these findings indicate a moderate security posture that needs improvement to protect against common cyber threats.\n\nTwo things are worth noticing. First, the three Medium findings all carry the identical CVSS score (4.3): they're the same underlying finding type (a missing email-authentication record), scored by the same deterministic rule, every time. That's the \"byte-identical numbers on re-run\" claim made concrete: same finding type, same score, no exceptions. Second, the paragraph above is the *only* part of this output the LLM touched. If Ollama had been unreachable during this run, the table wouldn't change at all: only the paragraph would, replaced by generic fallback text and flagged as such.\n\nA scan's status is a small state machine, and the interesting design decisions are all about what happens off the happy path.\n\n``` php\nstateDiagram-v2\n    [*] --> queued\n    queued --> running\n    running --> analysing: all 8 modules succeeded or partial\n    running --> awaiting_user_decision: a module failed or timed out\n    running --> failed: stuck-scan deadline exceeded\n    awaiting_user_decision --> running: operator retries failed modules\n    awaiting_user_decision --> analysing: operator continues without them\n    awaiting_user_decision --> cancelled: operator cancels\n    awaiting_user_decision --> failed: stuck-scan deadline exceeded\n    analysing --> complete\n    complete --> [*]\n    cancelled --> [*]\n    failed --> [*]\n```\n\nIf any of the eight modules reports `failed`\n\nor `timeout`\n\n, the pipeline doesn't quietly proceed without it: it pauses at `awaiting_user_decision`\n\nand surfaces the failure to the operator, who chooses to retry the failed modules, continue without them, or cancel the scan outright. That pause is a tested, load-bearing state, not just a box on a diagram: the retry/continue/cancel endpoints are part of the 690-test unit suite.\n\nSeparately, a stuck-scan reaper independently fails any scan that exceeds a hard deadline with no progress. That guards against a specific, easy-to-miss failure mode: a Celery hard time-limit that kills a task outright, before it ever gets the chance to report back as `failed`\n\n. Without the reaper, that scan would just sit at `running`\n\nforever, with nothing in the pipeline aware anything had gone wrong.\n\nONUS's report frames testing as three distinct exercises, kept apart from the implementation work on purpose:\n\nIt's worth being precise about what this establishes. All three levels test that the *pipeline* behaves correctly: that a known CVSS vector scores the way it should, that a failed module actually pauses the scan, that a login flow actually authenticates before crawling. None of them measure how often the underlying scanners correctly identify real vulnerabilities in an application ONUS has never seen before. That's a different, harder question, and the report doesn't claim to answer it. More on that a few sections down.\n\n| Metric | Value | Source |\n|---|---|---|\n| Automated backend tests | 690 |\n`pytest --collect-only` , live repo, verified August 2026 |\n| Scanning modules | 8 | `backend/tasks/` |\n| Distinct CVSS-scored finding types | 73 |\n`cvss_scorer.py` 's rule catalogue |\n| OWASP Top 10 (2021) categories actively mapped | 5 of 10 |\n`aggregator.py` 's category map |\n| Maximum concurrent scans | 5 (configurable) | `config.MAX_CONCURRENT_SCANS` |\n| Docker Compose services | 18 | `docker-compose.yml` |\nTotal lines of code (backend `.py` + frontend `.ts` /`.tsx` ) |\n15,810 |\n`wc -l` , excluding `node_modules` /`.next`\n|\n| Scans executed during development/validation | 79 | live `scans` table |\n| PDF reports generated during development/validation | 124 | live `reports` table |\n| Validation/practice targets used | 9 | `docs/test_findings.md` |\n\nOne number in that table is not the report's own figure, and it's worth flagging plainly rather than letting it blend in: **the test count**. The report states 438 automated tests. While preparing this article I cloned the live repository and ran `pytest --collect-only`\n\ndirectly against it; it collected 690 tests cleanly, with no errors. That's the number used throughout this piece from here on. Every other figure in the table above is exactly what the report states, unchanged.\n\nTwo more things here deserve a beat of interpretation, clearly marked as mine rather than the report's own framing:\n\n`dvwa.local`\n\ncompleted in as little as 0.9 minutes and as long as 5.8 minutes across different runs; `clinkl.in`\n\ncompleted in approximately 4.6 minutes; `nodegoat.local`\n\ntook approximately 13.5–14 minutes on its two full runs. A single \"average scan time\" would have been a punchier, more citable number, and a more misleading one, given that spread. Reporting the range instead is a small methodological choice worth noticing.Real validation surfaced two bug classes that a design review alone wouldn't have caught, and the report calls both out because they generalize past their specific fixes.\n\nThe IDOR (insecure direct object reference) detector built for NodeGoat's `/allocations/:userId`\n\nvulnerability initially found nothing when run through the full pipeline, despite working correctly when tested in isolation. The root cause: the crawler was following NodeGoat's own logout link mid-crawl, silently destroying the authenticated session for every remaining test in that run, not just the one that happened to hit logout.\n\nThe fix, excluding logout-shaped links from the crawl, improved every OWASP Top 10 test against authenticated targets, not just IDOR detection. The report's own framing of the generalizable lesson is worth keeping intact: a detector that \"works in isolation\" is not yet validated until it's run through the actual pipeline that will call it.\n\nSeparately, several external-tool integrations (testssl.sh, WHOIS lookups, WhatWeb) were, at different points, confirmed non-functional in ways that produced no error and no empty-result warning. They simply never contributed a finding, silently, for reasons ranging from missing system packages to an incorrect flag.\n\nThis is arguably the scarier bug class of the two, because wrong output is at least visible; no output and no error isn't. Both bug classes motivated the same structural response: a module's execution status, whether it found nothing, failed outright, or succeeded quietly, is now always visible in the report, tied back to the non-negotiable finding schema described earlier. That's a direct, traceable line from two specific production bugs to a specific architectural invariant: nothing about a module's run is allowed to be silent.\n\nA handful of structural guardrails run through every layer of the system:\n\n| Guardrail | Mechanism |\n|---|---|\n| Authorization | Every scan requires an explicit `authorized: true` confirmation, logged with a timestamp |\n| Network isolation | Private IP ranges (RFC 1918) and localhost are rejected at request validation |\n| Non-destructive testing | Active tests use read-only, proof-of-concept payloads only: no data modification, no denial-of-service payload |\n| Audit trail | Every scan (target, timestamp, operator) is permanently logged |\n| Rate limiting | Configurable cap on concurrent scans; duplicate-active-scan rejection for the same domain |\n| Data privacy | Zero external API calls: all analysis, including the LLM, runs on local infrastructure |\n\nThe report is candid about ONUS's functional scope (see Limitations below). It's worth being equally candid about what its evidence does and doesn't establish, since that's easy to blur in any project write-up, this one included.\n\nNone of this is a criticism unique to ONUS: most VAPT write-ups, open-source or commercial, don't publish precision/recall data either. But an article whose whole point is separating what's measured from what's designed should hold itself to the same standard.\n\nDirectly from the project report (open, acknowledged gaps in current functionality, not evidence against the deterministic/LLM boundary itself):\n\nFrom the report's own future-scope section:\n\nBeyond what the report proposes, the evaluation gaps above point to a few natural additions to that agenda:", "url": "https://wpnews.pro/news/where-the-llm-stops-deterministic-scoring-in-an-ai-assisted-vapt-pipeline", "canonical_source": "https://dev.to/maverickaayush/where-the-llm-stops-deterministic-scoring-in-an-ai-assisted-vapt-pipeline-4jcd", "published_at": "2026-08-22 12:01:52+00:00", "updated_at": "2026-08-22 12:13:19.138550+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-safety", "ai-products", "developer-tools"], "entities": ["ONUS", "IIT Kanpur", "Navpreet Singh", "Ollama", "OWASP"], "alternates": {"html": "https://wpnews.pro/news/where-the-llm-stops-deterministic-scoring-in-an-ai-assisted-vapt-pipeline", "markdown": "https://wpnews.pro/news/where-the-llm-stops-deterministic-scoring-in-an-ai-assisted-vapt-pipeline.md", "text": "https://wpnews.pro/news/where-the-llm-stops-deterministic-scoring-in-an-ai-assisted-vapt-pipeline.txt", "jsonld": "https://wpnews.pro/news/where-the-llm-stops-deterministic-scoring-in-an-ai-assisted-vapt-pipeline.jsonld"}}