{"slug": "scaling-security-reviews-at-1password-solving-the-context-and-nondeterminism", "title": "Scaling security reviews at 1Password: Solving the context and nondeterminism problems", "summary": "1Password has released SuperSAGE v2, an AI-powered security code review pipeline that solves context-window limits and nondeterminism by generating incremental, compressed scaffolding files per source directory. The system uses a bottom-up map-reduce pattern to summarize code structure, regenerating only directories with actual changes to avoid cascading rewrites from LLM output variability.", "body_md": "In [ our last post](https://1password.com/blog/scaling-security-reviews-ai-powered-pipeline), we shared how we began to scale our security code review process with SAGE. We discussed how we gathered historical Product Security (ProdSec) review records to create a 1Password-specific ruleset, the three-stage Finder/Critic/Judge pipeline, and the limitations of our v1 implementation. Above all, human ProdSec reviewers still had to bring full context to the findings: where the trust boundaries lie, which directories are sensitive, and whether mitigations exist elsewhere in the codebase.\n\nOur goal for v2 was to help SAGE understand our entire codebase. Many of our GitHub repositories are *huge*, including our client and server monorepos. That means we have way too much information to fit within any LLM’s context window. We had to find a way to let SAGE perform deeper reasoning about the PR diffs it reviews without the codebase itself.\n\nThere was another hurdle. As we built v2, we ran into a fundamental LLM trait: they can’t reliably produce the same output twice. We knew we had to do our best to manage this nondeterminism so we could trust SAGE to be a relatively consistent security reviewer.\n\nWe had two things to figure out: how to fit a lot of data into a context window, and how to get consistent output from inherently inconsistent tools. If we could solve those riddles, SAGE wouldn’t just know 1Password, it would finally understand it.\n\nAnd it would earn the name SuperSAGE.\n\nAs it turns out, our Security Research team had already developed a Python proof of concept designed to compress our code context. It was a set of LLM prompts that generated one SCAFFOLDING.md file per source directory. Those scaffolding files carried compressed structural context like sensitivity ratings, attack surfaces, trust boundaries, and file summaries. It was a great foundation;, we just had to productionize it as a Go rewrite on top of SAGE v1’s model-agnostic llm.Client harness.\n\nTo start, the PoC took inventory of our code structure. Any well-organized codebase is shaped like a tree: a root that branches into directories and subdirectories, all the way down to individual files. The PoC walked that tree once from the bottom up, so by the time it reached any given directory, everything beneath it had already been analyzed. This is a classic map-reduce pattern: each directory was summarized on its own (the map), and those summaries were folded upward, child into parent, all the way to the root (the reduce).\n\nThis worked well for a point-in-time snapshot of our codebase, but 1Password has hundreds of hard-working engineers, so there are sections of our code that change *every day.* On the other hand, other sections, like our cryptography layer, are trusted, well-vetted, and rarely touched. Re-running that full bottom-up walk every night, across a codebase the size of ours, would be slow and expensive. So we chose to make it incremental: only regenerate scaffolding for directories in which the code had *actually* changed.\n\nBut skipping unchanged directories is only half the picture. When a directory's code has changed and its scaffolding needs to be regenerated, the model hands back **new** prose everytime, even if nothing more than a line of whitespace was added. An LLM rarely, if ever, describes the same code the same way twice. Had we kept folding each directory's model-written summary into the one above it, those harmless rewordings would have piled up: a rephrased child summary would make its parent look changed, and that parent its own parent, all the way to the root, leaving us regenerating the whole tree every night to chase code changes that weren’t semantically different.\n\nIt left another problem sitting in front of us, even for a single directory. We'd been letting the model's wording alone decide what counted as a change. We needed a way to answer one question without depending on the prose at all: Did this directory *truly* change?\n\nThere it was: the nondeterminism problem.\n\nThe easiest fix for inconsistent LLM output is to force temperature=0, which helps reduce randomness from the generation. But two of the three models in our SAGE pipeline don’t support that setting at all, so we had to solve our nondeterminism issue architecturally.\n\nAt first we explored an alternative to plain-text comparison. Instead of checking if the new summary’s *text* was identical to the original, we considered checking if the new summary’s *meaning* was close enough to the original. It sounded good on paper, but ultimately we rejected the idea because a genuinely important security change (like adding a new endpoint that’s reachable from outside the trust boundary) might only shift the similarity score a small amount, and would be indistinguishable from ordinary LLM-rephrasing noise. For a security tool, silently missing that kind of change is worse than being too cautious.\n\nInstead, we narrowed what SAGE v2 considers a change. Rather than comparing everything within a directory's SCAFFOLDING.md file, we only track the SHA256 hashes of a small, fixed set of structural facts, — things like the files in the directory, their sensitivity ratings, and trust-boundary designations. If any of *those* hashes shift, we treat it as a real change worth propagating throughout the scaffolding. The written analysis generated for humans is intentionally left out of the comparison because it's the piece most likely to come back worded differently from one scan to the next without any meaningful changes.\n\nWe made one other deliberate choice: The LLM never gets to decide what counts as a change, either. The hashes themselves are computed by our Go harness, which is completely deterministic. The LLM no longer needs to infer what has changed, so we’re not asking the model to grade its own homework.\n\nLong story short, we let the structure decide, not the prose. A new file in a directory is a real signal; a reworded summary of the same code is just noise. And by eliminating that noise, SAGE keeps its understanding of our code fresh.\n\nImplementing context compression and solving for model nondeterminism are what let SAGE go from reviewing diffs in isolation to reasoning about the code around them: where a change lives, whyit matters, and whether a risk is real. And it can keep that knowledge current, day after day, without unnecessary costs incurred by LLM drift.\n\nThe results speak for themselves. SAGE now distills a directory’s raw source into a security summary a fraction of its size — often 30 times smaller — and refreshes that map every night for a few dollars, re-deriving only the small slice of code that actually moved while skipping the rest for free. It has *earned* the name SuperSAGE.\n\nWe’ve come a long way. But we still have work to do.\n\nTo this point, SAGE has only scanned select PRs: those voluntarily tagged with the sage-review label and those assigned to the ProdSec team for review. The next step is to roll it out to every PR across all repositories.\n\nBut scaling a tool is its own test. A reviewer that can't tell a real finding from a false positive is manageable on a handful of PRs, and a liability on *all* of them. Before we have SAGE scan every PR throughout the organization, we need to teach our AI-assisted reviewer which of its findings actually matter.\n\nWhich means SAGE needs to *learn*. And that’s exactly where we’re headed next.\n\nStay tuned for Part 3.\n\nOur Security team is hiring. If this work sounds exciting to you, we [encourage you to apply](https://jobs.ashbyhq.com/1password).", "url": "https://wpnews.pro/news/scaling-security-reviews-at-1password-solving-the-context-and-nondeterminism", "canonical_source": "https://1password.com/blog/scaling-security-reviews-solving-context-and-nondeterminism", "published_at": "2026-07-30 00:00:00+00:00", "updated_at": "2026-07-30 13:54:58.251583+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-tools", "ai-agents", "ai-safety"], "entities": ["1Password", "SuperSAGE", "SAGE", "GitHub", "ProdSec"], "alternates": {"html": "https://wpnews.pro/news/scaling-security-reviews-at-1password-solving-the-context-and-nondeterminism", "markdown": "https://wpnews.pro/news/scaling-security-reviews-at-1password-solving-the-context-and-nondeterminism.md", "text": "https://wpnews.pro/news/scaling-security-reviews-at-1password-solving-the-context-and-nondeterminism.txt", "jsonld": "https://wpnews.pro/news/scaling-security-reviews-at-1password-solving-the-context-and-nondeterminism.jsonld"}}