A good patch isn't the same as a fixed vulnerability.
Shopify operates systems that can generate dependency upgrades and draft fixes for first-party code, but they can’t drive remediation to closure. The repository can change while a PR waits for review; a developer might add context that changes what the fix should be; and passing CI on a PR doesn't prove the vulnerability has been removed from the repository head.
River is an AI agent that lives in our company Slack. As we described in Under the River, River works from the root of World, Shopify’s monorepo, using the same reproducible development environments, written-down skills, and engineering conventions as our developers. That shared substrate gives these security workflows Shopify-specific context from the start; we add the remediation logic rather than reteaching the agent how our codebase works.
In practice, engineers no longer have to shuttle state between code, CI, Slack, and the tracker; they can focus on the product and the risk decisions that require human judgement. River checks the original finding against current code, updates the patch when it can do so safely, brings in the right engineers when a decision needs their context, and follows the work after handoff. After a merge, it checks the repository head and the resulting dependency graph before calling the work ‘done.’ Now that we've operationalized the workflow for dependency and application vulnerabilities, we can share some results.
In the first 11 days of running the dependency workflow, the backlog of open issues fell by about 70%. Roughly two-thirds were direct merges, and the rest were confirmed by River as obsolete or already fixed elsewhere. Since launch, security merges through our freshness-gated merge queue went from about 10% to 80%.
This is the part of agentic patching we’ve seen discussed the least: operating the remediation pipeline around the diff. It’s also where the defensive asymmetry becomes concrete. Attackers can tolerate repeated failure but defenders, especially organizations, have to preserve production-intended behavior with every fix.
That asymmetry is why we’re sharing this work. Attackers reuse the same techniques across targets, and security gets stronger when we show our work. We’re hoping to help raise the floor for everyone building agentic security workflows so we don’t all have to learn the hard way.
The pipeline starts after the patch appears #
The workflows for dependency and first-party application vulnerability remediation begin with different inputs. Dependency work usually starts with a PR aimed at upgrading a vulnerable dependency. Most PRs link to a vulnerability item, but the workflow still has to verify that link and status. Application vulnerability remediation work begins with an agent-discovered security finding and a recommended fix that may need deeper investigation.
From there, the two workflows follow the same remediation loop. For application vulnerabilities, River creates one Slack thread per finding. For dependency vulnerabilities, it groups related upgrade PRs affecting the same area of the codebase into a shared thread. On one recent run, four threads went up within a minute of each other, covering 35 findings across several codebases, and each then proceeded independently—rebasing, retriggering CI, and answering questions without waiting on the others. In both types of workflows, River reads its work ledger, identifies eligible items, and verifies them against the live repository, PR, and vulnerability-tracker state before acting. The engineers who own the affected code make decisions that affect product behavior and remain responsible for approvals and merges. Later runs check back on thread statuses and update the ledger.
A Slack thread provides a transcript of a given run, but more than that it preserves the investigation and handoff. This means that in future sessions, River, security engineers, and code owners can pick up this shared context and apply it to future fixes.
Building the remediation loop #
The pipeline looks simple when drawn as boxes but this hides the hard parts. The real engineering is in the transitions.
Validate the live state
Each run starts by comparing the ledger against the live repository, pull-request, and vulnerability-tracker state. The ledger is useful memory, but it isn't the operational truth. It may list a PR as open after it’s merged, or show work in progress on a PR that closed without merging.
So River treats every ledger entry as a claim to be checked, not a fact. On one recent run, River reconciled the ledger against the actual state of the recorded PRs in GitHub and removed a large share of the apparent open dependency backlog. That didn’t mean every missing PR represented a completed fix: most had merged, but a few were closed out by the system due to the availability of recent upgrades.
Then River checks whether the vulnerability is still present before touching code. For a dependency, that means asking whether the affected codebase still resolves to the vulnerable version, or whether a sibling PR already fulfilled the upgrade. For an application vulnerability finding, it means locating the current repository, confirming at the repository head that the code still does what the finding says it does, and searching each available PR system for someone else's fix already in flight. When a system can't be searched exhaustively, the handoff says so. If another change has already fixed the issue, River records the evidence and stops. A technically correct patch built on a stale premise burns engineering time and risks changing production behavior for nothing.
Repair and iterate on the current head #
River only touches code after confirming the vulnerability is still active. For a stale dependency upgrade, it replays the upgrade on the repository head, regenerates the lockfile with the repository's own tooling, and repairs mechanical failures caused by the upgrade. For an application vulnerability finding, it prepares a narrowly scoped change in a draft PR.
In one batch of eight dependency PRs, River rebased seven, retriggered CI on each new head, and repaired a release-version check that the upgrade had tripped. Two things to note here:
First, the lockfile on repository head had moved while the upgrade PRs sat in review, so replaying each branch's copy would have reverted two unrelated upgrades. River grafted each upgrade into the current lockfile instead; three PRs shared that lockfile and all three landed without reverting an adjacent change.
And second, every rebase creates a commit that invalidates the last one's results. The CI-passed notifications that arrived moments after each force-push were for the pre-push heads, so River discarded them and waited for the real verdicts.
It didn't rebase the eighth PR. A human had taken over that branch with their own commits, and a second engineer had an open review on it. Rebasing would have overwritten someone's work and pre-empted a design question, so it went back to its author untouched.
Six of the seven PRs that River had rebased came out green and went to their stewarding teams needing only an approval. The seventh stayed red on a failure that needed a decision rather than a patch.
Hand off the decision, not the investigation #
When River reaches a decision it can't make, its handoff carries a lot of context: the current head, what it tried, what the evidence shows, and the smallest question that an owner has to answer. Owners supply product context, accept risk, review, and merge. They don't shuttle state between Git, CI, Slack, and the tracker.
The PR that stayed red needed two answers rather than one: whether to deploy the new version at all, and how a type change would behave at runtime. Both were context River could gather but had no standing to adjudicate, and when you can't define the outcome you want, or the evidence that would prove it correct, another edit is a bet rather than a fix.
Sometimes the handoff changes the answer. One finding recommended hardening a long-standing feature. River confirmed the behavior, prepared a change, and updated the tests. Then a developer stepped in to ask: what requirement was this feature originally designed to serve?
River went looking. The behavior turned out to be deliberate, introduced years earlier to solve a specific problem, and accompanied by its own security review at the time. Platform-level protections that didn't exist when it was written had since reduced the residual risk further. And the fix the finding prescribed would have broken the flow it was trying to preserve. So in the end, River recommended dropping its own patch.
That was the correct outcome, and it wasn’t a failure: the evidence changed, so the recommendation changed. A useful stop leaves the evidence in a shared place, names the boundary, and routes a decision that can resume the work. Unexplained pushes and premature handoffs are what cost you developer trust.
Verify closure #
When a security issue is handed off to a steward, ownership changes but the vulnerability remains open until the fix is shipped.
River keeps watching dependency PRs after review. If the code changes, earlier test results no longer apply. After a PR merges, River checks that the repository head no longer uses the vulnerable dependency, and that the vulnerability record and ledger both reflect the fix.
That’s an important distinction. We had two green dependency PRs, and both passed CI. One merged, removing the vulnerable version from the repository head and closing its backlog item. The other one was still open waiting for review, so the vulnerable version stayed on repository head and its backlog item remained open. Both fixes “worked,” but only the first one removed the vulnerability.
For this workflow, “done” means the source system records the outcome, the default branch reflects it, and the tracker and ledger agree. A superseding fix or a rejected finding can also be correct, but it needs the same evidence and attribution.
A workflow prompt isn't a control plane #
These workflows are driven by River prompts and skills that specify complete enumeration, draft-only behavior, and human merge authority. Prompts are the right place to express judgment and the wrong place to express a guarantee.
We make it explicit that a claim is a record, not a lock, and tells the agent to enumerate completely. On one run, River rechecked only the oldest in-flight claim and said so in its own report. That shows why "the prompt says so" isn't an invariant.
So decide which guarantees belong in code. Pagination, deduplication, current-head identity, and accounting should become deterministic as the workflow matures. Prompts stay useful for judgment calls: whether code still expresses the reported behavior, and why a stop is warranted.
In security, this matters more than usual. Any rule that determines whether a vulnerability is closed must be enforced in code.
What you should copy first #
Even without Shopify's internal tools, there are three ideas you can steal:
Revalidate before you edit: Security work is a claim about moving code. Compare it with the repository head, live PR state, and the tracker before sending an agent to churn on a patch.
Make sure evidence follows the code: Bind tests and CI to the current SHA. After merge, inspect the default branch and reconcile the tracker. PR count and green-branch count measure motion, not remediation.
Make handoff a deliberate part of the workflow: Let the agent complete routine work it can verify. When it reaches a decision requiring judgement, it should preserve the evidence, identify the right owner, and ask one clear question so the work can continue. Measure whether each finding reaches an evidence-supported outcome—fixed, rejected, or escalated—not how many patches the agent creates.
The patch is the beginning #
We’ve gotten pretty close to a fully autonomous remediation pipeline. River connects our specialized systems for finding vulnerabilities into a remediation loop. It takes care of the handoffs, and preserves context across systems and days, and stops when it should.
The durable advantage of this goes beyond producing more patches. It’s about defining an evidence-backed protocol for every transition in the pipeline: the expected input, the conditions that must remain true to continue, the actions an agent may take, and explicit stopping points. This protocol has turned our agentic patching into remediation we can trust at scale.