RESEARCH NOTE / INTERNAL PILOT
Many permission checks happen when a request arrives. AI agents keep working after that: a run can call tools, wait for an approval and write data well after it started. In an internal pilot, that gap showed up as a file written after the user who started the run had been suspended.
The finding: in an internal pilot using Scopehaven's approach, with synthetic data, new requests from a suspended test user were correctly refused, but in 2 of 14 test cases an agent run started before the suspension still wrote a file afterward.
01 / FINDING
What we saw. #
In an internal pilot, we used Scopehaven's approach on a pinned version of an open-source AI workspace on our own machine, with its real server, a local model and synthetic data. We suspended a test user. New requests from that user were correctly refused, but in 2 of 14 test cases an agent run started before the suspension still wrote a file afterward. We made a partial, local repair, and the same 14 checks then passed, including the ones confirming that normal use still works. We verified the repair only on the file-write path we tested; other tools and full session termination were not tested.
We're not naming the project. This is one finding in one pinned version, reproduced and partly repaired on our machine with synthetic data. It is not a safety certification or a claim about the project's current release.
- Target
- Pinned open-source AI workspace, not named
- Data
- Synthetic
- Run
- Our machine, real server, local model
- Check
- Suspending a user stops agent actions already in flight
- Before
- Fail: 2 of 14 cases wrote a file after the user was suspended
- Retest
- Pass: 14 of 14 checks passed after a partial, local repair, verified on the file-write path only
- Evidence
- HTTP status, tool approvals, file state on disk
02 / SEQUENCE
Refused at the door, still writing inside. #
- An agent run starts for a test user who is allowed to use it.
- The user is suspended while that run is still going.
- New requests from the user are refused. The obvious check passes.
- The run that was already in flight still writes a file. That happened in 2 of the 14 test cases.
Looking only at new requests, the suspension worked. The file on disk said otherwise. That gap is why, in this pilot, we read the file on disk, not the response to the next request.
03 / WHY
Why an agent run can outlive a permission change. #
This section describes a general pattern in agent systems, not that project's code. A web request is usually authorized once, when it arrives, and finishes in a moment. An agent run is different: it can keep working for seconds or minutes, calling tools, waiting for an approval or resuming from a queue. Each of those steps can act on permissions that were checked earlier.
- Checked at the start only. A run is allowed when it begins, and nothing checks again before it acts.
- d actions. An action approved, or waiting for approval, before the change can resume after it.
- Queued and scheduled work. Jobs created for the user can run later under the user's old access.
- Cached sessions and tokens. Credentials the run already holds keep working until they expire or are invalidated.
The usual remedy is to check the user's current access again right before each action takes effect, not only when the run starts. Whether that holds is something to test, not assume.
04 / WHY IT MATTERS
Where this could matter in real products. #
- Offboarding. An employee is removed while a long agent task they started is still running.
- Plan or role changes. A customer is downgraded, or a user loses a role, partway through a task that needs it.
- Account takeover. A suspicious account is suspended, but work it already queued could keep acting.
In each case it is natural to check that the user can no longer do anything new. The question that is easy to miss is what work already in flight does next.
05 / TEST IT
How to check your own agent. #
- Start a run that acts later. Pick a task that calls a tool or writes data after a delay, a tool call or an approval step.
- Change access partway through. Suspend the test user, remove a role or cancel the plan while the run is still going.
- Read the records, not the reply. Check the files, rows, messages or ledger entries the run could change, and their timestamps against the moment of the change.
- Vary the moment. Change access before a tool call, during an approval wait and just before the write. Each is a separate case.
- Keep a control. A run for a user whose access did not change must still finish, so a fix that stops everything does not pass.
- Repeat. Timing matters here, so one clean run is weak evidence. Run each case several times.
06 / LIMITS
What this does and does not show. #
- One pinned version of one open-source project, on our machine, with synthetic data and a local model.
- The repair was partial and local, and verified only on the file-write path we tested. Other tools and full session termination were not tested.
- It is not a claim about the project's current release, and not a safety certification.
- We measured the behaviour against our own expectation for agent runs: suspending a user should stop actions already in flight.
- We used Scopehaven's approach, not the shipped product; the 14 of 14 retest is a dated pilot result, not a current Scopehaven product pass.
Where Scopehaven fits #
Scopehaven is a local assessment workbench for testing the permissions you define, on the actions you check, and rerunning the same checks to show whether a fix held, including the actions that should still work. Runs and evidence stay on your machine; the workbench does not upload them to Scopehaven.
Our synthetic demo suite includes a matching check, MIDTASK-01: access removed mid-task stops the pending action.
On your own app, what a run can establish depends on the connection and the evidence available. Scopehaven does not certify your application or guarantee it is free from vulnerabilities.
Common questions #
Which project was it? #
We're not naming the project. The same kind of gap can occur in other agent systems, and the steps above are one way to check yours.
Is this a claim about that project's security? #
No. We measured one behaviour in one pinned version against our own expectation for agent runs: suspending a user should stop actions already in flight. It is not a claim about the project's current release.
How far did the repair go? #
It was a partial, local repair. The same 14 checks then passed, including the ones confirming that normal use still works. We verified it only on the file-write path we tested.
Does this cover hosted assistants? #
We tested one self-hosted app on our own machine; this note does not cover hosted assistants.