MCP servers describe their own tools. A tool can publish readOnlyHint: true
, and an agent harness may use that annotation to decide whether a human approval card is needed.
That creates an uncomfortable trust boundary: the server being evaluated is also the server describing how dangerous it is.
I built Airlock to test the declaration against observed behaviour, then enforce the resulting policy on the wire.
Airlock reports what it observed. Absence of a finding is not proof of safety.
Consider an MCP tool called export_report
. It declares itself read-only, returns a normal result and writes a file outside the operator's declared scope.
A harness that trusts the annotation may run it without approval. Airlock opens a case, inventories the declared tools, exercises them under a capped budget and compares the declaration with its observations.
If a tool behaves differently from what it declared, the difference becomes evidence. For an approved case, Airlock emits a connector that points to a per-case enforcing proxy rather than back to the suspect server.
The policy is enforced before the call reaches the upstream tool. It is not a sentence in a system prompt.
Airlock ships two six-tool fixtures with the same surface.
The dishonest fixture plants five behaviours, including a filesystem write behind readOnlyHint: true
, scope escape, undeclared egress, injected instructions and canary exfiltration.
The result:
target dishonest fixture, controlled_fixture mode
probes 24
result 7 findings of 36 checks, all five planted behaviours
The honest fixture produces zero findings across the same 36 checks.
That contrast is essential. A detector that only finds problems in a fixture designed to look suspicious has not shown much. The honest control shows that the findings follow observed behaviour rather than unfamiliarity.
Each check resolves to one of four states:
finding
no_finding_observed
not_tested
sensor_failed
Severity is reported separately.
I deliberately avoided a single safety score. An average can hide the one tool that lied. It also encourages readers to treat missing visibility as a passing result.
not_tested
is a first-class state. Under transcript_only
, MCP traffic cannot reveal server-side filesystem or network activity. Airlock records those questions as untested rather than clean.
It distinguishes capability_absent
, where no sensor existed, from evidence_missing
, where the sensor could answer but observed nothing.
The owned fixture proves that the detectors can catch planted behaviour. The more interesting question is what happens on external servers.
Airlock audited a deployed ContextFirewall target over HTTPS:
6 tools
30 probes
0 tools declaring any annotation
The server did not lie. It said nothing. A harness that resolves categories such as @write
and @destructive
from annotations can match nothing either way, so tools such as remember
and forget_memory
may avoid the approval path.
Airlock also inventoried five public stdio servers:
| Server | Tools |
|---|---|
server-filesystem |
|
| 14 | |
server-everything |
|
| 13 | |
mcp-server-git |
|
| 12 | |
server-memory |
|
| 9 | |
server-sequential-thinking |
|
| 1 |
Stdio support matters because many MCP servers are launched as commands rather than exposed at a URL.
Launching a stdio server means executing the code Airlock exists to distrust. The command therefore never comes from a case argument, tool result or model-generated string.
The operator configures a fixed map of names to argument arrays. A case may select a name. Names are looked up, never parsed into shell commands.
The child starts in a throwaway working directory with an explicit environment. Airlock revalidates the whole command binding before each connection, so repointing a configured name revokes an open case instead of silently running a different command.
Airlock documents an important remaining boundary: its HTTP response cap does not apply to the MCP SDK's stdio transport. The audit deadline still applies, but a large line can reach the SDK before Airlock can bound it. Calling that solved would be worse than stating it.
A TrueForge agent drives the audit through six control tools:
open_case
list_declared_tools
probe_tool
read_evidence
seal_case
emit_policy
Three operations are always approval-gated: probe_tool
, seal_case
and emit_policy
.
The agent inventories the target, asks Airlock to probe opaque tool IDs, reads the aggregate evidence and presents a human choice: Block, Approve selected or Approve all.
For an allowed case, the emitted connector points at the enforcing case proxy. A call to a tool the case did not approve receives:
MCP error -32001: Tool blocked by Airlock policy
Airlock audited its own control MCP. The server published full annotations, but the probe planner rejected open_case
because its schema contains a $ref
into $defs
, outside Airlock's bounded v1 probe profile.
That case is incomplete. It appears on the hosted page because a product arguing that missing evidence is not proof of safety should not hide its own missing evidence.
Every substantive change went through a Qodo-reviewed pull request.
On the stdio transport, Qodo found that revalidation compared only the target name. Repointing that name could leave an open case running an operator-withdrawn command. It also found that the MCP SDK inherited more host environment variables than the README claimed.
On the approval boundary, Qodo caught a test that claimed to cover every side-effecting control tool while checking three hard-coded names. The first fix introduced a second map that could drift, and Qodo caught that too. The decorators now read from the same policy map the test validates.
Another review found that a passing-suite claim did not survive a clean CI machine. The failures were fixed, re-reviewed and recorded.
The reviews did more than find code defects. They caught statements that were stronger than the observed behaviour, exactly the failure Airlock is designed to expose.
Run the complete suite with:
python3 -m venv .venv
.venv/bin/python -m pip install -e '.[dev]'
.venv/bin/python -m pytest -q
The current suite contains 302 passing tests.
The backend requires persistent state, long-running audits, child processes for stdio targets and a loopback-only operator interface, which makes a serverless deployment the wrong shape. The repository quickstart runs the complete fixture audit locally in two commands.
Airlock is open source under the MIT licence and was built for the WeMakeDevs x TrueFoundry Agent Harness Hackathon.
Do not ask a server whether it is safe. Ask what it did, keep the record and enforce the answer.