On January 3, 2026, a 3,000-user load test against IBM ContextForge left PostgreSQL connections stuck in idle in transaction until the pool ran dry. The cause was three lines of cleanup code. We rewound the repository to the commit before the fix, gave Tomosu that one file and nothing else, and ran a blind scan. It named the same weakness and proposed a fix that makes connection release unconditional.
| 42 to 81 | 120 s | 27 / 100 | Direct hit |
|---|---|---|---|
connections stuck idle in transaction |
held before a timeout released them | PRI on the blind scan | against a criterion sealed before scanning |
ContextForge is an open-source gateway and registry for MCP, A2A, REST and gRPC. Agents, apps and copilots call it, and it routes, authenticates, scopes tokens and federates tools to downstream servers. Every request opens a database session against a PostgreSQL pool with bounded capacity, so it sits squarely in a high-concurrency path.
The team ran 3,000 concurrent users to surface behavior ordinary development never hits. Requests looked healthy, but connections stopped cycling. Between 42 and 81 sat in idle in transaction: checked out, doing nothing, unavailable to anyone else. Each one stayed stuck until PostgreSQL's idle_in_transaction_session_timeout killed it at 120 seconds. Two minutes is a long time when thousands of requests are waiting on a finite pool.
The stuck query was a SELECT from the token-scoping middleware (issue #1885).
The investigation led to mcpgateway/middleware/token_scoping.py. It looks fine at a glance: a finally, a commit, a close.
finally:
db.commit() # can raise
db.close() # never reached if commit raises
finally guarantees the block starts, not that every statement in it finishes. db.commit() can fail (a PgBouncer timeout, a connection reset). When it raises, Python leaves the block and close() never runs. Under enough concurrency, three lines become pool exhaustion.
The issue lists the same pattern in four files, about 24 instances in total, with admin.py holding around 20 of them.
PR #1886 nests the commit in its own try so the close survives a failing commit:
finally:
try:
db.commit()
finally:
db.close() # always runs
Under the same 3,000-user load:
| Before the fix | After the fix | |
|---|---|---|
| Stuck connections | 42 to 81 | 0 |
| Connection release | Held up to 120 s | Cycling in 1 to 2 s |
| Pool state | Exhausted | Healthy |
The rule underneath: if you acquire a finite production resource, its release has to survive every failure path.
Everything the team learned came from production-scale conditions. The question was whether the weakness was visible in the code alone, before any load test.
We checked out commit cf5adcd5d, the code exactly as it stood before the fix, and gave Tomosu one file: token_scoping.py, 755 lines. Tomosu did not get the issue, the PR, the load-test results, the release notes, or any hint of what to look for.
Before scanning, we sealed the success criterion so the result couldn't be reinterpreted afterwards:
Identify unsafe database-session cleanup where an operation before close() can fail and prevent reliable connection release.
Direct hit: names the unsafe session lifecycle on this path. Adjacent: generic exception-handling or database advice. Miss: anything else.
One file, 1 minute 41 seconds, 5 findings, a Production Reliability Index of 27 out of 100. One of the five, rated high severity:
Unsafe database-session lifecycle. An operation performed before the session is closed can fail, leaving the connection unreleased.
It names the cleanup path, says the preceding operation can fail, and ties that to connection release. All three checks in the criterion pass.
Tomosu's suggested change went a different way from IBM's. This path only reads (the stuck query was a SELECT), so it drops the commit entirely:
finally:
db.close()
Three versions of the same block, one invariant:
finally.
It shows the weakness was readable from the source, with no incident context, before anyone ran a load test.
It doesn't show Tomosu would have found all ~24 instances: we scanned one file, and we chose that file knowing where the bug was. A full-repository blind scan at the same commit is the stronger test. Static analysis also doesn't replace load testing. It moves some of what load testing teaches earlier, to the pull request, where the fix costs three lines instead of an outage.
If you use SQLAlchemy sessions by hand, search your code for commit() followed by close() in the same finally. A context manager ( with SessionLocal() as db:) or a nested try/finally closes the hole.
Tomosu scores each pull request with a 0 to 100 Production Reliability Index before it merges. Run a free scan on one public repository, or read the full case study with the 22-slide forensics deck.
ContextForge is an open-source project. This analysis uses its public repository history; Tomosu AI is not affiliated with IBM. Originally published on tomosu.ai.