{"slug": "a-blind-scan-found-ibm-contextforge-s-connection-leak-from-one-file", "title": "A Blind Scan Found IBM ContextForge's Connection Leak From One File", "summary": "A 3,000-user load test against IBM's open-source ContextForge MCP gateway exposed a connection leak that left 42 to 81 PostgreSQL connections stuck in 'idle in transaction' until a 120-second timeout drained the pool. The culprit was three lines in mcpgateway/middleware/token_scoping.py, where a failing db.commit() inside a finally block prevented db.close() from running; PR #1886 nested the commit in its own try so close always executes, dropping stuck connections to zero and cutting release time to 1-2 seconds. A blind scan of the pre-fix file by a tool called Tomosu, given only the 755-line file and a sealed success criterion, independently flagged the unsafe session lifecycle as a high-severity finding.", "body_md": "On January 3, 2026, a 3,000-user load test against [IBM ContextForge](https://github.com/IBM/mcp-context-forge) left PostgreSQL connections stuck in `idle in transaction` until the pool ran dry. The cause was three lines of cleanup code. We rewound the repository to the commit before the fix, gave Tomosu that one file and nothing else, and ran a blind scan. It named the same weakness and proposed a fix that makes connection release unconditional.\n\n| 42 to 81 | 120 s | 27 / 100 | Direct hit | \n|---|---|---|---|\n| connections stuck `idle in transaction` | held before a timeout released them | PRI on the blind scan | against a criterion sealed before scanning | \n\nContextForge is an open-source gateway and registry for MCP, A2A, REST and gRPC. Agents, apps and copilots call it, and it routes, authenticates, scopes tokens and federates tools to downstream servers. Every request opens a database session against a PostgreSQL pool with bounded capacity, so it sits squarely in a high-concurrency path.\n\nThe team ran 3,000 concurrent users to surface behavior ordinary development never hits. Requests looked healthy, but connections stopped cycling. Between 42 and 81 sat in `idle in transaction`: checked out, doing nothing, unavailable to anyone else. Each one stayed stuck until PostgreSQL's `idle_in_transaction_session_timeout` killed it at 120 seconds. Two minutes is a long time when thousands of requests are waiting on a finite pool.\n\nThe stuck query was a `SELECT` from the token-scoping middleware ([issue #1885](https://github.com/IBM/mcp-context-forge/issues/1885)).\n\nThe investigation led to `mcpgateway/middleware/token_scoping.py`. It looks fine at a glance: a `finally`, a commit, a close.\n\n```\nfinally:\n    db.commit()   # can raise\n    db.close()    # never reached if commit raises\n```\n\n`finally` guarantees the block starts, not that every statement in it finishes. `db.commit()` can fail (a PgBouncer timeout, a connection reset). When it raises, Python leaves the block and `close()` never runs. Under enough concurrency, three lines become pool exhaustion.\n\nThe issue lists the same pattern in four files, about 24 instances in total, with `admin.py` holding around 20 of them.\n\n[PR #1886](https://github.com/IBM/mcp-context-forge/pull/1886) nests the commit in its own `try` so the close survives a failing commit:\n\n```\nfinally:\n    try:\n        db.commit()\n    finally:\n        db.close()   # always runs\n```\n\nUnder the same 3,000-user load:\n\n|  | Before the fix | After the fix | \n|---|---|---|\n| Stuck connections | 42 to 81 | 0 | \n| Connection release | Held up to 120 s | Cycling in 1 to 2 s | \n| Pool state | Exhausted | Healthy | \n\nThe rule underneath: if you acquire a finite production resource, its release has to survive every failure path.\n\nEverything the team learned came from production-scale conditions. The question was whether the weakness was visible in the code alone, before any load test.\n\nWe checked out commit `cf5adcd5d`, the code exactly as it stood before the fix, and gave Tomosu one file: `token_scoping.py`, 755 lines. Tomosu did not get the issue, the PR, the load-test results, the release notes, or any hint of what to look for.\n\nBefore scanning, we sealed the success criterion so the result couldn't be reinterpreted afterwards:\n\nIdentify unsafe database-session cleanup where an operation before `close()` can fail and prevent reliable connection release.\n\n**Direct hit:** names the unsafe session lifecycle on this path. **Adjacent:** generic exception-handling or database advice. **Miss:** anything else.\n\nOne file, 1 minute 41 seconds, 5 findings, a Production Reliability Index of 27 out of 100. One of the five, rated high severity:\n\n**Unsafe database-session lifecycle.** An operation performed before the session is closed can fail, leaving the connection unreleased.\n\nIt names the cleanup path, says the preceding operation can fail, and ties that to connection release. All three checks in the criterion pass.\n\nTomosu's suggested change went a different way from IBM's. This path only reads (the stuck query was a `SELECT`), so it drops the commit entirely:\n\n```\nfinally:\n    db.close()\n```\n\nThree versions of the same block, one invariant:\n\n`finally`.\nIt shows the weakness was readable from the source, with no incident context, before anyone ran a load test.\n\nIt doesn't show Tomosu would have found all ~24 instances: we scanned one file, and we chose that file knowing where the bug was. A full-repository blind scan at the same commit is the stronger test. Static analysis also doesn't replace load testing. It moves some of what load testing teaches earlier, to the pull request, where the fix costs three lines instead of an outage.\n\nIf you use SQLAlchemy sessions by hand, search your code for `commit()` followed by `close()` in the same `finally`. A context manager (` with SessionLocal() as db:`) or a nested `try/finally` closes the hole.\n\nTomosu scores each pull request with a 0 to 100 Production Reliability Index before it merges. [Run a free scan](https://tomosu.ai/start?utm_source=devto&utm_medium=post&utm_campaign=contextforge-case-study) on one public repository, or read the [full case study](https://tomosu.ai/case-studies/contextforge-connection-leak.html?utm_source=devto&utm_medium=post&utm_campaign=contextforge-case-study) with the 22-slide forensics deck.\n\n*ContextForge is an open-source project. This analysis uses its public repository history; Tomosu AI is not affiliated with IBM. Originally published on [tomosu.ai](https://tomosu.ai/case-studies/contextforge-connection-leak.html).*", "url": "https://wpnews.pro/news/a-blind-scan-found-ibm-contextforge-s-connection-leak-from-one-file", "canonical_source": "https://dev.to/parsa_m/a-blind-scan-found-ibm-contextforges-connection-leak-from-one-file-1cnh", "published_at": "2026-10-21 15:31:45+00:00", "updated_at": "2026-10-21 15:40:03.543074+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "mlops", "developer-tools", "ai-infrastructure"], "entities": ["IBM", "ContextForge", "PostgreSQL", "Tomosu", "PgBouncer", "MCP", "A2A"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/a-blind-scan-found-ibm-contextforge-s-connection-leak-from-one-file", "markdown": "https://wpnews.pro/news/a-blind-scan-found-ibm-contextforge-s-connection-leak-from-one-file.md", "text": "https://wpnews.pro/news/a-blind-scan-found-ibm-contextforge-s-connection-leak-from-one-file.txt", "jsonld": "https://wpnews.pro/news/a-blind-scan-found-ibm-contextforge-s-connection-leak-from-one-file.jsonld"}}