# The pool that got slower the more we shared it > Source: > Published: 2026-07-14 18:08:47+00:00 *How a 15-minute chat with an LLM found a hidden ArrayPool.Shared contention point that no static analyzer would ever flag — and why the fix had nothing to do with adding a lock.* "CPU on `market-data-gateway` climbed ~20% after we added a broadcast fan-out path last sprint. No new allocations, no GC pressure, nothing obviously wrong in the diff. Can someone take a look before we scale out more pods?" Nobody had touched anything that *looked* like a synchronization primitive. The diff that shipped the week before only touched a buffer-pooling helper — swapping a bespoke byte-array cache for the "obviously correct" BCL default, `ArrayPool.Shared` . That refactor was reviewed and merged specifically *because* it removed custom code in favor of a well-known, heavily optimized framework type. If anything, it should have been a free win. This is the kind of regression static analysis is structurally blind to: no analyzer rule says "using `ArrayPool.Shared` across threads is suspicious." It's *the recommended way* to pool buffers in .NET. The problem isn't in the code you'd read in a diff — it's in a runtime behavior that only shows up under a specific, easy-to-miss usage pattern. This is the real value proposition of [ dotnet-diagnostics-mcp](https://github.com/pedrosakuma/dotnet-diagnostics): **Step 1 — orient.** ``` user> Can you check what's going on with pid 41213? CPU looks high. assistant> Let me look at the process first. → inspect_process(view="triage", processId=41213) { "processId": 41213, "resolvedProcess": { "name": "market-data-gateway", "runtime": "net10.0" }, "summary": "CPU 78% (host has 16 cores). No memory growth trend. No recent GC pause anomalies. Consider a CPU sample to see where time goes.", "hints": [ "collect_sample(kind=\"cpu\", processId=41213, durationSeconds=10)" ] } ``` Nothing screaming "leak," nothing screaming "GC." CPU is just... busy. Time to see where. **Step 2 — sample.** ``` assistant> Taking a 10-second CPU sample. → collect_sample(kind="cpu", processId=41213, durationSeconds=10, topN=20) { "processId": 41213, "totalSamples": 18545, "summary": "Hottest self-time method: System.Threading.Monitor.Enter_Slowpath (2933 exclusive, 15.8% of samples). Rank self-time with query_snapshot(view=\"top-methods\") or walk the call path with view=\"call-tree\".", "handle": "A0CKKWH5VVS9Q68AH3GG", "hints": [ "query_snapshot(handle=\"A0CKKWH5VVS9Q68AH3GG\", view=\"caller-callee\", rootMethodFilter=\"Monitor.Enter_Slowpath\")" ] } ``` That's the first real surprise. **15.8% of all CPU samples land inside Monitor.Enter_Slowpath.** Nobody wrote a `lock` statement in this service.`grep -r "lock ("` across the whole repo returns nothing relevant. So where is**Step 3 — trace the lock back to its call site.** ``` assistant> There's no explicit lock in this codebase. Let's see who's calling into the monitor slow path. → query_snapshot(handle="A0CKKWH5VVS9Q68AH3GG", view="caller-callee", rootMethodFilter="Monitor.Enter_Slowpath") { "method": "System.Threading.Monitor.Enter_Slowpath(System.Object)", "exclusiveSamples": 2933, "exclusivePercent": 15.82, "callers": [ { "method": "MarketData.BufferPool.Return(byte[])", "samples": 1478, "percent": 7.97 }, { "method": "MarketData.Broadcaster+<>c__DisplayClass.b__0()", "samples": 1455, "percent": 7.85 } ] } ``` Both callers are *our* code — a thin wrapper method calling `.Rent()` on one side, and a per-connection consume loop calling `.Return()` on the other. Not a `ThreadPool` internal, not GC. The lock is being taken **inside ArrayPool.Shared itself**, and it's the **Step 4 — read the actual lock, not a guess about it.** `ArrayPool.Shared` 's internals are in `System.Private.CoreLib` — no source, no PDB for the private implementation type. This is exactly what the companion server, [ dotnet-assembly-mcp](https://github.com/pedrosakuma/dotnet-assembly-mcp), ``` assistant> Let's not guess. I'll look up the actual pool implementation in the runtime binary you're running. → find_method(mvidOrPath="System.Private.CoreLib.dll", namePattern="^Return$") { "summary": "5 match(es) for /^Return$/.", "matches": [ { "typeFullName": "System.Buffers.SharedArrayPool`1", "methodName": "Return", "signature": "void System.Buffers.SharedArrayPool`1.Return(!0[], bool)", "handle": "m:e8c78f6b-...:0x060038A7" } ] } assistant> Decompiling the real Return method for this exact runtime build. → decompile_method(moduleVersionId="e8c78f6b-...", metadataToken="0x060038A7") public override void Return(T[] array, bool clearArray = false) { // ... SharedArrayPoolThreadLocalArray[] array2 = t_tlsBuckets ?? InitializeTlsBucketsAndTrimming(); // ... same-thread fast-path slot swap ... if (array3 != null) { flag2 = (_buckets[num] ?? CreatePerCorePartitions(num)).TryPush(array3); } // ... } assistant> And the fallback path that pushes to a per-core partition: → find_method(mvidOrPath="System.Private.CoreLib.dll", namePattern="^TryPush$") → decompile_method(moduleVersionId="e8c78f6b-...", metadataToken="0x060038B3") [MethodImpl(MethodImplOptions.AggressiveInlining)] public bool TryPush(Array array) { bool result = false; Monitor.Enter(this); // ... push onto a small fixed-size array-backed stack ... Monitor.Exit(this); return result; } ``` There it is — ** Monitor.Enter(this)**, in the flesh, in the exact binary `ArrayPool.Shared` in this runtime build is `SharedArrayPool` , and`Monitor.Enter_Slowpath` : it's a real, ordinary`Return` misses the per-threadTo be clear: `dotnet/runtime` 's source is public on GitHub — nothing here was hidden. The point isn't that the source was unreachable, it's the *ergonomics* of the query. Getting the same answer the "normal" way means: know (or guess) which internal type backs `ArrayPool.Shared` today, find the right tag in `dotnet/runtime` that matches the exact SDK/runtime patch this process is actually running, navigate to the right file, and read enough surrounding code to trust you're looking at the live behavior and not a since-changed version. `decompile_method(moduleVersionId, metadataToken)` collapses all of that into one call, scoped to the exact assembly loaded in the exact process under investigation — no version-matching, no guessing a type name, no context-switch out of the chat. Checking the *application* code (only *now*, after both tools already pointed at the exact mechanism, not before): ``` // One dedicated broadcaster thread — the ONLY thread that ever Rents. while (running) { var buffer = ArrayPool.Shared.Rent(FrameSize); FillFromMarketFeed(buffer); channel.Writer.TryWrite(buffer); } // N per-connection consumer tasks — each may Return a buffer it never Rented. await foreach (var buffer in channel.Reader.ReadAllAsync()) { await connection.WriteAsync(buffer); ArrayPool.Shared.Return(buffer); // <-- always a different thread than Rent } ``` Exactly the pattern the tool output implied: **one thread rents, many other threads return.** `ArrayPool.Shared` isn't a naive shared bag — its current .NET 10 implementation, `SharedArrayPool` , keeps a **single-slot cache stored per-thread** ( `[ThreadStatic]` buckets): if the `Monitor` None of this is a bug in `ArrayPool` . It's working exactly as designed — optimized for the common case (rent-use-return on one thread, e.g. inside a single request handler), and gracefully degrading, but *not silently free*, for the cross-thread fan-out case. There is no compiler warning, no analyzer rule, and no code-review heuristic for "you are about to violate an implicit performance assumption of a BCL type you didn't write." The only way to know, for certain, what *this* process, running *this* runtime build, is actually doing is to watch where the CPU goes and read the exact binary it's running. The fix isn't "add a lock" (there's nothing to protect) — it's swap the pool implementation to one designed for the actual ownership pattern: many threads returning items nobody-in-particular rented, i.e., a plain lock-free queue. ``` public sealed class CrossThreadBufferPool { private readonly ConcurrentQueue _pool = new(); public byte[] Rent(int size) => _pool.TryDequeue(out var buf) ? buf : new byte[size]; public void Return(byte[] buffer) => _pool.Enqueue(buffer); } ``` **Isolated cost, before/after** (BenchmarkDotNet, 20,000 cross-thread `Return` calls, buffers pre-rented on a background thread outside the timed region so only `Return` itself is measured): | Scenario | Mean time (20,000 calls) | vs. contended baseline | |---|---|---| `ArrayPool.Shared.Return` , cross-thread (the bug) | ~4.3–5.1 ms | 1x (baseline) | `ConcurrentQueue` `Return` , cross-thread (the fix) | ~126–146 µs | ~34x faster | `ArrayPool.Shared` full Rent+Return, same thread — theoretical best case | ~115–150 µs | ~34–40x faster | The last row is the ceiling: the one scenario with genuinely zero synchronization, because the per-thread cache slot is always hit. Comparing it to the `ConcurrentQueue` fix shows exactly how much overhead our own fix still carries: **~126 µs vs. ~115 µs — roughly 10% above the theoretical best**, for a data structure built to be used safely by an arbitrary number A single cross-thread `Return` call on the *unfixed* pool costs more than an entire same-thread Rent+Return round trip. That's how disproportionate the fast-path miss is. **Live re-check after the fix** — same 2-thread (1 producer / 1 consumer) topology, same CPU sampler, before touching anything else: | Metric | `ArrayPool.Shared` (cross-thread) | Lock-free pool (cross-thread) | Same-thread control | |---|---|---|---| | Throughput | ~3.2M ops/s | ~6.6M ops/s | ~20M ops/s | `Monitor.Enter_Slowpath` in CPU sample | 15.8% exclusive | 0% — gone entirely | n/a (no shared pool contention path) | ``` assistant> With the lock-free pool under the same load: → collect_sample(kind="cpu", processId=41213, durationSeconds=8, topN=20) { "summary": "Hottest self-time method: System.Threading.Thread.Sleep(int32) (3972 exclusive, 40.6% of samples). No Monitor frames in the top 20 hotspots.", ... } ``` No `Monitor.Enter_Slowpath` anywhere in the profile. The contention didn't move — it disappeared, because the pool that produced it is gone. `ArrayPool.Shared` itself exists as an optimization over the naive baseline — just calling `new byte[size]` and letting the GC reclaim it. A fair objection to everything above: if the *shared* pool is this contended under cross-thread `Return` , maybe the honest fix is to drop pooling entirely, not swap in a different pool. Mean latency might even look competitive, since a plain allocation has zero synchronization of any kind — but the expectation is that Gen0/Gen1 GC pressure would show up in the tail (P99), not the mean. We tested it — same 2-thread producer/consumer topology, only the pool implementation swapped for one that allocates fresh on every `Rent` and does nothing on `Return` : | Scenario | Throughput | `Monitor` contention | |---|---|---| `ArrayPool.Shared` , cross-thread | ~3.2M ops/s | 15.8% (Monitor) | | Lock-free queue pool, cross-thread | ~6.6M ops/s | 0% | No pooling (`new byte[size]` every call) | ~180K ops/s | 0% | No pooling isn't a tail-latency-only regression — it's **~18x slower on raw throughput**, worse than the contended `ArrayPool.Shared` baseline we started ``` assistant> Let's check what removing the pool entirely does to GC. → collect_events(kind="gc", processId=41213, durationSeconds=10) { "summary": "134 collection(s), max pause 69.9ms, total pause 5118.1ms." } assistant> Breaking that down: → query_snapshot(handle="...", view="pauseHistogram") { "totalCollections": 134, "maxPauseTime": "00:00:00.0698743", "buckets": [ { "label": "<1ms", "count": 0 }, { "label": "1-10ms", "count": 0 }, { "label": "10-100ms", "count": 134 }, { "label": "100-1000ms","count": 0 }, { "label": ">=1s", "count": 0 } ] } assistant> And by generation: → query_snapshot(handle="...", view="byGeneration") { "generations": [ { "bucket": "gen1", "count": 134, "totalPause": "00:00:05.1180759", "meanPause": "00:00:00.0381945", "maxPause": "00:00:00.0698743" } ] } ``` **134 gen1 collections in a 10-second window — 13.4 per second, every single one landing in the 10–100ms bucket — for 5.12 seconds of accumulated pause time inside those 10 seconds.** Over half the wall clock is GC pause, not So the honest ranking, worst to best, for this specific cross-thread fan-out pattern is: **no pooling (worst, by both mean and tail) → shared pool with cross-thread Return (contended but bounded) → a pool built for the actual ownership shape (best on every axis measured)**. `ArrayPool.Shared` Static analysis, linters, and code review are excellent at catching *shapes* of bugs — a missing `await` , a `lock` around the wrong object, an `O(n²)` loop. They cannot catch a case where the code is a textbook-correct call into a well-designed BCL type, and the only thing wrong is a runtime ownership pattern that BCL type happens to be sensitive to. You either know this specific gotcha already, or you find it by watching the process — there is no third option. `dotnet-diagnostics-mcp` exists so that "watching the process" is a normal step in an LLM-driven investigation, not a specialized skill gated behind whoever on the team happens to know `dotnet-trace` by heart. Every dynamic step above — `inspect_process` , `collect_sample` , `query_snapshot` — is a single MCP tool call an assistant can make on its own, against a real running .NET process, with no code changes and no redeploy. And when the trail leads into a BCL type nobody on the team has source for, its companion server, [ dotnet-assembly-mcp](https://github.com/pedrosakuma/dotnet-assembly-mcp), The reproduction and all numbers in this post are in a small, standalone, open repro you can run yourself — no proprietary code involved. The investigation pattern itself (fix unrelated to this repro) is inspired by a real production change in [ B3MarketDataPlatform](https://github.com/pedrosakuma/B3MarketDataPlatform), `ac2dc9f` `dotnet-diagnostics-mcp` is open source: [github.com/pedrosakuma/dotnet-diagnostics](https://github.com/pedrosakuma/dotnet-diagnostics). It ships both an MCP server (for your LLM client of choice) and a standalone CLI (`dotnet-diagnostics-cli` ) for scripting the same investigations by hand.