The pool that got slower the more we shared it

A .NET developer discovered that using ArrayPool<byte>.Shared across multiple threads caused a 20% CPU increase in a market-data-gateway service. The contention was hidden inside the framework's own lock in the shared pool's Rent/Return methods, which static analysis could not detect. The dotnet-diagnostics-mcp tool identified the root cause by tracing CPU samples to Monitor.Enter_Slowpath, revealing the lock inside ArrayPool<byte>.Shared.

How a 15-minute chat with an LLM found a hidden ArrayPool<byte .Shared contention point that no static analyzer would ever flag — and why the fix had nothing to do with adding a lock. "CPU on market-data-gateway climbed ~20% after we added a broadcast fan-out path last sprint. No new allocations, no GC pressure, nothing obviously wrong in the diff. Can someone take a look before we scale out more pods?" Nobody had touched anything that looked like a synchronization primitive. The diff that shipped the week before only touched a buffer-pooling helper — swapping a bespoke byte-array cache for the "obviously correct" BCL default, ArrayPool<byte .Shared . That refactor was reviewed and merged specifically because it removed custom code in favor of a well-known, heavily optimized framework type. If anything, it should have been a free win. This is the kind of regression static analysis is structurally blind to: no analyzer rule says "using ArrayPool<T .Shared across threads is suspicious." It's the recommended way to pool buffers in .NET. The problem isn't in the code you'd read in a diff — it's in a runtime behavior that only shows up under a specific, easy-to-miss usage pattern. This is the real value proposition of dotnet-diagnostics-mcp https://github.com/pedrosakuma/dotnet-diagnostics : Step 1 — orient. user Can you check what's going on with pid 41213? CPU looks high. assistant Let me look at the process first. → inspect process view="triage", processId=41213 { "processId": 41213, "resolvedProcess": { "name": "market-data-gateway", "runtime": "net10.0" }, "summary": "CPU 78% host has 16 cores . No memory growth trend. No recent GC pause anomalies. Consider a CPU sample to see where time goes.", "hints": "collect sample kind=\"cpu\", processId=41213, durationSeconds=10 " } Nothing screaming "leak," nothing screaming "GC." CPU is just... busy. Time to see where. Step 2 — sample. assistant Taking a 10-second CPU sample. → collect sample kind="cpu", processId=41213, durationSeconds=10, topN=20 { "processId": 41213, "totalSamples": 18545, "summary": "Hottest self-time method: System.Threading.Monitor.Enter Slowpath 2933 exclusive, 15.8% of samples . Rank self-time with query snapshot view=\"top-methods\" or walk the call path with view=\"call-tree\".", "handle": "A0CKKWH5VVS9Q68AH3GG", "hints": "query snapshot handle=\"A0CKKWH5VVS9Q68AH3GG\", view=\"caller-callee\", rootMethodFilter=\"Monitor.Enter Slowpath\" " } That's the first real surprise. 15.8% of all CPU samples land inside Monitor.Enter Slowpath. Nobody wrote a lock statement in this service. grep -r "lock " across the whole repo returns nothing relevant. So where is Step 3 — trace the lock back to its call site. assistant There's no explicit lock in this codebase. Let's see who's calling into the monitor slow path. → query snapshot handle="A0CKKWH5VVS9Q68AH3GG", view="caller-callee", rootMethodFilter="Monitor.Enter Slowpath" { "method": "System.Threading.Monitor.Enter Slowpath System.Object ", "exclusiveSamples": 2933, "exclusivePercent": 15.82, "callers": { "method": "MarketData.BufferPool.Return byte ", "samples": 1478, "percent": 7.97 }, { "method": "MarketData.Broadcaster+< c DisplayClass.<ConsumeLoop b 0 ", "samples": 1455, "percent": 7.85 } } Both callers are our code — a thin wrapper method calling .Rent on one side, and a per-connection consume loop calling .Return on the other. Not a ThreadPool internal, not GC. The lock is being taken inside ArrayPool<byte .Shared itself , and it's the Step 4 — read the actual lock, not a guess about it. ArrayPool<byte .Shared 's internals are in System.Private.CoreLib — no source, no PDB for the private implementation type. This is exactly what the companion server, dotnet-assembly-mcp https://github.com/pedrosakuma/dotnet-assembly-mcp , assistant Let's not guess. I'll look up the actual pool implementation in the runtime binary you're running. → find method mvidOrPath="System.Private.CoreLib.dll", namePattern="^Return$" { "summary": "5 match es for /^Return$/.", "matches": { "typeFullName": "System.Buffers.SharedArrayPool 1", "methodName": "Return", "signature": "void System.Buffers.SharedArrayPool 1.Return 0 , bool ", "handle": "m:e8c78f6b-...:0x060038A7" } } assistant Decompiling the real Return method for this exact runtime build. → decompile method moduleVersionId="e8c78f6b-...", metadataToken="0x060038A7" public override void Return T array, bool clearArray = false { // ... SharedArrayPoolThreadLocalArray array2 = t tlsBuckets ?? InitializeTlsBucketsAndTrimming ; // ... same-thread fast-path slot swap ... if array3 = null { flag2 = buckets num ?? CreatePerCorePartitions num .TryPush array3 ; } // ... } assistant And the fallback path that pushes to a per-core partition: → find method mvidOrPath="System.Private.CoreLib.dll", namePattern="^TryPush$" → decompile method moduleVersionId="e8c78f6b-...", metadataToken="0x060038B3" MethodImpl MethodImplOptions.AggressiveInlining public bool TryPush Array array { bool result = false; Monitor.Enter this ; // ... push onto a small fixed-size array-backed stack ... Monitor.Exit this ; return result; } There it is — Monitor.Enter this , in the flesh, in the exact binary ArrayPool<byte .Shared in this runtime build is SharedArrayPool<T , and Monitor.Enter Slowpath : it's a real, ordinary Return misses the per-threadTo be clear: dotnet/runtime 's source is public on GitHub — nothing here was hidden. The point isn't that the source was unreachable, it's the ergonomics of the query. Getting the same answer the "normal" way means: know or guess which internal type backs ArrayPool<T .Shared today, find the right tag in dotnet/runtime that matches the exact SDK/runtime patch this process is actually running, navigate to the right file, and read enough surrounding code to trust you're looking at the live behavior and not a since-changed version. decompile method moduleVersionId, metadataToken collapses all of that into one call, scoped to the exact assembly loaded in the exact process under investigation — no version-matching, no guessing a type name, no context-switch out of the chat. Checking the application code only now , after both tools already pointed at the exact mechanism, not before : // One dedicated broadcaster thread — the ONLY thread that ever Rents. while running { var buffer = ArrayPool<byte .Shared.Rent FrameSize ; FillFromMarketFeed buffer ; channel.Writer.TryWrite buffer ; } // N per-connection consumer tasks — each may Return a buffer it never Rented. await foreach var buffer in channel.Reader.ReadAllAsync { await connection.WriteAsync buffer ; ArrayPool<byte .Shared.Return buffer ; // <-- always a different thread than Rent } Exactly the pattern the tool output implied: one thread rents, many other threads return. ArrayPool<T .Shared isn't a naive shared bag — its current .NET 10 implementation, SharedArrayPool<T , keeps a single-slot cache stored per-thread ThreadStatic buckets : if the Monitor None of this is a bug in ArrayPool<T . It's working exactly as designed — optimized for the common case rent-use-return on one thread, e.g. inside a single request handler , and gracefully degrading, but not silently free , for the cross-thread fan-out case. There is no compiler warning, no analyzer rule, and no code-review heuristic for "you are about to violate an implicit performance assumption of a BCL type you didn't write." The only way to know, for certain, what this process, running this runtime build, is actually doing is to watch where the CPU goes and read the exact binary it's running. The fix isn't "add a lock" there's nothing to protect — it's swap the pool implementation to one designed for the actual ownership pattern: many threads returning items nobody-in-particular rented, i.e., a plain lock-free queue. public sealed class CrossThreadBufferPool { private readonly ConcurrentQueue<byte pool = new ; public byte Rent int size = pool.TryDequeue out var buf ? buf : new byte size ; public void Return byte buffer = pool.Enqueue buffer ; } Isolated cost, before/after BenchmarkDotNet, 20,000 cross-thread Return calls, buffers pre-rented on a background thread outside the timed region so only Return itself is measured : | Scenario | Mean time 20,000 calls | vs. contended baseline | |---|---|---| ArrayPool.Shared.Return , cross-thread the bug | ~4.3–5.1 ms | 1x baseline | ConcurrentQueue<byte Return , cross-thread the fix | ~126–146 µs | ~34x faster | ArrayPool.Shared full Rent+Return, same thread — theoretical best case | ~115–150 µs | ~34–40x faster | The last row is the ceiling: the one scenario with genuinely zero synchronization, because the per-thread cache slot is always hit. Comparing it to the ConcurrentQueue fix shows exactly how much overhead our own fix still carries: ~126 µs vs. ~115 µs — roughly 10% above the theoretical best , for a data structure built to be used safely by an arbitrary number A single cross-thread Return call on the unfixed pool costs more than an entire same-thread Rent+Return round trip. That's how disproportionate the fast-path miss is. Live re-check after the fix — same 2-thread 1 producer / 1 consumer topology, same CPU sampler, before touching anything else: | Metric | ArrayPool.Shared cross-thread | Lock-free pool cross-thread | Same-thread control | |---|---|---|---| | Throughput | ~3.2M ops/s | ~6.6M ops/s | ~20M ops/s | Monitor.Enter Slowpath in CPU sample | 15.8% exclusive | 0% — gone entirely | n/a no shared pool contention path | assistant With the lock-free pool under the same load: → collect sample kind="cpu", processId=41213, durationSeconds=8, topN=20 { "summary": "Hottest self-time method: System.Threading.Thread.Sleep int32 3972 exclusive, 40.6% of samples . No Monitor frames in the top 20 hotspots.", ... } No Monitor.Enter Slowpath anywhere in the profile. The contention didn't move — it disappeared, because the pool that produced it is gone. ArrayPool<T .Shared itself exists as an optimization over the naive baseline — just calling new byte size and letting the GC reclaim it. A fair objection to everything above: if the shared pool is this contended under cross-thread Return , maybe the honest fix is to drop pooling entirely, not swap in a different pool. Mean latency might even look competitive, since a plain allocation has zero synchronization of any kind — but the expectation is that Gen0/Gen1 GC pressure would show up in the tail P99 , not the mean. We tested it — same 2-thread producer/consumer topology, only the pool implementation swapped for one that allocates fresh on every Rent and does nothing on Return : | Scenario | Throughput | Monitor contention | |---|---|---| ArrayPool.Shared , cross-thread | ~3.2M ops/s | 15.8% Monitor | | Lock-free queue pool, cross-thread | ~6.6M ops/s | 0% | No pooling new byte size every call | ~180K ops/s | 0% | No pooling isn't a tail-latency-only regression — it's ~18x slower on raw throughput , worse than the contended ArrayPool.Shared baseline we started assistant Let's check what removing the pool entirely does to GC. → collect events kind="gc", processId=41213, durationSeconds=10 { "summary": "134 collection s , max pause 69.9ms, total pause 5118.1ms." } assistant Breaking that down: → query snapshot handle="...", view="pauseHistogram" { "totalCollections": 134, "maxPauseTime": "00:00:00.0698743", "buckets": { "label": "<1ms", "count": 0 }, { "label": "1-10ms", "count": 0 }, { "label": "10-100ms", "count": 134 }, { "label": "100-1000ms","count": 0 }, { "label": " =1s", "count": 0 } } assistant And by generation: → query snapshot handle="...", view="byGeneration" { "generations": { "bucket": "gen1", "count": 134, "totalPause": "00:00:05.1180759", "meanPause": "00:00:00.0381945", "maxPause": "00:00:00.0698743" } } 134 gen1 collections in a 10-second window — 13.4 per second, every single one landing in the 10–100ms bucket — for 5.12 seconds of accumulated pause time inside those 10 seconds. Over half the wall clock is GC pause, not So the honest ranking, worst to best, for this specific cross-thread fan-out pattern is: no pooling worst, by both mean and tail → shared pool with cross-thread Return contended but bounded → a pool built for the actual ownership shape best on every axis measured . ArrayPool<T .Shared Static analysis, linters, and code review are excellent at catching shapes of bugs — a missing await , a lock around the wrong object, an O n² loop. They cannot catch a case where the code is a textbook-correct call into a well-designed BCL type, and the only thing wrong is a runtime ownership pattern that BCL type happens to be sensitive to. You either know this specific gotcha already, or you find it by watching the process — there is no third option. dotnet-diagnostics-mcp exists so that "watching the process" is a normal step in an LLM-driven investigation, not a specialized skill gated behind whoever on the team happens to know dotnet-trace by heart. Every dynamic step above — inspect process , collect sample , query snapshot — is a single MCP tool call an assistant can make on its own, against a real running .NET process, with no code changes and no redeploy. And when the trail leads into a BCL type nobody on the team has source for, its companion server, dotnet-assembly-mcp https://github.com/pedrosakuma/dotnet-assembly-mcp , The reproduction and all numbers in this post are in a small, standalone, open repro you can run yourself — no proprietary code involved. The investigation pattern itself fix unrelated to this repro is inspired by a real production change in B3MarketDataPlatform https://github.com/pedrosakuma/B3MarketDataPlatform , ac2dc9f dotnet-diagnostics-mcp is open source: github.com/pedrosakuma/dotnet-diagnostics https://github.com/pedrosakuma/dotnet-diagnostics . It ships both an MCP server for your LLM client of choice and a standalone CLI dotnet-diagnostics-cli for scripting the same investigations by hand.