{"slug": "the-pool-that-got-slower-the-more-we-shared-it", "title": "The pool that got slower the more we shared it", "summary": "A .NET developer discovered that using ArrayPool<byte>.Shared across multiple threads caused a 20% CPU increase in a market-data-gateway service. The contention was hidden inside the framework's own lock in the shared pool's Rent/Return methods, which static analysis could not detect. The dotnet-diagnostics-mcp tool identified the root cause by tracing CPU samples to Monitor.Enter_Slowpath, revealing the lock inside ArrayPool<byte>.Shared.", "body_md": "*How a 15-minute chat with an LLM found a hidden ArrayPool<byte>.Shared\ncontention point that no static analyzer would ever flag — and why the fix\nhad nothing to do with adding a lock.*\n\n\"CPU on\n\n`market-data-gateway`\n\nclimbed ~20% after we added a broadcast fan-out\n\npath last sprint. No new allocations, no GC pressure, nothing obviously\n\nwrong in the diff. Can someone take a look before we scale out more pods?\"\n\nNobody had touched anything that *looked* like a synchronization primitive.\n\nThe diff that shipped the week before only touched a buffer-pooling helper —\n\nswapping a bespoke byte-array cache for the \"obviously correct\" BCL default,\n\n`ArrayPool<byte>.Shared`\n\n. That refactor was reviewed and merged specifically\n\n*because* it removed custom code in favor of a well-known, heavily optimized\n\nframework type. If anything, it should have been a free win.\n\nThis is the kind of regression static analysis is structurally blind to: no\n\nanalyzer rule says \"using `ArrayPool<T>.Shared`\n\nacross threads is suspicious.\"\n\nIt's *the recommended way* to pool buffers in .NET. The problem isn't in the\n\ncode you'd read in a diff — it's in a runtime behavior that only shows up\n\nunder a specific, easy-to-miss usage pattern.\n\nThis is the real value proposition of [ dotnet-diagnostics-mcp](https://github.com/pedrosakuma/dotnet-diagnostics):\n\n**Step 1 — orient.**\n\n```\nuser> Can you check what's going on with pid 41213? CPU looks high.\n\nassistant> Let me look at the process first.\n→ inspect_process(view=\"triage\", processId=41213)\n{\n  \"processId\": 41213,\n  \"resolvedProcess\": { \"name\": \"market-data-gateway\", \"runtime\": \"net10.0\" },\n  \"summary\": \"CPU 78% (host has 16 cores). No memory growth trend. No recent\n               GC pause anomalies. Consider a CPU sample to see where time goes.\",\n  \"hints\": [\n    \"collect_sample(kind=\\\"cpu\\\", processId=41213, durationSeconds=10)\"\n  ]\n}\n```\n\nNothing screaming \"leak,\" nothing screaming \"GC.\" CPU is just... busy. Time to\n\nsee where.\n\n**Step 2 — sample.**\n\n```\nassistant> Taking a 10-second CPU sample.\n→ collect_sample(kind=\"cpu\", processId=41213, durationSeconds=10, topN=20)\n{\n  \"processId\": 41213,\n  \"totalSamples\": 18545,\n  \"summary\": \"Hottest self-time method: System.Threading.Monitor.Enter_Slowpath\n              (2933 exclusive, 15.8% of samples). Rank self-time with\n              query_snapshot(view=\\\"top-methods\\\") or walk the call path with\n              view=\\\"call-tree\\\".\",\n  \"handle\": \"A0CKKWH5VVS9Q68AH3GG\",\n  \"hints\": [\n    \"query_snapshot(handle=\\\"A0CKKWH5VVS9Q68AH3GG\\\", view=\\\"caller-callee\\\", rootMethodFilter=\\\"Monitor.Enter_Slowpath\\\")\"\n  ]\n}\n```\n\nThat's the first real surprise. **15.8% of all CPU samples land inside\nMonitor.Enter_Slowpath.** Nobody wrote a\n\n`lock`\n\nstatement in this service.`grep -r \"lock (\"`\n\nacross the whole repo returns nothing relevant. So where is**Step 3 — trace the lock back to its call site.**\n\n```\nassistant> There's no explicit lock in this codebase. Let's see who's calling\ninto the monitor slow path.\n→ query_snapshot(handle=\"A0CKKWH5VVS9Q68AH3GG\", view=\"caller-callee\",\n                  rootMethodFilter=\"Monitor.Enter_Slowpath\")\n{\n  \"method\": \"System.Threading.Monitor.Enter_Slowpath(System.Object)\",\n  \"exclusiveSamples\": 2933,\n  \"exclusivePercent\": 15.82,\n  \"callers\": [\n    {\n      \"method\": \"MarketData.BufferPool.Return(byte[])\",\n      \"samples\": 1478,\n      \"percent\": 7.97\n    },\n    {\n      \"method\": \"MarketData.Broadcaster+<>c__DisplayClass.<ConsumeLoop>b__0()\",\n      \"samples\": 1455,\n      \"percent\": 7.85\n    }\n  ]\n}\n```\n\nBoth callers are *our* code — a thin wrapper method calling `.Rent()`\n\non one\n\nside, and a per-connection consume loop calling `.Return()`\n\non the other.\n\nNot a `ThreadPool`\n\ninternal, not GC. The lock is being taken\n\n**inside ArrayPool<byte>.Shared itself**, and it's the\n\n**Step 4 — read the actual lock, not a guess about it.**\n\n`ArrayPool<byte>.Shared`\n\n's internals are in `System.Private.CoreLib`\n\n— no\n\nsource, no PDB for the private implementation type. This is exactly what the\n\ncompanion server, [ dotnet-assembly-mcp](https://github.com/pedrosakuma/dotnet-assembly-mcp),\n\n```\nassistant> Let's not guess. I'll look up the actual pool implementation\nin the runtime binary you're running.\n→ find_method(mvidOrPath=\"System.Private.CoreLib.dll\", namePattern=\"^Return$\")\n{\n  \"summary\": \"5 match(es) for /^Return$/.\",\n  \"matches\": [\n    { \"typeFullName\": \"System.Buffers.SharedArrayPool`1\", \"methodName\": \"Return\",\n      \"signature\": \"void System.Buffers.SharedArrayPool`1.Return(!0[], bool)\",\n      \"handle\": \"m:e8c78f6b-...:0x060038A7\" }\n  ]\n}\nassistant> Decompiling the real Return method for this exact runtime build.\n→ decompile_method(moduleVersionId=\"e8c78f6b-...\", metadataToken=\"0x060038A7\")\npublic override void Return(T[] array, bool clearArray = false)\n{\n    // ...\n    SharedArrayPoolThreadLocalArray[] array2 = t_tlsBuckets ?? InitializeTlsBucketsAndTrimming();\n    // ... same-thread fast-path slot swap ...\n    if (array3 != null)\n    {\n        flag2 = (_buckets[num] ?? CreatePerCorePartitions(num)).TryPush(array3);\n    }\n    // ...\n}\nassistant> And the fallback path that pushes to a per-core partition:\n→ find_method(mvidOrPath=\"System.Private.CoreLib.dll\", namePattern=\"^TryPush$\")\n→ decompile_method(moduleVersionId=\"e8c78f6b-...\", metadataToken=\"0x060038B3\")\n[MethodImpl(MethodImplOptions.AggressiveInlining)]\npublic bool TryPush(Array array)\n{\n    bool result = false;\n    Monitor.Enter(this);\n    // ... push onto a small fixed-size array-backed stack ...\n    Monitor.Exit(this);\n    return result;\n}\n```\n\nThere it is — ** Monitor.Enter(this)**, in the flesh, in the exact binary\n\n`ArrayPool<byte>.Shared`\n\nin this runtime build is `SharedArrayPool<T>`\n\n, and`Monitor.Enter_Slowpath`\n\n: it's a real, ordinary`Return`\n\nmisses the per-threadTo be clear: `dotnet/runtime`\n\n's source is public on GitHub — nothing here was\n\nhidden. The point isn't that the source was unreachable, it's the *ergonomics*\n\nof the query. Getting the same answer the \"normal\" way means: know (or guess)\n\nwhich internal type backs `ArrayPool<T>.Shared`\n\ntoday, find the right tag in\n\n`dotnet/runtime`\n\nthat matches the exact SDK/runtime patch this process is\n\nactually running, navigate to the right file, and read enough surrounding\n\ncode to trust you're looking at the live behavior and not a since-changed\n\nversion. `decompile_method(moduleVersionId, metadataToken)`\n\ncollapses all of\n\nthat into one call, scoped to the exact assembly loaded in the exact process\n\nunder investigation — no version-matching, no guessing a type name, no\n\ncontext-switch out of the chat.\n\nChecking the *application* code (only *now*, after both tools already\n\npointed at the exact mechanism, not before):\n\n```\n// One dedicated broadcaster thread — the ONLY thread that ever Rents.\nwhile (running)\n{\n    var buffer = ArrayPool<byte>.Shared.Rent(FrameSize);\n    FillFromMarketFeed(buffer);\n    channel.Writer.TryWrite(buffer);\n}\n\n// N per-connection consumer tasks — each may Return a buffer it never Rented.\nawait foreach (var buffer in channel.Reader.ReadAllAsync())\n{\n    await connection.WriteAsync(buffer);\n    ArrayPool<byte>.Shared.Return(buffer); // <-- always a different thread than Rent\n}\n```\n\nExactly the pattern the tool output implied: **one thread rents, many other\nthreads return.**\n\n`ArrayPool<T>.Shared`\n\nisn't a naive shared bag — its current .NET 10\n\nimplementation, `SharedArrayPool<T>`\n\n, keeps a **single-slot cache stored\nper-thread** (\n\n`[ThreadStatic]`\n\nbuckets): if the `Monitor`\n\nNone of this is a bug in `ArrayPool<T>`\n\n. It's working exactly as designed —\n\noptimized for the common case (rent-use-return on one thread, e.g. inside a\n\nsingle request handler), and gracefully degrading, but *not silently free*,\n\nfor the cross-thread fan-out case. There is no compiler warning, no analyzer\n\nrule, and no code-review heuristic for \"you are about to violate an\n\nimplicit performance assumption of a BCL type you didn't write.\" The only way\n\nto know, for certain, what *this* process, running *this* runtime build, is\n\nactually doing is to watch where the CPU goes and read the exact binary it's\n\nrunning.\n\nThe fix isn't \"add a lock\" (there's nothing to protect) — it's swap the pool\n\nimplementation to one designed for the actual ownership pattern: many threads\n\nreturning items nobody-in-particular rented, i.e., a plain lock-free queue.\n\n```\npublic sealed class CrossThreadBufferPool\n{\n    private readonly ConcurrentQueue<byte[]> _pool = new();\n    public byte[] Rent(int size) => _pool.TryDequeue(out var buf) ? buf : new byte[size];\n    public void Return(byte[] buffer) => _pool.Enqueue(buffer);\n}\n```\n\n**Isolated cost, before/after** (BenchmarkDotNet, 20,000 cross-thread\n\n`Return`\n\ncalls, buffers pre-rented on a background thread outside the timed\n\nregion so only `Return`\n\nitself is measured):\n\n| Scenario | Mean time (20,000 calls) | vs. contended baseline |\n|---|---|---|\n`ArrayPool.Shared.Return` , cross-thread (the bug) |\n~4.3–5.1 ms | 1x (baseline) |\n`ConcurrentQueue<byte[]>` `Return` , cross-thread (the fix) |\n~126–146 µs | ~34x faster |\n`ArrayPool.Shared` full Rent+Return, same thread — theoretical best case |\n~115–150 µs | ~34–40x faster |\n\nThe last row is the ceiling: the one scenario with genuinely zero\n\nsynchronization, because the per-thread cache slot is always hit. Comparing\n\nit to the `ConcurrentQueue`\n\nfix shows exactly how much overhead our own fix\n\nstill carries: **~126 µs vs. ~115 µs — roughly 10% above the theoretical\nbest**, for a data structure built to be used safely by an arbitrary number\n\nA single cross-thread `Return`\n\ncall on the *unfixed* pool costs more than an\n\nentire same-thread Rent+Return round trip. That's how disproportionate the\n\nfast-path miss is.\n\n**Live re-check after the fix** — same 2-thread (1 producer / 1 consumer)\n\ntopology, same CPU sampler, before touching anything else:\n\n| Metric |\n`ArrayPool.Shared` (cross-thread) |\nLock-free pool (cross-thread) | Same-thread control |\n|---|---|---|---|\n| Throughput | ~3.2M ops/s | ~6.6M ops/s | ~20M ops/s |\n`Monitor.Enter_Slowpath` in CPU sample |\n15.8% exclusive |\n0% — gone entirely |\nn/a (no shared pool contention path) |\n\n```\nassistant> With the lock-free pool under the same load:\n→ collect_sample(kind=\"cpu\", processId=41213, durationSeconds=8, topN=20)\n{\n  \"summary\": \"Hottest self-time method: System.Threading.Thread.Sleep(int32)\n              (3972 exclusive, 40.6% of samples). No Monitor frames in the\n              top 20 hotspots.\",\n  ...\n}\n```\n\nNo `Monitor.Enter_Slowpath`\n\nanywhere in the profile. The contention didn't\n\nmove — it disappeared, because the pool that produced it is gone.\n\n`ArrayPool<T>.Shared`\n\nitself exists as an optimization over the naive\n\nbaseline — just calling `new byte[size]`\n\nand letting the GC reclaim it. A\n\nfair objection to everything above: if the *shared* pool is this contended\n\nunder cross-thread `Return`\n\n, maybe the honest fix is to drop pooling\n\nentirely, not swap in a different pool. Mean latency might even look\n\ncompetitive, since a plain allocation has zero synchronization of any kind —\n\nbut the expectation is that Gen0/Gen1 GC pressure would show up in the tail\n\n(P99), not the mean.\n\nWe tested it — same 2-thread producer/consumer topology, only the pool\n\nimplementation swapped for one that allocates fresh on every `Rent`\n\nand does\n\nnothing on `Return`\n\n:\n\n| Scenario | Throughput |\n`Monitor` contention |\n|---|---|---|\n`ArrayPool.Shared` , cross-thread |\n~3.2M ops/s | 15.8% (Monitor) |\n| Lock-free queue pool, cross-thread | ~6.6M ops/s | 0% |\nNo pooling (`new byte[size]` every call) |\n~180K ops/s |\n0% |\n\nNo pooling isn't a tail-latency-only regression — it's **~18x slower on raw\nthroughput**, worse than the contended\n\n`ArrayPool.Shared`\n\nbaseline we started\n\n```\nassistant> Let's check what removing the pool entirely does to GC.\n→ collect_events(kind=\"gc\", processId=41213, durationSeconds=10)\n{\n  \"summary\": \"134 collection(s), max pause 69.9ms, total pause 5118.1ms.\"\n}\nassistant> Breaking that down:\n→ query_snapshot(handle=\"...\", view=\"pauseHistogram\")\n{\n  \"totalCollections\": 134,\n  \"maxPauseTime\": \"00:00:00.0698743\",\n  \"buckets\": [\n    { \"label\": \"<1ms\",      \"count\": 0 },\n    { \"label\": \"1-10ms\",    \"count\": 0 },\n    { \"label\": \"10-100ms\",  \"count\": 134 },\n    { \"label\": \"100-1000ms\",\"count\": 0 },\n    { \"label\": \">=1s\",      \"count\": 0 }\n  ]\n}\nassistant> And by generation:\n→ query_snapshot(handle=\"...\", view=\"byGeneration\")\n{\n  \"generations\": [\n    { \"bucket\": \"gen1\", \"count\": 134, \"totalPause\": \"00:00:05.1180759\",\n      \"meanPause\": \"00:00:00.0381945\", \"maxPause\": \"00:00:00.0698743\" }\n  ]\n}\n```\n\n**134 gen1 collections in a 10-second window — 13.4 per second, every single\none landing in the 10–100ms bucket — for 5.12 seconds of accumulated pause\ntime inside those 10 seconds.** Over half the wall clock is GC pause, not\n\nSo the honest ranking, worst to best, for this specific cross-thread\n\nfan-out pattern is: **no pooling (worst, by both mean and tail) → shared pool\nwith cross-thread Return (contended but bounded) → a pool built for the\nactual ownership shape (best on every axis measured)**.\n\n`ArrayPool<T>.Shared`\n\nStatic analysis, linters, and code review are excellent at catching *shapes*\n\nof bugs — a missing `await`\n\n, a `lock`\n\naround the wrong object, an `O(n²)`\n\nloop. They cannot catch a case where the code is a textbook-correct call into\n\na well-designed BCL type, and the only thing wrong is a runtime ownership\n\npattern that BCL type happens to be sensitive to. You either know this\n\nspecific gotcha already, or you find it by watching the process — there is no\n\nthird option.\n\n`dotnet-diagnostics-mcp`\n\nexists so that \"watching the process\" is a normal\n\nstep in an LLM-driven investigation, not a specialized skill gated behind\n\nwhoever on the team happens to know `dotnet-trace`\n\nby heart. Every dynamic\n\nstep above — `inspect_process`\n\n, `collect_sample`\n\n, `query_snapshot`\n\n— is a\n\nsingle MCP tool call an assistant can make on its own, against a real running\n\n.NET process, with no code changes and no redeploy. And when the trail leads\n\ninto a BCL type nobody on the team has source for, its companion server,\n\n[ dotnet-assembly-mcp](https://github.com/pedrosakuma/dotnet-assembly-mcp),\n\nThe reproduction and all numbers in this post are in a small, standalone,\n\nopen repro you can run yourself — no proprietary code involved. The\n\ninvestigation pattern itself (fix unrelated to this repro) is inspired by a\n\nreal production change in [ B3MarketDataPlatform](https://github.com/pedrosakuma/B3MarketDataPlatform),\n\n`ac2dc9f`\n\n`dotnet-diagnostics-mcp`\n\nis open source: [github.com/pedrosakuma/dotnet-diagnostics](https://github.com/pedrosakuma/dotnet-diagnostics).\n\nIt ships both an MCP server (for your LLM client of choice) and a standalone\n\nCLI (`dotnet-diagnostics-cli`\n\n) for scripting the same investigations by hand.", "url": "https://wpnews.pro/news/the-pool-that-got-slower-the-more-we-shared-it", "canonical_source": "https://dev.to/pedrosakuma/the-pool-that-got-slower-the-more-we-shared-it-57oh", "published_at": "2026-07-14 18:08:47+00:00", "updated_at": "2026-07-14 18:29:19.669817+00:00", "lang": "en", "topics": ["developer-tools", "artificial-intelligence"], "entities": ["dotnet-diagnostics-mcp", "ArrayPool<byte>.Shared", "System.Threading.Monitor.Enter_Slowpath", "MarketData.BufferPool", "MarketData.Broadcaster", ".NET"], "alternates": {"html": "https://wpnews.pro/news/the-pool-that-got-slower-the-more-we-shared-it", "markdown": "https://wpnews.pro/news/the-pool-that-got-slower-the-more-we-shared-it.md", "text": "https://wpnews.pro/news/the-pool-that-got-slower-the-more-we-shared-it.txt", "jsonld": "https://wpnews.pro/news/the-pool-that-got-slower-the-more-we-shared-it.jsonld"}}