How a 15-minute chat with an LLM found a hidden ArrayPool<byte>.Shared contention point that no static analyzer would ever flag — and why the fix had nothing to do with adding a lock.
"CPU on
market-data-gateway
climbed ~20% after we added a broadcast fan-out
path last sprint. No new allocations, no GC pressure, nothing obviously
wrong in the diff. Can someone take a look before we scale out more pods?"
Nobody had touched anything that looked like a synchronization primitive.
The diff that shipped the week before only touched a buffer-pooling helper —
swapping a bespoke byte-array cache for the "obviously correct" BCL default,
ArrayPool<byte>.Shared
. That refactor was reviewed and merged specifically
because it removed custom code in favor of a well-known, heavily optimized
framework type. If anything, it should have been a free win.
This is the kind of regression static analysis is structurally blind to: no
analyzer rule says "using ArrayPool<T>.Shared
across threads is suspicious."
It's the recommended way to pool buffers in .NET. The problem isn't in the
code you'd read in a diff — it's in a runtime behavior that only shows up
under a specific, easy-to-miss usage pattern.
This is the real value proposition of dotnet-diagnostics-mcp:
Step 1 — orient.
user> Can you check what's going on with pid 41213? CPU looks high.
assistant> Let me look at the process first.
→ inspect_process(view="triage", processId=41213)
{
"processId": 41213,
"resolvedProcess": { "name": "market-data-gateway", "runtime": "net10.0" },
"summary": "CPU 78% (host has 16 cores). No memory growth trend. No recent
GC anomalies. Consider a CPU sample to see where time goes.",
"hints": [
"collect_sample(kind=\"cpu\", processId=41213, durationSeconds=10)"
]
}
Nothing screaming "leak," nothing screaming "GC." CPU is just... busy. Time to
see where.
Step 2 — sample.
assistant> Taking a 10-second CPU sample.
→ collect_sample(kind="cpu", processId=41213, durationSeconds=10, topN=20)
{
"processId": 41213,
"totalSamples": 18545,
"summary": "Hottest self-time method: System.Threading.Monitor.Enter_Slowpath
(2933 exclusive, 15.8% of samples). Rank self-time with
query_snapshot(view=\"top-methods\") or walk the call path with
view=\"call-tree\".",
"handle": "A0CKKWH5VVS9Q68AH3GG",
"hints": [
"query_snapshot(handle=\"A0CKKWH5VVS9Q68AH3GG\", view=\"caller-callee\", rootMethodFilter=\"Monitor.Enter_Slowpath\")"
]
}
That's the first real surprise. 15.8% of all CPU samples land inside Monitor.Enter_Slowpath. Nobody wrote a
lock
statement in this service.grep -r "lock ("
across the whole repo returns nothing relevant. So where isStep 3 — trace the lock back to its call site.
assistant> There's no explicit lock in this codebase. Let's see who's calling
into the monitor slow path.
→ query_snapshot(handle="A0CKKWH5VVS9Q68AH3GG", view="caller-callee",
rootMethodFilter="Monitor.Enter_Slowpath")
{
"method": "System.Threading.Monitor.Enter_Slowpath(System.Object)",
"exclusiveSamples": 2933,
"exclusivePercent": 15.82,
"callers": [
{
"method": "MarketData.BufferPool.Return(byte[])",
"samples": 1478,
"percent": 7.97
},
{
"method": "MarketData.Broadcaster+<>c__DisplayClass.<ConsumeLoop>b__0()",
"samples": 1455,
"percent": 7.85
}
]
}
Both callers are our code — a thin wrapper method calling .Rent()
on one
side, and a per-connection consume loop calling .Return()
on the other.
Not a ThreadPool
internal, not GC. The lock is being taken
inside ArrayPool<byte>.Shared itself, and it's the
Step 4 — read the actual lock, not a guess about it.
ArrayPool<byte>.Shared
's internals are in System.Private.CoreLib
— no
source, no PDB for the private implementation type. This is exactly what the
companion server, dotnet-assembly-mcp,
assistant> Let's not guess. I'll look up the actual pool implementation
in the runtime binary you're running.
→ find_method(mvidOrPath="System.Private.CoreLib.dll", namePattern="^Return$")
{
"summary": "5 match(es) for /^Return$/.",
"matches": [
{ "typeFullName": "System.Buffers.SharedArrayPool`1", "methodName": "Return",
"signature": "void System.Buffers.SharedArrayPool`1.Return(!0[], bool)",
"handle": "m:e8c78f6b-...:0x060038A7" }
]
}
assistant> Decompiling the real Return method for this exact runtime build.
→ decompile_method(moduleVersionId="e8c78f6b-...", metadataToken="0x060038A7")
public override void Return(T[] array, bool clearArray = false)
{
// ...
SharedArrayPoolThreadLocalArray[] array2 = t_tlsBuckets ?? InitializeTlsBucketsAndTrimming();
// ... same-thread fast-path slot swap ...
if (array3 != null)
{
flag2 = (_buckets[num] ?? CreatePerCorePartitions(num)).TryPush(array3);
}
// ...
}
assistant> And the fallback path that pushes to a per-core partition:
→ find_method(mvidOrPath="System.Private.CoreLib.dll", namePattern="^TryPush$")
→ decompile_method(moduleVersionId="e8c78f6b-...", metadataToken="0x060038B3")
[MethodImpl(MethodImplOptions.AggressiveInlining)]
public bool TryPush(Array array)
{
bool result = false;
Monitor.Enter(this);
// ... push onto a small fixed-size array-backed stack ...
Monitor.Exit(this);
return result;
}
There it is — ** Monitor.Enter(this)**, in the flesh, in the exact binary
ArrayPool<byte>.Shared
in this runtime build is SharedArrayPool<T>
, andMonitor.Enter_Slowpath
: it's a real, ordinaryReturn
misses the per-threadTo be clear: dotnet/runtime
's source is public on GitHub — nothing here was
hidden. The point isn't that the source was unreachable, it's the ergonomics
of the query. Getting the same answer the "normal" way means: know (or guess)
which internal type backs ArrayPool<T>.Shared
today, find the right tag in
dotnet/runtime
that matches the exact SDK/runtime patch this process is
actually running, navigate to the right file, and read enough surrounding
code to trust you're looking at the live behavior and not a since-changed
version. decompile_method(moduleVersionId, metadataToken)
collapses all of
that into one call, scoped to the exact assembly loaded in the exact process
under investigation — no version-matching, no guessing a type name, no
context-switch out of the chat.
Checking the application code (only now, after both tools already
pointed at the exact mechanism, not before):
// One dedicated broadcaster thread — the ONLY thread that ever Rents.
while (running)
{
var buffer = ArrayPool<byte>.Shared.Rent(FrameSize);
FillFromMarketFeed(buffer);
channel.Writer.TryWrite(buffer);
}
// N per-connection consumer tasks — each may Return a buffer it never Rented.
await foreach (var buffer in channel.Reader.ReadAllAsync())
{
await connection.WriteAsync(buffer);
ArrayPool<byte>.Shared.Return(buffer); // <-- always a different thread than Rent
}
Exactly the pattern the tool output implied: one thread rents, many other threads return.
ArrayPool<T>.Shared
isn't a naive shared bag — its current .NET 10
implementation, SharedArrayPool<T>
, keeps a single-slot cache stored per-thread (
[ThreadStatic]
buckets): if the Monitor
None of this is a bug in ArrayPool<T>
. It's working exactly as designed —
optimized for the common case (rent-use-return on one thread, e.g. inside a
single request handler), and gracefully degrading, but not silently free,
for the cross-thread fan-out case. There is no compiler warning, no analyzer
rule, and no code-review heuristic for "you are about to violate an
implicit performance assumption of a BCL type you didn't write." The only way
to know, for certain, what this process, running this runtime build, is
actually doing is to watch where the CPU goes and read the exact binary it's
running.
The fix isn't "add a lock" (there's nothing to protect) — it's swap the pool
implementation to one designed for the actual ownership pattern: many threads
returning items nobody-in-particular rented, i.e., a plain lock-free queue.
public sealed class CrossThreadBufferPool
{
private readonly ConcurrentQueue<byte[]> _pool = new();
public byte[] Rent(int size) => _pool.TryDequeue(out var buf) ? buf : new byte[size];
public void Return(byte[] buffer) => _pool.Enqueue(buffer);
}
Isolated cost, before/after (BenchmarkDotNet, 20,000 cross-thread
Return
calls, buffers pre-rented on a background thread outside the timed
region so only Return
itself is measured):
| Scenario | Mean time (20,000 calls) | vs. contended baseline |
|---|---|---|
ArrayPool.Shared.Return , cross-thread (the bug) |
||
| ~4.3–5.1 ms | 1x (baseline) | |
ConcurrentQueue<byte[]> Return , cross-thread (the fix) |
||
| ~126–146 µs | ~34x faster | |
ArrayPool.Shared full Rent+Return, same thread — theoretical best case |
||
| ~115–150 µs | ~34–40x faster |
The last row is the ceiling: the one scenario with genuinely zero
synchronization, because the per-thread cache slot is always hit. Comparing
it to the ConcurrentQueue
fix shows exactly how much overhead our own fix
still carries: ~126 µs vs. ~115 µs — roughly 10% above the theoretical best, for a data structure built to be used safely by an arbitrary number
A single cross-thread Return
call on the unfixed pool costs more than an
entire same-thread Rent+Return round trip. That's how disproportionate the
fast-path miss is.
Live re-check after the fix — same 2-thread (1 producer / 1 consumer)
topology, same CPU sampler, before touching anything else:
| Metric |
ArrayPool.Shared (cross-thread) |
Lock-free pool (cross-thread) | Same-thread control |
|---|---|---|---|
| Throughput | ~3.2M ops/s | ~6.6M ops/s | ~20M ops/s |
Monitor.Enter_Slowpath in CPU sample |
15.8% exclusive |
0% — gone entirely |
n/a (no shared pool contention path) |
assistant> With the lock-free pool under the same load:
→ collect_sample(kind="cpu", processId=41213, durationSeconds=8, topN=20)
{
"summary": "Hottest self-time method: System.Threading.Thread.Sleep(int32)
(3972 exclusive, 40.6% of samples). No Monitor frames in the
top 20 hotspots.",
...
}
No Monitor.Enter_Slowpath
anywhere in the profile. The contention didn't
move — it disappeared, because the pool that produced it is gone.
ArrayPool<T>.Shared
itself exists as an optimization over the naive
baseline — just calling new byte[size]
and letting the GC reclaim it. A
fair objection to everything above: if the shared pool is this contended
under cross-thread Return
, maybe the honest fix is to drop pooling
entirely, not swap in a different pool. Mean latency might even look
competitive, since a plain allocation has zero synchronization of any kind —
but the expectation is that Gen0/Gen1 GC pressure would show up in the tail
(P99), not the mean.
We tested it — same 2-thread producer/consumer topology, only the pool
implementation swapped for one that allocates fresh on every Rent
and does
nothing on Return
:
| Scenario | Throughput |
Monitor contention |
|---|---|---|
ArrayPool.Shared , cross-thread |
~3.2M ops/s | 15.8% (Monitor) |
| Lock-free queue pool, cross-thread | ~6.6M ops/s | 0% |
No pooling (new byte[size] every call) |
~180K ops/s |
0% |
No pooling isn't a tail-latency-only regression — it's ~18x slower on raw throughput, worse than the contended
ArrayPool.Shared
baseline we started
assistant> Let's check what removing the pool entirely does to GC.
→ collect_events(kind="gc", processId=41213, durationSeconds=10)
{
"summary": "134 collection(s), max 69.9ms, total 5118.1ms."
}
assistant> Breaking that down:
→ query_snapshot(handle="...", view="Histogram")
{
"totalCollections": 134,
"maxTime": "00:00:00.0698743",
"buckets": [
{ "label": "<1ms", "count": 0 },
{ "label": "1-10ms", "count": 0 },
{ "label": "10-100ms", "count": 134 },
{ "label": "100-1000ms","count": 0 },
{ "label": ">=1s", "count": 0 }
]
}
assistant> And by generation:
→ query_snapshot(handle="...", view="byGeneration")
{
"generations": [
{ "bucket": "gen1", "count": 134, "total": "00:00:05.1180759",
"mean": "00:00:00.0381945", "max": "00:00:00.0698743" }
]
}
134 gen1 collections in a 10-second window — 13.4 per second, every single one landing in the 10–100ms bucket — for 5.12 seconds of accumulated time inside those 10 seconds. Over half the wall clock is GC , not
So the honest ranking, worst to best, for this specific cross-thread
fan-out pattern is: no pooling (worst, by both mean and tail) → shared pool with cross-thread Return (contended but bounded) → a pool built for the actual ownership shape (best on every axis measured).
ArrayPool<T>.Shared
Static analysis, linters, and code review are excellent at catching shapes
of bugs — a missing await
, a lock
around the wrong object, an O(n²)
loop. They cannot catch a case where the code is a textbook-correct call into
a well-designed BCL type, and the only thing wrong is a runtime ownership
pattern that BCL type happens to be sensitive to. You either know this
specific gotcha already, or you find it by watching the process — there is no
third option.
dotnet-diagnostics-mcp
exists so that "watching the process" is a normal
step in an LLM-driven investigation, not a specialized skill gated behind
whoever on the team happens to know dotnet-trace
by heart. Every dynamic
step above — inspect_process
, collect_sample
, query_snapshot
— is a
single MCP tool call an assistant can make on its own, against a real running
.NET process, with no code changes and no redeploy. And when the trail leads
into a BCL type nobody on the team has source for, its companion server,
The reproduction and all numbers in this post are in a small, standalone,
open repro you can run yourself — no proprietary code involved. The
investigation pattern itself (fix unrelated to this repro) is inspired by a
real production change in B3MarketDataPlatform,
ac2dc9f
dotnet-diagnostics-mcp
is open source: github.com/pedrosakuma/dotnet-diagnostics.
It ships both an MCP server (for your LLM client of choice) and a standalone
CLI (dotnet-diagnostics-cli
) for scripting the same investigations by hand.