# How I Used Claude Code to Cut My API's P99 Latency in Half > Source: > Published: 2026-08-12 14:31:56+00:00 Our checkout API's p99 latency crept up to 2.1s and nobody could pin down why. I paired with Claude Code to profile it, found two dumb mistakes hiding behind "it's probably the database," and cut p99 to under 900ms in an afternoon. This is the actual debugging session, not a cleaned-up retelling. A few weeks ago our checkout API started showing up in the "slow endpoints" dashboard. Not down, not erroring — just slow. p50 was fine at 180ms, but p99 had drifted to 2.1 seconds over about a month. Nobody noticed until a support ticket mentioned "checkout feels laggy sometimes." That "sometimes" is the annoying part. Intermittent tail latency is one of the worst debugging problems in software because: The first thing we actually tried, before any of this, was adding a database index on a column we suspected. It shaved maybe 40ms off p50 and did nothing to p99. That's usually a sign you're solving the wrong layer of the problem — if the tail is what hurts, the fix has to target whatever the tail-triggering requests specifically do differently, not the average-case query plan. I'd normally spend a day adding logging, deploying, waiting for traffic, and staring at traces. This time I decided to use Claude Code as an actual profiling partner instead of just a code-writing tool — feed it real data, let it form hypotheses, and have it write the instrumentation to test each one. First move was pulling the actual trace data instead of speculating. I exported a sample of slow requests (p95+) from our tracing backend as JSON and handed that to Claude Code along with the endpoint's source. ``` Here are 40 traces where checkout took >1.5s. Here's the handler code. Find the pattern — what do the slow ones have in common that the fast ones don't? ``` Within a couple minutes it flagged something I'd missed: every slow trace had a `line_items` count above 12. Orders with more than a dozen items were consistently the ones blowing past 1.5s. Small orders were always fast. That's a much better starting point than "the database is slow." Claude Code wrote a quick seed script to generate orders with varying `line_items` counts (1, 5, 12, 25, 50) and a small benchmark harness around the checkout handler: ``` python import time def bench_checkout(order): start = time.perf_counter() process_checkout(order) return time.perf_counter() - start for count in [1, 5, 12, 25, 50]: order = make_order(line_items=count) elapsed = bench_checkout(order) print(f"{count} items: {elapsed*1000:.1f}ms") ``` The output made the problem obvious immediately: ``` 1 items: 42.1ms 5 items: 58.3ms 12 items: 210.4ms 25 items: 980.7ms 50 items: 2340.1ms ``` That's not linear growth — that's quadratic. Something in the checkout path scales with the *square* of the line item count, not the count itself. I asked Claude Code to trace through `process_checkout` and flag anything with nested iteration over `line_items` . It found the culprit in about a minute — a discount-application function that, for every line item, re-scanned the *entire* line item list to check for bundle discounts: ``` python # The bug: O(n²) — for each item, rescan all items def apply_bundle_discounts(line_items): for item in line_items: for other in line_items: if is_bundle_pair(item, other): item.discount += bundle_discount(item, other) return line_items ``` This had been fine when average orders had 3-4 items. Nobody touched this function in months — it just quietly got worse as our average cart size grew with a new "bulk order" feature we'd shipped two months earlier. Classic case of a change in one part of the system (bulk orders) exposing a latent bug in a completely different part (discount logic) that nobody thought to re-check. The fix was to pre-index items by the attribute `is_bundle_pair` actually checks, so each item does a lookup instead of a full rescan: ``` python # O(n) — group once, then look up def apply_bundle_discounts(line_items): groups = index_by_bundle_key(line_items) for item in line_items: for other in groups.get(item.bundle_key, []): if other is not item: item.discount += bundle_discount(item, other) return line_items ``` Here's where having an agent do this really paid off. Instead of stopping at the one fix, I asked it to grep the codebase for the same *shape* of bug — nested loops over the same collection: ``` Search the codebase for any function with a nested loop over the same list variable (for x in items: for y in items:). Flag each one and tell me if it looks intentional or accidental. ``` It found three more instances. Two were genuinely fine (small, bounded lists, like a 3-item shipping options list). One was a second real bug in the inventory reservation path that hadn't shown up in traces yet because it hadn't been hit by a large-enough order — a landmine waiting to go off. Fixed that one too. What struck me about this step wasn't that the agent found the bug — grep can find nested loops. It's that it also gave me a plain-language reason for each hit, including the two false positives, so I didn't have to manually re-derive "is this actually bounded" for every match. That triage step is normally the tedious part that makes people skip the search entirely. Before calling it done, I re-ran the benchmark harness and also replayed a sample of real slow-trace orders against the fixed code: ``` 1 items: 41.8ms 5 items: 54.2ms 12 items: 71.6ms 25 items: 94.3ms 50 items: 138.9ms ``` 50-item orders went from 2.3 seconds to 139 milliseconds. After deploying, p99 for the checkout endpoint settled at 860ms within a day — most of the remaining tail was legitimate payment-provider network latency, not our code. **"It's probably the database" is a hypothesis, not a diagnosis.** I've lost count of how many times that phrase turned out to be wrong. Pull real trace data first and let the data pick the hypothesis for you. **Feeding an agent raw trace data beats describing the problem in prose.** When I gave Claude Code the actual slow traces instead of my own summary ("checkout is sometimes slow"), it found the `line_items` correlation faster than I would have staring at the same data manually. **Quadratic bugs hide in plain sight until your data shape changes.** This function was "fine" for two years because nobody had a 50-item cart. A completely unrelated feature (bulk orders) is what exposed it. Performance bugs are often dormant, not new. **Once you find one instance of a bug pattern, search for its siblings immediately.** The second nested-loop bug in inventory reservation would've been next month's incident. Pattern-matching across a whole codebase is exactly the kind of tedious, mechanical search an agent is good at and I'm bad at (I get bored after the third file). **Benchmark before AND after, with the same harness.** It's tempting to just ship the fix once it "looks right." The before/after numbers are what let me confidently say p99 actually moved, instead of hoping. **An index on the wrong column just moves the bottleneck, it doesn't remove it.** Our first, database-shaped attempt at a fix barely touched p99 because the actual cost was in application code, not the query. If a "fix" only moves p50, be suspicious that you're treating a symptom. I'm turning the "search for sibling bugs after every fix" step into a habit rather than a one-off — it's cheap and it already caught one live landmine. Next up is doing the same profiling pass on our search endpoint, which has a similar shape of tail-latency complaints. If you've got a "sometimes slow" endpoint that nobody's been able to pin down, pull the actual slow traces before you guess. Curious what other quadratic-bug war stories people have — drop them in the comments. If this was useful, follow me here on Dev.to — I write these build-log posts regularly.