# Cloudflare’s AI code reviewer got 68% cheaper by fixing one cache bug

> Source: <https://rews.cc/a/cloudflare-s-ai-code-reviewer-got-68-cheaper-by-fixing-one-c-b41e5d>
> Published: 2026-10-19 19:12:53+00:00

An average AI-generated code review at Cloudflare cost $0.77 in October. In August it cost $2.46, and over the company’s two most expensive weeks that summer it averaged $2.43. Cloudflare says engineers made no significant architectural change to the system in that stretch: the same coordinator agent, the same specialist reviewers, the same risk tiers. The 68% drop came from finding where five months of token spend had gone and cutting it, as Cloudflare describes in a [post on its engineering blog](https://blog.cloudflare.com/obsessive-inference-optimization/).

The reviewer is the system Cloudflare [first described in April](https://blog.cloudflare.com/ai-code-review/): a coordinator agent dispatches up to seven specialist reviewers, covering code quality, security, performance, documentation and internal standards, at one of several review-effort tiers depending on how big a merge request looks. In its first month of production, March 10 to April 9, it ran 131,246 reviews across 48,095 merge requests in 5,169 repositories, at a median cost of $0.98 and an average of $1.19. The system runs on OpenCode, an open-source coding-agent framework, with the heaviest reviewers on Claude models from Anthropic and lighter ones on cheaper models.

That $1.19 average climbed steadily with no redesign behind it: $1.22 in March, $1.44 in June, $1.74 in July, $2.32 in August, measured as a weekday average with the mix of review tiers held fixed so a day with more large diffs doesn’t skew the comparison. Part of the rise was mechanical — reviews landing in the most expensive, full-review tier went from 59% of the total in March to 65% in August, which Cloudflare attributes to engineers shipping bigger, more complex changes as the underlying coding models got more capable. The larger part came from switching providers: moving every reviewer onto Anthropic models in early August took the average job from $1.38 to $2.03, and moving the internal-standards reviewer specifically onto Claude Opus 4.7 took it from $0.25 to $0.88, making it pricier than the coordinator that oversees it.

**Average cost per AI code review** Weekday average, Cloudflare's internal AI reviewer, fixed review-tier mix, $

|  | Cost per review ($) | 
|---|---|
| Mar | 1.22 | 
| Jun | 1.44 | 
| Jul | 1.74 | 
| Aug | 2.32 | 
| Oct (after fixes) | 0.77 | 

Source: [Cloudflare](https://blog.cloudflare.com/obsessive-inference-optimization/)

Behind both increases sat a cache that wasn’t doing its job. Anthropic’s prompt caching, [documented on its API platform](https://platform.claude.com/docs/en/build-with-claude/prompt-caching), lets a model reuse context it has already processed instead of re-reading it on every call. Writing a block of a prompt into the cache costs 1.25 times the normal input-token price; reading it back later costs 0.1 times, a 90% discount. An entry expires after five minutes unless something reads it again, refreshing the clock. The saving only shows up on the parts of a conversation that stay identical across calls, which Anthropic calls prefixes, and a prefix is stored only up to the point where the request marks a cache breakpoint.

Before Cloudflare’s fixes, output tokens made up about 23% of the AI reviewer’s bill, cache writes 43% and cache reads 34% — most of the money was going toward context the model had already seen, or was about to see again, not toward generating anything new. The code-quality reviewer alone made 16.6 LLM calls per review on average, writing 72,000 tokens to cache and reading back 1.08 million, because each call resends the entire conversation so far as context. A session’s cost scales with both the number of calls and the size of that context, and both had crept up quietly for months.

## Why the cache kept missing

On September 14, Cloudflare started logging the token count of each reviewer’s first LLM call alongside a hash of the exact system block sent to the model. Every review session opens with the same tool definitions and the same agent prompt — 11,000 to 21,000 tokens that are identical no matter which of Cloudflare’s repositories is being reviewed — so nearly all of those first calls should have hit the cache. Instead, the data showed a hit rate of only 14% to 26%.

The cause was in how OpenCode v1 built the system prompt. It assembled one block containing the agent prompt followed by an environment section listing the working directory and the date, then placed the single cache breakpoint at the very end of that combined block. Because the environment section differs by repository, the whole block differed by repository, so a review of one codebase could never reuse what a review of a different codebase had already written to cache.

**Fixing the shared prefix** Where Cloudflare moved the cache breakpoint

| From | To | How | 
|---|---|---|
| LLM call begins | Tool definitions + agent prompt *(11k–21k tokens, identical across every repo)* |  | 
| Tool definitions + agent prompt *(11k–21k tokens, identical across every repo)* | New cache breakpoint *(added September 16)* | write once, read by every repo | 
| New cache breakpoint *(added September 16)* | Per-repo context *(working directory, date, merge-request diff)* |  | 
| Per-repo context *(working directory, date, merge-request diff)* | Original breakpoint |  | 
| Original breakpoint | Model response |  | 

Based on [Cloudflare](https://blog.cloudflare.com/obsessive-inference-optimization/)

On September 16, Cloudflare added a marker to the end of the agent prompt and a plugin hook that split the system block in two at that marker, giving the shared half its own breakpoint. Keeping that half byte-identical across repositories required removing anything repo-specific from in front of it: the list of available skills had to be disabled, and tool descriptions could no longer vary between jobs. The same day, the internal-standards reviewer’s reference index — a table of coding standards, a skill file and a list of reference files that had previously taken three tool calls to read on every single run — was folded directly into its prompt, cutting its LLM calls per run from 15.9 to 9.6 and its own cost from $1.12 to $0.72. Together, the two changes took the average review from $2.47 to $2.02 the day they shipped.

## Fewer calls, not just cheaper tokens

A second round of changes on September 21 and 23 shaved roughly $0.20 more off every review, none of them individually large. The most involved was sharing one prefix across three reviewers at once: the code-quality, security and performance reviewers now send an identical first request — same tools, same shared rules file, same merge-request diff — and only one of them writes it to cache while the other two read it back. Because a cache entry can’t be read until the response that wrote it has started streaming, the second and third reviewers wait up to 20 seconds for the first one to begin before sending their own request, a small latency cost for roughly nine cents saved per reviewer.

Cloudflare also replaced OpenCode v1 with a ground-up rewrite, OpenCode v2, after running every review in its canary repositories through both versions in shadow mode; the two agreed on the approval decision 98% of the time, which was enough to cut over in early October and delete the old code. Holding the model fixed, full-tier reviews run about 13% faster on v2 than v1.

## Where the rest of the 68% came from

Splitting the savings at every release and configuration change, Cloudflare attributes $0.65 of the roughly $1.67 total drop to code changes and $1.06 to switching models and effort levels — a comparison of weekday hours before and after each change rather than a controlled experiment. On October 3, the company ran a 50/50 split between Sonnet 5 and the newer Sonnet 5.5 and moved every Sonnet-based reviewer onto 5.5 two days later: cost per call fell 62% for the code-quality reviewer, 58% for performance and 47% for security, even though Anthropic charges the same price per token for both models. The entire saving came from the newer model needing far fewer calls to finish a review — the code-quality reviewer went from 6.2 calls per run at medium effort to 1.7, with more than half of reviews now finishing in a single call.

Cloudflare also tested effort settings against each other in randomized splits. Opus 5.5 at medium effort showed no measurable difference in review outcomes or critical findings against the older Opus 5, so the newer model became the default; full-tier jobs at medium effort finished 20% sooner than at high, and a one-day test found high effort cost 61% more per run while turning up only 58% more critical-and-warning findings — about two extra cents of signal per review. Across all of it, Cloudflare reports the drop in raw tokens without a drop in usefulness: over-severity findings, flagged critical or warning but judged by a blinded human review to be overrated, fell 70%, from 0.46 to 0.14 per review, and the share of code-quality findings engineers actually acted on rose from 34% to 59%. Latency improved alongside cost — median review time fell from 4 minutes 9 seconds to 1 minute 32 seconds, and the share of reviews taking over five minutes dropped from 37.9% to 0.7%.

The company says it is now experimenting with running the reviewer entirely on Cloudflare Durable Objects, a stateful serverless primitive, under a project it calls pi-durable, with more detail promised in a later post. Cloudflare has been running similar internal audits across its own infrastructure this autumn, from [trimming 36 GB/s off an analytics pipeline](https://rews.cc/a/cloudflare-trimmed-36-gb-s-off-its-pipeline-while-events-gre-9b878a) to [running AI models against its own firewall](https://rews.cc/a/cloudflare-attacks-its-own-firewall-with-ai-models-and-finds-4a9480) to look for gaps. Nothing in this case stopped the April architecture from quietly doubling its own bill over five months without anyone redesigning it; the open question is whether the next generation of models, getting better and more expensive at roughly the same pace, does the same thing again before the next obsessive pass catches it.