The Cheap Way to Add AI Review to CI: Small Local Models Plus Prompt Caching

A developer built a two-tier AI code review system for CI that uses a small local model for triage and a large model with prompt caching for high-risk diffs, cutting costs without sacrificing review quality. The triage model classifies diffs as low or high risk, short-circuiting most PRs, while the large model's prompt caching reduces per-review costs by reusing stable system prompts and repo context.

I wired an AI code reviewer into CI, felt clever, and then looked at the bill after a busy week of PRs. Every push, every commit, sending full diffs to a big frontier model. It added up faster than I expected, and most of what it was reviewing was formatting changes and README edits that did not need a genius reading them. So I rebuilt it as two tiers, and the cost dropped hard without losing the catches that mattered. The idea is simple. Do not send everything to the expensive model. Use a cheap model as a bouncer that decides what is even worth escalating, and reserve the big model with caching turned on for the files that could actually hurt you. The first tier looks at the diff and answers one question: is anything here risky enough to justify a real review? That is a classification task, and classification does not need a frontier model. I run this tier on a small local model. On my own machine that is Ollama with qwen2.5-coder, but in CI it can be a self-hosted small model or the cheapest cloud tier your provider offers. The job is triage, not judgment. The triage prompt is deliberately narrow. I do not ask it to review. I ask it to route: You are a triage filter for code review. For the diff below, output JSON only: { "risk": "low" | "high", "reason": "<one short phrase " } Mark "high" if the diff touches: auth, access control, money/token math, cryptography, SQL or shell string building, deserialization, file paths, network calls, or anything that changes who can do what. Mark "low" for formatting, comments, docs, tests, renames, and config bumps. DIFF: <diff here A small model is perfectly good at "does this touch auth or money." When it says low, CI posts a one-line "no high-risk changes detected" and stops. No expensive call. In practice that shortcuts the majority of PRs, because most PRs really are docs, tests, and plumbing. When tier one says high, the risky files get escalated to the strong model. This is where the real review happens, and this is where prompt caching earns its keep. The trick is that most of what you send the big model does not change between reviews. The system prompt your review rubric, your severity definitions, your house rules is identical every time. And your repo context the surrounding files, the conventions, the interfaces the changed code depends on is stable across the many small PRs that touch the same area. Caching that stable prefix means you pay full price for it once and a small fraction on every subsequent review that reuses it. So I structure the request as: cached system prompt, then cached repo context, then the fresh diff last. Anthropic-style caching keys on a stable prefix, so the ordering matters. Stable stuff first, volatile stuff last. pseudocode, provider-agnostic shape response = client.messages.create model="big-model", system= { "type": "text", "text": REVIEW RUBRIC, never changes "cache control": {"type": "ephemeral"}, }, { "type": "text", "text": repo context for files , changes rarely "cache control": {"type": "ephemeral"}, }, , messages= {"role": "user", "content": build review prompt diff }, changes every time , The rubric and the repo context ride the cache. Only the diff is fresh each run. On a repo where lots of small PRs hit the same modules, the cache hit rate is high and the per-review cost of the big model drops to mostly the fresh tokens. Wiring it together is not exotic. One job, two steps, conditional escalation: name: ai-review on: pull request permissions: contents: read pull-requests: write jobs: review: runs-on: ubuntu-latest steps: - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 v4.2.2 with: fetch-depth: 0 persist-credentials: false - name: Triage cheap model id: triage run: | git diff origin/${{ github.base ref }}...HEAD diff.patch python scripts/triage.py diff.patch triage.json echo "risk=$ jq -r .risk triage.json " "$GITHUB OUTPUT" - name: Deep review big model, cached if: steps.triage.outputs.risk == 'high' env: MODEL API KEY: ${{ secrets.MODEL API KEY }} run: python scripts/deep review.py diff.patch The if: on the second step is the whole cost story. Low-risk PRs never reach the expensive model. High-risk ones get the full treatment. Here is the part people get wrong, and it is the reason a naive AI reviewer gives you false confidence. If you only send the diff, the model cannot see cross-file bugs. Say a PR changes a function's return type or the meaning of one of its parameters. The diff looks locally fine. The reviewer, seeing only those lines, approves it. But three other files call that function and now pass the old assumptions, and those files are not in the diff at all. The bug is real, it is severe, and diff-only review is structurally blind to it. The cheap fix is to include the callers of changed functions in the context you send. You do not need to send the whole repo. You need the changed symbols plus everyone who touches them: find who calls the functions changed in this PR changed funcs=$ git diff origin/main...HEAD \ | grep -oP '^\+. function \K\w+' | sort -u for fn in $changed funcs; do grep -rl "$fn " src/ caller files.txt done sort -u caller files.txt You feed those caller files into the tier-two context the part that gets cached, since callers change slowly . Now when a change alters an interface, the model can see the code that relied on the old behavior and flag the mismatch. That single addition catches a class of bug that pure diff review will never see, and because callers are stable, they cost almost nothing after the first cached run. The economics come from routing, not from a cheaper model doing the hard job. The small model is a filter, and filtering is cheap and easy. The big model does the real work, but only on the small slice that deserves it, and it pays full token price only on the changing diff while the rubric and context ride the cache. I use pieces of this in how spectr-ai reasons about cost too: a fast local pass to decide what is worth a careful, expensive look, then the expensive look only where it counts. Running qwen2.5-coder locally for the triage tier means that stage is effectively free on hardware I already own. The failure mode to avoid is treating the cheap tier as the reviewer. It is not. It is the bouncer. If your triage model starts writing review comments, you have given a hard job to the wrong tool. Keep tier one narrow, keep tier two well-fed with callers and cached context, and the bill stays small while the catches stay real. If you run AI review in CI, are you sending bare diffs, or do you pull in the callers of what changed?