GFS Cost More Than 50 Cents a Run. Now Each Run Costs 4 Cents

Maneshwar, building LiveReview, cut the cost of an AI code-review run from over 50 cents to about 4 cents by replacing Gemini's hosted File Search with a local Chroma vector store embedded with Qwen3-Embedding-0.6B, dense-plus-BM25 retrieval fused via Reciprocal Rank Fusion, and DeepSeek V4 Flash on Atlas Cloud for model calls. The original pipeline paid a reasoning model to perform retrieval, with 8,493 of 9,529 output tokens spent thinking on a single search call; splitting search out so one search feeds every draft cut a run from roughly $0.70 to $0.12 before the local stack took it further.

Hello, I'm Maneshwar, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us https://github.com/HexmosTech/LiveReview/ to help devs discover the project, give it a try, and share your feedback to help improve the product. The bill was 50 cents a run. Not a month. Not a thousand runs. One run. Paste a design proposal in, get back a grounded review with precedent from our shelf of books and internal postmortems, and watch half a dollar evaporate. Part 1 was the happy version of this story: a Go binary, Gemini's hosted File Search, no vector database, and a free tier that made the whole thing look like a free lunch. Then the free tier ran out and the paid rates arrived, and the free lunch turned out to be a tasting menu. Five people on the team, several runs a day each, and a number that scaled with how hard the tool was thinking. This post is what we replaced it with, what we measured, and the one component we built, benchmarked, and then deleted. Here is the thing the pricing page tells you about File Search, and it is all true. Storage is free. Query-time embeddings are free. You pay once at indexing time. Here is the thing it does not put in bold. The search is performed by a model . It is a tool the model calls mid-answer. Which means the retrieved chunks land in that model's context as ordinary input tokens, and the model reasons its way through them before replying. And on Gemini 3.5 Flash https://ai.google.dev/gemini-api/docs/pricing , reasoning costs $9.00 per million output tokens. Look at that output row. 9,529 tokens out, of which 8,493 were the model thinking. For a call whose entire job is "find the relevant passages and hand them over." We paid a reasoning model to reason about which paragraphs to copy. The first fix was structural and it helped a lot. The draft step used to do its own search, which meant three drafts meant three searches. Splitting search out so one search feeds every draft took a run from about $0.70 to about $0.12. Six times cheaper, and still the wrong shape. Because the expensive part was never the searching. It was the thinking attached to it. Retrieval does not need a brain. It needs an index, a similarity function, and a tiebreaker. Every one of those is a thing you can run on a laptop while it charges. A local Chroma https://www.trychroma.com/ store, embedded with Qwen3-Embedding-0.6B https://huggingface.co/Qwen/Qwen3-Embedding-0.6B , searched with dense vectors plus BM25, fused with Reciprocal Rank Fusion. Every model call moved to DeepSeek V4 Flash on Atlas Cloud at $0.14 in and $0.28 out per million. Roughly ten times cheaper per token than what we were on, with a 1M context window and JSON mode. The retrieval service is Python, running as a child process of the Go binary, on a port nobody else talks to. It starts with make run and dies with it. I did try to talk myself out of that boundary. Go has no mature CUDA story. The honest route is exporting the model to ONNX and linking a Go runtime through cgo, matching driver and CUDA and cuDNN versions exactly. PyTorch is the path a million people have already debugged, including on WSL2, where CUDA passthrough breaks in creative ways. Adding a process boundary was cheaper than adding a whole new class of "works on my machine." Everyone talks about which embedding model to pick. Almost nobody talks about what you feed it, and that is where the wins were. Our corpus is 71 markdown files, and most of them are books that used to be PDFs. PDFs converted to markdown are full of things that are not text. The cleaner strips the converter's own footer, lines that are only a page number, picture-text blocks, and hyphenated line breaks that split a word across two lines. The one that surprised me is running headers. A converted book repeats the chapter title at the top of every single page, usually promoted to a markdown heading. Left in, it shreds the text into confetti, and a passage about paper reactors comes back with "Ship Project and Civilian Power" wedged into the middle of a sentence. The rule that fixed it is embarrassingly simple: a short line that appears five or more times in one file is page furniture, not prose. Then the chunking. About 300 words, hard cap 450, split on sentence boundaries, with two sentences carried into the next chunk so a quote that straddles a boundary survives in at least one piece. A real section heading starts a new chunk, once the current one has enough in it to stand alone. And then the part I would steal even if you take nothing else from this post. Embed more than you store. The chunk you store is the text the draft is allowed to quote. The string you embed has a header glued on top of it: EMBED MODEL = "Qwen/Qwen3-Embedding-0.6B" what goes into the embedding, per chunk f"{title} {kind} › {heading path}\n\n{text}" and the query side gets an instruction, because Qwen3-Embedding is instruction-tuned on queries only. documents are embedded as they are. QUERY PROMPT = "Instruct: Given an abstract pattern or principle, retrieve historical cases, " "documented examples, and named principles from books and essays that show " "the same pattern\nQuery: " A chunk in the middle of chapter nine might only say "he decided otherwise." With the header, that vector still knows it is Rickover, in a book about Rickover, in a section about paper reactors. Without it, it is a pronoun floating in space. The query instruction is the Part 1 idea, "search with the pattern, not the post," pushed one layer down into the embedding itself. The corpus is cases. The queries are patterns. Saying so out loud to a model that was trained to listen costs nothing. Vector search is great at "these two paragraphs mean the same thing" and oddly bad at "this paragraph contains the word Rickover." Keyword search is the reverse. So we run both, take 40 candidates each, and fuse them. php flowchart TD Q one query from call 1 -- DN dense top 40, Qwen3-Embedding Q -- BM BM25 top 40, title + text DN -- RRF Reciprocal Rank Fusion, k=60 BM -- RRF RRF -- SEL keep the best 4 per query SEL -- CAP{2 chunks from this file already?} CAP -- yes -- SKIP skip, so no book dominates CAP -- no -- KEEP keep it KEEP -- RR round-robin merge, 3 queries SKIP -- RR RR -- DUP{near-duplicate of one picked?} DUP -- yes -- DROP drop: a post synced twice DUP -- no -- OUT 12 passages go to the draft classDef decision fill: f4d35e,stroke: b8991f,color: 1a1a1a classDef start fill: e9ecef,stroke: 6c757d,color: 1a1a1a classDef dense fill: 9d8cff,stroke: 5b4bcc,color: 1a1a1a classDef lex fill: 6ea8ff,stroke: 2f5fc4,color: 1a1a1a classDef good fill: 5ee6c8,stroke: 1f9c86,color: 1a1a1a classDef bad fill: ff9a5c,stroke: c4602a,color: 1a1a1a class CAP,DUP decision class Q start class DN,RRF dense class BM,SEL,RR lex class KEEP,OUT good class SKIP,DROP bad php def hybrid self, query: str, k: int = CANDIDATES - list tuple int, float : score: dict int, float = {} for ranked in self.dense query, k , self.lexical query, k : for r, i in enumerate ranked : score i = score.get i, 0.0 + 1.0 / RRF K + r + 1 return sorted score.items , key=lambda x: -x 1 :k That is the whole fusion. Six lines. The reason it works is that it never compares the two scores. A cosine similarity of 0.82 and a BM25 score of 14.3 have nothing to say to each other. Ranks do. A document at position 3 in both lists beats one that is first in a single list, and the constant k 60 is the number the literature settled on keeps the top of each list from steamrolling everything else. BM25 indexes the title and heading alongside the text, same as the embedding header does, so naming a book in your query actually finds pages from that book. A few rules keep the final twelve honest. At most two chunks per file per query, so one 400-page book cannot fill every slot. Queries merge round robin, so each of the three contributes its best passage before any of them gets its fourth. And anything with a word-overlap above 0.6 against something already picked gets dropped, because our blog corpus has a couple of posts that got synced twice and they were politely returning themselves as two independent sources. Standard advice says: retrieve broadly, then rerank with a cross-encoder. So we did that, with bge-reranker-v2-m3 . Then a run took six and a half minutes and I went looking. The retrieval service's own log had it in one line: search: 3 queries - 8 chunks in 117.5s . Three queries times 40 candidates is 120 cross-encoder forward passes on a 4 GB GTX 1650 that is also drawing the desktop. It was not thrashing. It was just honest work on unfit hardware. Before ripping it out, we measured. The golden set builds itself out of finished sessions: take Call 1's retrieval queries, pair them with the files the accepted draft actually cited, and you have a retrieval test built from real usage rather than from vibes. The reranker won on MRR and hit@4 mailto:hit@4 . It genuinely put better passages nearer the top. It also did not change recall@12 at all, and recall@12 is the only number with a consumer. The draft call gets all twelve passages in its prompt. It reads all twelve. There is no top-4 cutoff downstream, no truncation, nothing that treats passage 1 differently from passage 9. So the reranker was spending 108 seconds improving an ordering that nothing downstream reads. It was optimising a metric we had accidentally chosen because it appears in every retrieval paper, not because our pipeline consumed it. Out it went, with the reasoning written into the top of retriever.py so the next person does not "fix" its absence. And the honest caveat, which lives there too: this is one golden session. A harder query might genuinely need reranking to pull the right passage into the top twelve. The code is in git history, the eval is a make target, and when the golden set is fat enough to mean something we will run it again. Measure before you delete. Also measure before you keep. 3,639 chunks at roughly 1.8 seconds each on a shared 4 GB GPU is about two hours, which is about one hour and fifty minutes more than anyone wants to wait. But embedding is deterministic and the corpus splits cleanly by file. So it parallelises across people, not just across cores. php flowchart TD C 71 files, 3,639 chunks -- S shard files: disjoint quarters S -- M four laptops, --shard i/4 M -- R{sha256 AND chunk count match?} R -- yes -- SK already embedded, skip R -- no -- EM embed, halve the batch on OOM EM -- G commit the shard, hand it back SK -- G G -- I rag/integrate.py I -- V{same embedding model everywhere?} V -- no -- X refuse: mixed vectors lie quietly V -- yes -- O{every file in exactly one shard?} O -- in two -- X O -- in none -- W warn, merge what arrived O -- yes -- F copy vectors into db/chroma classDef decision fill: f4d35e,stroke: b8991f,color: 1a1a1a classDef start fill: e9ecef,stroke: 6c757d,color: 1a1a1a classDef work fill: 9d8cff,stroke: 5b4bcc,color: 1a1a1a classDef box fill: 6ea8ff,stroke: 2f5fc4,color: 1a1a1a classDef good fill: 5ee6c8,stroke: 1f9c86,color: 1a1a1a classDef bad fill: ff9a5c,stroke: c4602a,color: 1a1a1a class R,V,O decision class C start class M,EM work class S,G,I,SK box class F good class X,W bad on four different machines, one quarter each uv run rag/ingest.py --shard 1/4 --out db/chroma-shards/1 uv run rag/ingest.py --shard 2/4 --out db/chroma-shards/2 ... then, once everyone commits their shard back uv run rag/integrate.py db/chroma-shards/ The merge does no embedding at all. It checks that every shard used the same embedding model, that no file landed in two shards, and that no file landed in none, then copies the vectors into one store. Those checks are not paranoia. Mixed embedding models do not crash. They return confidently wrong neighbours forever, which is a far worse failure than a stack trace. Building this also shook out a real bug in the incremental logic. Resume was deciding "already done" by comparing the file's sha256 against what was in the store. A run killed halfway through a file leaves that file's sha perfectly correct and its chunk count short, so it would have been marked done forever, silently missing half a book. The fix is one AND : sha256 and chunk count both have to match. Then we committed the finished store. db/chroma is 69 MB in git, which is nothing, and it means nobody else on the team ever embeds anything. Clone, run, search. The nice thing about killing your most expensive call is that the cheap calls get interesting. A draft is now a few tenths of a cent. So instead of drafting once and retrying on failure, the pipeline fires three or four drafts at once at temperature 0.7 and lets them race. Each reply gets resolved and checked in Go as it lands. Drafts that fail the checks are rejected on the spot. The first one that passes goes on to the review call. Only if all of them fail does the batch retry, with the closest draft's failures appended to the prompt. Every draft is kept and shown, including the rejected ones with the checks they failed, because "here are four attempts and why three of them were bad" is more useful to the person reading than one draft and a shrug. This did produce one genuinely dumb bug, which parallelism made much more likely. In one run the first batch produced a draft that passed every check. The reviewer then asked for a revision. Eight redrafts later, none of them passing, the pipeline shipped the closest failing redraft. It had a passing draft in hand and threw it away for a worse one. The fix is the obvious fallback: if no redraft passes, ship the draft that already did, with the reviewer's notes attached. Retries are cheap. Losing work you already paid for is not. Handing the passages in ourselves unlocked the thing I actually care about. When a hosted search tool returns grounding metadata, you know which chunks the model looked at. You do not know that the words it put in quotation marks are in any of them. Now the pipeline can prove it. Every history.sources .passage in the output has to appear word for word in a chunk from the file it names. Same for an authority's exact words when it cites one of the given files. Markdown, line wrapping and quote style are normalised away first, and a ... marks an omission, with the remaining pieces required to appear in order. A failure is not a warning. It is a check failure, exactly like a length violation or an unresolved law citation, and it feeds the same retry loop that everything else does. This is the difference between "the model had access to the right book" and "the model quoted the right book correctly," and only one of those is worth showing a reviewer. One real run, end to end, with 15 model calls including 12 drafts across three batches: Against $0.70 for the same pipeline shape on hosted search. Fifteen times cheaper, and the expensive part is now the part that does the actual writing, which is how it should be. Two honest footnotes on that number. Atlas reports most of each draft's input as cached, and we charge every input token at the full rate, so $0.045 is a ceiling, not an estimate. And latency went up, not down. Retrieval dropped from 117 seconds to half a second, but DeepSeek thinks hard before each draft, so a full run is still two to four minutes. We bought cost, not speed. For "paste a design doc, come back with a coffee," that is the right trade. For a chat box it would be the wrong one. If your retrieval is a model call, you are paying reasoning prices for a lookup. Pull it onto your own machine, spend the effort on cleaning and chunking rather than on model selection, embed a contextual header you never show anyone, fuse dense and lexical on ranks instead of scores, and check the quotes rather than trusting them. Then build the eval before you build the clever part. Ours told us to throw the clever part away, which saved 108 seconds a search and a permanent dependency we did not need. The best component in this system is the one that is not in it. Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production secure and reliable without slowing you down. I'm building LiveReview , a blast-radius aware AI code review built for your business-critical systems. Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters. Spend code review effort where business risk is highest — not spread evenly across every diff. ⭐ Star it on GitHub: LiveReview is an AI code reviewer that scores every hunk of a diff by blast radius : how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff. LiveReview's Blast Radius & Review Priority scoring, live in the diff viewer. | The exact math, not a black box | Visualize blast radius at a glance | Every factor that feeds the score | |---|---|---| Here's the goal: Click below to try LiveReview with your codebase: