{"slug": "gfs-cost-more-than-50-cents-a-run-now-each-run-costs-4-cents", "title": "GFS Cost More Than 50 Cents a Run. Now Each Run Costs 4 Cents", "summary": "Maneshwar, building LiveReview, cut the cost of an AI code-review run from over 50 cents to about 4 cents by replacing Gemini's hosted File Search with a local Chroma vector store embedded with Qwen3-Embedding-0.6B, dense-plus-BM25 retrieval fused via Reciprocal Rank Fusion, and DeepSeek V4 Flash on Atlas Cloud for model calls. The original pipeline paid a reasoning model to perform retrieval, with 8,493 of 9,529 output tokens spent thinking on a single search call; splitting search out so one search feeds every draft cut a run from roughly $0.70 to $0.12 before the local stack took it further.", "body_md": "*Hello, I'm Maneshwar, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. [Star us](https://github.com/HexmosTech/LiveReview/) to help devs discover the project, give it a try, and share your feedback to help improve the product.*\n\nThe bill was 50 cents a run.\n\nNot a month. Not a thousand runs. One run.\n\nPaste a design proposal in, get back a grounded review with precedent from our shelf of books and internal postmortems, and watch half a dollar evaporate.\n\nPart 1 was the happy version of this story: a Go binary, Gemini's hosted File Search, no vector database, and a free tier that made the whole thing look like a free lunch.\n\nThen the free tier ran out and the paid rates arrived, and the free lunch turned out to be a tasting menu.\n\nFive people on the team, several runs a day each, and a number that scaled with how hard the tool was thinking.\n\nThis post is what we replaced it with, what we measured, and the one component we built, benchmarked, and then deleted.\n\nHere is the thing the pricing page tells you about File Search, and it is all true.\n\nStorage is free. Query-time embeddings are free. You pay once at indexing time.\n\nHere is the thing it does not put in bold.\n\nThe search is performed **by a model**. It is a tool the model calls mid-answer. Which means the retrieved chunks land in that model's context as ordinary input tokens, and the model reasons its way through them before replying.\n\nAnd on [Gemini 3.5 Flash](https://ai.google.dev/gemini-api/docs/pricing), reasoning costs $9.00 per million output tokens.\n\nLook at that output row.\n\n9,529 tokens out, of which 8,493 were the model thinking. For a call whose entire job is \"find the relevant passages and hand them over.\"\n\nWe paid a reasoning model to reason about which paragraphs to copy.\n\nThe first fix was structural and it helped a lot. The draft step used to do its own search, which meant three drafts meant three searches. Splitting search out so one search feeds every draft took a run from about $0.70 to about $0.12.\n\nSix times cheaper, and still the wrong shape.\n\nBecause the expensive part was never the searching. It was the thinking attached to it.\n\n**Retrieval does not need a brain.**\n\nIt needs an index, a similarity function, and a tiebreaker. Every one of those is a thing you can run on a laptop while it charges.\n\nA local [Chroma](https://www.trychroma.com/) store, embedded with [Qwen3-Embedding-0.6B](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B), searched with dense vectors plus BM25, fused with Reciprocal Rank Fusion.\n\nEvery model call moved to DeepSeek V4 Flash on Atlas Cloud at $0.14 in and $0.28 out per million.\n\nRoughly ten times cheaper per token than what we were on, with a 1M context window and JSON mode.\n\nThe retrieval service is Python, running as a child process of the Go binary, on a port nobody else talks to. It starts with `make run` and dies with it.\n\nI did try to talk myself out of that boundary.\n\nGo has no mature CUDA story. The honest route is exporting the model to ONNX and linking a Go runtime through cgo, matching driver and CUDA and cuDNN versions exactly. PyTorch is the path a million people have already debugged, including on WSL2, where CUDA passthrough breaks in creative ways.\n\nAdding a process boundary was cheaper than adding a whole new class of \"works on my machine.\"\n\nEveryone talks about which embedding model to pick. Almost nobody talks about what you feed it, and that is where the wins were.\n\nOur corpus is 71 markdown files, and most of them are books that used to be PDFs. PDFs converted to markdown are full of things that are not text.\n\nThe cleaner strips the converter's own footer, lines that are only a page number, picture-text blocks, and hyphenated line breaks that split a word across two lines.\n\nThe one that surprised me is running headers.\n\nA converted book repeats the chapter title at the top of every single page, usually promoted to a markdown heading.\n\nLeft in, it shreds the text into confetti, and a passage about paper reactors comes back with \"Ship Project and Civilian Power\" wedged into the middle of a sentence.\n\nThe rule that fixed it is embarrassingly simple: a short line that appears five or more times in one file is page furniture, not prose.\n\nThen the chunking. About 300 words, hard cap 450, split on sentence boundaries, with two sentences carried into the next chunk so a quote that straddles a boundary survives in at least one piece. A real section heading starts a new chunk, once the current one has enough in it to stand alone.\n\nAnd then the part I would steal even if you take nothing else from this post.\n\n**Embed more than you store.**\n\nThe chunk you store is the text the draft is allowed to quote. The string you embed has a header glued on top of it:\n\n```\nEMBED_MODEL = \"Qwen/Qwen3-Embedding-0.6B\"\n\n# what goes into the embedding, per chunk\nf\"{title} ({kind}) › {heading_path}\\n\\n{text}\"\n\n# and the query side gets an instruction, because Qwen3-Embedding is\n# instruction-tuned on queries only. documents are embedded as they are.\nQUERY_PROMPT = (\n    \"Instruct: Given an abstract pattern or principle, retrieve historical cases, \"\n    \"documented examples, and named principles from books and essays that show \"\n    \"the same pattern\\nQuery: \"\n)\n```\n\nA chunk in the middle of chapter nine might only say \"he decided otherwise.\"\n\nWith the header, that vector still knows it is Rickover, in a book about Rickover, in a section about paper reactors. Without it, it is a pronoun floating in space.\n\nThe query instruction is the Part 1 idea, \"search with the pattern, not the post,\" pushed one layer down into the embedding itself. The corpus is cases. The queries are patterns. Saying so out loud to a model that was trained to listen costs nothing.\n\nVector search is great at \"these two paragraphs mean the same thing\" and oddly bad at \"this paragraph contains the word Rickover.\"\n\nKeyword search is the reverse.\n\nSo we run both, take 40 candidates each, and fuse them.\n\n``` php\nflowchart TD\n    Q[one query from call 1] --> DN[dense top 40, Qwen3-Embedding]\n    Q --> BM[BM25 top 40, title + text]\n    DN --> RRF[Reciprocal Rank Fusion, k=60]\n    BM --> RRF\n    RRF --> SEL[keep the best 4 per query]\n    SEL --> CAP{2 chunks from this file already?}\n    CAP -- yes --> SKIP[skip, so no book dominates]\n    CAP -- no --> KEEP[keep it]\n    KEEP --> RR[round-robin merge, 3 queries]\n    SKIP --> RR\n    RR --> DUP{near-duplicate of one picked?}\n    DUP -- yes --> DROP[drop: a post synced twice]\n    DUP -- no --> OUT[12 passages go to the draft]\n\n    classDef decision fill:#f4d35e,stroke:#b8991f,color:#1a1a1a\n    classDef start    fill:#e9ecef,stroke:#6c757d,color:#1a1a1a\n    classDef dense    fill:#9d8cff,stroke:#5b4bcc,color:#1a1a1a\n    classDef lex      fill:#6ea8ff,stroke:#2f5fc4,color:#1a1a1a\n    classDef good     fill:#5ee6c8,stroke:#1f9c86,color:#1a1a1a\n    classDef bad      fill:#ff9a5c,stroke:#c4602a,color:#1a1a1a\n\n    class CAP,DUP decision\n    class Q start\n    class DN,RRF dense\n    class BM,SEL,RR lex\n    class KEEP,OUT good\n    class SKIP,DROP bad\nphp\ndef hybrid(self, query: str, k: int = CANDIDATES) -> list[tuple[int, float]]:\n    score: dict[int, float] = {}\n    for ranked in (self.dense(query, k), self.lexical(query, k)):\n        for r, i in enumerate(ranked):\n            score[i] = score.get(i, 0.0) + 1.0 / (RRF_K + r + 1)\n    return sorted(score.items(), key=lambda x: -x[1])[:k]\n```\n\nThat is the whole fusion. Six lines.\n\nThe reason it works is that it never compares the two scores. A cosine similarity of 0.82 and a BM25 score of 14.3 have nothing to say to each other. Ranks do.\n\nA document at position 3 in both lists beats one that is first in a single list, and the constant `k` (60 is the number the literature settled on) keeps the top of each list from steamrolling everything else.\n\nBM25 indexes the title and heading alongside the text, same as the embedding header does, so naming a book in your query actually finds pages from that book.\n\nA few rules keep the final twelve honest. At most two chunks per file per query, so one 400-page book cannot fill every slot. Queries merge round robin, so each of the three contributes its best passage before any of them gets its fourth.\n\nAnd anything with a word-overlap above 0.6 against something already picked gets dropped, because our blog corpus has a couple of posts that got synced twice and they were politely returning themselves as two independent sources.\n\nStandard advice says: retrieve broadly, then rerank with a cross-encoder. So we did that, with `bge-reranker-v2-m3`.\n\nThen a run took six and a half minutes and I went looking.\n\nThe retrieval service's own log had it in one line: `search: 3 queries -> 8 chunks in 117.5s`.\n\nThree queries times 40 candidates is 120 cross-encoder forward passes on a 4 GB GTX 1650 that is also drawing the desktop. It was not thrashing. It was just honest work on unfit hardware.\n\nBefore ripping it out, we measured. The golden set builds itself out of finished sessions: take Call 1's retrieval queries, pair them with the files the accepted draft actually cited, and you have a retrieval test built from real usage rather than from vibes.\n\nThe reranker won on MRR and [hit@4](mailto:hit@4). It genuinely put better passages nearer the top.\n\nIt also did not change recall@12 at all, and recall@12 is the only number with a consumer.\n\nThe draft call gets all twelve passages in its prompt. It reads all twelve. There is no top-4 cutoff downstream, no truncation, nothing that treats passage 1 differently from passage 9.\n\nSo the reranker was spending 108 seconds improving an ordering that nothing downstream reads. It was optimising a metric we had accidentally chosen because it appears in every retrieval paper, not because our pipeline consumed it.\n\nOut it went, with the reasoning written into the top of `retriever.py` so the next person does not \"fix\" its absence.\n\nAnd the honest caveat, which lives there too: this is one golden session. A harder query might genuinely need reranking to pull the right passage into the top twelve. The code is in git history, the eval is a make target, and when the golden set is fat enough to mean something we will run it again.\n\nMeasure before you delete. Also measure before you keep.\n\n3,639 chunks at roughly 1.8 seconds each on a shared 4 GB GPU is about two hours, which is about one hour and fifty minutes more than anyone wants to wait.\n\nBut embedding is deterministic and the corpus splits cleanly by file. So it parallelises across people, not just across cores.\n\n``` php\nflowchart TD\n    C[71 files, 3,639 chunks] --> S[shard_files: disjoint quarters]\n    S --> M[four laptops, --shard i/4]\n    M --> R{sha256 AND chunk count match?}\n    R -- yes --> SK[already embedded, skip]\n    R -- no --> EM[embed, halve the batch on OOM]\n    EM --> G[commit the shard, hand it back]\n    SK --> G\n    G --> I[rag/integrate.py]\n    I --> V{same embedding model everywhere?}\n    V -- no --> X[refuse: mixed vectors lie quietly]\n    V -- yes --> O{every file in exactly one shard?}\n    O -- in two --> X\n    O -- in none --> W[warn, merge what arrived]\n    O -- yes --> F[copy vectors into db/chroma]\n\n    classDef decision fill:#f4d35e,stroke:#b8991f,color:#1a1a1a\n    classDef start    fill:#e9ecef,stroke:#6c757d,color:#1a1a1a\n    classDef work     fill:#9d8cff,stroke:#5b4bcc,color:#1a1a1a\n    classDef box      fill:#6ea8ff,stroke:#2f5fc4,color:#1a1a1a\n    classDef good     fill:#5ee6c8,stroke:#1f9c86,color:#1a1a1a\n    classDef bad      fill:#ff9a5c,stroke:#c4602a,color:#1a1a1a\n\n    class R,V,O decision\n    class C start\n    class M,EM work\n    class S,G,I,SK box\n    class F good\n    class X,W bad\n# on four different machines, one quarter each\nuv run rag/ingest.py --shard 1/4 --out db/chroma-shards/1\nuv run rag/ingest.py --shard 2/4 --out db/chroma-shards/2\n# ... then, once everyone commits their shard back\nuv run rag/integrate.py db/chroma-shards/*\n```\n\nThe merge does no embedding at all. It checks that every shard used the same embedding model, that no file landed in two shards, and that no file landed in none, then copies the vectors into one store.\n\nThose checks are not paranoia. Mixed embedding models do not crash. They return confidently wrong neighbours forever, which is a far worse failure than a stack trace.\n\nBuilding this also shook out a real bug in the incremental logic. Resume was deciding \"already done\" by comparing the file's sha256 against what was in the store.\n\nA run killed halfway through a file leaves that file's sha perfectly correct and its chunk count short, so it would have been marked done forever, silently missing half a book.\n\nThe fix is one `AND`: sha256 and chunk count both have to match.\n\nThen we committed the finished store. `db/chroma` is 69 MB in git, which is nothing, and it means nobody else on the team ever embeds anything. Clone, run, search.\n\nThe nice thing about killing your most expensive call is that the cheap calls get interesting.\n\nA draft is now a few tenths of a cent. So instead of drafting once and retrying on failure, the pipeline fires three or four drafts at once at temperature 0.7 and lets them race.\n\nEach reply gets resolved and checked in Go as it lands. Drafts that fail the checks are rejected on the spot. The first one that passes goes on to the review call. Only if all of them fail does the batch retry, with the closest draft's failures appended to the prompt.\n\nEvery draft is kept and shown, including the rejected ones with the checks they failed, because \"here are four attempts and why three of them were bad\" is more useful to the person reading than one draft and a shrug.\n\nThis did produce one genuinely dumb bug, which parallelism made much more likely.\n\nIn one run the first batch produced a draft that passed every check. The reviewer then asked for a revision. Eight redrafts later, none of them passing, the pipeline shipped the closest failing redraft.\n\nIt had a passing draft in hand and threw it away for a worse one. The fix is the obvious fallback: if no redraft passes, ship the draft that already did, with the reviewer's notes attached.\n\nRetries are cheap. Losing work you already paid for is not.\n\nHanding the passages in ourselves unlocked the thing I actually care about.\n\nWhen a hosted search tool returns grounding metadata, you know which chunks the model looked at. You do not know that the words it put in quotation marks are in any of them.\n\nNow the pipeline can prove it.\n\nEvery `history.sources[].passage` in the output has to appear word for word in a chunk from the file it names. Same for an authority's `exact_words` when it cites one of the given files. Markdown, line wrapping and quote style are normalised away first, and a `...` marks an omission, with the remaining pieces required to appear in order.\n\nA failure is not a warning. It is a check failure, exactly like a length violation or an unresolved law citation, and it feeds the same retry loop that everything else does.\n\nThis is the difference between \"the model had access to the right book\" and \"the model quoted the right book correctly,\" and only one of those is worth showing a reviewer.\n\nOne real run, end to end, with 15 model calls including 12 drafts across three batches:\n\nAgainst $0.70 for the same pipeline shape on hosted search. Fifteen times cheaper, and the expensive part is now the part that does the actual writing, which is how it should be.\n\nTwo honest footnotes on that number.\n\nAtlas reports most of each draft's input as cached, and we charge every input token at the full rate, so $0.045 is a ceiling, not an estimate.\n\nAnd latency went up, not down. Retrieval dropped from 117 seconds to half a second, but DeepSeek thinks hard before each draft, so a full run is still two to four minutes.\n\nWe bought cost, not speed. For \"paste a design doc, come back with a coffee,\" that is the right trade. For a chat box it would be the wrong one.\n\nIf your retrieval is a model call, you are paying reasoning prices for a lookup.\n\nPull it onto your own machine, spend the effort on cleaning and chunking rather than on model selection, embed a contextual header you never show anyone, fuse dense and lexical on ranks instead of scores, and check the quotes rather than trusting them.\n\nThen build the eval before you build the clever part. Ours told us to throw the clever part away, which saved 108 seconds a search and a permanent dependency we did not need.\n\nThe best component in this system is the one that is not in it.\n\nYour team's attention is limited, and the deluge of AI-generated code is making it harder to keep production secure and reliable without slowing you down.\n\nI'm building **LiveReview**, a blast-radius aware AI code review built for your business-critical systems.\n\nInstead of presenting every diff with equal emphasis, **LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.**\n\nSpend code review effort where business risk is highest — not spread evenly across every diff.\n\n⭐ Star it on GitHub:\n\nLiveReview is an AI code reviewer that scores every hunk of a diff by **blast radius**: how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff.\n\n*LiveReview's Blast Radius & Review Priority scoring, live in the diff viewer.*\n\n| The exact math, not a black box | Visualize blast radius at a glance | Every factor that feeds the score | \n|---|---|---|\n\n**Here's the goal:**\n\n**Click below to try LiveReview with your codebase:**", "url": "https://wpnews.pro/news/gfs-cost-more-than-50-cents-a-run-now-each-run-costs-4-cents", "canonical_source": "https://dev.to/lovestaco/gfs-cost-more-than-50-cents-a-run-now-it-costs-4-cents-3lf8", "published_at": "2026-09-26 18:02:42+00:00", "updated_at": "2026-09-26 18:31:18.430577+00:00", "lang": "en", "topics": ["ai-tools", "ai-infrastructure", "large-language-models", "mlops", "developer-tools"], "entities": ["LiveReview", "HexmosTech", "Gemini 3.5 Flash", "Chroma", "Qwen3-Embedding-0.6B", "DeepSeek V4 Flash", "Atlas Cloud", "Maneshwar"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/gfs-cost-more-than-50-cents-a-run-now-each-run-costs-4-cents", "markdown": "https://wpnews.pro/news/gfs-cost-more-than-50-cents-a-run-now-each-run-costs-4-cents.md", "text": "https://wpnews.pro/news/gfs-cost-more-than-50-cents-a-run-now-each-run-costs-4-cents.txt", "jsonld": "https://wpnews.pro/news/gfs-cost-more-than-50-cents-a-run-now-each-run-costs-4-cents.jsonld"}}