Stop building RAG systems without doing evals

Bootstrapped software company QX Labs built an evaluation system for its enterprise AI brain product in a few days, enforcing that only code with measurable positive impact on AI performance is shipped. Founder Jai argues that evals, often seen as complex and reserved for big labs like OpenAI and Anthropic, are cheap and easy for small teams and critical for maintaining velocity and trajectory in AI development. The eval harness, which uses a dataset, graded questions, and a scorer, has dictated critical product decisions for the solo-founded company.

AI Evals Aren't Just for Big Tech How a small bootstrapped team measures its AI the way frontier labs do: a practical approach to evals that you can copy. 👋 Hey, Jai here I’m a software engineer and tech investor turned solo founder, now bootstrapping a software company. I write about using AI to build faster, sell smarter, and run a whole company solo. New here? Join me: Or if you don’t like my work, send me hate mail. In my last post I described how we overhauled our enterprise “AI brain” product https://www.minimumviablefounder.com/p/why-ai-company-brains-fail . During the development process, there was one critical constraint: we only shipped code that had a measurable positive impact on our AI’s performance. Today I’ll dive into the eval system we built to enforce this. Evals short for evaluations measure whether an AI system performs well by grading its responses to a fixed set of questions or tasks. They help steer product development by telling you whether new features improve or regress results, and by pointing to the gaps that you should prioritise. Evals have a reputation for being a complex and unwieldy discipline reserved for companies with big budgets; something that OpenAI and Anthropic do with dedicated teams, annotation pipelines, and fancy in-house platforms. But in reality the inverse is true. Evals are incredibly useful for small teams. They’re cheap and easy to build while your product is still small. My company QX Labs https://www.qxlabs.com is fully bootstrapped: I founded it solo and built our knowledge management system with one other engineer. Our eval harness took a few days to build and tune, and it has since dictated critical product decisions. Here’s my underlying thesis. In business you need to solve for two things: velocity and trajectory. The most valuable thing you can build in the AI age isn’t the system itself, but the feedback loop that tells you whether your system works and how to improve it. Build the loop and you can sustain velocity without hitting a wall, because you’ll always know your next step. Evals are one of those loops. By the end of this post you’ll have: A working mental model of what evals are and how the biggest AI companies use them A copyable approach to evals for small teams Real before/after numbers from our RAG evals What an eval actually is An eval is made up of three things: A dataset that represents your real workload documents, tickets, conversations, or whatever your system operates on A set of graded questions or tasks with pre-defined correct answers “golden” questions A scorer that runs every question through your system and outputs numbers You run it before and after a change, and the delta tells you the result of that change. The closest analogy is an engineering test suite, but with one key difference: unit tests are binary they pass or fail , while evals are statistical and directional AI can get better for one set of questions and worse for another . That’s the trap with LLM systems: they don’t fail by erroring, but instead by confidently delivering mediocre results. A retrieval system that misses half the relevant documents will write an authoritative response with gaping holes. Without evals, you’d never even know. What the experts recommend Before building our eval harness, I looked at how the top labs run theirs. Surprisingly, most of their advice is highly suited to small teams. Greg Brockman President of OpenAI is a strong advocate: Here’s what they say: Gate your evals like a continuous integration CI pipeline. OpenAI open-sourced their evals framework https://github.com/openai/evals back in 2023, calling evals “one of the most impactful things you can do” when building with LLMs. They name the practice eval-driven development . “Evaluate early and often. Write scoped tests at every stage.” Start small and cover real failures. Anthropic’s engineering guidance https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents says: “We see teams delay building evals because they think they need hundreds of tasks. In reality, 20–50 simple tasks drawn from real failures is a great start.” That’s totally feasible for a small team or a solopreneur. Eyeball your system’s outputs and errors before coming up with metrics. Hamel Husain’s “ Your AI Product Needs Evals https://hamel.dev/blog/posts/evals/ ” is worth a read. He insists that the first step is to read your system’s actual outputs one-by-one and categorise the failures. Only then should you come up with the appropriate metrics to measure against those failures. Similar to Anthropic he recommends starting with ~30 hand-graded examples. In short: use your own product like your customers would, manually log and categorise the failures, then feed that data into your eval system. Avoid using LLM “judges” for grading. The big shops need AI judges to grade open-ended outputs at scale, but they calibrate them against human graders because research shows that LLM-graded evals can otherwise be very flimsy. The MT-Bench paper https://arxiv.org/abs/2306.05685 Zheng et al. outlines their biases: position bias judges favour the first response rather than the best one , verbosity bias judges favour longer responses 90% of the time regardless of quality , and self-enhancement models score their own outputs higher . The lesson for a small team is not “don’t use judges,” it’s “don’t start with judges.” Stick to mechanical, deterministic scoring first. In the end big-tech evaluation is expensive due to scale and not the underlying to process. A small team doesn’t need thousands of evals in CI, expensive annotation or judge calibration pipelines. They just need to run a process of getting fixed data, creating graded questions and scoring every change. That costs almost nothing. The small-team version: our recipe for evals First, some context: one of our products https://www.qxlabs.com at QX ingests a company’s documents Drive, SharePoint, Granola, Notion, etc. and lets AI agents answer questions across them. When we originally built the MVP it used a simple RAG pipeline for indexing and retrieval, but we wanted to turn it into a full knowledge graph. However, we needed a way to measure whether the changes we implemented were actually improving the agent’s ability to query data. So before doing anything, we built the eval harness. Here’s how we approached it. 1. Build the eval around your customer archetype Our dataset uses ~1,000 real documents that represent one hypothetical customer’s universe. In our case we mimicked an investment firm’s internal data: PDFs, spreadsheets, decks, memos, meeting notes, scanned pages. Specifically, we wanted things to look multimodal, messy, and organised into a folder hierarchy an actual firm might accumulate over time. Having raised my owned fund and worked as an investor for 10 years, I had a very good mental model of what this should look like. That right there is the secret sauce to good evals: model them closely on your end-customers’ files and queries. Synthetic data is appealing because it’s easier to create, but it’s too clean. Real documents have tables that span pages, scans, duplicates, vague project codenames, folders whose names carry information the documents don’t, and so on. Modelling all of those nuances in a synthetic dataset is practically impossible. And the gap between real and synthetic data is meaningful: Jason Liu found https://jxnl.co/writing/2024/05/22/systematically-improving-your-rag/ that RAG systems had 97% recall on synthetic questions, then dropped to 55-65% on real user queries. If you truly have no real data, dig out some real public documents and then create synthetic documents woven around the same entities so that cross-document structure exists. One decision that made it much cheaper for us to run evals is we froze the document extraction step. Extracting text from 1,000 multimodal files with vision models, table parsing, etc. is slow and expensive but broadly deterministic. So we did it up-front and committed the extracted text to its own git repo. That way, we could run every re-ingest from already-extracted text and only needed to pay for embedding + indexing the text. The eval loop became ~100x cheaper and similarly faster. It also enabled us to isolate document extraction from indexing + retrieval two distinct processes that should be eval’d separately . 2. Segment the questions by your product’s logic Instead of writing 100 random questions, we wrote ~20 questions in each of five classes that 1 matched customer behaviour 2 captured the spectrum of capabilities a “company brain” should have. The question categories broke down as follows: Needle questions that pull out one fact . Example: “What discount rate are we assuming in our DCF analysis for Acme?” Entity questions that require the complete document set for one thing . Example: “What do we know about Acme Inc?” Multi-part questions that require documents for different entities to co-appear . Example: “Compare Corp A and Corp B’s valuation metrics.” Aggregation questions that need exact lists or counts . Example: “Do we have any expert calls discussing AI regulation in Europe?” Thematic questions that broadly coverage a topic . Example: “What are the recurring risks across our food-delivery investments?” NB: each of our question categories had their own unique set of metrics that we evaluated such as recall@20, mean reciprocal rank and F1 score . Those metrics are very specific to retrieval and recommendation systems, which you can read about here. Naturally your metrics should be well thought out and reflect what it is that you’re actually evaluating. Why segment like this? Because a single overall score hides the nuance . Our baseline system scored a decent 0.83 overall, but scored only 0.59 on thematic questions and 0.69 on aggregation. The average score would have you believe that things were pretty good. But the category breakdown revealed significant failures for important use-cases. Different question classes also improve for different reasons, which helps you connect metric movements back to specific engineering decisions. If your product has distinct usage patterns then each mode deserves its own question class. 3. Make scoring mechanical no judges All of our questions were scored using deterministic measures, for example: Document-recall against expected document lists “these specific documents should appear” Substring checks against expected evidence “the retrieved chunk must contain 43%” Set precision/recall for list questions “this many results should appear” Coverage of expected string groups for thematic questions “these words or sentences should appear” We didn’t use an LLM judge at all. This kept it free and fast, whereas a judge would have added a second AI system to tune and eval Shankar et al. address this in “ Who Validates the Validators https://arxiv.org/abs/2404.12272 ” . Add a judge only when a metric you need is actually unmeasurable without one. And when you do, calibrate it against your own judgment. One other pointer: make sure your eval system gracefully handles version changes new questions, new files etc . We version both the questions and the final eval reports in git. Our harness auto-generates reports with deltas against the previous run and can gracefully handle drift in the underlying questions and documents. 4. Curate the questions yourself We semi-automated the question generation: an LLM read the ingested corpus and proposed ~250 candidate questions along with ground truth. I then spent a couple hours removing bad questions, sharpening good ones, and drafting my own where I felt there were gaps. I ended up with 100 questions. That couple of hours is crucial: the golden set is where you encode what “good” means for your customers. That’s a judgment only someone who properly understands the customer can make. Big labs pay annotation teams to do this, but frankly you have a better understanding of your customer than any outsourced annotator ever will, so make the most of it. It’s one of the few genuine advantages a small team has over a big one. How it played out for us: real numbers We ran the eval at three checkpoints when improving our knowledge management system https://www.minimumviablefounder.com/p/why-ai-company-brains-fail : Phase 0 baseline; our old RAG system implemented in 2024 Phase 1 improved document ingestion by adding contextual chunk headers, upgraded metadata extraction + filtering Phase 2 built the entity layer that turned our system into a knowledge graph A few things stood out in the results: The biggest win was also the easiest to implement. Thematic recall jumped from 0.59 to 0.83 in Phase 1, the largest single movement we saw. Most of it came from some really easy changes, most notably prepending a two-sentence AI-generated context header to every chunk before embedding it saying something like “This text is from document X and talks about Y” . Anthropic discuss that approach here https://www.anthropic.com/engineering/contextual-retrieval . The fancy entity layer we were so excited about in Phase 2 had a much smaller impact. Without the eval we would have credited the wrong work. The eval caught a regression that we would have otherwise missed. Multi-part recall actually dropped in Phase 2 from 0.88 to 0.82. It turned out that adding new retrieval tools to our agent impacted how results merged for comparison questions. This kind of degradation would have been invisible without evals, creating a hidden pile of technical debt. The eval also told us what not to build. Our original research mapped out a Phase 3: community detection and cluster summaries over the knowledge graph. But by Phase 2 our thematic scores had already hit 0.91 via much cheaper means so we parked it. Sometimes less work is good work; that’s the ethos of The Minimum Viable Founder . It’s just a loop If you look at what we’re doing with evals, it’s just a feedback loop: Decide what “good” looks like i.e. set the golden questions and choose what metrics to measure Make measurement cheap so that the loop can run frequently Change one thing then measure the result Decide what to ship or not based on the result These loops show up everywhere in business: pricing experiments, ad management, onboarding funnels, sales scripts. It’s nothing new. The startup checklist for building an eval harness Use a real corpus that reflects your customers. Only use synthetic files to fill gaps. Segment questions based on usage patterns. Write the golden set yourself AI-assisted, not AI-generated . Freeze any steps in your eval that are expensive and invariant. In our case, document extraction. Do mechanical scoring. Add an LLM judge if it’s the only way to measure your chosen metrics. Make running the eval super easy one command . Otherwise you’ll never run it. Commit eval reports next to code. E.g. an /evals folder in your project root so that it’s easy to track history. Introduce new metrics as the product evolves. Don’t let the eval become dated and obsolete. If you’re building your own eval system or want to better understand how we approach things, I’d love to exchange notes.