{"slug": "stop-building-rag-systems-without-doing-evals", "title": "Stop building RAG systems without doing evals", "summary": "Bootstrapped software company QX Labs built an evaluation system for its enterprise AI brain product in a few days, enforcing that only code with measurable positive impact on AI performance is shipped. Founder Jai argues that evals, often seen as complex and reserved for big labs like OpenAI and Anthropic, are cheap and easy for small teams and critical for maintaining velocity and trajectory in AI development. The eval harness, which uses a dataset, graded questions, and a scorer, has dictated critical product decisions for the solo-founded company.", "body_md": "# AI Evals Aren't Just for Big Tech\n\n### How a small bootstrapped team measures its AI the way frontier labs do: a practical approach to evals that you can copy.\n\n*👋 Hey, Jai here! I’m a software engineer and tech investor turned solo founder, now bootstrapping a software company. I write about using AI to build faster, sell smarter, and run a whole company solo. New here? Join me:*\n\n*Or if you don’t like my work, send me hate mail.*\n\nIn my last post I described [how we overhauled our enterprise “AI brain” product](https://www.minimumviablefounder.com/p/why-ai-company-brains-fail). During the development process, there was one critical constraint: we only shipped code that had a measurable positive impact on our AI’s performance.\n\nToday I’ll dive into the eval system we built to enforce this.\n\nEvals (short for evaluations) measure whether an AI system performs well by grading its responses to a fixed set of questions or tasks. They help steer product development by telling you whether new features improve or regress results, and by pointing to the gaps that you should prioritise.\n\nEvals have a reputation for being a complex and unwieldy discipline reserved for companies with big budgets; something that OpenAI and Anthropic do with dedicated teams, annotation pipelines, and fancy in-house platforms.\n\nBut in reality the inverse is true. **Evals are incredibly useful for small teams.** They’re cheap and easy to build while your product is still small.\n\nMy company [QX Labs](https://www.qxlabs.com) is fully bootstrapped: I founded it solo and built our knowledge management system with one other engineer. Our eval harness took a few days to build and tune, and it has since dictated critical product decisions.\n\nHere’s my underlying thesis. In business you need to solve for two things: velocity and trajectory. The most valuable thing you can build in the AI age isn’t the system itself, but the feedback loop that tells you whether your system works and how to improve it. Build the loop and you can sustain velocity without hitting a wall, because you’ll always know your next step. Evals are one of those loops.\n\nBy the end of this post you’ll have:\n\nA working mental model of what evals are and how the biggest AI companies use them\n\nA copyable approach to evals for small teams\n\nReal before/after numbers from our RAG evals\n\n## What an eval actually is\n\nAn eval is made up of three things:\n\n**A dataset** that represents your real workload (documents, tickets, conversations, or whatever your system operates on)**A set of graded questions or tasks** with pre-defined correct answers (“golden” questions)**A scorer** that runs every question through your system and outputs numbers\n\nYou run it before and after a change, and the delta tells you the result of that change.\n\nThe closest analogy is an engineering test suite, but with one key difference: unit tests are binary (they pass or fail), while evals are statistical and directional (AI can get better for one set of questions and worse for another).\n\nThat’s the trap with LLM systems: they don’t fail by erroring, but instead by confidently delivering mediocre results. A retrieval system that misses half the relevant documents will write an authoritative response with gaping holes. Without evals, you’d never even know.\n\n## What the experts recommend\n\nBefore building our eval harness, I looked at how the top labs run theirs. Surprisingly, most of their advice is highly suited to small teams. Greg Brockman (President of OpenAI) is a strong advocate:\n\nHere’s what they say:\n\n**Gate your evals like a continuous integration (CI) pipeline.** OpenAI [open-sourced their evals framework](https://github.com/openai/evals) back in 2023, calling evals “one of the most impactful things you can do” when building with LLMs. They name the practice *eval-driven development*. “Evaluate early and often. Write scoped tests at every stage.”\n\n**Start small and cover real failures.** [Anthropic’s engineering guidance](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) says: “We see teams delay building evals because they think they need hundreds of tasks. In reality, 20–50 simple tasks drawn from real failures is a great start.” That’s totally feasible for a small team or a solopreneur.\n\n**Eyeball your system’s outputs and errors before coming up with metrics.** Hamel Husain’s “[Your AI Product Needs Evals](https://hamel.dev/blog/posts/evals/)” is worth a read. He insists that the first step is to read your system’s actual outputs one-by-one and categorise the failures. Only then should you come up with the appropriate metrics to measure against those failures. Similar to Anthropic he recommends starting with ~30 hand-graded examples. In short: use your own product like your customers would, manually log and categorise the failures, then feed that data into your eval system.\n\n**Avoid using LLM “judges” for grading.** The big shops need AI judges to grade open-ended outputs at scale, but they calibrate them against human graders because research shows that LLM-graded evals can otherwise be very flimsy. [The MT-Bench paper](https://arxiv.org/abs/2306.05685) (Zheng et al.) outlines their biases: position bias (judges favour the first response rather than the best one), verbosity bias (judges favour longer responses >90% of the time regardless of quality), and self-enhancement (models score their own outputs higher). The lesson for a small team is not “don’t use judges,” it’s “don’t *start* with judges.” Stick to mechanical, deterministic scoring first.\n\nIn the end **big-tech evaluation is expensive due to scale and not the underlying to process.** A small team doesn’t need thousands of evals in CI, expensive annotation or judge calibration pipelines. They just need to run a process of getting fixed data, creating graded questions and scoring every change. That costs almost nothing.\n\n## The small-team version: our recipe for evals\n\nFirst, some context: one of [our products](https://www.qxlabs.com) at QX ingests a company’s documents (Drive, SharePoint, Granola, Notion, etc.) and lets AI agents answer questions across them. When we originally built the MVP it used a simple RAG pipeline for indexing and retrieval, but we wanted to turn it into a full knowledge graph. However, we needed a way to measure whether the changes we implemented were actually improving the agent’s ability to query data.\n\nSo before doing anything, we built the eval harness. Here’s how we approached it.\n\n### 1. Build the eval around your customer archetype\n\nOur dataset uses ~1,000 real documents that represent one hypothetical customer’s universe. In our case we mimicked an investment firm’s internal data: PDFs, spreadsheets, decks, memos, meeting notes, scanned pages. Specifically, we wanted things to look multimodal, messy, and organised into a folder hierarchy an actual firm might accumulate over time. Having raised my owned fund and worked as an investor for 10 years, I had a very good mental model of what this should look like.\n\nThat right there is the secret sauce to good evals: **model them closely on your end-customers’ files and queries.** Synthetic data is appealing because it’s easier to create, but it’s too clean. Real documents have tables that span pages, scans, duplicates, vague project codenames, folders whose names carry information the documents don’t, and so on. Modelling all of those nuances in a synthetic dataset is practically impossible.\n\nAnd the gap between real and synthetic data is meaningful: [Jason Liu found](https://jxnl.co/writing/2024/05/22/systematically-improving-your-rag/) that RAG systems had 97% recall on synthetic questions, then dropped to 55-65% on real user queries. If you truly have no real data, dig out some real public documents and then create synthetic documents woven around the same entities so that cross-document structure exists.\n\nOne decision that made it much cheaper for us to run evals is we **froze the document extraction step.** Extracting text from 1,000 multimodal files (with vision models, table parsing, etc.) is slow and expensive but broadly deterministic. So we did it up-front and committed the extracted text to its own git repo. That way, we could run every re-ingest from already-extracted text and only needed to pay for embedding + indexing the text. The eval loop became ~100x cheaper and similarly faster. It also enabled us to isolate document extraction from indexing + retrieval (two distinct processes that should be eval’d separately).\n\n### 2. Segment the questions by your product’s logic\n\nInstead of writing 100 random questions, we wrote ~20 questions in each of five classes that (1) matched customer behaviour (2) captured the spectrum of capabilities a “company brain” should have. The question categories broke down as follows:\n\n**Needle** questions (that pull out one fact). Example: “What discount rate are we assuming in our DCF analysis for Acme?”**Entity** questions (that require the complete document set for one thing). Example: “What do we know about Acme Inc?”**Multi-part** questions (that require documents for different entities to co-appear). Example: “Compare Corp A and Corp B’s valuation metrics.”**Aggregation** questions (that need exact lists or counts). Example: “Do we have any expert calls discussing AI regulation in Europe?”**Thematic** questions (that broadly coverage a topic). Example: “What are the recurring risks across our food-delivery investments?”\n\n*NB: each of our question categories had their own unique set of metrics that we evaluated (such as recall@20, mean reciprocal rank and F1 score). Those metrics are very specific to retrieval and recommendation systems, which you can read about here. Naturally your metrics should be well thought out and reflect what it is that you’re actually evaluating.*\n\nWhy segment like this? Because a single overall score hides the nuance**.** Our baseline system scored a decent 0.83 overall, but scored only 0.59 on thematic questions and 0.69 on aggregation. The average score would have you believe that things were pretty good. But the category breakdown revealed significant failures for important use-cases.\n\nDifferent question classes also improve for different reasons, which helps you connect metric movements back to specific engineering decisions. If your product has distinct usage patterns then each mode deserves its own question class.\n\n### 3. Make scoring mechanical (no judges)\n\nAll of our questions were scored using deterministic measures, for example:\n\nDocument-recall against expected document lists (“these specific documents should appear”)\n\nSubstring checks against expected evidence (“the retrieved chunk must contain 43%”)\n\nSet precision/recall for list questions (“this many results should appear”)\n\nCoverage of expected string groups for thematic questions (“these words or sentences should appear”)\n\nWe didn’t use an LLM judge at all. This kept it free and fast, whereas a judge would have added a second AI system to tune and eval (Shankar et al. address this in “[Who Validates the Validators](https://arxiv.org/abs/2404.12272)”). Add a judge only when a metric you need is actually unmeasurable without one. And when you do, calibrate it against your own judgment.\n\nOne other pointer: **make sure your eval system gracefully handles version changes** (new questions, new files etc). We version both the questions and the final eval reports in git. Our harness auto-generates reports with deltas against the previous run and can gracefully handle drift in the underlying questions and documents.\n\n### 4. Curate the questions yourself\n\nWe semi-automated the question generation: an LLM read the ingested corpus and proposed ~250 candidate questions along with ground truth. I then spent a couple hours removing bad questions, sharpening good ones, and drafting my own where I felt there were gaps. I ended up with 100 questions.\n\nThat couple of hours is crucial: **the golden set is where you encode what “good” means for your customers.** That’s a judgment only someone who properly understands the customer can make. Big labs pay annotation teams to do this, but frankly you have a better understanding of your customer than any outsourced annotator ever will, so make the most of it. It’s one of the few genuine advantages a small team has over a big one.\n\n## How it played out for us: real numbers\n\nWe ran the eval at three checkpoints when [improving our knowledge management system](https://www.minimumviablefounder.com/p/why-ai-company-brains-fail):\n\n**Phase 0**(baseline; our old RAG system implemented in 2024)** Phase 1**(improved document ingestion by adding contextual chunk headers, upgraded metadata extraction + filtering)** Phase 2**(built the entity layer that turned our system into a knowledge graph)\n\nA few things stood out in the results:\n\n**The biggest win was also the easiest to implement.** Thematic recall jumped from 0.59 to 0.83 in Phase 1, the largest single movement we saw. Most of it came from some really easy changes, most notably prepending a two-sentence AI-generated context header to every chunk before embedding it (saying something like “This text is from document X and talks about Y”). Anthropic discuss that approach [here](https://www.anthropic.com/engineering/contextual-retrieval). The fancy entity layer we were so excited about in Phase 2 had a much smaller impact. Without the eval we would have credited the wrong work.\n\n**The eval caught a regression that we would have otherwise missed.** Multi-part recall actually dropped in Phase 2 from 0.88 to 0.82. It turned out that adding new retrieval tools to our agent impacted how results merged for comparison questions. This kind of degradation would have been invisible without evals, creating a hidden pile of technical debt.\n\n**The eval also told us what not to build.** Our original research mapped out a Phase 3: community detection and cluster summaries over the knowledge graph. But by Phase 2 our thematic scores had already hit 0.91 via much cheaper means so we parked it. Sometimes less work is good work; that’s the ethos of *The Minimum Viable Founder*.\n\n## It’s just a loop\n\nIf you look at what we’re doing with evals, it’s just a feedback loop:\n\nDecide what “good” looks like (i.e. set the golden questions and choose what metrics to measure)\n\nMake measurement cheap so that the loop can run frequently\n\nChange one thing then measure the result\n\nDecide what to ship (or not) based on the result\n\nThese loops show up everywhere in business: pricing experiments, ad management, onboarding funnels, sales scripts. It’s nothing new.\n\n## The startup checklist for building an eval harness\n\n**Use a real corpus that reflects your customers.** Only use synthetic files to fill gaps.**Segment questions based on usage patterns.****Write the golden set yourself**(AI-assisted, not AI-generated).** Freeze any steps in your eval that are expensive and invariant.**In our case, document extraction.** Do mechanical scoring.**Add an LLM judge if it’s the only way to measure your chosen metrics.** Make running the eval super easy (one command).**Otherwise you’ll never run it.** Commit eval reports next to code.**E.g. an /evals folder in your project root so that it’s easy to track history.** Introduce new metrics as the product evolves.**Don’t let the eval become dated and obsolete.\n\n*If you’re building your own eval system or want to better understand how we approach things, I’d love to exchange notes.*", "url": "https://wpnews.pro/news/stop-building-rag-systems-without-doing-evals", "canonical_source": "https://www.minimumviablefounder.com/p/ai-evals-arent-just-for-big-tech", "published_at": "2026-07-29 18:40:58+00:00", "updated_at": "2026-07-29 18:52:25.612071+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-tools", "ai-products", "ai-research", "ai-infrastructure"], "entities": ["QX Labs", "OpenAI", "Anthropic", "Greg Brockman", "Jai"], "alternates": {"html": "https://wpnews.pro/news/stop-building-rag-systems-without-doing-evals", "markdown": "https://wpnews.pro/news/stop-building-rag-systems-without-doing-evals.md", "text": "https://wpnews.pro/news/stop-building-rag-systems-without-doing-evals.txt", "jsonld": "https://wpnews.pro/news/stop-building-rag-systems-without-doing-evals.jsonld"}}