# How we help your agent find what it’s looking for > Source: > Published: 2026-11-12 00:00:00+00:00 # How we help your agent find what it’s looking for November 12 Connect an agent to your meeting notes and ask it to tell you everything a customer said about pricing in the last month, and it'll give you a confident answer, but not a correct one. It will search the note, which captured the final decisions made by those in the room, but not the transcript, where the numerical details were hammered out, and where the real discussion happened. The Granola MCP had a way for your agent to read a transcript, but it had to work for it: find the right meeting, open the note, then pull the full, long transcript into their context window, one meeting at a time. We knew there was a hole to fill because users kept telling us their workaround, copying transcripts out of Granola by hand and pasting them manually into whatever agent they were using. So over the last few months we rebuilt how agents retrieve meeting context and information from Granola. We knew we wanted to improve two main things: how agents find the right meeting, and how they find what was said in it. ## Before we started this work, connecting an agent to Granola using the MCP meant using a tool called `list_meetings`, which filtered by date and nothing else. We had better retrieval tools in our in-app chat (keyword search, people search), but none of them had been ported to the MCP yet (Remember when nobody knew whether MCP would take off?), so beyond finding the date, agents had to do everything themselves. Unfortunately, they didn't: faced with 30 meeting notes, an agent will either confidently answer based on the six or seven it has read so far, or tell you it couldn't find what you were looking for. So we gave the agent a way to search. We'd been running a note search tool internally for a little while called `search_meetings`, which provided a date-ordered listing if the agent provided no keywords, or if it did, a BM25 search over an index in Turbopuffer. ## Early on, we noticed that asking the same question with different clients (Claude Desktop, Codex, etc.) returned different answers, so our first job when building our MCP eval harness was turning all of these observations into numbers. For questions where we knew the answer was in the transcript (rather than the note), we needed to know how often an agent actually chose to read a transcript, and how often it included what it found in its answer. For every fact the agent needed, we checked three crucial factors: ### This was a check on whether the fact (for the sake of argument, 'everything a customer said about pricing in the last month') ever even reached the agent. We checked this deterministically: the reference note or transcript passage either made it into one of the agent's tool responses for that run, or it didn't. ### Did the answer the agent gave actually use that fact? We used ROUGE-L recall to check what share of the fact's words showed up in the answer, in the same order, then calibrated on 90 hand-labelled examples. ### We had an LLM judge check whether every required fact was present, backed by two checks that didn't trust our judge. We checked if the answer cited a meeting (note and/or transcript) the agent hadn't read, and whether it put quotation marks around words that weren't in any tool response. Both of those were deterministic string matches; either one would fail the trial completely. We split it this way because each of these – retrieval, retainment, accuracy – pointed to a different fix. If the fact never got to the agent, search needed improving. If it made it to the agent but got dropped, it was either because of the model, or the way we surfaced search results. If it made it to the model but the answer was still wrong, we had either given it too much extraneous information, or the agent had done something wrong along the way. ## We hill-climb on our own data. We share almost everything with the whole company, which gives us a rich corpus of real meeting and transcript data that we can query using eval agents, and write test cases for. Because LLMs are non-deterministic, we ran everything in evals at n=10 (10 repeats) and even small improvements in the results led to significant improvements with users. For example, a 4% absolute improvement meant that there were now 16 additional questions the MCP could answer accurately. ## A one-hour meeting yields 60,000-80,000 characters, so a month's worth of meeting transcripts wouldn't fit in a single context window. So we built an index. Each workspace got a BM25 Turbopuffer namespace, and when a meeting finished we cut the transcript into chunks and identified the meeting attendees, which of them spoke in each chunk, and a version string. ### We optimistically thought we'd be able to launch this immediately, but the first eval run proved us wrong. In cases where the agent had to get the answer from the transcript, 34% of agents never called transcript search at all. The index was there, but the agent didn't use it. It read the note only, then gave a confident wrong answer. So we stopped hill climbing on improving the index, and set about improving the reader – an interesting thing to do via an MCP, as you have very little control, beyond tool descriptions and (sometimes) server instructions. But we tried. We gave an agent an experiment ledger and had it run autonomously overnight, making small changes, running evals, and appending findings to the ledger. Twenty variants later, we had 88 batches across five datasets, four models and two MCP clients. ### We told agents returning with results from notes that, "The summary is a condensed overview; the transcript is the record", which raised transcript use from 6% to about 45% on the 14 questions only the transcript could answer. Adding to the server instructions that, "Notes are an index, not the answer" took it to 83%. Required facts from the transcripts went from about a 35% pass rate to a 50% one. Then we had a new problem. Agents reading transcripts were tidying the quotations up, so as many as 40% of answers now had a lightly reworded (and thus unsupported) quote. In our eval harness, one unsupported quote failed the case, so pass rate on note cases fell from 72% to as low as 41%. We'd made the hard cases better, but the easy cases worse. The fix we landed on was to tell agents to paraphrase by default, and quote only when asked for a direct quote. Unsupported quotes fell back to 6% and note questions recovered to 72%. We gave back some of the transcript gain to get there (required facts settled at 41%), but on the full set of 65 transcript questions, Sonnet's pass rate still went from 24% to 37%. Always sending agents to the transcript had one more cost. Asked for a recap, they answered from a few searched passages and missed things the summary already covered. So we now let them answer recaps from the summary and save the transcript for specifics, which kept the transcript gain and won back the general questions. ## If you're building retrieval systems for agents you need to measure each stage on its own: did the agent retrieve it, did it retain it, and did it answer accurately. Give the agent less than it wants; tell it where the rest is. Run the easy cases every time you run the hard ones. Building the index is the easy part. More important is communicating clearly to agents, giving them more precise tool descriptions and instructions. Finally, because we came to see how agents' reliance on notes over transcripts led to inaccurate answers, we tried one more experiment. What would happen, we wondered, if we took away notes entirely? When agents went exclusively to the transcript, their answers got worse. The note provided something invaluable: meeting context that the transcript would never state plainly, such as who someone is, or what a project name is referring to. That essential context meant that agents needed to be pointed to the note first, before going to the transcript for the rest. What we're shipping next will show just how useful that meeting context can be. [Josh Tuddenham](https://www.granola.ai/authors/josh), AI Engineer