{"slug": "when-ai-writes-the-code-show-your-judgment", "title": "When AI Writes the Code, Show Your Judgment", "summary": "A case study published by an unnamed author documents an AI workflow that extracted five years of past customers from a United Kingdom HVAC company's email, arguing that detailed public case studies are replacing screenshots as the way builders demonstrate judgment. The HVAC client chose to analyze mailbox data before it reaches a CRM, betting on re-engaging customers with expiring warranties to secure repeat servicing, since individual service visits rarely turn a profit but bundling jobs in the same neighborhood or pairing them with a nearby installation cuts travel and transport costs. The author contends that judgment must compound into a predictable pattern for others to trust it, requiring builders to share thinking processes they normally keep private.", "body_md": "Showing Taste in Judgment: An AI Workflow Case Study in Mailbox Analysis\n\nI'm betting that detailed case studies are becoming the way to show skill. This one records the judgment behind an AI workflow that pulled five years of past customers out of a company's email.\n\nIf the assistant can’t open the study, use Copy page and paste it into your chat.\n\nIntroduction\n\nThere’s no shortage of opinions right now on how software should be built. Like most of us on X, I’m watching from the sidelines, testing out new workflows and exploring the ideas posted. There are already great discussions happening about this, but there is always room to explore further how we build the underlying skills and core judgment, and then communicate it, and get better at it in a way that outlasts any workflow.\n\nIn short, the solution I am betting on is a shift toward mainstream, detailed case studies, much like the one I share below, so that judgment can be displayed, evaluated, and improved in public. Consultants have been doing this for a while, and many senior software engineering roles already rely on it during hiring. But the days of showing just two or three screenshots and a simple link to previous work and a short description have come to an end.\n\nShowing the outcome of a product in these times may not be enough. People used to immediately realize how much effort and thought you put into a project when you showed them a final walkthrough. Now, you are left open to being second-guessed on whether the project was actually well thought out. The person viewing your product is tasked with understanding the domain and putting in the effort to see if you have sound judgment, something they are unlikely to do, because it is so hard to test a person's judgment from an isolated result. Judgment needs to compound and form a predictable pattern so that others can truly believe you can implement the same pattern in their projects. Ultimately, this means we need to share parts of our thinking process that we are not usually familiar with, which can certainly feel a bit vulnerable, but it gives us an opportunity to communicate our judgment, open a window to the public, and allow for valuable feedback to sharpen and improve it.\n\nFind the goal\n\nThe business and customer journey\n\nOur case study is for an HVAC company based in the United Kingdom. They have been operating for five years installing and servicing heating and cooling equipment on businesses, real estate, and private property. They communicate with customers mainly through email and keep their data in spreadsheets. As the business grows, keeping track of customers has become harder over the years.\n\nThe owner has considered using a standard CRM featuring an email pipeline that syncs conversations and automatically populates customer names. However, the client bets on the fact that more can be done if we are able to analyze data within the email before it reaches a CRM.\n\nTo go deeper into what the client plans to understand from their mailbox, we have to understand how the business operates. The enterprise functions as one large continuous loop, stretching from generating work and receiving payments to reinvesting revenue into generating more work.\n\nWithin this operational cycle, every job follows a specific path from start to finish. The flow diagram below outlines the complete progression of how an incoming opportunity turns into a completed job.\n\nThe client’s bet\n\nThe client bets on re-engaging customers with expiring warranties to secure repeat servicing. Although individual service visits rarely turn a profit on their own, profitability becomes achievable when bundling multiple jobs in the same neighborhood or pairing them with a nearby installation, which cuts down on travel time and transportation expenses.\n\nAfter getting all this background on the business and the client's bet, we need to set a goal that lets us constantly check if we are actually moving toward it. Establishing this goal brings structure and clarity, sitting right in the sweet spot of not being too broad or too narrow, which gives us the flexibility to switch bets if the current one doesn't pan out.\n\nSo below is the goal.\n\nOur goal\n\nThe business has five years of customers sitting unreachable in the Gmail mailboxes of several people. The goal is to get them out in a form the business can act on.\n\nIt clearly defines what we plan to do, not too narrow, but not too broad either.\n\nElements, connectors and bottlenecks\n\nOur goal functions as a distinct unit within a system. My notes in Systems Thinking give me a way to break this down into three key concepts:\n\nelements (actors). The partakers in the system.\n\nconnectors. The relationships between those elements.\n\nbottlenecks within the connectors. The friction points that affect how those relationships actually function.\n\nMake it work\n\nKickoff\n\nOur order of work is borrowed from Sam Bhagwat’s Principles of Building AI Agents: make it work, make it right, make it fast, a philosophy widely attributed to Kent Beck.\n\nmake it work. Get an email thread through the workflow and produce a record I can inspect.\n\nmake it right. Check the analyzed records against its sources and the client’s expectations. Find the mistakes and revise the system.\n\nmake it fast. Run more email threads through it and find which constraints appear under load and find ways it can scale to process more email threads.\n\nmake it useful. Run continuous Product Discovery through interviews, and make bets based on what the underlying system is capable of handling.\n\nThe last part (Make it useful) is my addition to Kent Beck's structure. Product discovery happens throughout the project, but giving it its own section makes room to look for opportunities where the extracted data can produce value. Similar parallels exist: some put 'Make it Useful' at the top, some at the end, and some treat it as continuous.\n\nFor this part, we divide the process into six stages and go through them one by one.\n\nPrepare\n\nPreparing the data covers getting the export and storing it in a form we can manipulate and analyze.\n\nPrepare (current step)\n\nFilter\n\nGroup\n\nExtract details\n\nAnalyze\n\nShow\n\nSo we ask, how do we get the client’s email without disrupting their day-to-day work?\n\nWe have several opportunities, each with their own pros and cons, which are mapped out in the opportunity tree below.\n\nAsking for an .mbox file gave all the access we wanted and the freedom to store and manipulate the data at hand. This is the first of multiple .mbox files to be acquired.\n\nWhich language should I use?\n\nI wanted to use a language I am comfortable with. I found no reason to pay the cost of switching languages. Python would also fit the import and processing work, but I just enjoy reading TypeScript, though I can't claim I write it anymore.\n\nDatabase design\n\nChoosing SQLite to store our data naturally leads to the next question, how should we design our database model? Drawing from Michael J. Hernandez’s Database Design for Mere Mortals, the process starts with a clear mission statement and objective, which aligns with our core extraction goals.\n\nmission objectives. Focused on extracting specific business requirements, identifying equipment installations, service locations, quote and invoice histories, and warranty timelines, to allow repeat job bundling.\n\nlegacy database and storage. Consists of email back-and-forth and spreadsheets. When inspecting these current systems, Hernandez cautions against copying spreadsheet layouts directly into table structures, since these layouts often combine multiple distinct subjects that should instead be separated into relational tables.\n\nbusiness workflow. Mapped across five core stages, enquiries → site visits → quotes → invoices → completed jobs.\n\nDatabase design is inherently iterative and refined continuously throughout the design phases, but our logical design phase is now complete after multiple feedback loops, ending up with the database diagram below.\n\nFilter\n\nPrepare (done)\n\nFilter (current step)\n\nGroup\n\nExtract details\n\nAnalyze\n\nShow\n\nIn the filtering phase, we need to remove emails unrelated to our core goal. These filtering opportunities are apparent by simply reviewing the email archive.\n\nRead the options\n\nGmail-labeled\n\nEmails Gmail has already labeled Promotions, Purchases, Travel or Spam.\n\nTransactional\n\nBusiness emails that pass the labels but are obvious transactional emails, like no-reply logins.\n\nEverything else\n\nEmails that are not transactional and not caught by the Gmail labels, but are still not related to the customer. These are the harder ones to identify.\n\nFigure 6Filtering non-customer email. Three categories to drop. The last is the hardest to identify.\n\nThe bigger question is the age-old debate of deterministic versus AI filters. While the modern consensus favors a hybrid approach, we still have to ask: where should we draw the line, and what parts of the task can AI actually take over from traditional deterministic filters?\n\nInstead of letting that assumption bias our approach again, let's keep it front and center while we first evaluate what a purely deterministic check can achieve and how far it can take us. Specifically, deterministic checks can easily filter out labeled Gmail messages and routine transactional business emails.\n\nThe mailbox after the deterministic checks\n\n27.1%\n\n72.9%\n\nDeterministic checksTransactional, promotions and marketing emails27.1% removed\n\nLeft for extractionThe remaining emails that are ready for extraction72.9% kept\n\nFigure 7Without an AI filter. What the deterministic checks alone removed, as a share of the whole mailbox.\n\nRather than adding a separate AI filter to clean out the remaining noise, I chose to let the extraction agent absorb it, trusting that it would only surface records when actual job data was present. We are not optimizing to eliminate all noise entirely; the goal is simply to extract relevant details, not to achieve zero noise.\n\nFast-forwarding to the results, letting the email extraction agent handle the remaining data paid off. While the agent was busy extracting relevant details, it indirectly categorized 11% of the emails as unrelated noise.\n\nThe mailbox after extraction\n\n27.1%\n\n61.8%\n\nDeterministic checksTransactional, promotions and marketing emails27.1% removed\n\nIndirectly marked by the email agentEmails the deterministic checks missed, but the email agent indirectly caught11.1% flagged later\n\nCustomer mailThe remaining emails that are ready for extraction61.8% kept\n\nFigure 8With the email agent filtering. The checks removed 27.1%. The agent marked another 11.1% unrelated while extracting, without a classifier of its own.\n\nGroup\n\nPrepare (done)\n\nFilter (done)\n\nGroup (current step)\n\nExtract details\n\nAnalyze\n\nShow\n\nWe can group the separate emails into threads using their unique X-GM-THRID headers found within the exported mbox data.\n\nExtract details\n\nPrepare (done)\n\nFilter (done)\n\nGroup (done)\n\nExtract details (current step)\n\nAnalyze\n\nShow\n\nTo gather enough context regarding the data we need to extract, we ask ourselves and the client, what specific fields does the business need to know about a past customer or job, and exactly where within the messages are they located?\n\nFieldDetailsSource\n\nPeopleDetailsNames and organizations of customers, staff, installers, suppliers and referrersSourceEmailAttachment\n\nRolesDetailsCustomer, staff, installer, supplier or referrerSourceEmailDerived\n\nContact detailsDetailsEmail addresses and phone numbersSourceEmail\n\nLead sourceDetailsReferral person or platform, only when supported by evidence; otherwise UnknownSourceEmailDerived\n\nJob stageDetailsLatest supported state of the job across the conversation, including planned versus completed workSourceDerived\n\nJobDetailsDescription and service typeSourceEmailAttachment\n\nService addressDetailsFull work address and address componentsSourceEmailAttachment\n\nEquipmentDetailsBrand, model, quantity, capacity, rooms and statusSourceEmailAttachment\n\nEventsDetailsInquiry, site visit, quote acceptance, installation scheduled or completed, rejection and cancellationSourceEmailAttachment\n\nSending and paymentDetailsQuote sent, invoice sent, deposit paid and remaining balance paidSourceEmail\n\nEvent detailsDetailsDate, status, description, amount and currencySourceEmailAttachment\n\nEmail typeDetailsCustomer-related or unrelated; relevant messages may contain an enquiry, quote or invoiceSourceDerived\n\nReview notesDetailsMissing information, uncertainty and conflictsSourceDerived\n\nEmail\n\nRead from the body, subject or headers.\n\nAttachment\n\nExtracted from files in the sandbox.\n\nDerived\n\nInterpreted from evidence across emails and attachments.\n\nTable 2Fields to extract. Derived fields need interpretation. Missing evidence stays unknown.\n\nBased on our findings, the required fields are buried across different parts of the thread, some reside in the email body, others are located within attachments, and a few must be inferred from the conversation.\n\nTo figure out how to approach this set of extractions, we first needed to determine what pattern to use, adapting to the reality that certain groups of extractions could be handled by deterministic logic while others required an agent. As Antonio Gulli’s Agentic Design Patterns highlights, we shouldn't treat these patterns as an either-or choice; instead, we approach them as modular blocks you build upon along the way. In our case, we ended up using all the patterns, starting from basic prompt chaining to parallelization, routing, and finally agent-directed execution.\n\nFundamentally, this aligns with Anthropic’s perspective on building effective agents, where a workflow is defined by paths set in code. We know the desired end result because the output and the broad sequence are known beforehand, while the model is left to handle the interpretation within each step.\n\nHow much control should the model have?\n\nPrompt chaining. Each model call works on the output of the one before it.Attachment-reader findings feed the final extraction prompt. Email-only conversations can use a single extraction prompt. The ordinary sequence of code steps doesn’t itself count as prompt chaining.\n\nParallelization. Independent pieces of work run at the same time.Multiple conversations are extracted concurrently, two by default and configurable up to 40. Customer and job reconciliation runs sequentially.\n\nRouting. A step sorts each input and sends it down its own path.Email rules decide what reaches extraction, and attachment rules and a classifier decide which documents to read. There’s no separate AI router classifying every email before extraction.\n\nAgent-directed execution. The model chooses its own next steps and tools.Within bounded tasks, the reader can request more text and submit findings, and the extraction agent can retrieve attachment pages. Code controls the overall workflow, available tools and limits.\n\nFigure 9How much control the model gets. Four patterns, each in its own bounded place. Sandboxing is a technique, not a fifth. Select one to see how.\n\nAt a high level, the proposed workflow is the following.\n\nAttachments · SandboxEmail · AI agentAnalyze · AI agent\n\nFigure 10Extraction pipeline. Read attachments in a sandbox, read the email, then interpret the two together.\n\nThis orchestration can be done bare bones or with a framework, or even with a no-code tool.\n\nTool to build the workflow\n\nPlain code. Direct model calls and ordinary functions. Few abstractions, but I would build the tracing, retries and experiment tooling myself.\n\nn8n. A visual workflow with integrations and custom-code steps. Useful when much of the work is connecting services.\n\nLangChain. An ecosystem in Python and JavaScript for composing model calls, tools and workflows.\n\nMastra. TypeScript agents, workflows, traces and evals in one development environment.It fit the language I knew and the short inspect-change-rerun loop I wanted.\n\nFigure 11Tool to build the workflow. Several tools fit. I chose around the development loop I expected to live in.\n\nI preferred using TypeScript since we already had the mailbox scripts built with it. LangChain (which also has a TypeScript package now) and no-code solutions could still work, but it gets a bit subjective at this point. I just really like Mastra for its trace viewers, evals UI, and their open-source philosophy. Plus, they put out great content for the community. Two of the key resources I used to understand agents more came from Sam Bhagwat's books, Principles of Building AI Agents and Patterns for Building AI Agents, and they also put out podcasts and workshops that are well worth checking out.\n\nSo, Mastra it is. Once we have a filtered golden set of grouped emails and threads, the first step Mastra handles is dealing with the attachments based on our proposed pattern in Figure 10.\n\nExtracting data from attachments\n\nReading every attachment in every email thread would consume unnecessary time, compute power, and money. Our focus is strictly on extracting data from quotes and invoices, as all the attachment-sourced fields listed in Table 2 reside within them.\n\nHowever, we still need a way to filter out quotes and invoices since there are many unrelated attachments, and their filenames don't follow any specific naming convention that we can filter out deterministically.\n\nExample email · fictional attachmentsYour installation quoteHi Sarah,I’ve attached the quote for the new system, along with the invoice for the deposit. Let me know if anything needs changing and I’ll book the install.Thanks, Daniel\n\nQuote.pdf\n\nBrochure.pdf\n\nINV-2210.pdf\n\nIMG_2041.jpg\n\nFigure 12An email with attachments. A fictional installation email. Only the quote and the invoice carry what the workflow needs.\n\nIntroducing Jev\n\nRight on cue, System 1 models were introduced (Jev calls them System 1 models rather than classifiers, apparently it's a big deal). But, given primitives for score, choice, and noul, I could use the choice primitive, passing in the filenames, file size and the entire email thread, to classify each attachment as a quote, invoice, or other.\n\nI used Daytona for the sandbox, though several providers could have done the job. The server decodes the selected attachments from the email archive and uploads them through Daytona’s file API. Mastra Workspaces then gives the agent access to those files through the sandbox’s filesystem.\n\nReading the attachments inside the sandbox can be done in multiple ways. It is actually quite nuanced, as there are several tool options available, each tailored to different scenarios.\n\nRead the options\n\npypdf\n\nA Python library that extracts the text page by page.Best for: Quick text from simple, digital PDFs.Advantage: Pure Python, with no system tools to install, so it is the lightest to set up.\n\npdfplumber\n\nExtracts text with its layout, and pulls tables out row by row.Best for: Quotes and invoices with tables and line items.Advantage: Keeps the layout and pulls tables out row by row.\n\npdftotext\n\nA command line tool from poppler-utils.Best for: Reading many PDFs quickly.Advantage: Can retain layout or emit plain text without a model call.\n\nOCR\n\npytesseract with pdf2image, reading each page as an image.Best for: Scanned PDFs and photos with no text in them.Advantage: Reads text the other three cannot see.\n\nFigure 13Reading a PDF. A digital text layer and a scanned page need different treatment.\n\nLuckily, Anthropic provides a PDF skill that I can hand off to the agent, letting it use its own judgment to pick the best package for any given PDF to read. This required pre-installing all the necessary libraries and CLI tools so the agent can call them.\n\nKept as a quote or invoice\n\nFigure 14Attachment pipeline. Attachments are decided first, then read one at a time in the sandbox using a method suited to each PDF.\n\nRead the email body\n\nWith the attachments read, I can pass their extracted details to an AI agent alongside the email body to return structured information as shown in Table 2. Much of that information can be read directly, with service locations, quote IDs, and invoice IDs found in the attachments while the rest reside in the email itself.\n\nThe email content agent brings the email body and extracted attachment data together to recover customer details and identify the signals described in Analyze.\n\nEmailAttachment data\n\nEmail content agent\n\nStructured dataSignalsJob stage · sales outcome · lead source\n\nFigure 15Email content agent. Email and attachment details enter together. The output keeps recorded details apart from interpreted signals.\n\nAnalyze\n\nPrepare (done)\n\nFilter (done)\n\nGroup (done)\n\nExtract details (done)\n\nAnalyze (current step)\n\nShow\n\nOur analysis has three steps. We find evidence through job signals and lead-source clues, interpret what it supports, then record the result in the structured fields shown in Table 2.\n\nStepJob progressLead source\n\n1. Find evidence through signalsJob progressAn email says, ‘Your installation is booked for Tuesday.’Lead sourceA Bark notification introduces the customer’s request.\n\n2. Interpret itJob progressThe installation is scheduled. Completion is still unconfirmed.Lead sourceThe lead came through Bark.\n\n3. Record itJob progressSave the scheduled event, its date and supporting evidence. Leave completion unknown.Lead sourceSave Bark as the lead source, linked to the supporting email.\n\nTable 3Three steps of analysis. From evidence to interpretation to a recorded conclusion, using a fictional booking and lead notification.\n\nThe job signals and lead-source clues we look for in the first step are listed below.\n\nJob signals\n\nInquiry.The first contact about a particular job, including a customer request, a platform lead or the business's opening message.\n\nSite visit.A survey or assessment visit is proposed, arranged or has happened.\n\nQuote sent.A quote was sent, or its delivery is explicitly planned or uncertain.\n\nQuote accepted.The customer accepted the quote. Under the current business rule, a formal numbered accounting quote also establishes acceptance.\n\nInvoice sent.The invoice was delivered to the customer. Having an invoice attachment alone does not establish this.\n\nDeposit paid.There is explicit confirmation that the initial deposit was paid.\n\nRemaining balance paid.There is explicit confirmation that the final balance was paid.\n\nInstallation scheduled.An installation date or slot has been agreed with the customer.\n\nInstallation completed.There is explicit confirmation that the work has been completed.\n\nRejection.The customer declined the quote before committing to the work.\n\nCancellation.Previously agreed or booked work was called off. A postponement or changed appointment alone does not count.\n\nLead source\n\nBark.A Bark lead notification in the conversation.\n\nGoogle Ads.An explicitly identified Google Ads lead submission.\n\nWebsite.A message identifying the website enquiry form.\n\nGoogle Forms.A form notification that identifies the intake channel.\n\nUnknown.No supporting evidence. A form alone does not prove which advert brought the customer.\n\nNow the question becomes, do we need a dedicated agent for analysis, or can the agent that extracts these details and signals also interpret them and return the analysis?\n\nWhere should analysis happen?\n\nSeparate analysis agent. A dedicated prompt and independently testable stage, with another handoff and model call.\n\nCombined email content agent. Read and interpret in the same context. Fewer handoffs, but a broader output to evaluate.\n\nFigure 16Where the analysis happens. Two architectures, weighed on usefulness, reliability and the cost of another call.\n\nThe best way to choose between them is simply to try both and check which provides better results. It is best to decrease failure points. In the event that a single agent handles both correctly, it’s preferable to pick that approach.\n\nFigure 17Working loop. Small changes, checked against evals, rerun cheaply.\n\nI kept extraction and analysis in a single agent. Feedback loops and evals gave me enough confidence to go this path\n\nEmailAttachment data\n\nEmail content agent\n\nStructured dataSignalsJob stage · sales outcome · lead source\n\nFigure 15Email content agent. Email and attachment details enter together. The output keeps recorded details apart from interpreted signals.\n\nThat leaves a harder question: how do I tell whether the records are reliable enough to use?\n\nMake it right\n\nEvals\n\nSetting up evals helps us test how reliable our agent is. Is the extracted information correct, and does it meet the client’s expectations?\n\nThere are 3 ways we can do evals:\n\nRead the options\n\nDeterministic\n\nCode checks the output against an expected value.Use when: The expected answer is known and acceptable differences can be defined in code.Judges: Quote and invoice numbers, the customer name and the service address.\n\nLLM as a judge\n\nA second model scores the output against a rubric.Use when: The answer is free text, or a judgement that can be worded many ways.Judges: The job events inferred from the conversation, compared with a separate reading of the source.\n\nHuman review\n\nA person checks the output and labels what is right.Use when: The answer depends on business knowledge, or the evidence conflicts.Judges: Business rules, uncertain customer/job matches, and the labels used by the deterministic scorers.\n\nFigure 18Types of evals. Deterministic scorers, LLM as a judge and human review, and what each is for.\n\nDeterministic scorers\n\nFields like customer name, service address, quote number, and invoice number are good candidates to score using deterministic evaluations provided with grounded data.\n\nAcquiring a grounded dataset requires the painstaking process of manually reviewing a subset of our mailbox and extracting the scored fields. I looked through 20 threads, extracted the fields, and created a dataset in Mastra so we can run multiple evaluations on these emails.\n\nEach scorer compares the agent’s output against the manually reviewed answer, scoring a match as 1 and a mismatch as 0. Flexible matching rules accommodate minor variations (like a dropped middle name or alternate address formatting) while still catching actual errors. Correctly identifying the absence of a quote or invoice on a thread also earns a full score.\n\nScorerPasses whenExampleScore\n\nCustomer nameThe customer the agent found in the thread.Passes whenThe first and last name match. Case, punctuation, accents and a dropped middle name are ignored.ExampleSam R. Taylor to Sam TaylorScore 1\n\nService addressWhere the work was done.Passes whenThe address matches after line breaks, postcode spacing, Rd and Road, and the order of the parts are evened out. A full postcode has to match exactly.Example12 Mill Road, Oxford OX1 1AA to 12 Mill Rd, Oxford OX11AAScore 1\n\nQuote numberThe ID on the job's formal quote.Passes whenOne of the quote numbers the agent returned matches. If the thread has no quote, it has to return none.Example1042 to 0000014A2BScore 0\n\nInvoice numberThe ID on the job's invoice.Passes whenOne of the invoice numbers the agent returned matches. If the thread has no invoice, it has to return none.ExampleNone to NoneScore 1\n\nTable 4Field scorers. What each scorer compares and when it passes, with one worked example.\n\nThe first headline score looked encouraging at 0.909 for the customer name. But only eleven of the twenty conversations had a score. Six had errored out and three finished without one. While the average represented ten correct names out of those eleven scored examples, it completely obscured the nine missing results.\n\nFigure 19First scored run. 0.909 on Customer name, but only 11 of 20 threads were scored. The average hid 6 errors.\n\nThe first reaction should not be to try a better prompt or a stronger model. Instead, it is to follow a framework that allows us to work in a loop, particularly the loop below.\n\nFigure 20Debugging failed evaluations. Group failures before fixing them, and prioritize the recurring, consequential ones. Some low scores are acceptable.\n\nMany failures came from the evidence gates the coding agent kept creating in code. I had required exact quotations to keep the model grounded, but the rigid checks were discarding correct answers as well.\n\nline-break mismatches: An address written on a single line failed to match the exact same address split across three lines in a PDF.\n\ncitation bottlenecks: A single failed citation would reject an entire conversation.\n\nlabel restrictions: Service addresses required an explicit installation address label that most quotes did not contain.\n\nunreadable files: A correct quote number found in a filename was rejected simply because the PDF itself could not be read.\n\nSwitching between models provided this evidence.\n\nModel testedObserved outcomeTradeoff\n\nMistralObserved outcomeThe initial model sometimes failed the required JSON schema.TradeoffAn inexpensive starting point, with extra work needed around structured output.\n\nOpen-source GPT (variant unrecorded)Observed outcomePerformed worse in my tests and also broke output-format rules.TradeoffA model change did not remove the need to inspect failures.\n\nGPT Luna (name from my notes)Observed outcomePerformed better in my tests.TradeoffA stronger candidate, considered alongside the cost of the input-heavy workload.\n\nDeepSeekObserved outcomePerformed better and became the selected extractor.TradeoffIts input cost mattered more to me than output cost because the returned records were short.\n\nTable 5Model candidates. What each model did on the same twenty conversations, and what choosing it trades off. Prompts, code and dataset labels changed during development.\n\nI chose DeepSeek for the combination of observed performance and input cost. The short records it returned made output cost a smaller part of the decision.\n\nAfter multiple iterations, the score averages fluctuated up and down. The table below shows how the results progressed from the lowest scores to the highest\n\nScorerFirst scored baseline to Improved runChange\n\nCustomer name0.75 to 0.95+0.20\n\nService address0.70 to 0.90+0.20\n\nQuote number0.65 to 0.95+0.30\n\nInvoice number0.70 to 0.95+0.25\n\nTable 6Field scores after the loops. Each field scorer on the baseline and the improved run, on the same examples.\n\nLLM as judge\n\nOur LLM-as-a-Judge evaluation is used to measure the accuracy and completeness of the email content agent. The judge, which is a different model from DeepSeek (currently GPT 5.6 Luna), checks whether the job signal events described in Table 2 (such as quote acceptance, payment, and installation booking) are supported by the original emails and PDF attachments, whether their details are correct, and whether any events were missed.\n\nOur approach follows a QAG-style pattern, described in this article, where we break the evaluation into individual judgments:\n\nevent identification: First, the judge identifies expected events from the source material without seeing the extractor's output.\n\nevidence comparison: It then compares each extracted event against that evidence, classifying it as supported, incorrect, unsupported, or not assessable.\n\nmetric calculation: Application code uses these findings to calculate precision, recall, and overall performance metrics.\n\nSupportedIncorrectMissedGPT-5.6 Luna via OpenRouter\n\nFigure 21Signal judge. The events the agent found, compared against the events in the thread.\n\nI applied the same failure-driven loop to the judge. Its first pass sometimes returned no events, making the extractor look wrong even when it had found something real. Later failures exposed disagreements in the rules.\n\nFailure categoryDiagnosisFix\n\nMissing reference eventsDiagnosisThe first pass produced an empty assessment.FixRepair the source-reading/validation path before trusting comparisons.\n\nInquiry definitionDiagnosisOne side required a customer message; the other counted the business’s opening message.FixUse the same earliest-job-contact definition on both sides.\n\nPlanned vs. confirmedDiagnosisAn inquiry that had happened was labeled planned.FixTreat an observed inquiry as confirmed; reserve planned for future actions.\n\nInstallation bookingDiagnosisThere was no event for an agreed installation date.FixAdd installation scheduled rather than force the evidence into another event.\n\nDocument referencesDiagnosisA manufacturer’s reference was confused with a quote ID.FixShare the document-number rules and accept supported filename evidence.\n\nRepeating a bad responseDiagnosisAn identical retry repeated the mistake.FixInclude the validation failure in the bounded retry.\n\nTable 7What the judge’s failures showed. Each failure, its cause, and the fix applied before the next run.\n\nInitialEvaluation score≈0.060Coverage / interpretation17 scored conversations; rounded from 0.0588. The source-reading pass could return no events.\n\nIntermediateEvaluation score0.515Coverage / interpretationAverage across 19 scored conversations.\n\nFinalEvaluation score0.750Coverage / interpretationAll 20 conversations scored. Extractor and judge had both changed.\n\nTable 8The judge’s score across the loop. Score and coverage as the judge was fixed. The final figure includes extractor changes too.\n\nHuman in the loop\n\nIncluding a human in the loop requires us to streamline the process of giving feedback between the domain expert and the person developing the system, with as little friction as possible. We focus this approach on our riskiest agent, the reconciliation agent. This is a new agent, separate from our email content agent, called at a different place, and used for a completely different problem.\n\nA customer may appear under different names or email addresses, and a job may have a quote in one conversation and a booking in another. Matching those records is a completely different decision from extraction our email content agent does and much more riskier in a sense a wrong merge can incorrectly combine two people's histories.\n\nBecause of this, the reconciliation agent lacks access to update the database directly and only suggests possible matching jobs or customers for review. We initially scaled back its role because we lacked enough reviewed examples to establish its accuracy, but the workflow below gives us a way to build that evidence while keeping people firmly responsible for those final decisions.\n\nThe AI finds potentially related records and presents them in a dashboard alongside its reasons. A domain expert then decides whether the records describe the same customer or job, should remain separate, or need further investigation.\n\n1 / 2\n\nFigure 22Reviewing possible duplicates. Names blurred.\n\nEach decision is saved with the exact records and evidence the reviewer saw. This creates a dataset for evaluating future reconciliation decisions, including cases where similar looking records should stay separate. We use data to refine the system and test it against held out cases. Once we have enough audit history and evaluations, we can use this human judgment to automatically notice and merge similar records automatically.\n\nFigure 23Human in the loop. The AI suggests, the domain expert decides, and each saved decision becomes a case to test the next version against.\n\nMake it fast\n\nFind the bottleneck\n\nWe have reached the stage where we can scale this workflow to handle all relevant email threads we can get. That means being able to handle all email extraction, attachment extractions, and analysis.\n\nTo find opportunities in scaling, we need to push our current minimal workflow and see where it breaks. In doing so, we found three key opportunities:\n\nincrease branching: Increase the number of branches that Mastra enables us to use to parallelize the process.\n\noptimize attachment processing: Minimize the time it takes to read and extract data from attachments in the sandbox.\n\nimplement caching: Investigate caching opportunities so extractions and analyses can be rerun from cached results.\n\nScaling is orchestration\n\nThis part of the work reminds me of The Goal by Eliyahu M. Goldratt and Jeff Cox, where systems thinking plays a big role in implementing these types of tasks. It teaches us how we should treat workflow optimization the same way we treat optimizing factories. The book highlights scenarios where over-optimizing a single part of a system simply creates a bottleneck for the next step. Instead, we need to view optimization as orchestrating multiple moving parts and treat the entire system as a whole.\n\nThe same reasoning applies to our Mastra workflow. Increasing the number of parallel threads allows more tasks to request sandbox execution at once, but it doesn't automatically increase how many tasks the sandboxes can actually process. When those requests outpace available capacity, work starts backing up or competing for resources, meaning we might have more tasks running concurrently even while individual jobs take longer to finish.\n\nThreads\n\nWorking on its email\n\nWaiting for a sandbox\n\nWaiting over 4 s\n\nIts PDF is being read\n\nPDFs waiting for a sandbox\n\nA PDF\n\nWaiting over 4 s\n\nSandboxes\n\nA read, filling as it goes\n\nOut of memory, at three or more reads\n\nThe bottleneck, on any stage\n\nFinished a minute\n\n1,414 threads20× scenario 1Only threads without a PDF finish. 999 PDFs a minute fail and are lost.Held back by the sandboxesThreads and sandboxes in balance\n\nTime per thread\n\n1.7 sno time waitingWorkingWaiting\n\nSandbox load\n\n79%keeping up0100%200%\n\nMemory per read\n\n384 MiBof 768Enough for a read\n\nTime per read\n\n1.5×two reads share the vCPUA sandbox reads 1.3× as many PDFs\n\nFigure 24Scaling funnel. Raise threads alone and the line grows while each thread takes longer. Add sandboxes and more work gets through. A simulation, not a benchmark.\n\nBecause of this, we need to tune our thread concurrency to match the actual limits of our sandbox pool. This balance depends on the total number of sandboxes, the resources allocated to each one, and the heavy demands of each individual task. That doesn't strictly mean a one-to-one ratio of threads to sandboxes. Instead, it means opening up just enough parallel work to fully utilize our available capacity, while holding any excess work in a queue until resources free up. At the end of the day, our true measure of success is how much total work the entire workflow can push through reliably over time.\n\nReduce work per PDF\n\nOnce we dial in the knobs for concurrency and the number of sandboxes to an optimal level, we see another opportunity to cut down the time it takes to read attachments from the sandboxes, based on what we discovered in the traces\n\nChangeObserved problemRationale / effect\n\nTrim layout paddingObserved problemSome extracted pages were mostly spaces; a thread exceeded the input-size limit.Rationale / effectKeep the words and useful line breaks without padding to the visual page width.\n\nReduce AI round tripsObserved problemThe model requested a guide the application already had and selected methods page by page.Rationale / effectCapture page text in code and give it to the address-reading agent directly.\n\nRemove the PDF skill from this pathObserved problemFetching skill files and using a general reading procedure added overhead for routine digital PDFs.Rationale / effectUse the project’s extraction path for these files; leave interpretation to the model.\n\nMake OCR conditionalObserved problemA logo triggered OCR even when the page had readable text.Rationale / effectExtract native text first, check whether it is usable, and use OCR only for pages that need it.\n\nTable 9Work removed from each PDF read. What the traces showed each read doing, and the change that removed it.\n\nBefore fixes25 min 41 s\n\nAfter fixes11 min 54 s\n\nAbout 54% less time\n\nFigure 25Development run time. Before and after the PDF changes. Several changes landed together, so this isolates none of them.\n\nCaching\n\nCaching pays off especially on repeated runs as we tweak the prompts. I might change the email prompt several times without changing the attachments, and re-reading every unchanged PDF makes that feedback loop unnecessarily slow. It also pays off by lowering compute costs for the sandboxes. Ultimately, there are various caching opportunities we can leverage throughout the entire process of extracting content from attachments\n\nPipeline stageWhat is reusedWhen it helps\n\nExtraction runWhat is reusedA saved result for the same thread and extractor version.When it helpsReplay an unchanged run; a new extractor version must run again.\n\nUploadWhat is reusedAn already uploaded file within a run; skip upload when a complete cached reading is available.When it helpsAvoid sending the same attachment repeatedly.\n\nPDF captureWhat is reusedPage text keyed by file hash and extraction version.When it helpsReuse the read while testing a different email prompt or model.\n\nService-address readingWhat is reusedAn address result keyed by file and reader configuration.When it helpsReuse only while its model, prompt and relevant reader code still match.\n\nSandbox setupWhat is reusedA snapshot with the PDF tools installed.When it helpsStart workers from a prepared environment; this is environment reuse rather than a cached answer.\n\nTable 10What each cache reuses. What each stage keeps, reused only while the inputs it depends on are unchanged.\n\nBaseline25 min 41 s\n\nOptimized, warm cache2 min 59 s\n\nAbout 88% less time\n\nFigure 26Combined development result. The same 20 pilot conversations, at the baseline and optimized with a warm cache.\n\nThe complete workflow\n\nIf I had to map out this whole system in one go, it might look something like this, at least, as best as I can picture it.\n\nYou’ve got Jev feeding attachments into a cluster of Daytona sandboxes, and DeepSeek turning that raw evidence into clean records. Throw in our scorers, the signal judge, and human review, and you've got quality control keeping everything grounded. Thanks to the anatomy skill we can generate this factory floor view.\n\nFig 27The whole workflowHover or tap an icon30 branches · 0 conversations processed · simulation", "url": "https://wpnews.pro/news/when-ai-writes-the-code-show-your-judgment", "canonical_source": "https://www.eyosias.dev/case-studies/ai-mailbox-analysis", "published_at": "2026-11-01 11:44:25+00:00", "updated_at": "2026-11-01 11:49:41.879323+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-tools"], "entities": ["HVAC company", "United Kingdom"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/when-ai-writes-the-code-show-your-judgment", "markdown": "https://wpnews.pro/news/when-ai-writes-the-code-show-your-judgment.md", "text": "https://wpnews.pro/news/when-ai-writes-the-code-show-your-judgment.txt", "jsonld": "https://wpnews.pro/news/when-ai-writes-the-code-show-your-judgment.jsonld"}}