{"slug": "from-email-export-to-agent-memory-the-pipeline-i-actually-wanted", "title": "From Email Export to Agent Memory: The Pipeline I Actually Wanted", "summary": "A developer built Waada, a deal-continuity system that turns exported sales history from email, Slack, transcripts, audio and CRM JSON into durable, queryable agent memory. The pipeline normalizes all sources into a Zod-backed Interaction model, makes ingestion idempotent via a manifest of retained sourceIds, and retains account-scoped documents in the Hindsight memory layer so the LLM generates only from bounded retrieved evidence.", "body_md": "From Email Export to Agent Memory: The Pipeline I Actually Wanted\n\nThe interesting part of an AI application is often not the model call. It is the pipeline that decides what the model gets to remember.\n\nI built Waada as a deal-continuity system, and the engineering problem that kept surfacing was straightforward: how do I turn messy account history into durable, queryable memory without making every downstream feature understand every source format?\n\nThe answer was a strict pipeline: normalize first, retain second, recall by purpose, and generate only from bounded evidence.\n\nThe architecture starts before the LLM\n\nWaada accepts exported sales history from several sources:\n\n.eml email\n\nSlack day JSON\n\ntext, Markdown, and VTT transcripts\n\naudio\n\nCRM JSON\n\nThe application is a TypeScript pnpm workspace. packages/core owns the data model, parsers, ingestion, memory adapter, LLM layer, and agent functions. apps/web is a TanStack Start application exposing those capabilities through server functions.\n\nThe architecture is:\n\nUploaded files\n\n      |\n\n      v\n\n parseFiles()\n\n      |\n\n      v\n\n Interaction[]\n\n      |\n\n      v\n\n   ingest()\n\n      |\n\n      +-------------------+\n\n      |                   |\n\n      v                   v\n\nlocal normalized data   Hindsight\n\n                            |\n\n                            v\n\n                       Memory.search()\n\n                            |\n\n                            v\n\n                      bounded evidence\n\n                            |\n\n                            v\n\n                           LLM\n\nI like this shape because it makes the LLM the last stage rather than the center of the application.\n\nStep one: normalize everything\n\nThe core model is a Zod-backed Interaction:\n\nInteraction = {\n\n  account: string;\n\n  sourceId: string;\n\n  type: \"call\" | \"email\" | \"slack\" | \"meeting\" | \"note\";\n\n  date: string;\n\n  title: string;\n\n  participants: string[];\n\n  content: string;\n\n  source: \"eml\" | \"slack_export\" | \"transcript\" | \"audio\" |\n\n          \"gmail\" | \"slack_api\" | \"hubspot\" | \"meet\";\n\n}\n\nOnce a file becomes an Interaction, downstream code doesn't care about its original syntax.\n\nThat matters because source formats are full of special cases.\n\nEmail parsing has headers, quoted replies, HTML fallbacks, and message IDs.\n\nSlack exports have channel/day structure and optional user metadata.\n\nTranscripts may have front matter or require metadata extraction.\n\nAudio needs transcription before it can enter the same pipeline.\n\nI wanted all of those problems to end at the parser boundary.\n\nStep two: make ingestion idempotent\n\ningest() validates and sorts interactions by date, checks .waada/manifest.json, ensures the account bank, retains new material, and persists normalized interactions for the comparison baseline.\n\nThe manifest tracks retained sourceId values.\n\nThat gives me:\n\nsame import twice\n\n       |\n\n       v\n\nsame sourceId\n\n       |\n\n       v\n\nmanifest match\n\n       |\n\n       v\n\nskip duplicate retain\n\nThe Hindsight document ID provides another layer of stable identity.\n\nThis matters more than it initially looks.\n\nSales exports are likely to be re-exported. Users may upload the same data again. If ingestion is not idempotent, retrieval quality can degrade through duplicated evidence.\n\nThe system therefore keeps the import path deterministic wherever possible.\n\nStep three: retain into account-scoped memory\n\nI use Hindsight as the durable memory layer.\n\nThe Hindsight adapter is isolated in packages/core/src/memory/, and every account maps to a bank derived from its slug:\n\nconst bankId = bankIdFor(account);\n\nFor each interaction, the adapter retains:\n\ncontent\n\ninteraction date\n\ncontext\n\ndocument ID\n\nstring metadata\n\nThe Hindsight GitHub repository is the implementation reference, and the Hindsight documentation covers the memory API.\n\nFor Waada, Hindsight's job is not to decide what a commitment means. It provides the durable account history from which the agent can retrieve evidence.\n\nThat separation is important.\n\nMemory storage and reasoning are different responsibilities.\n\nStep four: recall according to the operation\n\nOnce the data is retained, I don't have one generic getAccountHistory() function.\n\nThe agent has different retrieval jobs.\n\nThe commitment ledger searches for promise-related evidence.\n\nThe landmine extractor searches for objections and resolved agreements.\n\nThe brief recalls stakeholder and recent-change evidence.\n\nThe Ask function uses the user's exact question.\n\nThis is the pattern:\n\nagent operation\n\n      |\n\n      v\n\npurpose-specific query\n\n      |\n\n      v\n\nHindsight recall\n\n      |\n\n      v\n\nMemoryHit[]\n\n      |\n\n      v\n\nevidence selection\n\nThis is where persistent memory differs from a static summary.\n\nA summary has already decided what deserves attention.\n\nA memory system postpones that decision until the question is known.\n\nThe commitment ledger is a retrieval pipeline of its own\n\ncommitmentLedger() performs multiple promise-oriented queries, deduplicates evidence, chunks it, runs structured extraction, merges duplicates, and sorts the result.\n\nThe resulting Commitment contains the deliverable, parties, date, optional due date, status, evidence, and source.\n\nThe sorting policy is straightforward:\n\nopen before unclear\n\nunclear before delivered\n\noverdue open items first\n\nthen upcoming deadlines\n\nthen newer/undated items\n\nThe important part is that the status is derived.\n\nI don't want a permanently stored open value to become stale when a later interaction says the work was delivered.\n\nThe source history remains the evidence.\n\nThe landmine pipeline solves the opposite problem\n\nA landmine is a resolved objection that should not be casually reopened.\n\nThe landmine function recalls objections, sensitive topics, and accepted agreements. It sends bounded evidence into structured extraction and returns an object containing the topic, what happened, resolution, date, guidance, and source.\n\nThat means the agent has two different operational memories:\n\ncommitment -> unfinished work\n\nlandmine   -> settled history\n\nThis is a useful distinction for any system where continuity matters.\n\nA handoff isn't only about preserving tasks.\n\nIt is also about preserving decisions.\n\nStep five: budget before generation\n\nRecall can return more information than an LLM should see.\n\nWaada therefore uses evidence caps and a shared prompt budget. The configured maximum is 5,000 input tokens, approximated conservatively through character counts.\n\nI intentionally don't call this an exact tokenizer budget.\n\nThe implementation uses it as a safety boundary.\n\nThe resulting pipeline is:\n\nHindsight recall\n\n      |\n\n      v\n\ndeduplicate\n\n      |\n\n      v\n\nselect relevant hits\n\n      |\n\n      v\n\nchunk\n\n      |\n\n      v\n\nprompt budget\n\n      |\n\n      v\n\nLLM\n\nThis was one of the most useful design constraints in the project.\n\nWithout a budget, every retrieval function tends toward “send more context.”\n\nWith a budget, every query has to earn its place.\n\nWhy the brief is sequential\n\nbrief() runs the ledger and landmine stages sequentially. It separately recalls stakeholder and recent-change evidence before generating the final brief.\n\nThe reason is not conceptual purity.\n\nIt is provider capacity.\n\nThe live implementation uses a constrained provider environment, and parallel model calls can collide with a shared rate window. Sequential work makes the behavior more predictable.\n\nIt is a good example of a practical engineering trade-off: the theoretically faster architecture is not always the architecture that survives its provider limits.\n\nAsk is the simplest complete path\n\nThe Ask operation makes the whole architecture easy to explain.\n\nUser:\n\nWhat changed since July?\n\nSystem:\n\nquestion\n\n   -> Hindsight recall\n\n   -> bounded evidence\n\n   -> LLM\n\n   -> answer + citations\n\nThe implementation returns the recalled contexts and document IDs with the answer.\n\nA successful live evaluation recorded the Q3-to-Q4 go-live change for the Acme scenario.\n\nThat is the kind of evidence I want the system to expose: not only a sentence, but the historical contexts that informed it.\n\nThe LLM layer has its own boundary\n\nThe LLM layer supports multiple providers through the AI SDK and has separate operations for chat, structured extraction, and transcription.\n\nStructured extraction is treated as untrusted.\n\nconst parsed = schema.safeParse(modelOutput);\n\nif (parsed.success) {\n\n  return parsed.data;\n\n}\n\nconst repaired = await repair(modelOutput);\n\nreturn schema.safeParse(repaired).success\n\n  ? schema.parse(repaired)\n\n  : null;\n\nThe model can produce malformed output.\n\nThe application should survive that.\n\nThat sounds obvious, but it becomes important once generated data starts controlling downstream behavior.\n\nThe comparison path helped expose the real value\n\nWaada compares three context sources:\n\nCRM-only\n\nraw chronological summary\n\nmemory-aware Waada\n\nThe CRM baseline reads only imported CRM data.\n\nThe summary baseline reads chronological normalized interactions without memory.\n\nThe Waada version can use Hindsight recall.\n\nThis isn't a benchmark against CRM vendors. It is a controlled comparison of context availability.\n\nThe live evaluation record is deliberately mixed. It contains successful timeline and landmine behaviors as well as structured-output variability, rate-limit pressure, intermittent memory failures, and a run where summary-only scored higher on its checks.\n\nThat distinction is important.\n\nThe architecture establishes a different context path.\n\nIt does not establish universal accuracy.\n\nWhat I would reuse from this pipeline\n\nEnd the messy-source problem at the parser boundary\n\nDownstream code should consume one schema.\n\nMake source identity explicit\n\nStable IDs and manifests are part of retrieval quality, not just ingestion bookkeeping.\n\nKeep memory account-scoped\n\nThe account is the natural storage and retrieval boundary.\n\nPreserve timestamps\n\nTemporal questions are common in sales.\n\nSeparate retrieval from generation\n\nHindsight supplies evidence. The LLM interprets bounded evidence.\n\nValidate generated structures\n\nModel output is an external dependency.\n\nTreat budgets as architecture\n\nPrompt limits affect query design, chunking, sequencing, and error handling.\n\nThe pipeline is the product\n\nWhen I look at Waada now, the most interesting component isn't the final brief.\n\nIt is the path that creates the brief:\n\nmessy exports\n\n    -> canonical interactions\n\n    -> idempotent ingest\n\n    -> account memory\n\n    -> purpose-specific recall\n\n    -> bounded evidence\n\n    -> validated generation\n\nThe model is only useful because the pipeline gives it the right history.\n\nThat is the engineering lesson I would carry into another memory-heavy application: don't start by asking which model should summarize the data.\n\nStart by deciding what information must survive, how it will be identified, how it will be retrieved, and how much of it the model is allowed to see.\n\nThen call the model.", "url": "https://wpnews.pro/news/from-email-export-to-agent-memory-the-pipeline-i-actually-wanted", "canonical_source": "https://dev.to/nani0000/from-email-export-to-agent-memory-the-pipeline-i-actually-wanted-5c9o", "published_at": "2026-09-29 11:05:14+00:00", "updated_at": "2026-09-29 11:16:44.895895+00:00", "lang": "en", "topics": ["ai-agents", "large-language-models", "ai-tools", "developer-tools"], "entities": ["Waada", "Hindsight", "TanStack Start", "Zod", "pnpm", "Slack", "HubSpot"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/from-email-export-to-agent-memory-the-pipeline-i-actually-wanted", "markdown": "https://wpnews.pro/news/from-email-export-to-agent-memory-the-pipeline-i-actually-wanted.md", "text": "https://wpnews.pro/news/from-email-export-to-agent-memory-the-pipeline-i-actually-wanted.txt", "jsonld": "https://wpnews.pro/news/from-email-export-to-agent-memory-the-pipeline-i-actually-wanted.jsonld"}}