"Last time" meant last upload, until I fixed Hindsight timestamps

A developer built Tareekh, a memory layer for litigators that ingests photographed diary pages, order sheets and deeds via Gemini on Vertex AI and stores one memory per hearing in Hindsight, an open-source agent memory engine. The key fix was stamping each memory with the hearing date rather than the upload time, since parallel backlog uploads otherwise collapsed an eighteen-month case timeline into a single evening and made "last time" mean the most recently uploaded file. The system is tested against a practice with five cases and 79 uploads spanning April 2025 to September 2026.

The most common question a litigator asks before walking into court is five words long: "What happened last time?" It sounds like the easiest question an agent with memory could get. It turned out to be the one that forced me to think hardest about time. When I first wired up the memory layer, "last time" quietly meant "the file you uploaded most recently." Those are different things, and in a law practice the gap between them can be eighteen months. The setup I built Tareekh, a memory for a litigator's practice. The lawyer photographs pocket-diary pages, uploads certified order sheets, drops in deeds and notices, or types a quick note. Tareekh reads each upload with Gemini on Vertex AI, splits it into one entry per hearing, files each entry under the right case, and retains it in Hindsight, an open-source memory engine for agents. An agent answers questions from that memory and cites its sources. Figure: Tareekh's architecture. Each hearing becomes one Hindsight memory in a single bank per lawyer. The practice I test with belongs to Adv. Aditya Varma and his junior Divya. There are five cases and 79 uploads covering April 2025 to September 2026. How lawyers actually enter data Nobody uploads a hearing on the day it happens. The real pattern is a backlog. A lawyer who starts using the tool has a year of diary pages in a drawer, so they photograph forty of them one evening. After that, uploads come in bursts: a week of notes on a Sunday, an order sheet when the certified copy finally arrives three weeks after the hearing, a scanned deed from 2011 when a dispute about it comes up. My backlog loader makes this worse, and makes the problem impossible to miss. It sends the uploads oldest first, but it runs five workers in parallel because each upload spends most of its time waiting on OCR: with ThreadPoolExecutor a.workers as pool: for i, up, m, r in enumerate pool.map one, manifest , 1 : ... So the order in which entries reach Hindsight is roughly chronological, with the details decided by whichever OCR call finishes first. If a memory's time is "when it was stored," the timeline is noise. Figure: Stamped at upload time, 79 uploads land on one evening and "last time" means whichever file finished last. Stamped with the hearing date, they spread across eighteen months and "last time" means the actual last hearing. What Hindsight does with time Hindsight treats time as a first-class part of a memory. Every item you retain can carry a timestamp, and recall results come back with occurred start, the time the fact is about. Recall runs a temporal path alongside semantic and keyword search, so a query that implies a time "last hearing", "in June" can lean on it. That's exactly what you want, as long as the timestamp means what you think it means. If you don't pass one, the reasonable default is "now." For a chat assistant that's correct. For a legal record it's wrong in a subtle way: nothing errors, the answers look plausible, and they're about the wrong hearing. The fix: the hearing date is the timestamp Every Tareekh memory is stamped with the date of the hearing it describes. The rule is short enough to live in a comment: cid, date = entry "case id" , entry "hearing date" return { "content": header + "\n" + entry "text" , "timestamp": f"{date}T10:30:00+05:30", the HEARING date, never the upload time "metadata": {"source file": source file, "upload id": upload id, "case id": cid, "hearing date": date, "author": author}, "document id": f"{upload id}:{cid}:{date}", } 10:30 IST is roughly when district courts start sitting, so every hearing gets the same plausible time of day. The date also goes into metadata, because metadata comes back with every recalled fact and that's what the UI shows on a citation chip. And it's part of document id, so uploading the same diary photo twice replaces the memory rather than duplicating it. The hard part is getting the date right, not storing it. Where the hearing date comes from A diary page prints the date at the top. An order sheet has rows like 22.06.2026: Counsel for defendant again sought time.... A typed note might say "12/9 Gorle partition" and nothing else. And almost every note contains a second date that is not the hearing date: the next date. The segmentation prompt is explicit about that trap: Figure: A diary page from the test practice. The date is printed at the top, and the file is named IMG 20250618 114747.jpg, which agrees with it. One exception: things said in chat Tareekh also remembers durable things the lawyer says in chat: "we won't settle for Srinivas's half only," "insist on 18% interest from June 2023." Those are stamped with the moment they were said, because that's when the decision was made. They're labelled CHAT MEMORY ... not a court record in their content and tagged type:chat memory, and the agent ranks them below records and notes. Two kinds of time, stored deliberately, both in the same bank. Before and after Take the Seabreeze specific-performance suit. Its most recent hearing is 24 September 2026, recorded in Divya's typed note "objections to IA 1187/2026 amendment : draft 60% done" . But the most recently uploaded file touching that case could easily be an older order sheet whose certified copy took weeks to arrive, or the 2024 agreement of sale that someone scanned last. With upload-time stamps, "what happened last time in Seabreeze SP suit?" leans toward whatever file finished processing last. The answer is fluent, it's cited, and it describes the wrong day. That combination is the worst possible failure for a tool a lawyer reads while standing in court. With hearing-date stamps, the Today page and the agent agree. The cause-list card shows "Previous hearing, 24 Sept · Divya's note," and the answer cites that note. The case timeline tool sorts by the same hearing date in metadata, so asking for the history of a case gives you the hearings in the order they happened, whatever order you uploaded them in. Figure: The Today page. Each listed matter shows its previous hearing by hearing date, not by upload date. What I'd pass on Decide what time a memory is about before you store the first one. "When it happened" and "when I learned it" are both valid, but you have to pick on purpose. Hindsight gives you the field; it can't guess which one you meant. Put the date in three places. In the timestamp so retrieval understands time. In metadata so every citation can show it. In the document id so re-uploads are idempotent per hearing. Batches get re-run, and without that you pile up duplicates. Teach the extractor which dates to ignore. Next dates, filing dates, dates in the body of a deed: a legal note is full of dates, and exactly one of them is the hearing. A single sentence in the prompt did more than any post-processing. Cap confidence on guessed dates. A fallback is fine. A fallback that skips human review is how a wrong date becomes a confidently wrong memory. It's not perfect. A photo taken the morning after a hearing gets the next day's date from its filename, and a lawyer who photographs a week of pages on Sunday gets seven entries dated Sunday unless the page header is readable. The review screen catches most of this. It won't catch all of it. If you're adding memory to an agent that works over records with their own dates tickets, medical notes, incidents, transactions , this is probably the first design decision to get right. The Hindsight docs cover retain timestamps and how recall uses them, and Vectorize's overview of agent memory is a good read on why "when" is part of what a memory is, not an afterthought.