# What Building Anamnesis Taught Me About Giving AI a Memory > Source: > Published: 2026-07-04 10:53:29+00:00 *How Cognee turned a pile of medical PDFs into a living clinical memory graph, and why "storage" and "memory" turned out to be completely different problems.* A patient walks into a clinic. Behind them: years of blood reports, prescriptions, discharge summaries, imaging scans. In front of them: a doctor who has maybe four minutes to reconstruct that entire history before making a decision that affects the next one. That's not a data problem. Every one of those documents is sitting in some folder, some PDF, some scanned image. The information *exists*. What's missing is the connection between it: the fact that the kidney function flagged in March relates to the medication started in January, which was prescribed because of a diagnosis made the previous year. A filing cabinet doesn't know that. Neither does a database that just stores rows. That distinction, between storing information and *remembering* it, is the whole thesis behind Anamnesis, the project I built for wemakedevs.org's Cognee hackathon ("The Hangover Part AI: Where's My Context?"). And it's the reason I built it on [Cognee](https://github.com/topoteretes/cognee) instead of a conventional RAG pipeline. Most "AI over your documents" tools follow the same recipe: chunk the text, embed it, stuff the closest chunks into a prompt when someone asks a question. That works fine for FAQ bots. It falls apart for clinical history, because clinical facts aren't independent, they're relational. A lab value only means something in the context of the medication that might be causing it and the diagnosis that explains why the medication was prescribed in the first place. Flat vector similarity search doesn't reason about relationships; it just finds text that *sounds* related. What Anamnesis actually needed was a *memory*: something that ingests new facts, connects them to what's already known, revises itself when new information contradicts old information, and lets old information decay or retire without being deleted outright. That's a different shape of problem than search, and it turns out Cognee had already built the primitives for exactly that shape. Cognee organizes memory around four operations, and mapping Anamnesis onto them was the core design exercise of the whole build: ** remember()**: A clinician uploads a blood report, a prescription, a discharge summary. Anamnesis runs it through Gemini's vision-based OCR to pull out structured entities (diagnoses, medications, precise lab values with units), then commits those entities to the patient's Cognee graph. This isn't "save the file," it's "extract the facts and let them become nodes and edges that connect to everything already known about this patient." ** recall()**: When a doctor asks "why is kidney function declining?", Anamnesis doesn't search documents, it traverses the graph. The answer comes back with a traceable evidence chain: which document, which date, which exact snippet supports each claim. This traceability mattered more than almost anything else in the build. A clinician will never trust an answer they can't audit down to the source. ** improve()** (Cognee's `memify()` ): Medicine is not append-only. When a clinician marks a condition as "ruled out" or a medication as "discontinued," that correction has to propagate through the graph, not just get logged as a new fact sitting next to the old, now-wrong one. `improve()` restructures the relationships so the graph reflects the corrected understanding. This is the operation that makes the memory feel less like a log and more like an evolving belief system.** forget()**: When the same blood report type gets uploaded twice for the same date, Anamnesis doesn't want two contradictory copies of the same fact living in the graph forever. `forget()` intelligently retires the duplicate or superseded information while preserving the historical record: memory that can prune itself without becoming amnesia.Building around these four verbs, instead of hand-rolling my own retrieval and update logic, is what let a solo hackathon build produce something that behaves less like a chatbot wrapper and more like an actual second brain for a patient's chart. Patient data is PHI. Anamnesis runs entirely on **self-hosted, open-source Cognee**: a GCP VM running the open-source stack behind Caddy/TLS, backed by Postgres+pgvector for vectors and an embedded Kuzu graph store, with every API call authenticated and every byte of patient data staying inside infrastructure I control. Nothing patient-related ever touches a third-party managed data store. That constraint is also why the hackathon track fit: "Best Use of Open Source" isn't a checkbox here, it's the actual architecture. (Yes, all data used anywhere in the build is synthetic. Real PHI doesn't touch this system until compliance work, signed BAAs, a security review, a real retention/deletion policy, is actually done. That's a gate, not a formality, and it's true in the code, not just in this post.) The unlock, in practice, showed up in small moments during testing. Ask "what changed since the last consultation?" and the answer isn't a document dump, it's a diff over the graph, phrased the way a colleague would phrase it. Correct a diagnosis and the *next* query about that diagnosis already reflects the correction, without re-ingesting anything. Upload a duplicate report and the system quietly resolves the conflict instead of presenting two contradictory truths side by side. None of that requires cleverness on my part. It requires trusting the memory primitives to do their job, and designing the product to expose them rather than hide them. Anamnesis has a live operations panel showing every `remember` /`recall` /`improve` /`forget` call as it happens, because the whole point was to prove the memory is real, not a black box. The hackathon build (Phases 0 to 4: infra, Remember, Recall, Improve, Forget against a single demo patient) is done and live. It's now becoming a real product: multi-tenant clinics, real clinician rosters, hardened security, because the underlying bet held up under a week of building. Clinical software should behave like a memory, not a filing cabinet. Every upload should make the system *more* useful, not just *bigger*. That's the difference between storing a patient's story and remembering it. Cognee is what made the second one possible. **Live app:** [anamnesisai.vercel.app](https://anamnesisai.vercel.app) **Demo video:** [youtu.be/IH7V3kgMxo8](https://youtu.be/IH7V3kgMxo8) **Code:** [github.com/Naazimsnh02/Anamnesis](https://github.com/Naazimsnh02/Anamnesis) **Built with:** [Cognee](https://github.com/topoteretes/cognee) (self-hosted), Next.js, Clerk, Gemini, Postgres/Kuzu