{"slug": "maintaining-an-organizational-knowledge-graph-with-an-llm-and-event-sourcing", "title": "Maintaining an organizational knowledge graph with an LLM and event sourcing", "summary": "Arkency has built Planet Arkency, a multi-tenant knowledge graph with a closed ontology, using Rails Event Store and an LLM to maintain organizational knowledge from unstructured sources like meeting transcripts and Slack threads. The system, inspired by Obie Fernandez's NEXUS and Andrej Karpathy's LLM Wiki, ingests content through a single endpoint and uses LLM extraction to identify entities and relations, storing them in PostgreSQL tables.", "body_md": "# Maintaining an organizational knowledge graph with an LLM and event sourcing\n\n… and check why [5600+ Rails engineers read also this](#cbcac110c1)\n\n# Maintaining an organizational knowledge graph with an LLM and event sourcing\n\nOrganizations are surprisingly good at forgetting.\n\nDecisions are made on calls, insights get buried in Slack threads, and a month later no one remembers why things are the way they are.\n\nAt Arkency, I had a feeling that some things slip away from us too from time to time.\n\nWeekly calls, ad-hoc meetings, our book clubs, Slack discussions, GitHub mentions, e-mail inbox - we could use some support in organizing all those signals.\n\nThen [Ruby Community Conference 2026](https://rubycommunityconference.com) happened in March.\n\nIn Kraków, **Obie Fernandez** showed some parts of his NEXUS system.\n\nHe had already described it [on his blog](https://obie.medium.com/what-used-to-take-months-now-takes-days-cc8883cc21e9) back in January, but the conference was where I first came across it.\n\nThat was the push I needed to start building our own software.\n\nWhen it was already taking shape, Andrej Karpathy published his [LLM Wiki](https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f) note.\n\nInstead of a RAG system rediscovering your documents on every query, an LLM incrementally maintains a persistent wiki: interlinked markdown pages, immutable sources underneath, and a human curating the loop.\n\nIt was quite exciting to realize I was working on something that had just become one of the hottest topics in the industry.\n\nWe ended up with *Planet Arkency* - a multi-tenant **knowledge graph with a closed ontology**, built on Rails Event Store.\n\nIn this post, I want to walk you through the design decisions I made.\n\n## Unstructured input is where LLMs actually shine\n\nFor structured data, you could have built such a system like twenty years ago.\n\nWebhooks, forms, integrations - parsing structured input into a graph is a solved problem.\n\nBut the most interesting knowledge lives in the input no parser could ever handle: meeting transcripts, Slack discussions, emails, or anything coming from an integration nobody has built yet.\n\nThis is where LLMs changed the game for us.\n\nEverything flows into the system through a single ingestion endpoint.\n\nTranscripts, Slack threads someone flagged with a dedicated emoji reaction, emails arriving at a bridge inbox, RSS feeds, calendar invites, personal notes.\n\n**We don’t even write code for the integration points.**\n\nTools like Zapier or n8n watch the sources and push the content to that single endpoint.\n\nEvery ingested piece of content then goes through an **extraction** - the heart of the system.\n\nAn LLM reads the content and works out what it means for our knowledge: which entities appear in it, what we learned about them, and how they relate to each other.\n\nMost of this post is about what happens around that single step.\n\n## Why a graph?\n\nThe same names keep coming back in our conversations: people, projects, clients, tools, decisions.\n\nWhat changes from week to week is what we know about them and how they relate to each other.\n\nThat maps naturally to a graph: entities with attributes, connected by typed relations.\n\nWho works on what.\n\nWho made which decision, and when.\n\nWhich project depends on which tool.\n\nThis is where we differ most from the LLM Wiki approach.\n\nIn a wiki, the fact that someone works on some project is written down in a sentence on a page, at best with a link between the two pages.\n\nThe knowledge is there, but only a reader can make use of it.\n\nIn a typed graph, `person --works_on--> project`\n\nis a piece of data: you can query it, traverse it, count it.\n\nThe graph itself sits on PostgreSQL: a `nodes`\n\ntable, an `edges`\n\ntable with a unique `(source, target, relation)`\n\ntriple, `jsonb`\n\nattributes on both.\n\nNo rocket science here.\n\nDedicated graph databases (Neo4j, triple stores like the one NEXUS uses) could be a better fit for some specific workloads, like deep multi-hop traversal.\n\nBut that is a storage detail - the kind you could change later by writing another adapter for the data layer.\n\n### The ontology\n\nWhich kinds of nodes and relations may exist is defined in an **ontology**, stored in a plain YAML file:\n\n```\n# from config/ontology.yml\nnode_kinds:\n  - kind: person\n    description: \"team member, candidate, client contact, external person\"\n  - kind: decision\n    description: \"formal decision requiring group verdict — for casual suggestions use idea\"\nedge_relations:\n  - relation: works_on\n    signature: \"person --works_on--> project\"\n```\n\nThe ontology is closed - if a kind or relation is not on the list, the model cannot use it.\n\nInitially I was thinking about an open ontology, where the LLM could introduce its own types.\n\nIt brought complete chaos into the graph surprisingly fast.\n\nIn my opinion, it is better to tell the model upfront what to look for.\n\n### Not one graph, but many\n\n*“The organizational knowledge graph”* suggests one universal graph for all different purposes.\n\nWe don’t believe in that, and DDD practitioners will recognize why.\n\nWe use multi-tenant architecture to maintain separate graphs with their own ontologies, which really means their own ubiquitous languages.\n\nOur internal Arkency graph speaks in people, projects and decisions - a domain quite close to a CRM.\n\nThe graph we run as Rails Event Store maintainers speaks in releases, known problems and community content:\n\nDifferent domains, different vocabularies, the same machinery underneath.\n\nThe boundaries of a bounded context tell you where one graph ends and another begins.\n\n## What comes out of an extraction\n\nThe ontology is rendered into the extraction prompt as markdown tables and into the schema as enums.\n\n```\n(from app/lib/prompts/extraction.md.erb)\n\nYou are an organizational knowledge analyst for <%= Tenancy.current_tenant.name %>. We are building an internal knowledge graph.\n\nExtract a knowledge graph from the provided content: nodes and edges. The graph should allow full reconstruction of the provided content.\n\n## Nodes\n\nEach node has: name, kind, short_description, description, attrs (optional key-value pairs).\n\nAllowed kinds:\n\n| kind | what it represents | typical attrs |\n|---|---|---|\n<% ontology.node_kinds.each do |k| -%>\n| <%= k.fetch(\"kind\") %> | <%= k.fetch(\"description\") %> | <%= k.fetch(\"attrs\", []).then { |attrs| attrs.empty? ? \"—\" : attrs.map { |a| a.is_a?(Hash) ? (a[\"values\"] ? \"#{a[\"name\"]} (#{a[\"values\"].join(\", \")})\" : a[\"name\"]) : a }.join(\", \") } %> |\n<% end -%>\n\n## Edges\n\nEach edge has: source, target, relation, context, attrs (optional key-value pairs).\n\nAllowed relations:\n\n| relation | source kind | target kind | hint | attrs |\n|---|---|---|---|---|\n<% ontology.edge_relations.each do |r| -%>\n<%\n  sig = Ontology.parse_signature(r.fetch(\"signature\"))\n  source_kind = sig[:source].join(\" / \")\n  target_kind = sig[:target].join(\" / \")\n  hint = r[\"hint\"] || \"—\"\n  attrs = r.fetch(\"attrs\", []).then { |a| a.empty? ? \"—\" : a.map { |at| at.is_a?(Hash) ? (at[\"values\"] ? \"#{at[\"name\"]} (#{at[\"values\"].join(\", \")})\" : at[\"name\"]) : at }.join(\", \") }\n-%>\n| <%= r.fetch(\"relation\") %> | <%= source_kind %> | <%= target_kind %> | <%= hint %> | <%= attrs %> |\n<% end -%>\n\n...\n```\n\nEach extraction ends with the model returning **one structured result**: the entities it found in the content, the relations between them, and how the existing graph should change to reflect them.\n\nWe use [RubyLLM’s schema support](https://rubyllm.com/chat/#using-rubyllmschema-recommended) for that.\n\n```\n# from app/lib/extraction_result_schema.rb\narray :nodes, description: \"Entities to create or update. Each name must be unique — no duplicate nodes.\" do\n  object do\n    string :status, enum: [\"new\", \"existing\"], description: \"'existing' iff the node was returned by search_nodes/list_nodes_by_kind/get_node_edges and you are reusing it. 'new' if you are introducing it. The system verifies the canonical name and aborts on mismatch.\"\n    string :name, description: \"Entity name. For 'existing' nodes use the EXACT canonical name from the tool call result. For 'new' nodes the canonical name you are introducing.\"\n    string :new_name, required: false, description: \"Optional. Set ONLY for 'existing' nodes when the content reveals a more explicit canonical form (e.g. acronym → full term, diminutive → full name). The node is looked up by `name` and renamed to `new_name`.\"\n    string :kind, description: \"Must be one of: #{kind_names}\"\n    string :short_description, description: \"Stable synthesis of what this entity is (for search). General and identity-focused, not episode-specific. Max 15 words.\"\n    string :description, description: \"For new nodes: brief description based on the content. For existing nodes: synthesize prior description with new information. Rewriting for clarity is fine, but preserve prior facts.\"\n    array :attrs, description: \"Key-value attributes. Only include what is known from the content.\" do ... end\n    array :aliases, required: false, description: \"Optional. Alternative surface forms (diminutives, acronyms, full vs short forms) under which this entity was referred to in the content, or — when renaming via `new_name` — the old canonical if it remains a valid surface form. Only include NEW aliases not already present on the existing node. An alias is the SAME entity under another name — never a separate entity.\" do ... end\n  end\nend\n\narray :edges, description: \"ALL relationships. Be thorough and precise.\" do\n  object do\n    string :source, description: \"Source node name (exact match — existing or newly created)\"\n    string :target, description: \"Target node name (exact match — existing or newly created)\"\n    string :relation, description: \"Must be one of: #{relation_names}\"\n    string :context, description: \"Briefly explain why this relationship exists, grounded in the content\"\n    array :attrs, description: \"Key-value attributes for this edge (e.g. since, weight)\" do ... end\n  end\nend\n```\n\nActual data operations (create or update, with the exact field-level diff) are derived server-side.\n\nWe load or initialize an ActiveRecord model, assign what the LLM returned, and let dirty tracking do the rest:\n\n```\n# from app/handlers/propose_graph_change.rb\nnode = Node.find_or_initialize_by(name: data[:name])\nenforce_status!(data[:name], data[:status], node) # raises when the model's new/existing claim disagrees with the DB\n\nwas_new = node.new_record?\nnode.assign_attributes(short_description: ..., description: ..., attrs: node.attrs.merge(attrs))\nchanges = node.changes.except(\"updated_at\", \"created_at\", \"kind\", \"slug\")\n\n{ op: was_new ? \"create\" : \"update\", node_id: node.persisted? ? node.id : nil, changes: changes }\n```\n\n`node.changes`\n\ngives us `{field => [before, after]}`\n\npairs for free, and this before/after snapshot becomes the wire format of the graph change proposal.\n\nEdges get exactly the same treatment - looked up by their `(source, target, relation)`\n\ntriple and diffed with dirty tracking.\n\nWe also don’t blindly trust what the LLM claims.\n\nIt has to declare each node as `new`\n\nor `existing`\n\n, and a validator cross-checks it against the database.\n\nOn mismatch, the LLM gets natural-language feedback and another attempt on the same conversation.\n\n## Identity resolution is the hard part\n\nI just wrote that the model has to declare each node as `new`\n\nor `existing`\n\n.\n\nBut how would it know?\n\nDo we load the whole graph into LLM context?\n\nNo - this is where **tool calls** come in.\n\nAnd it is harder than a simple lookup.\n\n“Piotrek”, “Piotr Jurewicz” and whatever Zoom’s transcription makes out of my name are the same person.\n\nIf you create a node per surface form, your graph turns into garbage within a week.\n\nWe handle it on three levels.\n\n**First, the model must look before it writes.**\n\nDuring extraction it has access to read-only tools like `search_nodes`\n\nor `get_node_edges`\n\n.\n\nThe extraction prompt is explicit about it:\n\n```\n(from app/lib/prompts/extraction.md.erb)\n- Before creating any node, use search_nodes to check if it already exists. (...)\n- If search_nodes returns no results, the node does not exist yet — proceed to create it. (...)\n- If search_nodes returns ambiguous results, or you need broader context to make extraction decisions, use get_node_edges to inspect the node's connections.\n- After finding nodes with search_nodes, use get_node_edges to see their existing relationships before deciding how to connect them.\n```\n\n**Second, aliases are the identity mechanism.**\n\nEach node has one canonical name and any number of aliases.\n\nThe schema instructs the model that an alias is the same entity under another name - never a separate entity.\n\nWhen the content reveals a better canonical form, the model sets `new_name`\n\nand the old name stays as an alias, so future fuzzy searches still resolve it.\n\n**Third, the search is hybrid.**\n\nTrigram similarity (pg_trgm with GIN indexes) over node names *and* aliases catches misspellings.\n\nEmbedding search catches semantic matches which share no characters:\n\n```\n# from app/models/node.rb\ndef self.hybrid_search(query, limit: 10)\n  # fuzzy match on canonical names and aliases, powered by pg_trgm\n  by_name  = where(\"similarity(nodes.name, ?) > 0.3\", query)\n  by_alias = joins(:aliases).where(\"similarity(node_aliases.name, ?) > 0.3\", query)\n  trigram_results = union_by_best_similarity(by_name, by_alias)\n\n  response = RubyLLM.embed(query, model: \"bge-m3\", provider: :ollama)\n  semantic_results =\n    nearest_neighbors(:embedding, response.vectors, distance: \"cosine\")\n      .select { |n| n.neighbor_distance < SEMANTIC_THRESHOLD }\n\n  merge_and_rank(trigram_results, semantic_results, limit)\nend\n```\n\nThe embeddings come from a self-hosted `bge-m3`\n\nmodel on Ollama, stored in pgvector.\n\n## Every fact has a source\n\nA graph edited by an AI is only trustworthy if you can audit every change.\n\nFor every node and edge we can answer: which extraction created you, which extractions updated you, and what exactly changed each time.\n\nProvenance lives in join tables (`node_extractions`\n\nand `edge_extractions`\n\n): one row per extraction and entity pair, holding the operation, the status, and the field-level `diff`\n\nproduced by the dirty tracking described before.\n\nStarting from any node, you can walk back through these rows to the extraction that touched it, and from the extraction to the ingested content it was based on.\n\nEvery fact in the graph traces back to its source.\n\nWe also record something we call the **read set**.\n\nEvery tool call the model makes during extraction is published as an `ExtractionToolCalled`\n\nevent and projected into `tool_invocations`\n\n, linked to the nodes and edges the call returned.\n\nSo we know not only what an extraction wrote, but also what it read before deciding.\n\nWhen you wonder “why did the model merge these two people?”, the answer is on the extraction page: here is the search it ran, and here is what came back.\n\nEach node’s page shows its full history: created in, last updated in, read by N extractions.\n\n## Keeping an eye on the costs\n\nBesides auditing changes in the graph, we also track how much each extraction costs: token usage and the resulting price.\n\nWhen you work with an LLM API, it is worth keeping a finger on the pulse here.\n\nA transcript of a few hours of conversation, processed in multiple rounds interleaved with tool calls, can generate significant costs.\n\n[Prompt caching](https://rubyllm.com/chat/#anthropic-prompt-caching) helps a lot - the system prompt and the content stay identical between rounds, so most of the input is billed at the cache-read rate.\n\nThe exact numbers depend on the model you run the extraction on, but most of ours cost well under a dollar.\n\n## Human in the loop\n\nWe don’t let the LLM write to the graph directly.\n\nExtraction produces a **proposal** with the before/after diffs, and applying it to the graph is a separate step.\n\nProposals can sit in a review window before they get applied.\n\nAs soon as an extraction completes, we get a short summary of it on Slack.\n\nA human can inspect the diff, apply it early, or just let it flow after the configured delay.\n\nTime passes between propose and apply, so the graph may have moved in the meantime.\n\nWhen the current state no longer matches what the proposal was based on, the apply stops and the affected rows get marked as conflicted, with a human-readable explanation.\n\n## Event sourcing ties it all together\n\nYou may have noticed that every mechanism above was described in terms of events.\n\nWell, this is an Arkency blog after all.\n\nThe whole pipeline is an event flow: `TranscriptIngested`\n\n→ `ExtractionRequested`\n\n→ `KnowledgeExtracted`\n\n→ `GraphChangeProposed`\n\n→ `GraphChangeApplied`\n\n(or `GraphChangeConflicted`\n\n).\n\nTwo small aggregates guard the invariants: one per ingestion (no two concurrent extractions of the same content), one per extraction (the propose → apply state machine).\n\nEverything you see in the UI (ingestions, extractions, diffs, tool invocations) is a read model built from these events.\n\nIn this architecture, the review window is just one more state in the aggregate’s state machine, and provenance is just one more read model built from an event we already had.\n\nI cannot understand people claiming that event sourcing makes things more complex ;)\n\n## The graph can feed itself\n\nOne feature shows the value of a uniform pipeline well.\n\nFrom any node you can request research.\n\nA job asks a model equipped with Anthropic’s server-side `web_search`\n\nand `web_fetch`\n\ntools to compile a brief about the entity:\n\n```\n# from app/jobs/research_topic.rb\nchat = RubyLLM\n  .chat(model: MODEL)\n  .with_params(tools: [\n    { type: \"web_search_20250305\", name: \"web_search\", max_uses: 10 },\n    { type: \"web_fetch_20250910\", name: \"web_fetch\", max_uses: 10 }\n  ])\n  .with_schema(ResearchBriefSchema.build)\n```\n\nThe prompt grounds the research in what the graph already knows about the entity, and tells the model when to give up:\n\n```\n# from app/jobs/research_topic.rb\nResearch \"#{topic}\". Use web_search and web_fetch as needed to gather facts.\n\nIn our knowledge base this entity is currently described as:\n- Kind: ...\n- Short description: ...\n- Attributes: ...\n\nWhen you can produce a useful brief, return status=\"completed\" and put the\nbrief in `brief` as Markdown. (...) Cover identity, key facts a knowledgeable\nreader should know, recent activity worth recording, and relationships to\nother named entities. Include source URLs inline next to claims that come\nfrom a specific page. Keep it factual; do not speculate.\n\nReturn status=\"aborted\" instead — with `abort_reason` naming the specific\nproblem — when any of these holds:\n- The topic is ambiguous and you cannot confidently pick the intended\n  interpretation from the disambiguation context above.\n- You cannot find substantive, verifiable information about this exact\n  entity (...)\n\nDo not pad an aborted result with related-but-different information.\n```\n\nThe resulting brief is not applied to the graph directly.\n\nIt gets published as a regular `TranscriptIngested`\n\nevent with its own kind, and flows through the same extraction, proposal and review pipeline as any other input.\n\n## The graph speaks MCP\n\nThe graph is not locked inside its own UI.\n\nWe expose it over **MCP**, so any AI assistant with access to our server can search it by asking questions in natural language - and answer from the graph, with sources.\n\n## Final thoughts\n\nWorking on Planet Arkency taught me a lot.\n\nAbout graphs, about LLMs, and about concepts I had never even heard of before: ontologies, identity resolution, provenance.\n\nI hope some of that knowledge stays with you after reading this post.\n\nIt also reassured me about the tools we have been using at Arkency for years.\n\nEvent-driven architecture and Rails Event Store carried this project naturally.\n\nI still have a head full of ideas on where to take this project next.\n\nWorking with [RubyLLM](https://rubyllm.com) was a pure pleasure - credits to Carmine Paolino for this gem.\n\nIf you are thinking about organizational memory for your company, or want us to help you build one, [get in touch](https://arkency.com/hire-us/).", "url": "https://wpnews.pro/news/maintaining-an-organizational-knowledge-graph-with-an-llm-and-event-sourcing", "canonical_source": "https://blog.arkency.com/maintaining-an-organizational-knowledge-graph-with-an-llm-and-event-sourcing/", "published_at": "2026-08-12 16:56:43+00:00", "updated_at": "2026-08-12 17:13:00.408245+00:00", "lang": "en", "topics": ["artificial-intelligence"], "entities": ["Arkency", "Planet Arkency", "Rails Event Store", "Obie Fernandez", "NEXUS", "Andrej Karpathy", "LLM Wiki", "PostgreSQL"], "alternates": {"html": "https://wpnews.pro/news/maintaining-an-organizational-knowledge-graph-with-an-llm-and-event-sourcing", "markdown": "https://wpnews.pro/news/maintaining-an-organizational-knowledge-graph-with-an-llm-and-event-sourcing.md", "text": "https://wpnews.pro/news/maintaining-an-organizational-knowledge-graph-with-an-llm-and-event-sourcing.txt", "jsonld": "https://wpnews.pro/news/maintaining-an-organizational-knowledge-graph-with-an-llm-and-event-sourcing.jsonld"}}