{"slug": "incremental-graph-updates-for-corporate-knowledge-graphs-three-problems-batch-t", "title": "Incremental Graph Updates for Corporate Knowledge Graphs: Three Problems Batch Pipelines Can't Solve", "summary": "A developer building 2asy.ai, a Graph-RAG system for East Asian corporate intelligence, found that batch pipelines fail for incremental graph updates, causing duplicate entities and conflicting facts. They propose resolving entities against the live graph before insertion and modeling property values as time-bounded assertions to handle supersession.", "body_md": "Corporate knowledge graphs break in production the moment you treat them as a batch artifact.\n\nI ran into this building [2asy.ai](https://2asy.ai), a Graph-RAG system for East Asian corporate intelligence. The initial architecture was standard: ingest a document set, run NER and entity resolution across the batch, write the graph. Clean, deterministic, fast to prototype.\n\nThen the first update cycle hit.\n\nThe batch pipeline handled entity resolution using blocking keys across the entire document corpus. With incremental updates, you only have the incoming batch. The blocking keys no longer reach the existing graph nodes.\n\nIn practice this means: a new trade document arrives with \"Samsung Elec. Co., Ltd.\" The existing graph has \"Samsung Electronics Co., Ltd.\" as a canonical node with 847 existing edges. Without live-graph entity resolution, you insert a second node and start building a disconnected duplicate.\n\nHere is the resolution-at-boundary pattern that works:\n\n``` python\nfrom dataclasses import dataclass\nfrom typing import Optional\nimport numpy as np\n\n@dataclass\nclass EntityCandidate:\n    raw_name: str\n    normalized_name: str\n    language: str\n    doc_id: str\n    embedding: Optional[np.ndarray] = None\n\ndef resolve_against_live_graph(\n    candidate: EntityCandidate,\n    graph_client,  # neo4j.GraphDatabase or similar\n    embed_fn,\n    threshold: float = 0.85,\n) -> Optional[str]:\n    \"\"\"\n    Run entity resolution against existing graph nodes before inserting.\n    Returns the existing node ID if a match is found, else None.\n    \"\"\"\n    # Step 1: Blocking — retrieve candidate neighbors by name prefix\n    blocking_query = \"\"\"\n    MATCH (e:Entity)\n    WHERE e.blocking_key STARTS WITH $prefix\n    RETURN e.id AS id, e.canonical_name AS name, e.name_embedding AS emb\n    LIMIT 50\n    \"\"\"\n    prefix = candidate.normalized_name[:4].upper()\n    neighbors = graph_client.run(blocking_query, prefix=prefix).data()\n\n    if not neighbors:\n        return None\n\n    # Step 2: Embedding similarity against blocked candidates\n    if candidate.embedding is None:\n        candidate.embedding = embed_fn(candidate.normalized_name)\n\n    best_score = 0.0\n    best_id = None\n    for row in neighbors:\n        if row[\"emb\"] is None:\n            continue\n        existing_emb = np.array(row[\"emb\"])\n        score = float(np.dot(candidate.embedding, existing_emb) /\n                      (np.linalg.norm(candidate.embedding) * np.linalg.norm(existing_emb) + 1e-9))\n        if score > best_score:\n            best_score = score\n            best_id = row[\"id\"]\n\n    return best_id if best_score >= threshold else None\n```\n\nIf `resolve_against_live_graph`\n\nreturns a node ID, you merge onto that node. If it returns None, you insert as new. The full entity resolution run happens at the boundary, not after insertion.\n\nA company relocates its headquarters. The new filing contains the correct address. The old record is already in the graph. Naive append gives you both — which means every Graph-RAG query touching that entity gets conflicting facts. The LLM has to guess which one is current.\n\nThe fix is to model each property value as a time-bounded assertion rather than a mutable attribute:\n\n``` python\nfrom datetime import date\nfrom neo4j import GraphDatabase\n\ndef upsert_fact_with_supersession(\n    driver: GraphDatabase,\n    entity_id: str,\n    property_name: str,\n    new_value: str,\n    valid_from: date,\n    source_doc_id: str,\n) -> None:\n    \"\"\"\n    Write a new fact, terminating any currently-open assertion for the same property.\n    Old assertions are retained for audit; only valid_to is set.\n    \"\"\"\n    with driver.session() as session:\n        session.run(\"\"\"\n            // Terminate the previous open assertion for this property\n            MATCH (e:Entity {id: $entity_id})-[:HAS_FACT]->(f:Fact)\n            WHERE f.property_name = $property_name\n              AND f.valid_to IS NULL\n            SET f.valid_to = date($valid_from) - duration('P1D')\n        \"\"\", entity_id=entity_id, property_name=property_name,\n             valid_from=str(valid_from))\n\n        session.run(\"\"\"\n            // Insert the new assertion with open valid_to\n            MATCH (e:Entity {id: $entity_id})\n            CREATE (e)-[:HAS_FACT]->(f:Fact {\n                property_name: $property_name,\n                value:          $new_value,\n                valid_from:     date($valid_from),\n                valid_to:       null,\n                source_doc_id:  $source_doc_id\n            })\n        \"\"\", entity_id=entity_id, property_name=property_name,\n             new_value=new_value, valid_from=str(valid_from),\n             source_doc_id=source_doc_id)\n```\n\nQueries then specify a point-in-time context:\n\n```\n// Get headquarters address as of 2024-03-01\nMATCH (e:Entity {id: $entity_id})-[:HAS_FACT]->(f:Fact)\nWHERE f.property_name = 'headquarters_address'\n  AND f.valid_from <= date('2024-03-01')\n  AND (f.valid_to IS NULL OR f.valid_to >= date('2024-03-01'))\nRETURN f.value AS address\n```\n\nThis is the pattern that makes Graph-RAG `as-of`\n\nqueries reliable. Without it, you are relying on insertion order, which is neither guaranteed nor queryable.\n\nWhen two entity nodes merge (because `resolve_against_live_graph`\n\nfound a match), the incoming node's edges need to be transferred to the surviving node. If your update pipeline does not model this explicitly, you end up with a partially disconnected graph where relationships point to deprecated node IDs.\n\n``` python\ndef merge_nodes_and_transfer_edges(\n    driver: GraphDatabase,\n    surviving_id: str,\n    deprecated_id: str,\n) -> int:\n    \"\"\"\n    Transfer all edges from deprecated node to surviving node,\n    then mark deprecated node as merged.\n    Returns the number of edges transferred.\n    \"\"\"\n    with driver.session() as session:\n        # Transfer outgoing edges\n        result = session.run(\"\"\"\n            MATCH (deprecated:Entity {id: $deprecated_id})-[r]->(target)\n            WHERE NOT target.id = $surviving_id\n            MATCH (surviving:Entity {id: $surviving_id})\n            CALL apoc.refactor.from(r, surviving)\n            YIELD input, output\n            RETURN count(*) AS transferred\n        \"\"\", deprecated_id=deprecated_id, surviving_id=surviving_id)\n        outgoing = result.single()[\"transferred\"]\n\n        # Transfer incoming edges\n        result = session.run(\"\"\"\n            MATCH (source)-[r]->(deprecated:Entity {id: $deprecated_id})\n            WHERE NOT source.id = $surviving_id\n            MATCH (surviving:Entity {id: $surviving_id})\n            CALL apoc.refactor.to(r, surviving)\n            YIELD input, output\n            RETURN count(*) AS transferred\n        \"\"\", deprecated_id=deprecated_id, surviving_id=surviving_id)\n        incoming = result.single()[\"transferred\"]\n\n        # Mark deprecated node\n        session.run(\"\"\"\n            MATCH (e:Entity {id: $deprecated_id})\n            SET e.merged_into = $surviving_id,\n                e.merged_at   = datetime()\n        \"\"\", deprecated_id=deprecated_id, surviving_id=surviving_id)\n\n    return outgoing + incoming\n```\n\nThe `merged_into`\n\nproperty lets you build a forwarding index. Any query that arrives with a deprecated node ID can be redirected to the surviving node without a full graph scan.\n\nAfter a merge, the direct neighbors of the deprecated node may have consistency issues — edges that now point through the merged node incorrectly, or facts that reference the deprecated entity in their source context.\n\nThe lightweight fix is a dirty-neighbor queue:\n\n``` python\ndef flag_dirty_neighbors(\n    driver: GraphDatabase,\n    node_id: str,\n    reason: str,\n) -> list[str]:\n    \"\"\"\n    Flag the direct neighbors of a node for consistency recheck\n    in the next ingestion cycle.\n    \"\"\"\n    with driver.session() as session:\n        result = session.run(\"\"\"\n            MATCH (n:Entity {id: $node_id})-[*1]-(neighbor:Entity)\n            WHERE NOT neighbor.id = $node_id\n            SET neighbor.dirty = true,\n                neighbor.dirty_reason = $reason,\n                neighbor.dirty_at = datetime()\n            RETURN collect(DISTINCT neighbor.id) AS neighbor_ids\n        \"\"\", node_id=node_id, reason=reason)\n        return result.single()[\"neighbor_ids\"]\n```\n\nAt the start of each ingestion cycle, you pull all dirty neighbors and run a consistency check before processing new documents. This handles cascade propagation without a full graph traversal on every update.\n\nBatch rebuild works in a prototype. Production knowledge graphs need an explicit update model from the start:\n\nAdding these after the fact means rebuilding the schema and reprocessing historical documents. The three problems above are not edge cases — they are the production steady state for any graph that receives continuous updates.\n\n*I work on entity resolution and agentic infrastructure at er-api.hannune.ai. The Graph-RAG system this is based on runs against East Asian corporate registries where transliteration variance and entity deduplication are the hard problems.*", "url": "https://wpnews.pro/news/incremental-graph-updates-for-corporate-knowledge-graphs-three-problems-batch-t", "canonical_source": "https://dev.to/hannune/incremental-graph-updates-for-corporate-knowledge-graphs-three-problems-batch-pipelines-cant-solve-2phm", "published_at": "2026-08-04 02:38:14+00:00", "updated_at": "2026-08-04 03:10:28.489029+00:00", "lang": "en", "topics": ["artificial-intelligence", "developer-tools"], "entities": ["2asy.ai", "Samsung Electronics", "Neo4j"], "alternates": {"html": "https://wpnews.pro/news/incremental-graph-updates-for-corporate-knowledge-graphs-three-problems-batch-t", "markdown": "https://wpnews.pro/news/incremental-graph-updates-for-corporate-knowledge-graphs-three-problems-batch-t.md", "text": "https://wpnews.pro/news/incremental-graph-updates-for-corporate-knowledge-graphs-three-problems-batch-t.txt", "jsonld": "https://wpnews.pro/news/incremental-graph-updates-for-corporate-knowledge-graphs-three-problems-batch-t.jsonld"}}