# LLM Wiki Maintenance: Drift, Contradictions and Review > Source: > Published: 2026-07-16 09:43:56+00:00 # LLM Wiki Maintenance: Drift, Contradictions and Review Keep compiled knowledge trustworthy An LLM Wiki fails when old facts remain plausible, contradictions become polished, and generated summaries drift from their sources. Maintenance is the real product of any compiled knowledge system. Creating wiki pages is straightforward compared with keeping them trustworthy across months of ingest, edits, rewrites, and new sources. This article covers the operational side of LLM Wiki systems: drift detection, contradiction checks, citation discipline, linting, Git review, and maintenance workflows. It assumes you already understand the basic pattern described in [LLM Wiki - Compiled Knowledge That RAG Cannot Replace](https://www.glukhov.org/knowledge-management/knowledge-systems-architectures/compiled-knowledge/what-is-llm-wiki/): raw sources are compiled into durable Markdown pages that humans and agents can query later. The calm but opinionated view is that an LLM Wiki without maintenance is just a nicer-looking knowledge graveyard. ## What Makes LLM Wiki Maintenance Different Traditional wikis rot because people stop updating them. RAG systems drift because the corpus changes, chunks get stale, metadata is weak, and retrieval keeps finding plausible but outdated fragments. The deeper reason is that [retrieval and representation solve different problems](https://www.glukhov.org/knowledge-management/foundations/retrieval-vs-representation/): retrieval can be re-run against fresh data, but a representation you already compiled has to be actively kept honest. An LLM Wiki has a different failure mode. It can look clean even when it is wrong. The pages may be well formatted. The links may work. The summaries may sound balanced. But underneath that neat surface, the system may have dropped critical facts, merged incompatible concepts, cited summaries instead of sources, or preserved an old decision as if it still applied. That is why LLM Wiki maintenance must check both structure and meaning. ## The Maintenance Goal The goal is not to make every page perfect. That is too expensive, and it usually leads to abandoned systems. The goal is to keep the wiki useful, inspectable, and recoverable. A maintained LLM Wiki should make it easy to answer these questions: - What sources support this claim? - When was this page last reviewed? - Has this concept changed? - Are there conflicting pages? - Is this summary still current? - Did the agent rewrite more than it should? - Can we roll back a bad update? - Can a human understand why the page says what it says? If the answer is no, the problem is not only content quality. It is system design. ## A Simple Maintenance Loop A useful LLM Wiki needs a repeatable loop. The loop should be simple enough to run often and strict enough to catch drift before it becomes invisible. This loop is not glamorous. That is the point. A knowledge system becomes durable through boring maintenance: source preservation, explicit review, predictable structure, and small safe updates. The same instinct drives [evergreen notes](https://www.glukhov.org/knowledge-management/methods/evergreen-notes/): a note or a wiki page only compounds in value if someone keeps refining it instead of leaving it to rot after the first draft. ## Types of Drift in an LLM Wiki Knowledge drift is not one thing. Different drift types require different checks. A good maintenance system should name them clearly. ### Source Drift Source drift happens when the underlying source material changes. For example, a tool releases a new version, a policy is updated, an API changes, or a vendor deprecates an old feature. The old wiki page may still be accurate for the previous version, but wrong for current use. Source drift is dangerous because the old claim may still be true in historical context. The problem is not that the claim is fake. The problem is that it no longer answers the current question. **Maintenance response:** - Record source dates - Record last reviewed dates - Mark version-specific pages clearly - Link old pages to superseding pages - Avoid mixing old and new versions without labels ### Concept Drift Concept drift happens when the meaning of a term changes over time. This is common in AI and software architecture. Terms like “agent”, “memory”, “RAG”, “workflow”, “tool use”, “structured output”, and “context engineering” can shift meaning quickly. A wiki can accidentally preserve several meanings of the same term without explaining the difference. That creates confusing pages that sound coherent but combine incompatible ideas. **Maintenance response:** - Maintain glossary pages - Add “meaning in this wiki” sections - Separate overloaded concepts into distinct pages - Link related meanings explicitly - Avoid letting the model silently merge terms ### Terminology Drift Terminology drift is smaller than concept drift but still harmful. It happens when the wiki uses multiple names for the same thing: “LLM Wiki”, “compiled knowledge base”, “AI-maintained wiki”, “agent-maintained wiki”, and “Markdown knowledge base”. Some variation is fine. Too much variation breaks search, linking, and review. **Maintenance response:** - Define canonical page names - Keep aliases in front matter or page metadata - Redirect duplicate pages - Lint for near-duplicate titles - Use consistent anchor text in internal links ### Decision Drift Decision drift happens when a past decision remains documented but no longer reflects current practice. For example, a page might say that the project uses vector RAG for all document search, while newer pages describe an LLM Wiki workflow. Both statements may be historically true, but the wiki must show which one is current. This matters for architecture notes, engineering processes, content strategy, and internal tooling. The same discipline applies to [decision records in AI-driven development](https://www.glukhov.org/app-architecture/documentation/decision-records-ai-driven-development/): a decision page is only trustworthy if superseded choices are marked as superseded rather than silently overwritten. **Maintenance response:** - Mark decisions as proposed, accepted, superseded, or rejected - Keep decision dates - Link superseded decisions to replacements - Preserve historical context - Avoid deleting old decisions without trace ### Citation Drift Citation drift happens when a page cites a source, but the claim no longer matches what the source says. This can happen after a rewrite. The citation remains in place, but the sentence around it changes. The page still looks sourced, yet the citation no longer supports the claim. This is one of the most serious LLM Wiki failure modes because it creates false confidence. **Maintenance response:** - Check claim-level citation support - Avoid citing only summaries - Keep raw sources - Require citations for important claims - Flag paragraphs with sources but no direct support - Review citation changes in Git diffs ### Structure Drift Structure drift happens when the wiki slowly loses navigability. New pages are created instead of updating old ones. Index pages fall behind. Duplicate pages appear. Orphan pages accumulate. Related pages stop linking to each other. The wiki still contains useful knowledge, but finding and trusting it becomes harder. **Maintenance response:** - Lint for orphan pages - Maintain index pages - Detect duplicate topics - Require backlinks for canonical pages - Archive rather than scatter - Keep folder roles clear ## The Core Maintenance Files An LLM Wiki should not rely on memory, vibes, or repeated prompting. It needs operating files that tell humans and agents how to maintain it. A practical structure can look like this: ``` llm-wiki/ raw/ sources/ transcripts/ documents/ wiki/ index.md concepts/ entities/ projects/ decisions/ maintenance/ review-log.md lint-report.md stale-pages.md contradiction-report.md rules/ AGENTS.md schema.md style-guide.md citation-policy.md source-policy.md ``` The exact names do not matter. The roles do. The wiki needs source storage, compiled pages, review artifacts, and rules that survive across sessions. ### AGENTS.md `AGENTS.md` should explain how the agent is allowed to work. It should answer questions like: - Should the agent update existing pages before creating new ones? - When should it ask for review? - How should it cite sources? - Can it reorganize folders? - Can it rewrite old pages? - What should it do with contradictions? - What should it never delete? A useful rule is: ``` Prefer small, source-backed updates over broad rewrites. ``` That one sentence prevents many bad maintenance habits. It also draws a useful boundary relative to [agent memory](https://www.glukhov.org/ai-systems/memory/memory-systems-in-ai-assistants/): memory shapes how an agent behaves in the moment, while the wiki’s rules file shapes what the agent is allowed to change in the shared knowledge base. ### schema.md `schema.md` should define the page structure. For example: ``` # Page Title ## Summary Short current summary. ## Key Claims - Claim with source reference. - Claim with source reference. ## Current Status Current, historical, superseded, draft, or uncertain. ## Details Main compiled explanation. ## Related Pages Internal links. ## Sources Raw source references. ## Review Notes Last reviewed date and reviewer. ``` A schema does not need to be rigid. But without a schema, the wiki becomes a pile of inconsistent essays. The schema gives the agent something to preserve. ### citation-policy.md The citation policy should say what requires a source. At minimum, require citations for: - Technical claims - Comparisons - Product behavior - Version-specific statements - Benchmark results - Pricing - Legal or compliance claims - Current status claims - Claims copied or derived from a source A good citation policy also says what not to do. Do not cite a compiled page as if it were the original source. Do not attach citations to paragraphs they do not support. Do not keep a citation after rewriting a claim unless the source still supports it. Citation discipline is the difference between a useful LLM Wiki and decorative confidence. ### source-policy.md The source policy should define how raw sources are stored. For example: - Keep original files when possible - Preserve URLs and access dates - Store copied notes separately from generated summaries - Record source type - Mark low-confidence sources - Avoid overwriting raw material - Keep superseded sources when they explain historical decisions This policy protects the wiki from a common mistake: summarizing a source, discarding the original, and then treating the summary as evidence. That is not knowledge management. That is lossy compression. ## What an LLM Wiki Linter Should Check A linter should check structure first. Structural problems are easier to automate and often reveal deeper semantic problems. Useful structural checks include: - Broken internal links - Orphan pages - Duplicate titles - Missing source sections - Missing review dates - Pages without backlinks - Empty or placeholder sections - Very long pages without section structure - Pages with no incoming links from an index - Inconsistent naming conventions - Invalid front matter - Stale “current” pages older than a review threshold These checks do not prove the knowledge is true. They prove the wiki is still maintainable. That is a necessary starting point. ## Semantic Checks Semantic checks are harder but more valuable. Useful semantic checks include: - Claims without direct source support - Two pages making incompatible claims - Old decisions presented as current - Duplicate concepts with different names - One concept page mixing several meanings - Summaries that omit known constraints - “Best” or “recommended” claims without criteria - Version-specific claims without version labels - Pages that contradict newer sources These checks should not automatically rewrite the wiki. They should usually produce a report for review. The safer pattern is: ``` Detect automatically. Explain clearly. Update deliberately. Review risky changes. ``` ## Contradiction Detection Contradiction detection is not just asking an LLM whether two pages contradict each other. That can help, but it is too vague. A better approach is to compare claims. A contradiction workflow can look like this: This works better because many apparent contradictions are not real contradictions. One page may describe version 1.0 and another version 2.0. One page may describe personal use and another enterprise use. One page may describe the design goal and another the implementation reality. The contradiction report should include: - Conflicting claims - Pages involved - Source references - Date or version context - Likely explanation - Suggested resolution - Whether human review is required Do not let the agent silently resolve contradictions by blending both claims into a vague compromise. That creates smooth nonsense. ## How to Resolve Contradictions There are several valid ways to resolve a contradiction. - If one claim is outdated, mark it as superseded and link to the newer page. - If both claims are true in different contexts, split the context clearly. - If the sources disagree, preserve the disagreement and explain it. - If the wiki page invented or overgeneralized a claim, remove or narrow the claim. - If the contradiction reflects a real unresolved decision, create or update a decision page. The goal is not to remove all tension. Some tension is useful. A good wiki shows where knowledge is settled and where it is still uncertain. ## Stale Page Review Every compiled page should have a review status. A simple metadata block can be enough: ``` status: current last_reviewed: 2026-07-10 review_after: 2026-10-10 source_confidence: medium ``` For fast-moving topics, use short review windows. For stable concepts, use longer windows. | Page type | Review interval | |---|---| | Tool version pages | 30 to 90 days | | Pricing or availability pages | 7 to 30 days | | Architecture principles | 6 to 18 months | | Historical decision records | Only when superseded | | Glossary pages | 3 to 12 months | | Source summaries | When source changes | | Comparison pages | 30 to 180 days | The interval is less important than the habit. A page without a review date is a page that will eventually lie quietly. ## Source-Aware Updates When a new source arrives, the agent should not simply write a new summary page. It should ask: - Does this update an existing concept? - Does it supersede a previous source? - Does it contradict an existing page? - Does it add a new entity or term? - Does it change a recommendation? - Does it require an index update? - Does it affect decision pages? A good ingest workflow updates the wiki as a graph, not as a stack of isolated summaries. This is why one source may touch many pages. That is normal. The maintenance challenge is making those updates small, reviewable, and traceable. ## Review with Git Diffs Git is one of the best maintenance tools for an LLM Wiki. Not because Git is fashionable, but because generated knowledge needs reviewable change history. A Git diff shows what the agent changed, deleted, moved, or rephrased. Use Git to review: - New pages - Deleted sections - Changed claims - Changed citations - Renamed pages - Index updates - Link changes - Status changes - Broad rewrites The most important review habit is to inspect deletions. LLMs often remove details while making prose cleaner. Clean prose is not always better knowledge. ## Safe Commit Strategy Avoid giant commits like: ``` Update wiki ``` Use commits that explain the operation: ``` ingest: add source notes for Qwen embedding release maint: mark old vector store comparison as superseded lint: fix broken internal links in RAG pages review: update LLM Wiki page citations refactor: split agent memory concept from long-term memory ``` This gives future humans and agents a useful history. Commit messages are part of the knowledge system. They explain why the wiki changed. ## Review Levels Not every page needs the same review level. Use risk-based review. **Low-risk changes:** - Broken link fixes - Formatting cleanup - Adding backlinks - Adding index entries - Correcting typos - Adding aliases **Medium-risk changes:** - Adding source summaries - Creating new concept pages - Updating recommendations - Merging duplicate pages - Changing page status **High-risk changes:** - Deleting claims - Rewriting canonical pages - Changing decision records - Updating security, legal, pricing, or benchmark claims - Resolving contradictions - Marking content as current or superseded High-risk changes should get human review. That may be you, a maintainer, or a domain owner. The agent can prepare the change, but it should not always be the final authority. ## A Maintenance Dashboard A simple maintenance dashboard can live in Markdown. Example: ``` # LLM Wiki Maintenance Dashboard ## Needs Review - wiki/concepts/agent-memory.md - review overdue - wiki/projects/search-index.md - conflicting claims found ## Stale Pages - wiki/tools/ollama.md - last reviewed 120 days ago - wiki/concepts/rag.md - fast-moving topic, review due ## Broken Links - wiki/index.md -> wiki/concepts/old-page.md ## Orphan Pages - wiki/entities/vendor-x.md ## Contradiction Reports - reports/contradictions/2026-07-10-agent-memory.md ## Recent Ingests - raw/sources/2026-07-09-new-paper.md ``` This is not a fancy admin panel. It is a working surface. For many solo or small-team systems, a Markdown dashboard is enough. ## Metrics for a Healthy LLM Wiki You can track wiki health without overengineering it. Useful metrics include: - Number of pages - Pages without sources - Pages without review dates - Pages past review date - Broken internal links - Orphan pages - Duplicate titles or aliases - Contradiction reports open - Contradiction reports resolved - Average page age by cluster - Raw sources not yet ingested - Pages changed without human review - Canonical pages changed in the last 30 days These metrics are not vanity numbers. They tell you whether the system is becoming easier or harder to trust. A growing wiki with rising orphan pages and stale reviews is not compounding knowledge. It is compounding maintenance debt. ## Maintenance Prompts Maintenance prompts should be narrow. Broad prompts produce broad rewrites. ### Structural Lint Prompt ``` Review the wiki structure. Check for: - broken internal links - orphan pages - duplicate or near-duplicate page titles - pages missing source sections - pages missing last_reviewed metadata - index pages that do not link to new pages Do not rewrite content. Produce a Markdown report with findings and suggested fixes. ``` ### Citation Review Prompt ``` Review citations for the selected page. For each important claim: - identify the supporting source - check whether the source directly supports the claim - flag claims with weak, missing, or mismatched support - do not rewrite the page yet Return a table with: claim, current citation, support level, issue, suggested action. ``` ### Contradiction Check Prompt ``` Check this page for contradictions against related pages. Steps: 1. Extract the main claims from the page. 2. Find related pages through links, backlinks, aliases, and search. 3. Extract potentially conflicting claims. 4. Classify each conflict as: - real contradiction - version difference - scope difference - terminology difference - unresolved uncertainty 5. Recommend a resolution. Do not edit files until the contradiction report is reviewed. ``` ### Stale Page Review Prompt ``` Review this page for staleness. Check: - dates - version-specific claims - tool behavior - recommendations - links to superseded pages - sources newer than the page review date - claims marked current without recent support Return: - keep current - update needed - mark historical - mark superseded - split into versioned pages ``` ### Safe Update Prompt ``` Update the selected wiki page using the provided source. Rules: - preserve existing useful structure - do not remove sourced claims unless the source is superseded - add citations for new claims - mark uncertainty explicitly - update related pages only when necessary - add a review note explaining what changed - keep the diff small ``` These prompts are not magic. They are guardrails. The real value comes from making the maintenance action explicit. ## Common Maintenance Mistakes The first mistake is letting the agent rewrite too much. Large rewrites feel productive because they make pages smoother. They also make it harder to see what changed and easier to lose sharp details. The second mistake is treating generated summaries as sources. A summary can be useful, but it is not the evidence. Keep raw sources and cite them when claims matter. The third mistake is creating new pages instead of updating old ones. This produces duplicate concepts, conflicting recommendations, and search confusion. The agent should search first, update second, and create new pages only when the concept is genuinely new. The fourth mistake is deleting historical context. Old decisions, failed experiments, and superseded recommendations can be valuable. Mark them clearly instead of erasing them. The fifth mistake is forgetting indexes. A page that is not linked from the right place is half-lost. Index maintenance is not clerical work. It is navigation architecture. The sixth mistake is using the same review policy for every page. A stable conceptual page and a fast-moving tool comparison should not have the same review schedule. ## Archive, Supersede, or Delete Most stale pages should not be deleted immediately. | Outcome | Use when | |---|---| | Archive | The page is historical but still useful | | Supersede | A newer page replaces the old answer | | Delete | The page is duplicate, empty, wrong, or unrecoverable | A superseded page should link to the replacement. For example: ``` Status: superseded Superseded by: wiki/concepts/agent-memory-architecture.md Reason: This page used an older definition of agent memory before the project separated session memory, user memory, and compiled knowledge. ``` This preserves context and reduces confusion. Deletion should be rare and reviewable. Knowledge systems need pruning, but invisible pruning is dangerous. ## Canonical Pages Need Extra Care Every LLM Wiki eventually develops canonical pages. These are pages that define core concepts, architecture choices, workflows, or project vocabulary. They are more important than ordinary notes because many other pages depend on them. Examples: - What is LLM Wiki - RAG vs compiled knowledge - Agent memory - Source policy - Review workflow - Project architecture - Glossary - Index Changes to canonical pages should require stricter review. A small error on an obscure page is local. A small error on a canonical page can distort the whole wiki. ## How Often Should You Run Maintenance For an active LLM Wiki, run small maintenance frequently and deep maintenance occasionally. | Frequency | Maintenance task | |---|---| | Every ingest | Update links, sources, indexes, and review notes | | Weekly | Run structural lint checks | | Monthly | Review stale pages and orphan pages | | Monthly | Check contradictions in active clusters | | Quarterly | Review canonical pages | | Quarterly | Archive or supersede old pages | | Before major use | Run citation checks on pages used for reports, articles, or decisions | The schedule should match risk. A personal research wiki can be lighter. A wiki used for technical publishing, customer support, or engineering decisions needs more discipline. ## Using an LLM Wiki for Technical Publishing For a technical blog or documentation site, the LLM Wiki can become the knowledge layer behind article planning. It can track: - Canonical explanations - Repeated definitions - Internal link opportunities - Stale comparisons - Article gaps - Topic clusters - Source notes - Decisions about structure - Claims that need verification This is useful, but it raises the maintenance bar. If the wiki feeds published articles, then stale wiki pages can become stale public content. If the wiki contains contradictions, those contradictions may leak into articles. If citations are weak, generated drafts will inherit that weakness. The scoped summaries, schema-based extraction, and human review loops described in [AI for Knowledge Management: Real Workflows That Hold Up](https://www.glukhov.org/knowledge-management/ai-augmented-knowledge/ai-for-knowledge-management-workflows/) are a good template for keeping that editorial pipeline honest. For publishing workflows, treat the LLM Wiki as an editorial system, not just a private notebook. A [digital garden](https://www.glukhov.org/knowledge-management/methods/digital-gardening/) makes this explicit by showing readers which pages are still growing and which are mature, which is exactly the kind of status signal a well-maintained LLM Wiki should also expose. ## Maintenance Is Not Optional An LLM Wiki is attractive because it promises compounding knowledge. But compounding only works if old knowledge remains useful. Otherwise, the system compounds noise, not insight. The maintenance burden does not disappear because an agent is involved. It changes shape. Humans should not have to do every tedious task. Agents can check links, find stale pages, compare claims, draft reports, and propose updates. But humans still need to set policy, review risky changes, and decide what the wiki is allowed to mean. That is the real bargain. ## Conclusion LLM Wiki maintenance is about keeping compiled knowledge honest. The important practices are not complicated: preserve sources, cite claims, keep indexes current, detect contradictions, review Git diffs, mark stale pages, and avoid broad unreviewed rewrites. A good LLM Wiki does not pretend to be magically self-correcting. It makes correction easier. That is the standard worth aiming for: not perfect knowledge, but inspectable knowledge that can be reviewed, repaired, and trusted over time.