{"slug": "how-to-keep-an-ai-context-layer-from-going-stale", "title": "How to Keep an AI Context Layer From Going Stale", "summary": "A September Build Lab session outlined four feedback loops — local review, CI, scheduled audits, and channel feedback — for keeping an AI context layer current, with each loop required to produce an owner, a tracked request, and a reviewed change merged by a person. The session, phase four of onboarding an agent like a new hire, warns that stale context is worse than none because it looks correct, and that agent memory does not fix the problem since it is private to one session or person. The recommended CI loop runs bruin validate, bruin format --fail-if-changed, and a custom context_audit.sh script on every pull request.", "body_md": "**Quick answer:** an AI context layer starts going stale the day you write it. Code changes, business rules change, and the correction someone made in a Slack thread scrolls away. Four feedback loops keep it current: local review, CI, scheduled audits, and channel feedback. Each one catches a different kind of drift, and they all end the same way, with an owner, a tracked request, and a reviewed change. At no point does the agent silently edit context.\n\nThis comes from the September [Build Lab](https://getbruin.com/build-lab/) session on context for agents. It is phase four of [onboarding an agent like a new hire](https://getbruin.com/blog/onboard-ai-data-agent-like-a-new-hire/), where the agent is hired, real teams use it, and maintenance becomes part of the job.\n\n## [Why context goes stale](#why-context-goes-stale)\n\nNobody has to be careless for this to happen. Three ordinary things wear it down.\n\nThe code moves. A backend release renames an order status, adds a column, or changes a unit, and the asset description, the check, and the glossary all still describe last month.\n\nThe business moves too. Finance decides refunds come out of net revenue in the period they happen, and the semantic model still says otherwise.\n\nAnd corrections evaporate. Someone tells the agent \"that spike is expected on the first of the month\". That sentence lives in a thread, so the next person and the next agent make the same mistake.\n\nStale context is worse than none, because it looks correct. Agent memory does not fix this either. Memory is private to one session or one person, nobody reviewed it, and a new session or a different tool loses it. Write it where the team reads it, which is the repo.\n\n## [The rule for every loop](#the-rule-for-every-loop)\n\nEvery loop has to produce the same three things:\n\n1. **An owner** - the person who can say what the definition should be.\n2. **A tracked request** - an issue or a pull request. A chat message does not count.\n3. **A reviewed change** - merged by a person, in the same surface you use for code.\n\nThis is the usual permission rule for agent tools, made concrete. The agent gets broad read access so it can see drift anywhere, and narrow write access so the most it can do is propose.\n\n## [Loop 1: local review](#loop-1-local-review)\n\nThe cheapest loop runs while someone is working. When a person corrects the agent during a task, the agent updates the context in that same task, in the same pull request as the fix.\n\nWrite this into `AGENTS.md` as an instruction:\n\n```\n## When you are corrected\n- Find the context that misled you: the asset description, a column, a check,\n  the glossary, or the semantic model.\n- Update it in the same pull request as the fix, and say what you changed and why.\n- Ask why before writing it down. \"Stop flagging it\" and \"this is expected on the\n  first of the month\" are different rules.\n- If the correction changes a metric definition, do not write it. Flag the owner.\n```\n\nThis catches errors before they reach anyone else. A correction today is context every agent has tomorrow.\n\n## [Loop 2: CI on every pull request](#loop-2-ci-on-every-pull-request)\n\nThe second loop reviews changes before they get a chance to break something. On every pull request, you validate the context the same way you validate the code.\n\n```\n# .github/workflows/context.yml\nname: context\non: pull_request\n\njobs:\n  context:\n    runs-on: ubuntu-latest\n    steps:\n      - uses: actions/checkout@v4\n      - uses: bruin-data/setup-bruin@main\n      - name: Write Bruin config\n        env:\n          BRUIN_CONFIG: ${{ secrets.BRUIN_CONFIG }}\n        run: printf '%s' \"$BRUIN_CONFIG\" > .bruin.yml\n\n      # Assets, dependencies, and checks are valid\n      - run: bruin validate .\n      # Asset files are in canonical format, so diffs stay readable\n      - run: bruin format --fail-if-changed .\n      # Your own audit: compare asset definitions with what the warehouse holds\n      - run: ./scripts/context_audit.sh\n```\n\n`bruin validate` and `bruin format --fail-if-changed` are [Bruin CLI](https://github.com/bruin-data/bruin) commands. The last step is not. It is a script you write, or an agent you call, that compares what the asset says with what the data shows. For example, it can flag a status value that appears in the warehouse but is missing from the column's `accepted_values` check.\n\nWhen it finds something, it drafts a candidate change and explains it in plain language, so the reviewer does not have to reverse-engineer a diff:\n\n*Illustrative mockup. The `custom-context-audit` check is custom logic, not a built-in Bruin command.*\n\nI call it a *candidate* on purpose. A value observed in the warehouse is not automatically a valid business value. `refunded` showing up in `stg_orders` might be a real new status, or it might be a bug in the app. The owner confirms the definition before merge.\n\n## [Loop 3: scheduled audits](#loop-3-scheduled-audits)\n\nCI only sees what changes in the repo. Drift that starts somewhere else (in the source app, in the warehouse, in who owns what) needs something that goes looking for it on a schedule.\n\nIn Bruin Cloud this is a [scheduled agent](https://getbruin.com/docs/bruin/cloud/ai-agents/scheduled.html): an existing AI agent given a task that runs on a recurring schedule. Each run uses the agent's connections, integrations, and CLI access, and posts to Slack, Teams, WhatsApp, or the Bruin Cloud chat. The audit is the task you give it, meaning what to scan, what counts as drift, and how to report it.\n\nA useful audit sorts what it finds into two buckets:\n\n*Illustrative mockup. The audit logic and the message format are custom; Bruin does not decide which context changes to make.*\n\n- **Suggested changes.** Mechanical drift, like a new value, an owner change, or a missing column, drafted as pull requests through your CI or GitHub automation.\n- **Needs a human.** Anything that changes business meaning, such as a revenue definition that may now exclude refunds. The agent flags it and stops there.\n\nThat separation is the trust model. The scheduled agent delivers the report, and the reviewable change goes through the same path as any other code.\n\nFor pure schema drift there is a simpler version of this loop. A weekly job that runs `bruin import database` and `bruin ai enhance` and opens a pull request with the diff turns new tables and columns into a reviewable change. `ai enhance` is additive, so it fills gaps without overwriting what people wrote. The [AI context layer post](https://getbruin.com/blog/build-ai-context-layer-data-warehouse/) covers that setup.\n\nOne cost to watch: a scheduled audit runs queries whether or not anything changed, and on most bills the warehouse spend matters more than the model tokens. Scope the audit to the assets that matter, and pick a cadence that matches how fast they change.\n\n## [Loop 4: channel feedback](#loop-4-channel-feedback)\n\nThe last loop catches what no audit can, which is knowledge that only comes up when a person reacts to an answer.\n\n*Illustrative mockup. Issue #482 is fictitious.*\n\nThe order of these steps is what keeps it safe:\n\n1. **Report.** A data consumer says`revenue_daily` looks off and may still count refunded orders.\n2. **Verify.** The agent checks the SQL, the canonical definition of net revenue, and whether a known-question eval fails. At this stage the report is a lead, nothing more.\n3. **Draft an issue.** Configured automation opens an issue with the evidence and a proposed fix, written as a question for the owner rather than a confirmed bug.\n4. **Review.** The metric owner decides what the definition should be.\n5. **Pull request.** The fix lands with the glossary entry or semantic model updated, plus a check so the same gap cannot quietly open again.\n\nThe issue should say what was checked and what is being proposed. If it presents an unverified cause as fact, that is exactly how a context layer collects confident mistakes.\n\n## [What each loop catches](#what-each-loop-catches)\n\n| Loop | Triggered by | Catches | \n|---|---|---|\n| Local review | a person correcting the agent mid-task | errors before anyone else sees them | \n| CI | every pull request | invalid context, formatting drift, definitions that disagree with the warehouse | \n| Scheduled audit | a cadence you choose | drift from outside the repo: new values, new tables, ownership changes | \n| Channel feedback | a user reacting to an answer | wrong business meaning nobody wrote down | \n\nCI and scheduled audits keep the context matching reality. Channel feedback keeps it matching what the business means. You want all four, because they catch different failures.\n\n## [Recycle while you are there](#recycle-while-you-are-there)\n\nKeeping context current also means taking things out. Every loop is a chance to delete something.\n\nIf a loop finds two sources that disagree, pick one home for the fact and remove or rewrite the other copy (a conflict is worse than a gap, and [The Best Context Is No Context](https://getbruin.com/blog/best-context-is-no-context/) explains why). If an audit turns up tables, reports, or dashboards nobody uses, remove them rather than documenting them. And if a page is wrong, delete it. A wrong page is worse than no page.\n\n## [Measure that it is working](#measure-that-it-is-working)\n\nKeep the known-question set from the internship phase in the repo and rerun it after context changes. If accuracy holds or goes up, the loops are doing their job. If it drops after a merge, the last change made the context worse, and you know exactly which pull request to look at.\n\nEvery incident and every correction should leave something behind: a check, a description, or a rule. When one leaves nothing, expect to see it again.\n\n## [FAQ](#faq)\n\n### [How do I keep AI agent context up to date?](#how-do-i-keep-ai-agent-context-up-to-date)\n\nRun four feedback loops. Local review: when an agent is corrected during a task, it updates the context in the same pull request. CI: every pull request validates the context and flags drift between the asset definitions and the warehouse. Scheduled audits: an agent checks for drift on a cadence and reports what changed. Channel feedback: when a user reports a wrong answer in Slack or Teams, it is verified and turned into a tracked issue. All four end in a reviewed change with an owner.\n\n### [Should an AI agent edit its own context automatically?](#should-an-ai-agent-edit-its-own-context-automatically)\n\nNo. The agent can detect drift and draft a change, but it should never silently edit context. It proposes a pull request or an issue in the same place you review code, and a person who owns the definition approves it. A value observed in the warehouse is not automatically a valid business value.\n\n### [Why not just use the AI agent's memory for context?](#why-not-just-use-the-ai-agents-memory-for-context)\n\nAgent memory is private to one session or one person, unversioned, and easy to lose with a new session or a different tool. Nobody reviewed it, and you cannot tell when it was learned or whether it is still true. Context written into the repository is reviewed, versioned, and read by every engineer and every agent.\n\n### [Can Bruin run a scheduled context audit?](#can-bruin-run-a-scheduled-context-audit)\n\nBruin Cloud has scheduled agents: an existing AI agent given a task that runs on a recurring schedule, using that agent's connections, integrations, and CLI access, and posting results to Slack, Teams, WhatsApp, or the Bruin Cloud chat. The audit logic itself is yours to write as the agent's instructions; Bruin does not decide which context changes to make.\n\n### [What should a context audit check?](#what-should-a-context-audit-check)\n\nMechanical drift first: tables or columns in the warehouse with no asset definition, values present in the data but missing from `accepted_values` checks, owners who left, and descriptions that contradict the query. Then flag, but do not fix, anything that changes business meaning, such as a metric that may now include refunds. Those go to the metric owner.\n\n## [Related Reading](#related-reading)\n\n- [How to Onboard an AI Data Agent Like a New Hire](https://getbruin.com/blog/onboard-ai-data-agent-like-a-new-hire/) - the four phases that come before these loops.\n- [The Best Context Is No Context](https://getbruin.com/blog/best-context-is-no-context/) - what to write down, what to leave out, and why one fact needs one home.\n- [How to Run Data Pipelines in CI/CD](https://getbruin.com/blog/data-pipelines-ci-cd-github-actions/) - the CI setup this loop builds on.\n- [What Is a Self-Healing Data Pipeline?](https://getbruin.com/blog/what-is-a-self-healing-data-pipeline/) - the same write-it-back loop, applied to incidents.", "url": "https://wpnews.pro/news/how-to-keep-an-ai-context-layer-from-going-stale", "canonical_source": "https://getbruin.com/blog/keep-ai-context-layer-current/", "published_at": "2026-10-05 00:00:00+00:00", "updated_at": "2026-10-05 16:16:56.287192+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "mlops"], "entities": ["Build Lab", "Bruin", "GitHub Actions"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/how-to-keep-an-ai-context-layer-from-going-stale", "markdown": "https://wpnews.pro/news/how-to-keep-an-ai-context-layer-from-going-stale.md", "text": "https://wpnews.pro/news/how-to-keep-an-ai-context-layer-from-going-stale.txt", "jsonld": "https://wpnews.pro/news/how-to-keep-an-ai-context-layer-from-going-stale.jsonld"}}