{"slug": "config-drift-hits-shared-ai-workbenches-too-an-illustrative-audit-plan-for-model", "title": "Config Drift Hits Shared AI Workbenches Too: An Illustrative Audit Plan for Model Aliases, MCP Tools, and Access Scopes", "summary": "A developer outlined an illustrative, unexecuted audit plan for detecting configuration drift in shared AI workbenches, covering model alias resolution, MCP tool reachability, and role access scopes. The method exports one JSON manifest per workspace and mechanically diffs the model aliases, MCP tool fingerprints, and role scopes, flagging that self-hosting does not remove external data paths to model providers or MCP servers. The writeup also notes the MonkeyCode repository's main README documents MonkeyAI while a separate file covers the earlier MonkeyCode platform, and discloses it was prepared as product outreach.", "body_md": "Most teams can describe their AI tooling in a paragraph and cannot answer three questions about it on the same afternoon. Which provider model does the `fast` alias actually resolve to in each workspace? Which MCP tools are reachable by a given role right now? Which of those tools sends data to a third party that your team never formally approved? A wiki page answers these the week it is written, and then quietly stops being true.\n\nThis post outlines an illustrative audit plan for that gap. Nothing below was executed against a live deployment, and no results are claimed. The goal is a repeatable method you can adapt to whatever your own workbench actually exposes.\n\nThe official repository at [github.com/chaitin/MonkeyCode](https://github.com/chaitin/MonkeyCode) needs a small naming clarification before it is useful. Its current main README describes **MonkeyAI**, an open-source team AI workbench covering office, coding, and design work, with shared team resources, configurable models, MCP integrations, access management, and self-hosting. A separate `README.monkeycode.md` documents the earlier **MonkeyCode** AI development platform.\n\nThose are two different documents describing two different things, and treating them as interchangeable is the fastest way to write a runbook that describes features nobody has. Read the file that matches the version you deployed, and quote it in your own documentation rather than paraphrasing from memory.\n\nDisclosure: This article was prepared as part of MonkeyCode's product outreach.\n\nOne caution that does not depend on which version you run: self-hosting moves the orchestration, not the network boundary. If a configured model provider or MCP server is external, that provider can still receive the data your workspace sends it. Confirm the data path with each provider's own documentation instead of assuming the deployment model settles it.\n\nBefore any script, decide what one auditable record looks like. A workable manifest shape for a single workspace is small enough to review by eye and specific enough to diff:\n\nThat last field is the one teams skip. It is also the only field that lets the next person tell a deliberate change apart from an accident.\n\nThe method is deliberately boring: export or hand-transcribe one manifest per workspace, then compare them mechanically. Transcribing by hand is acceptable for a first pass because it forces someone to read the values, but record the source on every line.\n\nThe following script is **illustrative and was not executed**. It assumes you have produced two JSON files in the shape described above, and it only compares the sections it knows about.\n\n``` python\n# illustrative only: not executed, not tested against any deployment\nimport hashlib\nimport json\nimport sys\nfrom pathlib import Path\n\ndef load(path):\n    return json.loads(Path(path).read_text(encoding=\"utf-8\"))\n\ndef fingerprint(tools):\n    # sort first: MCP servers may list tools in a different order per export\n    joined = \"\\n\".join(sorted(tools))\n    return hashlib.sha256(joined.encode(\"utf-8\")).hexdigest()[:12]\n\ndef summarize(manifest):\n    models = {a[\"alias\"]: a[\"target\"] for a in manifest.get(\"model_aliases\", [])}\n    mcp = {s[\"name\"]: fingerprint(s.get(\"tools\", [])) for s in manifest.get(\"mcp_servers\", [])}\n    roles = {r[\"name\"]: sorted(r.get(\"scopes\", [])) for r in manifest.get(\"roles\", [])}\n    return {\"models\": models, \"mcp\": mcp, \"roles\": roles}\n\ndef diff(left, right):\n    rows = []\n    for section in (\"models\", \"mcp\", \"roles\"):\n        for key in sorted(set(left[section]) | set(right[section])):\n            a, b = left[section].get(key), right[section].get(key)\n            if a != b:\n                rows.append((section, key, a, b))\n    return rows\n\nif __name__ == \"__main__\":\n    a, b = summarize(load(sys.argv[1])), summarize(load(sys.argv[2]))\n    rows = diff(a, b)\n    if not rows:\n        print(\"no drift in compared sections\")\n    for section, key, left, right in rows:\n        print(f\"{section}\\t{key}\\t{left!r}\\t{right!r}\")\n    sys.exit(1 if rows else 0)\n# illustrative invocation; your manifest paths and key names will differ\npython audit_manifests.py workspace_a.json workspace_b.json\n```\n\nThe exit code matters more than the printed table. A non-zero exit lets you wire the same check into a scheduled job later, and a fingerprint keeps the MCP tool list comparable even when the export order changes.\n\nFull tool lists in a diff are noisy and reviewers stop reading them. A short hash tells you *that* the surface changed, and the manifest itself tells you *how*. Pair every hash change with a mandatory human note explaining which tool was added or removed and who approved it.\n\nReading a permission table is not verification. The check that actually tells you something is a request that should fail, performed from a role that should not be able to make it. Write the procedure down before you run it, including the exact identity, the exact tool, and the response you expect.\n\nStep 4 is where most audits dissolve. If you write the expectation after seeing the result, you have created a description, not a test.\n\nAn MCP integration is a capability grant, and capability grants deserve a row each. The table below is a recording template, not a finding.\n\n| Field | Why it earns a column | \n|---|---|\n| Server name and transport | Distinguishes a local process from a remote endpoint | \n| Owner and review date | Gives the next reviewer a person and a deadline | \n| Tool list fingerprint | Detects silent surface changes between exports | \n| Data categories touched | Forces the egress question into the review | \n| Roles allowed to invoke | Connects the grant to your access model | \n| Disable procedure | Proves you can revoke before you need to | \n\nFill the last row honestly. A capability you cannot turn off during an incident is a capability you have not finished evaluating.\n\nThis approach earns its cost in some situations and wastes time in others.\n\nThe manifest is only as good as its source of truth. If you transcribe by hand, you have created a second copy that can drift from the deployment, which is the original problem wearing a new hat. Automate the export before you trust the comparison.\n\nThe diff compares sections you defined. A new category of configuration, such as a newly added integration type, appears in neither manifest and therefore never shows up as drift. Review the manifest shape itself on a schedule, not only its contents.\n\nFinally, model aliases resolve through providers you do not control. A provider deprecating a model identifier can change what your alias means without any edit inside your workspace, so treat provider announcements as an input to this process rather than an exception to it.\n\nIf you take one step from this post, take the manifest shape and the provenance field, and fill them in for a single workspace by hand. If you want the surrounding product context and the current capability list, start from the [official repository](https://github.com/chaitin/MonkeyCode) and read the README that matches the version you actually run.", "url": "https://wpnews.pro/news/config-drift-hits-shared-ai-workbenches-too-an-illustrative-audit-plan-for-model", "canonical_source": "https://dev.to/codepy_1473/config-drift-hits-shared-ai-workbenches-too-an-illustrative-audit-plan-for-model-aliases-mcp-haj", "published_at": "2026-11-10 04:09:28+00:00", "updated_at": "2026-11-10 04:20:58.465674+00:00", "lang": "en", "topics": ["ai-agents", "agent-protocols", "ai-tools", "mlops"], "entities": ["MonkeyCode", "MonkeyAI", "Chaitin", "MCP"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/config-drift-hits-shared-ai-workbenches-too-an-illustrative-audit-plan-for-model", "markdown": "https://wpnews.pro/news/config-drift-hits-shared-ai-workbenches-too-an-illustrative-audit-plan-for-model.md", "text": "https://wpnews.pro/news/config-drift-hits-shared-ai-workbenches-too-an-illustrative-audit-plan-for-model.txt", "jsonld": "https://wpnews.pro/news/config-drift-hits-shared-ai-workbenches-too-an-illustrative-audit-plan-for-model.jsonld"}}