Config Drift Hits Shared AI Workbenches Too: An Illustrative Audit Plan for Model Aliases, MCP Tools, and Access Scopes

A developer outlined an illustrative, unexecuted audit plan for detecting configuration drift in shared AI workbenches, covering model alias resolution, MCP tool reachability, and role access scopes. The method exports one JSON manifest per workspace and mechanically diffs the model aliases, MCP tool fingerprints, and role scopes, flagging that self-hosting does not remove external data paths to model providers or MCP servers. The writeup also notes the MonkeyCode repository's main README documents MonkeyAI while a separate file covers the earlier MonkeyCode platform, and discloses it was prepared as product outreach.

Most teams can describe their AI tooling in a paragraph and cannot answer three questions about it on the same afternoon. Which provider model does the fast alias actually resolve to in each workspace? Which MCP tools are reachable by a given role right now? Which of those tools sends data to a third party that your team never formally approved? A wiki page answers these the week it is written, and then quietly stops being true. This post outlines an illustrative audit plan for that gap. Nothing below was executed against a live deployment, and no results are claimed. The goal is a repeatable method you can adapt to whatever your own workbench actually exposes. The official repository at github.com/chaitin/MonkeyCode https://github.com/chaitin/MonkeyCode needs a small naming clarification before it is useful. Its current main README describes MonkeyAI , an open-source team AI workbench covering office, coding, and design work, with shared team resources, configurable models, MCP integrations, access management, and self-hosting. A separate README.monkeycode.md documents the earlier MonkeyCode AI development platform. Those are two different documents describing two different things, and treating them as interchangeable is the fastest way to write a runbook that describes features nobody has. Read the file that matches the version you deployed, and quote it in your own documentation rather than paraphrasing from memory. Disclosure: This article was prepared as part of MonkeyCode's product outreach. One caution that does not depend on which version you run: self-hosting moves the orchestration, not the network boundary. If a configured model provider or MCP server is external, that provider can still receive the data your workspace sends it. Confirm the data path with each provider's own documentation instead of assuming the deployment model settles it. Before any script, decide what one auditable record looks like. A workable manifest shape for a single workspace is small enough to review by eye and specific enough to diff: That last field is the one teams skip. It is also the only field that lets the next person tell a deliberate change apart from an accident. The method is deliberately boring: export or hand-transcribe one manifest per workspace, then compare them mechanically. Transcribing by hand is acceptable for a first pass because it forces someone to read the values, but record the source on every line. The following script is illustrative and was not executed . It assumes you have produced two JSON files in the shape described above, and it only compares the sections it knows about. python illustrative only: not executed, not tested against any deployment import hashlib import json import sys from pathlib import Path def load path : return json.loads Path path .read text encoding="utf-8" def fingerprint tools : sort first: MCP servers may list tools in a different order per export joined = "\n".join sorted tools return hashlib.sha256 joined.encode "utf-8" .hexdigest :12 def summarize manifest : models = {a "alias" : a "target" for a in manifest.get "model aliases", } mcp = {s "name" : fingerprint s.get "tools", for s in manifest.get "mcp servers", } roles = {r "name" : sorted r.get "scopes", for r in manifest.get "roles", } return {"models": models, "mcp": mcp, "roles": roles} def diff left, right : rows = for section in "models", "mcp", "roles" : for key in sorted set left section | set right section : a, b = left section .get key , right section .get key if a = b: rows.append section, key, a, b return rows if name == " main ": a, b = summarize load sys.argv 1 , summarize load sys.argv 2 rows = diff a, b if not rows: print "no drift in compared sections" for section, key, left, right in rows: print f"{section}\t{key}\t{left r}\t{right r}" sys.exit 1 if rows else 0 illustrative invocation; your manifest paths and key names will differ python audit manifests.py workspace a.json workspace b.json The exit code matters more than the printed table. A non-zero exit lets you wire the same check into a scheduled job later, and a fingerprint keeps the MCP tool list comparable even when the export order changes. Full tool lists in a diff are noisy and reviewers stop reading them. A short hash tells you that the surface changed, and the manifest itself tells you how . Pair every hash change with a mandatory human note explaining which tool was added or removed and who approved it. Reading a permission table is not verification. The check that actually tells you something is a request that should fail, performed from a role that should not be able to make it. Write the procedure down before you run it, including the exact identity, the exact tool, and the response you expect. Step 4 is where most audits dissolve. If you write the expectation after seeing the result, you have created a description, not a test. An MCP integration is a capability grant, and capability grants deserve a row each. The table below is a recording template, not a finding. | Field | Why it earns a column | |---|---| | Server name and transport | Distinguishes a local process from a remote endpoint | | Owner and review date | Gives the next reviewer a person and a deadline | | Tool list fingerprint | Detects silent surface changes between exports | | Data categories touched | Forces the egress question into the review | | Roles allowed to invoke | Connects the grant to your access model | | Disable procedure | Proves you can revoke before you need to | Fill the last row honestly. A capability you cannot turn off during an incident is a capability you have not finished evaluating. This approach earns its cost in some situations and wastes time in others. The manifest is only as good as its source of truth. If you transcribe by hand, you have created a second copy that can drift from the deployment, which is the original problem wearing a new hat. Automate the export before you trust the comparison. The diff compares sections you defined. A new category of configuration, such as a newly added integration type, appears in neither manifest and therefore never shows up as drift. Review the manifest shape itself on a schedule, not only its contents. Finally, model aliases resolve through providers you do not control. A provider deprecating a model identifier can change what your alias means without any edit inside your workspace, so treat provider announcements as an input to this process rather than an exception to it. If you take one step from this post, take the manifest shape and the provenance field, and fill them in for a single workspace by hand. If you want the surrounding product context and the current capability list, start from the official repository https://github.com/chaitin/MonkeyCode and read the README that matches the version you actually run.