Give Muse Code an On-Call MCP Toolkit with Kong AI Gateway A developer built an on-call MCP toolkit that lets Meta's Muse Code terminal coding agent perform the first ten minutes of an incident investigation while enforcing per-identity tool access at Kong AI Gateway 2.0. The gateway converts an existing REST ops API into MCP tools and filters tools/list per identity, so an oncall-investigator identity never sees the rollback-deployment tool while an oncall-operator does — a boundary the developer notes Muse's own sandbox and approval settings do not apply to MCP tool calls. I wanted to find out whether a coding agent could do the first ten minutes of an incident investigation. Not fix anything. Just the part where you open four tabs, line up a deploy timeline against an error timeline, and work out which change to blame. Muse Code , Meta's terminal coding agent, connects to remote MCP servers over streamable http . So the tools are easy. The part that stopped me is in Meta's own documentation: MCP tools operate outside the sandbox and approval mechanisms. They run as unrestricted child processes or network connections. That matters more than it sounds. You can set shell execute and file write to ask in your Muse settings and feel covered. Those settings do not put a prompt in front of an MCP tool call. If one of your MCP tools can roll back a production deployment, the agent can roll back a production deployment, and nothing in the client will stop it. So the question was never "can the agent investigate". It was "where does the boundary go". I put it in front of the tools, at Kong AI Gateway , and gave the same agent two identities to prove it holds. Everything below runs. The gateway config, the mock ops API, the verification script, and the real agent transcripts are all here: github.com/tejakummarikuntla/kong-muse-oncall-toolkit https://github.com/tejakummarikuntla/kong-muse-oncall-toolkit I looked at writing a bespoke MCP server that checks a role on every handler. That works, and it means every permission decision lives in code I have to test, and the tool list is the same for everyone regardless of who is calling. I looked at just putting the tools behind an API key. That is all or nothing. Anyone with the key gets every tool. I went with Kong AI Gateway 2.0 because of one specific behavior: it converts an existing REST API into MCP tools, and it applies per-tool access control lists per identity, so tools/list itself is filtered. An investigator identity does not see a rollback tool. It cannot propose calling something it was never shown. That is a different property from "the call gets rejected", and I wanted both. Kong AI Gateway sits between the agent and the operational APIs. It authenticates the caller, decides per tool whether that identity may use it, converts allowed calls into ordinary HTTP requests against the REST API, and writes an audit entry for every decision. Two identities: | Tool | oncall-investigator | oncall-operator | |---|---|---| | get-service-health | visible | visible | | list-deployments | visible | visible | | get-error-summary | visible | visible | | get-error-samples | visible | visible | | search-runbooks | visible | visible | | get-runbook | visible | visible | | create-incident | visible | visible | | rollback-deployment | not visible | visible | Note that this is not a read versus write split. The investigator can write. It opens the incident. What it cannot do is change production. That distinction is the whole point, and a plain read-only key gets it wrong. The ops API is an ordinary REST service. Deployments, error summaries, error samples, runbooks, incidents. Kong's conversion-listener maps each endpoint to an MCP tool, so I did not write an MCP server at all. ai gateway mcp servers: - ref: oncall-mcp ai gateway: lookup {id: env AI GATEWAY ID} name: oncall-mcp display name: "On-call toolkit" type: conversion-listener enabled: true config: url: http://host.docker.internal:9110/v1 route: paths: - /ops-mcp tools: - name: get-error-summary description: - Error counts for a service in five-minute buckets, broken down by error type. Use this to find the exact onset time and to see which error type is actually growing rather than which is merely loudest. method: GET path: /ops-mcp/errors/summary annotations: read only hint: true idempotent hint: true parameters: - name: service in: query required: false schema: type: string description: Service name. Defaults to "checkout". - name: minutes in: query required: false schema: type: integer description: Window length in minutes. Defaults to 60. Two things here are easy to get wrong. The tool path must include the route prefix. Route /ops-mcp plus url: .../v1 plus tool path /ops-mcp/errors/summary resolves to http://host:9110/v1/errors/summary . If you write path: /errors/summary you get a 404. The description is not decoration. It is the only thing the agent reads when deciding which tool to reach for. I wrote each one as an instruction to a colleague, including when to use it, and the quality of the investigation moved noticeably. ai gateway auth strategies: - ref: oncall-key-auth ai gateway: lookup {id: env AI GATEWAY ID} name: oncall-key-auth display name: "On-call Key Auth" type: key-auth config: key names: - apikey key in header: true hide credentials: true ai gateway consumers: - ref: oncall-investigator ai gateway: lookup {id: env AI GATEWAY ID} name: oncall-investigator display name: "On-call Investigator" type: api-key credentials: - ref: oncall-investigator-key ai gateway consumer: ref oncall-investigator id name: oncall-investigator-key display name: "On-call Investigator Key" type: api-key api key: secret {source: env INVESTIGATOR KEY} ai gateway consumer groups: - ref: incident-response ai gateway: lookup {id: env AI GATEWAY ID} name: incident-response display name: "Incident Response" consumers: - ref oncall-investigator name The operator is the same shape, in a group called sre-oncall . Write-only fields like api key need secret , not a plain string. A baseline on the server, then an override on the one tool that changes production. access: acl attribute type: consumer auth strategies: - ref oncall-key-auth name default tool acls: allow: - incident-response - sre-oncall - name: rollback-deployment description: - Roll a service back to the deployment that preceded the one named. This changes production. Only run it when a runbook calls for it and an incident is already open. method: POST path: /ops-mcp/deployments/{deployment id}/rollback annotations: destructive hint: true idempotent hint: false access: acls: allow: - sre-oncall parameters: - name: deployment id in: path required: true schema: type: string description: The deployment to roll back. It must currently be live. acl attribute type is required whenever you set access . Leave it out and the apply is rejected. A per-tool ACL fully replaces default tool acls rather than merging with it. rollback-deployment lists only sre-oncall , so incident-response loses it. On create-incident I had to restate both groups for exactly this reason. Apply it: kongctl apply -f oncall-gateway.yaml --region in Use apply , not sync . apply only creates and updates. sync reconciles every resource type named in the file, which means it deletes entities of those types that are not in your file. config: logging: audits: true payloads: true ai gateway policies: - ref: oncall-audit-log ai gateway: lookup {id: env AI GATEWAY ID} name: oncall-audit-log display name: "On-call audit log" type: http-log enabled: true global: false config: http endpoint: http://host.docker.internal:9111 method: POST I pointed http-log at a small local sink that prints the interesting fields. Every line below came from Kong. Muse Code reads ~/.config/muse/settings.json : { "schema version": 1, "provider": "meta", "model": "muse-spark-1.3-contributor", "mcp servers": { "oncall": { "transport": "streamable http", "url": "http://localhost:8000/ops-mcp", "headers": { "apikey": "inv your investigator key" }, "enabled": true, "mode": "required" } } } Then the prompt an on-call engineer would actually type: muse exec "Checkout errors increased after the last deployment. Investigate why, using the on-call tools available to you. Show the evidence for your conclusion, then tell me what should be done about it." Nothing in that prompt names a tool, a service version, or a cause. Here is what came back, unedited: Conclusion: the last checkout deployment dep-482 caused the spike. Evidence: - checkout is degraded since 2026-09-24T16:58:46Z : error rate 8.4% vs SLO 0.5%, p99 4180ms vs baseline 310ms, live version 2026.9.24-3 . - Last deployment is dep-482 at 16:55:46Z — 2026.9.24-3 , "checkout: reuse pooled connections for payments client" — ~3 min before degradation started. - Error breakdown: PaymentProviderTimeout was 0/5min before, then 125 in 16:56:46-17:01:46Z , rising to sustained 235/5min 1500 total . Other types flat: CardDeclined ~55-60, CartValidationError 15, SessionExpired 10. - Samples on 2026.9.24-3 : "payments client: pool exhausted, waited 3000ms for a free connection", pool max size: 8, pool active: 8, pool idle: 0 , stack PaymentsClient.acquire payments/pool.js:114 . - Dependency ruled out: payments-provider-api healthy, error 0.1% vs SLO 1.0%, p99 288ms vs baseline 275ms. - Runbook rb-204 "Payment provider timeouts in checkout": timeouts here almost always mean pool saturation, not provider down; if a deploy landed within ~10 min of onset, treat it as cause. Incident opened: INC-1001 sev2 . What should be done, per rb-204 : 1. Roll back checkout from dep-482 to dep-481 first, debug afterwards. Rollback is safe/idempotent. 2. Do not hot-patch pool sizing under load — a saturated pool does not recover on its own while traffic continues. 3. Do not page payments-oncall / fail open to queued-authorization — that path is only if provider latency is elevated, and it is not. The step I care about most is "dependency ruled out". The runbook says to check the provider before blaming it, and the agent checked the provider before blaming it. It also declined the escalation path on the grounds that its precondition was not met. And here is Kong's record of that same run: time consumer tool result 17:31:47 oncall-investigator initialize HTTP 200 6ms 17:31:47 oncall-investigator tool list ALLOW HTTP 200 group incident-response 17:31:52 oncall-investigator get-error-summary ALLOW HTTP 200 group incident-response 17:31:52 oncall-investigator list-deployments ALLOW HTTP 200 group incident-response 17:31:52 oncall-investigator get-service-health ALLOW HTTP 200 group incident-response 17:31:57 oncall-investigator get-service-health ALLOW HTTP 200 group incident-response 17:31:57 oncall-investigator get-error-samples ALLOW HTTP 200 group incident-response 17:31:57 oncall-investigator search-runbooks ALLOW HTTP 200 group incident-response 17:32:05 oncall-investigator get-runbook ALLOW HTTP 200 group incident-response 17:32:23 oncall-investigator create-incident ALLOW HTTP 200 group incident-response Ten tool calls, every one allowed, and no rollback-deployment anywhere. The agent recommended a rollback in prose and never attempted the call, because Kong never showed it the tool. Now the same binary, same prompt style, operator key: muse exec "Incident INC-1001 is open for the checkout service. Read the incident and runbook rb-204, confirm from the evidence whether a rollback is justified, and if it is, perform the rollback and report the result." php Rollback justified and completed. Action: rolled back dep-482 - dep-481 version 2026.9.24-2 under INC-1001 . Result: status: completed , rollback id: rb-912229ed , restored: dep-481 , completed 2026-09-24T17:33:55Z. Checkout health now recovering . 17:33:26 oncall-operator tool list ALLOW HTTP 200 group sre-oncall 17:33:39 oncall-operator get-service-health ALLOW HTTP 200 group sre-oncall 17:33:39 oncall-operator get-runbook ALLOW HTTP 200 group sre-oncall 17:33:39 oncall-operator list-deployments ALLOW HTTP 200 group sre-oncall 17:33:42 oncall-operator get-error-samples ALLOW HTTP 200 group sre-oncall 17:33:55 oncall-operator rollback-deployment ALLOW HTTP 200 group sre-oncall 17:33:57 oncall-operator get-service-health ALLOW HTTP 200 group sre-oncall Same endpoint, same agent, different key. The eighth tool appears and works. Tool filtering stops a cooperative agent from trying. It is not a security boundary on its own, because a client can call a tool name it was never offered. So I wrote a script that speaks the MCP handshake directly and deliberately calls rollback-deployment with the investigator key. php anonymous no credential PASS anonymous - 401 investigator key in group incident-response PASS tools/list shows 7 tools PASS rollback-deployment absent from tools/list PASS get-service-health - degraded PASS create-incident - allowed | incident=INC-1001 PASS rollback-deployment - 403 operator key in group sre-oncall PASS tools/list shows 8 tools PASS rollback-deployment - completed | restored=dep-481 audit Kong's http-log output PASS exactly one deny recorded PASS deny names the caller | oncall-investigator identifier=username PASS denied call never reached the ops API | upstream status='' proxy latency=-1 PASS same tool allowed for sre-oncall 16/16 passed The denial record is worth looking at directly: "ai": { "mcp": { "audit": { "primitive name": "rollback-deployment", "primitive": "tool", "consumer": { "name": "oncall-investigator", "identifier": "username" }, "action": "deny" } } }, "upstream status": "", "latencies": { "proxy": -1 } upstream status is empty and latencies.proxy is -1. Kong never opened a connection to the ops API. The refusal happened at the gateway, and the ops API has no record that anyone tried. That is two layers doing two different jobs. Filtering keeps the agent from forming the intent. The ACL handles the client that forms it anyway. Environment variables do not interpolate into MCP headers. The Muse docs describe ${VAR} interpolation. In Muse Code 1.3.0 it did not apply to MCP header values. My config sent the literal string ${INVESTIGATOR KEY} , Kong returned 401, and Muse reported this: Required MCP server oncall failed during startup: it requires an OAuth sign-in; run muse mcp login oncall and restart. There is no OAuth anywhere in this setup. Muse turns any 401 into an OAuth prompt, which sends you a long way in the wrong direction. Putting the literal key in the file connected immediately. If you hit that error against a key-auth server, check the header value before you touch OAuth. There is no --settings flag. Muse reads ~/.config/muse/settings.json and nothing else. To run two identities without overwriting my real config, I used XDG CONFIG HOME , which it does respect: XDG CONFIG HOME=/tmp/muse-investigator muse exec "..." Copy auth.json and trust.json into that directory alongside your settings.json or the run will not authenticate. Test the wiring with the echo provider. muse exec --provider echo "ping" starts the session and connects every MCP server with mode: required , but never calls the model. Every configuration mistake above was found this way, at zero cost. Request bodies collapse into one argument. Path and query parameters get prefixed, so deployment id in a path becomes path deployment id and service in a query becomes query service . A request body does not get expanded into named arguments at all. It arrives as a single body object. Check tools/list before you assume an argument name. The request body shape is not in the docs. The reference says tools accept request body in OpenAPI JSON format and gives no example. The OpenAPI 3 requestBody object works: request body: required: true content: application/json: schema: type: object required: title properties: title: type: string hide credentials does not apply to the log. I had hide credentials: true on the auth strategy, which strips the key from the request Kong forwards upstream. The http-log payload still contains the headers the client sent, raw key included. I found 26 copies of a working API key sitting in my log file. If you forward http-log to a hosted log service, that is where your keys end up. Scrub the sensitive headers at the sink, or before the sink: SENSITIVE HEADERS = "apikey", "authorization", "x-api-key", "cookie" def redact entry : for section in "request", "response" : headers = entry.get section or {} .get "headers" if isinstance headers, dict : for h in list headers : if h.lower in SENSITIVE HEADERS: headers h = "