{"slug": "didit-copilot-shipping-an-ai-agent-that-lives-inside-the-app", "title": "Didit Copilot: shipping an AI agent that lives inside the app", "summary": "Didit shipped Copilot, an AI agent embedded in its Business Console that lets users manage identity verification workflows via natural language. The agent operates the console in real time, using three tool families: MCP server tools for data, WebMCP browser tools for UI actions, and DOM fallback for unexposed interactions. The architecture relies on the open-source gui-agent library, which exposes page actions as structured tools via React hooks.", "body_md": "A few months ago I wrote that [AI agents shouldn't control your apps; they should be the app](https://aralroca.com/blog/ai-agents-should-be-the-app). That post was the thesis. This one is the proof.\n\nWe just shipped **Didit Copilot**: an AI agent embedded in the [Didit](https://didit.me) Business Console, the dashboard our customers use to manage identity verification (KYC/KYB, AML screening, biometrics, workflows). You open a chat panel, ask in natural language, and the agent operates the console **in front of you**: it queries your data, opens pages, applies filters, and builds verification workflows on the visual editor; node by node, with a glow over everything it touches.\n\nNo screenshots. No robot moving your mouse. The app itself exposes its actions to the agent, and the agent composes them.\n\nThe whole architecture boils down to a single decision: the LLM sees **three families of tools** and picks the right one for each step.\n\n** didit_* — MCP server tools.** Business data lives behind an MCP server that acts as the signed-in user: sessions, analytics, users, billing. These run on the backend with the user's own token, so the agent can never see more than the user can.\n\n** ui_* — WebMCP browser tools.** The console registers its own actions as structured tools on the page: open this session, apply these filters, add a workflow step on the canvas. These run\n\n**DOM fallback.** When nothing is exposed for the job, the agent reads a text snapshot of the page and clicks/fills by element ref — the same trick [page-agent](https://github.com/alibaba/page-agent) popularized, no multimodal model needed.\n\nThe routing rule the agent follows: prefer a purpose-built `ui_*`\n\ntool, fall back to the DOM for anything visible on the page, and go to the MCP server for data. Let's see each one working; every video below is a real recording against our console.\n\nAsk a data question and the agent chains MCP tools (`didit_context_get`\n\n→ `didit_analytics`\n\n→ `didit_session_search`\n\n), then streams the answer with real tables:\n\nNothing exotic here, this is what MCP was designed for. The interesting part is the other two families.\n\nThis is where [gui-agent](https://github.com/aralroca/gui-agent) comes in, an open-source library I built exactly for this pattern ([ @aralroca/gui-agent](https://www.npmjs.com/package/@aralroca/gui-agent) on npm). It implements the emerging\n\n`document.modelContext`\n\n, and any WebMCP agent, yours, or eventually the browser's native one, can call them.With the React bindings, exposing an action is one hook:\n\n``` js\nimport { useTool } from \"@aralroca/gui-agent/react\";\n\nfunction WorkflowEditor() {\n  // Registered while mounted; auto-unregistered on unmount.\n  useTool({\n    name: \"ui_create_workflow_step\",\n    description: \"Add a verification step to the workflow on the canvas\",\n    inputSchema: {\n      type: \"object\",\n      properties: {\n        feature: { type: \"string\", description: \"Step type, e.g. AML or PHONE_VERIFICATION\" },\n        after: { type: \"string\", description: \"Node id to insert after\" },\n      },\n      required: [\"feature\"],\n    },\n    execute: ({ feature, after }) => workflowStore.addStep(feature, after),\n  });\n  return /* … */;\n}\n```\n\nRegistration is scoped to the page the user is on: mount the component, the tool exists; navigate away, it's gone. The agent's tool list always mirrors what the user can actually do right now.\n\nHere's the agent building a workflow on the React Flow canvas, notice the subtle highlight ring over the node it creates (that's gui-agent's visualizer, themed to our console's own focus ring via CSS variables), and the **confirmation card** before the mutating tool runs:\n\nYou'll never cover every micro-interaction with first-class tools, and you shouldn't try. For the long tail, gui-agent synthesizes six DOM tools (`read_page`\n\n, `click`\n\n, `fill`\n\n, `select_option`\n\n, `wait_for_text`\n\n, `upload_file`\n\n). `read_page`\n\nreturns a **text outline** of the interactive elements with stable refs:\n\n```\n[e1] button \"Switch application\" expanded=false\n[e4] link \"Home\"\n[e5] link \"Users\"\n[e13] button \"Columns\" haspopup=menu\n```\n\nThe model reasons over that outline and acts by ref. Text in, text out, cheap, fast, and it works with any model:\n\nThe brain runs in a small Node service built on [Mastra](https://mastra.ai). The agent itself is a few lines, model via OpenRouter, conversation memory in Postgres:\n\n``` js\nimport { Agent } from \"@mastra/core/agent\";\nimport { Memory } from \"@mastra/memory\";\nimport { createOpenRouter } from \"@openrouter/ai-sdk-provider\";\n\nconst openrouter = createOpenRouter({ apiKey: process.env.OPENROUTER_API_KEY });\n\nexport const copilot = new Agent({\n  name: \"didit-copilot\",\n  instructions: COPILOT_INSTRUCTIONS, // the 3-family routing rules live here\n  model: openrouter(\"google/gemini-3.1-flash-lite\"),\n  memory: new Memory({ options: { lastMessages: 20 } }),\n});\n```\n\nThe MCP connection is **per request**, created with the calling user's Bearer token, that's what keeps the agent inside the user's permissions:\n\n``` js\nimport { MCPClient } from \"@mastra/mcp\";\n\nfunction diditMcpFor(accessToken: string) {\n  return new MCPClient({\n    servers: {\n      didit: {\n        url: new URL(\"https://mcp.didit.me/mcp\"),\n        requestInit: {\n          headers: { Authorization: `Bearer ${accessToken}` },\n        },\n      },\n    },\n  });\n}\n```\n\nAnd here's the trick that makes the browser tools work. Each turn, the console sends the JSON specs of whatever `ui_*`\n\ntools are registered on the current page. The server turns them into AI SDK tools **without an execute**:\n\n``` js\nimport { tool, jsonSchema } from \"ai\";\n\n// An execute-less tool is not run on the server — its call is forwarded\n// down the stream, the browser executes it via gui-agent's registry,\n// and useChat sends the output back so the model continues.\nclientTools[spec.name] = tool({\n  description: spec.description,\n  inputSchema: jsonSchema(spec.inputSchema),\n});\n```\n\nThe chat route hands both families to the agent and streams an AI SDK UI-message response, byte-compatible with `useChat`\n\non the frontend:\n\n``` js\nconst mcp = diditMcpFor(userToken);\n\nconst stream = await copilot.stream(messages, {\n  toolsets: await mcp.getToolsets(), // didit_* (server-executed)\n  clientTools,                       // ui_* + DOM (browser-executed)\n});\n```\n\nRound trip: model calls `ui_create_workflow_step`\n\n→ the call streams down → `onToolCall`\n\nin the browser executes it through gui-agent → the output goes back up → the model continues. The app is the runtime.\n\nMutating tools (`ui_create_*`\n\n, `ui_delete_*`\n\n, `upload_file`\n\n, anything `*_delete`\n\n/`*_publish`\n\n) require the user's explicit confirmation, that's the card you saw in the workflow video. One design decision I'd defend hard: **the gate is by tool name**, a boring regex both sides agree on. No model judgment involved in deciding what's dangerous.\n\nAnd a second lesson from recording these demos: we originally marked destructive client tools with the AI SDK's `needsApproval`\n\nflag on the server and trusted the framework to pause. It doesn't — approval gates run at execution time, and execute-less browser tools are never executed server-side, so the flag was silently ignored and mutating actions ran unconfirmed. We caught it on camera. The fix: the console now enforces the same by-name contract **client-side**, parking the call and showing the confirmation card before anything runs. If your agent executes tools in the browser, gate them in the browser, defense in depth is not optional when a framework sits between you and the pause button.\n\nSame argument as [last time](https://dev.toai-agents-should-be-the-app), now with production mileage:\n\n**1. Structured beats guessed.** A `ui_*`\n\ntool with a JSON schema succeeds or fails loudly. A bot guessing pixel coordinates degrades silently with every redesign.\n\n**2. The user sees everything.** The glow, the chips, the confirmation cards, the agent works in the same UI the user is looking at. Trust comes from visibility.\n\n**3. Permissions come free.** The MCP server acts as the signed-in user and the browser tools run in their session. There is no \"agent account\" with god-mode credentials.\n\n**4. The fallback covers the tail.** You expose your 30 core actions as WebMCP tools and let DOM inference handle the rest. You don't need 100% coverage on day one.\n\nIf you want to build this for your own product: register your app's actions with [gui-agent](https://github.com/aralroca/gui-agent) (or any WebMCP implementation), put your data behind an MCP server, and wire a Mastra agent between them with execute-less tools for the browser side. That's the whole recipe.\n\nThe browser is turning into an agent runtime. The question is no longer whether an AI will operate your app, it's whether it will operate it **through the front door you built for it**, or by rattling every window. I know which one I'm building for.", "url": "https://wpnews.pro/news/didit-copilot-shipping-an-ai-agent-that-lives-inside-the-app", "canonical_source": "https://dev.to/aralroca/didit-copilot-shipping-an-ai-agent-that-lives-inside-the-app-51b2", "published_at": "2026-07-09 19:05:08+00:00", "updated_at": "2026-07-09 19:35:34.411727+00:00", "lang": "en", "topics": ["ai-agents", "ai-products", "ai-tools", "developer-tools", "large-language-models"], "entities": ["Didit", "Didit Copilot", "gui-agent", "Aral Roca", "WebMCP", "MCP", "React"], "alternates": {"html": "https://wpnews.pro/news/didit-copilot-shipping-an-ai-agent-that-lives-inside-the-app", "markdown": "https://wpnews.pro/news/didit-copilot-shipping-an-ai-agent-that-lives-inside-the-app.md", "text": "https://wpnews.pro/news/didit-copilot-shipping-an-ai-agent-that-lives-inside-the-app.txt", "jsonld": "https://wpnews.pro/news/didit-copilot-shipping-an-ai-agent-that-lives-inside-the-app.jsonld"}}