{"slug": "letting-an-llm-operate-vs-code-without-letting-it-run-whatever-it-wants", "title": "Letting an LLM operate VS Code without letting it run whatever it wants", "summary": "The maintainer of Slashi, a VS Code extension, built a constrained agent that lets an LLM execute editor actions without giving it a shell. The model selects only from a pre-ranked shortlist of catalog items and settings, and every step is validated against known IDs and keys before execution, with the plan shown to the user first. In a test run, Claude Opus 5.5 correctly returned two setting steps (editor.fontSize to 16 and editor.wordWrap to on) from an 87-entry shortlist.", "body_md": "*The popup is the real Slashi webview. The progress notification and the confirmation dialog are VS Code's own UI, recreated from a real run whose output appears below.*\n\nIn [Slashi](https://github.com/meness/slashi), the VS Code extension I maintain, you can type `/do make the font bigger and turn on word wrap` and it happens. That's a small agent: a model reads a request and changes your editor.\n\nThe usual way to build this is to give the model tools, often including a shell, and let it decide. I didn't want a model with a shell sitting one keystroke away in every editor. So `/do` works differently: the model can only choose from a list of actions Slashi already knows, every choice is checked against that list, and you see the plan before it runs.\n\nCode is from Slashi 0.6.0, October 2026.\n\nSlashi already has a catalog of a few hundred things it can do: VS Code commands with slash names, settings, project scripts, Git actions, file actions. Sending all of it to the model on every request would be slow and noisy, so the request is first run through the same fuzzy ranker the popup uses, in a mode where *any* word may match instead of all of them:\n\n``` js\nconst query = parseQuery(task, []);\nconst ranked = rank(indexItems(all), query, { limit: 60, mode: 'any' });\n```\n\nSettings that make the shortlist are added again by their exact key, with their type or allowed values, so the model can set a precise value. For \"make the font bigger and turn on word wrap\", the shortlist in my test run (against the catalog Slashi's smoke test captures) had 87 entries, including `editor.fontSize (number)` and `editor.wordWrap (one of off | on | wordWrapColumn | bounded)`.\n\nThe system prompt describes exactly three kinds of step, and a few rules:\n\n```\n{\"reply\": \"one short sentence describing what you will do\",\n \"steps\": [\n {\"type\": \"item\", \"id\": \"\", \"args\": \"\", \"why\": \"\"},\n {\"type\": \"setting\", \"key\": \"\", \"value\": , \"why\": \"\"},\n {\"type\": \"terminal\", \"command\": \"\", \"why\": \"\"}\n ]}\nRules:\n- Prefer catalog items over terminal commands. Use a terminal step only when no catalog item fits.\n- Never invent ids or setting keys. If nothing fits, return an empty steps array and explain in reply.\n- Keep plans minimal: usually 1–3 steps, in execution order.\n```\n\nThe user message is the shortlist (`id | kind | title`) followed by the request, plus a line of context: the OS, the current file's language, whether text is selected, and the detected project stack.\n\nTo check this post against reality, I ran the exact prompt (built with Slashi's own `buildIntentPrompt` from the 87-entry shortlist) through Claude Opus 5.5. The raw response:\n\n```\n{\"reply\": \"I'll increase the editor font size to 16 and turn on word wrap.\",\n \"steps\": [\n {\"type\": \"setting\", \"key\": \"editor.fontSize\", \"value\": 16, \"why\": \"bigger font\"},\n {\"type\": \"setting\", \"key\": \"editor.wordWrap\", \"value\": \"on\", \"why\": \"enable word wrap\"}\n ]}\n```\n\nTwo settings, real keys, sensible values. Most requests look like this.\n\nThe model's output is parsed (the parser pulls the first JSON object out of whatever surrounds it, since models sometimes add a sentence or a code fence) and then checked:\n\n```\n/** Drops steps that reference unknown items or settings so the model can't invent actions. */\nexport function validatePlan(plan: IntentPlan, knownItemIds: ReadonlySet, knownSettingKeys: ReadonlySet): ValidatedPlan {\n const steps: PlanStep[] = [];\n const dropped: string[] = [];\n for (const step of plan.steps) {\n if (step.type === 'item' && !knownItemIds.has(step.id)) dropped.push(`unknown action “${step.id}”`);\n else if (step.type === 'setting' && !knownSettingKeys.has(step.key)) dropped.push(`unknown setting “${step.key}”`);\n else steps.push(step);\n }\n return { reply: plan.reply, steps: steps.slice(0, 10), dropped };\n}\n```\n\n(From [`src/core/intent.ts`](https://github.com/meness/slashi/blob/main/src/core/intent.ts).) A step naming an action or setting Slashi doesn't have is dropped and logged, not guessed at. A plan longer than 10 steps is cut off. If nothing valid is left, you get the model's reply as a message and nothing runs.\n\nBy default every plan is shown before it runs: the model's one-sentence reply, then each step in plain words with the model's \"why\":\n\n```\nI'll increase the editor font size to 16 and turn on word wrap.\n1. Set Editor: Font Size = 16 (bigger font)\n2. Set Editor: Word Wrap = \"on\" (enable word wrap)\n [Cancel] [Run Plan]\n```\n\nYou can turn that confirmation off (`slashi.ai.confirmBeforeRun`) for plans that only flip settings or run editor commands. You can't turn it off for steps that can reach a shell. Those are:\n\n`terminal.`, `task.`, `security.`, `http.`, `chat.tools.` or Slashi's own settings, and any key ending in `path`, `executable`, `command`, `shell` or `tsdk`.\nThat last group is the one people forget. \"Set `python.defaultInterpreterPath` to `/tmp/x`\" is a settings change, and it decides which binary runs next time you run Python.\n\nThe second request I tried was \"delete node_modules, reinstall and run the tests\". The same model returned:\n\n```\n{\"reply\": \"I'll delete node_modules, reinstall from the lockfile with npm ci, then run the Vitest unit tests in one terminal command so each step waits for the one before it.\",\n \"steps\": [{\"type\": \"item\", \"id\": \"term:run\", \"args\": \"rm -rf node_modules && npm ci && npm test\", \"why\": \"clean reinstall, then run tests\"}]}\n```\n\nThat's a reasonable answer to what I asked, and it contains `rm -rf`. The prompt tells the model to avoid destructive commands *unless explicitly requested*, and I did request it. `term:run` reaches a shell, so this plan is always shown, and the command appears in full before anything runs. Rules in a prompt shape what a model tends to do; only the check in code decides what can run without you.\n\n`/do` in one project changes all of them.\nIf you're putting a model in charge of a tool, a catalog plus validation plus a confirmation that can't be turned off for dangerous steps gets you most of what an agent with a shell does, for the everyday requests people actually type, without letting the model run something you didn't see.\n\n*I'm the author of Slashi. It's free and MIT-licensed: [Marketplace](https://marketplace.visualstudio.com/items?itemName=meness.slashi) · [Open VSX](https://open-vsx.org/extension/meness/slashi) · [GitHub](https://github.com/meness/slashi).*", "url": "https://wpnews.pro/news/letting-an-llm-operate-vs-code-without-letting-it-run-whatever-it-wants", "canonical_source": "https://dev.to/meness/letting-an-llm-operate-vs-code-without-letting-it-run-whatever-it-wants-5b0m", "published_at": "2026-10-22 22:07:37+00:00", "updated_at": "2026-10-22 22:11:32.894559+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "large-language-models", "ai-safety"], "entities": ["Slashi", "VS Code", "Claude Opus 5.5", "GitHub"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/letting-an-llm-operate-vs-code-without-letting-it-run-whatever-it-wants", "markdown": "https://wpnews.pro/news/letting-an-llm-operate-vs-code-without-letting-it-run-whatever-it-wants.md", "text": "https://wpnews.pro/news/letting-an-llm-operate-vs-code-without-letting-it-run-whatever-it-wants.txt", "jsonld": "https://wpnews.pro/news/letting-an-llm-operate-vs-code-without-letting-it-run-whatever-it-wants.jsonld"}}