{"slug": "beyond-clicking-and-shell-commands-api-native-computer-control", "title": "Beyond Clicking and Shell Commands: API-Native Computer Control", "summary": "An AI researcher proposes API-native computer control, where agents write short JavaScript programs against validated application APIs instead of relying solely on GUI automation or shell commands. This approach reduces inference steps by handling loops and conditions locally, and is most effective when applications already have structured data and domain rules.", "body_md": "[← All writing](/)\n\nEssay\n\n# Beyond Clicking and Shell Commands: API-Native Computer Control\n\nA practical exploration of letting AI agents write short JavaScript programs against narrow, validated application APIs.\n\nAn AI agent can draft an email, summarize a repository, or propose edits. The more difficult question is what happens next: how should it operate an application?\n\nGUI automation and shell access are two practical answers. I use both, but I have also been experimenting with another option: give the agent a small application API and let it write a short JavaScript program for each task.\n\nI call this **API-native computer control**. The name is more ambitious than the current implementation. I do not know whether it is the best general interface for agents, but it seems promising when an application already has structured data, domain rules, and a meaningful API.\n\n## When tool calls become the control loop\n\nTool calling often exposes actions such as:\n\n```\ncreate_rectangle(...)\nmove_object(...)\nset_fill_color(...)\ndelete_object(...)\n```\n\nThis works well for a few independent actions. It becomes less convenient when a task needs iteration or branching. Imagine a slide editor, canvas, or UI board where the user asks:\n\nFind every text object smaller than 12 px, increase it to 12 px, and move any overflow into a new text box below the original.\n\nIf the model calls one tool for every object, every observation and action may require another inference step. A short program keeps the ordinary computation local:\n\n``` js\nconst objects = canvas.listObjects({ type: \"text\" });\n\nfor (const object of objects) {\n  if (object.fontSize >= 12) continue;\n\n  const result = canvas.updateText(object.id, { fontSize: 12 });\n\n  if (result.overflowText) {\n    canvas.createText({\n      content: result.overflowText,\n      x: object.x,\n      y: object.y + object.height + 8,\n      fontSize: 12,\n    });\n  }\n}\n```\n\nThe model still chooses the operation, but loops, conditions, and intermediate values do not each need another model turn. Generated code is not a replacement for tools; it is a way to compose approved tools.\n\n## GUI control recovers semantics from pixels\n\nGraphical interfaces are excellent for people. We can scan a canvas, recognize an icon, and point at an object without naming every part of the scene.\n\nFor an agent, the same action often takes a longer translation path:\n\n``` php\nflowchart LR\n  subgraph GUI[\"GUI control\"]\n    A[Render] --> B[Interpret pixels] --> C[Find control] --> D[Click or type] --> E[Check result]\n  end\n  subgraph API[\"Semantic API control\"]\n    F[Read state] --> G[Call named action] --> H[Verify result]\n  end\n  E --> S[(Application state)]\n  H --> S\n  classDef gui fill:#fff0df,stroke:#c76b16,color:#17202a\n  classDef api fill:#e8f0fb,stroke:#2563a7,color:#17202a\n  class A,B,C,D,E gui\n  class F,G,H api\n```\n\nGUI control is indispensable when no structured interface exists. Vision also remains important when appearance is the result, such as editing slides, graphics, or a web page. My concern is using pixels and coordinates as the primary control plane when the application already knows the content, bounds, transform, and identity of each object.\n\nA useful split is to use vision to understand and judge the output, while using semantic operations to change application state when available.\n\n## Why the shell works so well\n\nThe shell removes much of the visual interpretation work. It is textual, scriptable, composable, and supported by decades of public examples. Models have seen large amounts of Bash, `git`\n\n, `ffmpeg`\n\n, and similar tools during training. Some of that operational experience is encoded in the model’s learned parameters, so common commands can feel almost native.\n\nThat is a substantial advantage:\n\n```\nffmpeg -i input.mov -vf \"scale=1920:-2,fps=30\" -c:v libx264 -crf 20 output.mp4\n```\n\nA model familiar with `ffmpeg`\n\nmay produce this without first studying a manual. The shell is therefore hard to beat for developer environments and established utilities.\n\nThe tradeoff is authority and precision. A process launcher plus filesystem access is broader than a narrowly scoped application operation:\n\n```\nvideo.resize({\n  assetId,\n  width: 1920,\n  frameRate: 30,\n  destination: approvedOutput,\n});\n```\n\nCLI syntax also depends on conventions that are not fully captured by a machine-checkable schema.\n\nContainers and operating-system sandboxes still matter. A narrow API does not replace them, but it can reduce the authority placed inside the boundary. For application-level automation, that may also allow a lighter runtime than a general shell environment.\n\n## From fixed handlers to generated programs\n\nIn a React application, a button usually invokes code that a developer prepared in advance:\n\n``` js\nfunction AlignButton({ selectedIds }) {\n  const onClick = () => {\n    const objects = selectedIds.map((id) => editor.getObject(id));\n    const left = Math.min(...objects.map((object) => object.x));\n\n    for (const object of objects) {\n      editor.updateTransform(object.id, { x: left });\n    }\n\n    editor.commitHistory(\"Align left\");\n  };\n\n  return <button onClick={onClick}>Align left</button>;\n}\n```\n\nThe button is a human-friendly handle for a predefined code path. This is reliable, but fixed: the developer must anticipate the action and prepare the handler beforehand.\n\nAn agent can instead generate a short program on the fly from a finite set of lower-level application APIs:\n\n``` js\nconst objects = canvas.getSelection();\nconst left = Math.min(...objects.map((object) => object.bounds.x));\n\nfor (const object of objects) {\n  canvas.move(object.id, { x: left, y: object.bounds.y });\n}\n```\n\nFrequent actions should probably remain buttons. The more flexible case is an operation that was not anticipated as one fixed handler. The application developer defines the finite API vocabulary and its limits; the agent combines those primitives for the current request. The developer does not need to enumerate every useful sequence beforehand.\n\n## How the API-native architecture fits together\n\nGenerating a program for each request raises two immediate questions: how does the agent learn the application’s API, and how can that generated code run without receiving unrestricted access?\n\nThe architecture I am exploring in [ CogCore](https://github.com/carsonDB/CogCore) answers those questions with an API retrieval path and a guarded execution path:\n\nThe application team owns the TypeScript API, product permissions, and final state changes shown in orange. [ CogCore](https://github.com/carsonDB/CogCore) provides the blue agent and runtime components. A chat agent delegates the task to a code agent. The code agent retrieves only the relevant API manual, writes a short JavaScript program, and sends it to the guarded runtime.\n\nThe application developer implements the API in TypeScript. A build step uses [ ts-morph](https://ts-morph.com/) to extract signatures and type relationships from selected\n\n`*.api.ts`\n\nentry points into a searchable graph. TypeScript remains the source of truth; the graph becomes the agent’s manual.Generated JavaScript receives named globals chosen by the host application, rather than automatic access to the DOM, shell, network, or application internals:\n\n```\nclass CanvasForAI {\n  listObjects(filter?: ObjectFilter): ObjectSummary[];\n  getObject(id: string): CanvasObject | null;\n  createShape(input: ShapeInput): CanvasObject;\n  createText(input: TextInput): CanvasObject;\n  updateGeometry(id: string, input: GeometryUpdate): CanvasObject;\n  updateAppearance(id: string, input: AppearanceUpdate): CanvasObject;\n  removeObjects(ids: string[]): void;\n}\n```\n\nThis is a generic example, not an announcement of a particular editor project.\n\nThe finite surface is the important constraint. Generated code can combine these APIs in ways the developer did not predefine, but it should not automatically receive the DOM, shell, network, or internal application state.\n\n## Why JavaScript needs runtime validation\n\nWhy generate JavaScript rather than TypeScript? A task program has a short lifecycle: write it, run it, verify it, and discard it. Requiring imports and type annotations adds syntax, while the actual application API is already defined in TypeScript.\n\nMore importantly, TypeScript checking would still not be enough. Its types are erased when code becomes JavaScript, and broad static types do not express many runtime rules:\n\n```\ntype UpdateInput = {\n  color: string;\n  svgPath: string;\n  opacity: number;\n};\n```\n\nAll three fields are statically valid, but the application still needs to ask:\n\n``` php\ncolor   -> Is it a valid color accepted by this renderer?\nsvgPath -> Is it valid SVG path data within a complexity limit?\nopacity -> Is it finite and between 0 and 1?\n```\n\nFor this reason, [ CogCore](https://github.com/carsonDB/CogCore) starts with plain JavaScript and adds checks at the boundary. JavaScript\n\n`Proxy`\n\nwrappers restrict access to the declared object surface and reject unknown properties. Zod schemas validate arguments and results against application rules. A worker adds separation and timeouts, although it is not equivalent to a VM, container, or operating-system sandbox.The goal is not to make generated code automatically correct. It is to make failures constrained and repairable:\n\n``` php\nflowchart LR\n  A[Generate] --> B{Validate}\n  B -->|Valid| C[Execute]\n  B -->|Structured error| A\n  C --> D{Verify result}\n  D -->|Accepted| E[Commit]\n  D -->|Repair feedback| A\n  classDef action fill:#e8f0fb,stroke:#2563a7,color:#17202a\n  classDef check fill:#fff0df,stroke:#c76b16,color:#17202a\n  class A,C,E action\n  class B,D check\n```\n\nAPI design carries much of the safety model. Compare a broad shell capability:\n\n```\nsystem.exec(command);\n```\n\nwith a narrow application capability:\n\n```\nproject.deleteGeneratedPreview({ previewId });\n```\n\nThe first delegates meaning to a shell with ambient authority. The second can validate an opaque identifier, check ownership, record history, and support undo. The narrower operation is safer and easier for the model to use correctly.\n\n## The central limitation of API-native control\n\nAPI-native control offers semantic operations, narrow permissions, strong multi-step composition, and the possibility of a lightweight application sandbox. Its central weakness is not execution power but unfamiliarity.\n\nThe same training advantage that helps shell agents exposes this weakness. Models have seen enormous amounts of shell code, so common shell skills are already encoded in their learned parameters. After training, those parameters remain fixed until the model is updated or fine-tuned. A larger model may reason better from context, but size alone does not give it native experience with a newly created application API.\n\nA manual can supply current information at inference time, yet models do not always use it reliably. They may ignore part of it, invent a familiar-sounding method, or fall back to stale patterns learned during training. Retrieval and repair help, but they also add latency.\n\nCompared with API-native control, GUI automation has greater reach because it can operate software that exposes no structured API, but it pays for that reach by recovering semantics from pixels. Shell control has much stronger model familiarity and an excellent ecosystem, but it often exposes broader authority and fits structured application state less precisely. Individual tool calls retain narrow schemas, but long tasks can turn the model itself into the control loop.\n\nAPI-native control is trying to keep the useful properties together: semantic access, finite authority, local program composition, and adaptation through an updated manual. The unresolved part is that an updated manual is not the same as an internalized skill.\n\nThis is why external skills matter. Instead of placing every operational skill permanently in model weights, an agent system could retain concise lessons from accepted runs and revise them as the application changes. [ CogCore](https://github.com/carsonDB/CogCore) includes an early experiment with reusable skill tips, but this is not a solved learning system.\n\nThe open problem is not only how to expose a safer interface. It is how to make an agent reliably adapt to an interface it has never seen before, without letting old habits override the application’s current rules. Until that works consistently, API-native control remains a useful direction with a very real limit.", "url": "https://wpnews.pro/news/beyond-clicking-and-shell-commands-api-native-computer-control", "canonical_source": "https://carsondb.github.io/blog/2026-06-api-native/", "published_at": "2026-06-22 07:28:25+00:00", "updated_at": "2026-06-22 07:41:02.355227+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "generative-ai"], "entities": ["JavaScript", "Bash", "ffmpeg", "git"], "alternates": {"html": "https://wpnews.pro/news/beyond-clicking-and-shell-commands-api-native-computer-control", "markdown": "https://wpnews.pro/news/beyond-clicking-and-shell-commands-api-native-computer-control.md", "text": "https://wpnews.pro/news/beyond-clicking-and-shell-commands-api-native-computer-control.txt", "jsonld": "https://wpnews.pro/news/beyond-clicking-and-shell-commands-api-native-computer-control.jsonld"}}