Show HN: Desktopmcp – MCP server for the Linux desktop

Desktopmcp, a new MCP server for the Linux desktop, gives AI models access to the desktop through 144 tools via XDG Desktop Portals, AT-SPI, and D-Bus, enabling visual and semantic interaction with the UI. The open-source tool runs on Wayland compositors and requires user permission for sensitive operations.

MCP server for the Linux desktop. It gives AI models access to the Linux desktop. desktopmcp connects AI assistants to the Linux desktop through three system interfaces: XDG Desktop Portals for sandboxed desktop operations, AT-SPI for semantic UI understanding, and D-Bus for low-level system access. It exposes 144 tools over the Model Context Protocol, letting an AI see what's on screen, understand UI structure, move cursor, click buttons, type text, manage files, and interact with desktop services. AI models are blind to the desktop. They can't see a window, read a button label, click a menu item, or control the desktop. desktopmcp solves this by giving AI models two complementary ways to interact with the desktop: Visual : screenshots, screen capture, color picking, mouse and keyboard input Semantic : the full accessibility tree AT-SPI , where every UI element has a name, role, position, and set of actions the AI can invoke directly The semantic path means an AI can call find element role="push button", name="Save" instead of scanning pixels to find where "Save" might be. It can read text content, check which element is focused, traverse the UI tree, and perform actions like "click" on any element -- all without needing a screenshot. Everything goes through XDG Desktop Portals, so the user gets permission dialogs, and a single binary runs on GNOME, KDE, Sway, or any Wayland compositor. 144 tools organized into five groups: | Category | Tools | What they do | |---|---|---| Remote Desktop & Input | 13 | Start sessions, take screenshots, move/click mouse, type text, touch events, scroll | XDG Portals | 35 | Notifications, file dialogs, clipboard, wallpaper, network status, location, email, printing, camera, settings, global shortcuts, secrets, power profile, game mode, and more | Dynamic Launcher | 8 | Install, launch, inspect, and remove .desktop application launchers | AT-SPI Accessibility | 76 | Full accessibility tree: find elements, read text, click buttons, get/set values, table and hyperlink inspection, document metadata, event subscriptions, collection search | D-Bus Bridge | 12 | Call any D-Bus method, read/write properties, introspect services, subscribe to signals | AI model Claude, etc. | | MCP protocol JSON-RPC over stdio or HTTP v desktopmcp Rust, async, single binary | |--- XDG Desktop Portals ashpd -- screenshots, input, files, settings, ... |--- AT-SPI bus zbus -- UI tree, element actions, text, events |--- D-Bus session/system zbus -- arbitrary service access |--- PipeWire pipewire-rs -- screen capture frames v Linux desktop GNOME / KDE / Sway / ... XDG Portals ensure every sensitive operation screen capture, input injection, file access goes through a user-facing permission dialog. The AI cannot act without the user granting consent. - Linux with a Wayland compositor GNOME 40+, KDE Plasma 5.27+, Sway, or similar xdg-desktop-portal and a desktop-specific backend xdg-desktop-portal-gnome , -kde , -wlr - PipeWire for screen capture - AT-SPI2 at-spi2-core , typically pre-installed on any desktop with accessibility support - D-Bus session bus present on all modern Linux desktops Build requirements: - Rust 1.85+ edition 2024 - System libraries: libdbus , libpipewire-0.3 , at-spi2-core , pkg-config - LLVM/Clang for pipewire-sys bindgen Run directly without cloning: nix run github:varbhat/desktopmcp -- --t http or stdio Or clone and build locally. The repository includes a flake.nix that provides all system dependencies automatically: git clone <repo-url cd desktopmcp nix develop cargo build --release We use Github Actions to build and release AppImages automatically. Grab the latest AppImage from the releases page https://github.com/varbhat/desktopmcp/releases . Mark it as executable and run it. It's built using nix-appimage https://github.com/ralismark/nix-appimage which bundles the nix derivation and all its dependencies into a single-file executable, and hence the size of the AppImage is very big It's something we'll optimize in upcoming releases—but hey, it lets you try it . Install system dependencies, then build: Debian / Ubuntu sudo apt install libdbus-1-dev libpipewire-0.3-dev at-spi2-core \ pkg-config libclang-dev Fedora sudo dnf install dbus-devel pipewire-devel at-spi2-core-devel \ pkg-config clang-devel Arch sudo pacman -S dbus pipewire at-spi2-core pkgconf clang Build cargo build --release The binary is at target/release/desktopmcp . stdio mode default; Your MCP Client will do this for you desktopmcp HTTP mode -- for remote-mcp You need to run this beforehand desktopmcp --transport http --bind 127.0.0.1:3000 In stdio mode, MCP messages are exchanged over stdin/stdout JSON-RPC . Logs go to stderr. In HTTP mode, the server listens at http://<bind /mcp using Streamable HTTP with Server-Sent Events. You can configure your MCP Client to use desktopmcp in stdio mode by specifying the binary path. Or you can run the desktopmcp in HTTP remote-mcp mode and configure your MCP Client to use it by specifying the URL. Ask AI : "What windows are open on my desktop?" The AI calls get window list and gets back a structured list of every window with title, application name, position, and size -- no screenshot needed. Ask AI : "Click the Save button in Firefox" php 1. find element role="push button", name="Save", app name="Firefox" - returns element ID with position and available actions 2. atspi do action id="...", action name="click" - button is clicked via the accessibility API Ask AI : "Take a screenshot and tell me what you see" php 1. start session devices= "pointer" , with screencast=true - user approves the permission dialog once 2. take screenshot session id="...", format="jpeg" - AI receives a base64-encoded image Ask AI : "Type my email address into the login form" php 1. find element role="entry", name="Email" - finds the text field 2. type into id="...", text="user@example.com" - text is entered via the EditableText accessibility interface Ask AI : "Send me a notification when the download finishes" 1. send notification id="dl-done", title="Download Complete", body="file.zip is ready" - desktop notification appears These tools require an active session created with start session . The user sees a one-time permission dialog. | Tool | Description | |---|---| start session | Create a remote desktop session optionally with screencast and clipboard | stop session | End a session and release resources | simple screenshot | One-shot screenshot via the Screenshot portal no session needed | take screenshot | Capture a frame from an active screencast session PNG or JPEG | pick color | Pick a color from anywhere on screen | mouse move | Move the mouse by a relative offset | mouse move absolute | Move the mouse to an absolute screen position | mouse click | Click a mouse button left, right, middle | mouse scroll | Scroll the mouse wheel smooth | mouse scroll discrete | Scroll the mouse wheel discrete click-by-click steps | keyboard key | Press, release, or tap a key by keycode | keyboard type | Type a text string as a sequence of key taps | touch event | Send touch events down, motion, up for touchscreen simulation | These tools use sandboxed XDG Desktop Portal APIs. Most work without a session. | Tool | Description | |---|---| send notification | Send a desktop notification | open uri | Open a URI in the default application | open file | Open a local file in its default application | open directory | Open a directory in the file manager | scheme supported | Check if a URI scheme https, ftp, etc. is handled | open file dialog | Show an open-file dialog | save file dialog | Show a save-file dialog | save files dialog | Show a save-multiple-files dialog | clipboard read | Read text from the clipboard requires session | clipboard write | Write text to the clipboard requires session | get appearance | Get color scheme and accent color | read setting | Read a specific XDG setting by namespace and key | read all settings | Read all settings in one or more namespaces | network status | Check network connectivity, metered status | can reach | Test reachability of a hostname:port | set wallpaper | Set wallpaper from a URI | set wallpaper file | Set wallpaper from a local file | trash file | Move a file to the system trash | get user information | Get current user's ID, name, avatar | request background | Request permission to run in the background | set background status | Set a background status message | check camera | Check if a camera is available | compose email | Open the default email client with a pre-composed message | get location | Get geographic coordinates requires user permission | get memory warning | Poll for low-memory warnings | get power profile | Check if power-saver mode is active | get proxy | Resolve proxy settings for a URI | retrieve secret | Get the app secret from the system keyring | game mode status | Check GameMode status | list shortcuts | List registered global keyboard shortcuts | bind shortcuts | Register global keyboard shortcuts | print file | Print a file via the system print dialog | session inhibit | Prevent logout, suspend, or idle | get available device types | Query available input device types | get screencast capabilities | Query available cursor modes and source types | Install and manage .desktop application launchers. | Tool | Description | |---|---| launcher supported types | Query supported launcher types application, web app | launcher prepare install | Show install dialog for user confirmation returns a token | launcher install | Install a .desktop launcher using a token | launcher request token | Get an install token without a dialog | launcher uninstall | Remove an installed launcher | launcher launch | Launch an app by its .desktop file ID | launcher get desktop entry | Read the .desktop file content | launcher get icon | Get the launcher icon as base64 | Semantic access to every UI element on the desktop. No screenshot or coordinate guessing required. Tree traversal & element inspection: | Tool | Description | |---|---| atspi get desktop | Get the accessibility tree root and list all applications | atspi get applications | List running applications with toolkit info | atspi get element | Get full properties of an element by ID | atspi get children | Get child elements | atspi get child at index | Get a specific child by index | atspi get parent | Get the parent element | atspi get properties | Get name, role, states, interfaces, position, size | atspi get attributes | Get key-value attributes CSS properties, etc. | atspi get relation set | Get relations labelled-by, controlled-by, etc. | atspi get extended properties | Get Locale, AccessibleId, HelpText | atspi get application info | Get toolkit name/version, AT-SPI version | Search & UI tree: | Tool | Description | |---|---| find element | Search by role, name, and/or application high-level | find focused | Get the currently focused element | get ui tree | Build a hierarchical UI tree to a given depth | get window list | List open windows with titles, positions, sizes | wait for element | Poll until an element appears or timeout | refresh ui cache | Refresh the application list | atspi collection get matches | Fast server-side search by role, interfaces, attributes | atspi collection get matches to | Backwards search from a given element | atspi get active descendant | Get the focused item in a container | Actions & interaction: | Tool | Description | |---|---| click element | Find an element and perform a click action high-level | type into | Find a text field and type into it high-level | atspi get actions | List available actions on an element | atspi do action | Perform an action by name or index | atspi get action details | Get name, description, key binding for one action | atspi grab focus | Move keyboard focus to an element | Text: | Tool | Description | |---|---| read element text | Find an element and read its text high-level | atspi get text | Read text content, character count, caret offset | atspi set text | Replace text content | atspi edit text | Cut, copy, paste, insert, delete operations | atspi set caret offset | Move the text cursor | atspi get text at offset | Get word/sentence/line at a character offset | atspi get string at offset | Get text segment by granularity | atspi get character extents | Get screen bounding box of a character | atspi get offset at point | Get character offset at screen coordinates | atspi get text selections | Get active text selection ranges | atspi set text selection | Add, modify, or remove text selections | atspi get text attributes | Get font, size, style at an offset | atspi get text attribute value | Get a single named text attribute | atspi get attribute run | Get the uniform attribute run at an offset | atspi get default attribute set | Get default attributes for the whole text | atspi get range extents | Get bounding box of a text range | atspi get bounded ranges | Find text ranges within a screen rectangle | atspi scroll substring to | Scroll a text range into view | atspi scroll substring to point | Scroll a text range to a specific screen point | Component geometry, scroll, focus : | Tool | Description | |---|---| atspi get position | Get screen position and size of an element | atspi get size | Get width and height | atspi get layer | Get rendering layer and alpha transparency | atspi contains | Hit-test: check if a point is inside an element | atspi get accessible at point | Find the element at screen coordinates | atspi scroll to | Scroll an element into the viewport | atspi scroll to point | Scroll to a specific point | atspi set geometry | Move or resize an element position, size, or extents | Value, Selection, Table, Hyperlinks, Document, Image: | Tool | Description | |---|---| atspi get value / atspi set value | Read/write numeric widget values sliders, spinners | atspi get selection / atspi select item / atspi deselect item | Manage selections in lists and combo boxes | atspi select all / atspi clear selection / atspi is child selected | Bulk selection operations | atspi get table info / atspi get table cell | Read table dimensions, cell content, row/column spans | atspi table select row / atspi table select column | Select table rows/columns | atspi get table selection counts | Get count of selected rows/columns | atspi get hyperlinks | List hyperlinks with URIs and character ranges | atspi get document info | Get document locale, attributes URL, MIME type , page count | atspi get document text selections / atspi set document text selections | Cross-element document selections | atspi get image info | Get image description, locale, position, and size | Events: | Tool | Description | |---|---| atspi subscribe events | Subscribe to AT-SPI event categories or specific events | atspi unsubscribe events | Cancel a subscription | atspi get pending events | Poll buffered events | atspi get registered events | List all currently registered event listeners | atspi get status | Check if AT-SPI is enabled and if a screen reader is active | Direct access to any D-Bus service on the session or system bus. The AI can introspect and interact with arbitrary desktop services. | Tool | Description | |---|---| dbus list names | List all D-Bus service names | dbus introspect | Introspect a service's interfaces, methods, signals, properties | dbus list objects | Walk the object tree of a service | dbus call method | Call any D-Bus method with JSON arguments | dbus get property | Get a property value | dbus get all properties | Get all properties of an interface | dbus set property | Set a property value | dbus subscribe signal | Subscribe to D-Bus signals with a match rule | dbus unsubscribe signal | Cancel a signal subscription | dbus get signals | Poll buffered signals | dbus list subscriptions | List active subscriptions | dbus get name owner | Resolve a well-known name to a unique bus name | nix develop enter dev shell with all dependencies cargo build compile cargo clippy lint cargo run run in stdio mode cargo run -- -t http run in HTTP mode Verify portals are running: systemctl --user status xdg-desktop-portal systemctl --user status pipewire Enable debug logging: RUST LOG=debug cargo run Test with a raw MCP message: echo '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"test","version":"1"}}}' \ | cargo run --quiet 2 /dev/null Permission dialog does not appear Restart the portal service: systemctl --user restart xdg-desktop-portal PipeWire connection fails Check that PipeWire is running: systemctl --user status pipewire systemctl --user restart pipewire AT-SPI returns no applications Make sure the AT-SPI bus is running and accessibility is enabled: Check AT-SPI status dbus-send --session --print-reply --dest=org.a11y.Bus \ /org/a11y/bus org.freedesktop.DBus.Properties.GetAll \ string:org.a11y.Status Enable accessibility on Gnome if disabled gsettings set org.gnome.desktop.interface toolkit-accessibility true Mouse/keyboard input has no effect - Call start session first with the required devices "keyboard", "pointer" - Accept the permission dialog when it appears - Verify the session is still active with a valid session id desktopmcp is designed around the XDG Desktop Portal security model: - Every sensitive operation requires explicit user consent via a system dialog - The server works inside Flatpak and other sandboxed environments - D-Bus and AT-SPI access follows standard Linux desktop permissions - No root privileges are needed - The user can deny or revoke access at any time That said, this server gives AI models significant control over your desktop. Use it with trusted models, review what the AI does, and be aware that an unrestricted AI could perform unintended actions. | Component | Library | Version | |---|---|---| | MCP protocol | rmcp | 1.7 | | XDG Portals | ashpd | 0.13 | | D-Bus / AT-SPI | zbus + zvariant | 5 | | Screen capture | pipewire-rs | 0.10 | | Async runtime | tokio | 1 | | Image encoding | image-rs | 0.25 | | Serialization | serde + serde json | 1 | | CLI | clap | 4 |