{"slug": "show-hn-ui-inspector-for-tauri", "title": "Show HN: UI Inspector for Tauri", "summary": "Tauri 2 has released a new UI Inspector plugin that lets developers select elements inside a running webview and generate durable @ui_<ULID> references for coding agents, capturing native window pixels and exact element crops without rebuilding the page. The plugin, available as tauri-plugin-ui-inspector 0.1, includes a CLI, framework-neutral frontend, and adapters for Svelte 5, React, and Vue 3, with support for canvas, WebGL, fonts, shadows, and overlays. The tool is designed to improve agent-based UI testing and debugging by providing stable references that include DOM metadata, accessibility semantics, ranked locators, and source file mappings.", "body_md": "Select an element inside a running Tauri 2 webview and turn it into a durable `@ui_<ULID>`\n\nreference for coding agents.\n\nA reference records the selected DOM node, accessibility semantics, ranked locators, optional framework source metadata, native window pixels, and an exact element crop. The native backend captures what the desktop compositor rendered, including canvas, WebGL, fonts, shadows, and overlays. It does not rebuild the page with a DOM-to-image library.\n\nThe plugin is framework-neutral. Development adapters map Svelte 5, React, and Vue 3 elements back to source files; apps without an adapter still get DOM metadata, locators, and screenshots.\n\nRun the checked-in Svelte fixture from a clone:\n\n```\npnpm install\ncargo install --path crates/ui-inspector\npnpm dev\n```\n\nIn a second terminal:\n\n```\nui-inspector pick\n```\n\nHover a control and click it. The fixture creates output like this:\n\n```\nWaiting for UI selection...\nSelected @ui_01M0...\nCreateWorkspaceButton: button 'Create workspace' at src/lib/CreateWorkspaceButton.svelte:9:1\nsrc/lib/CreateWorkspaceButton.svelte:9:1\n.ui-inspector/refs/ui_01M0.../element.png\n```\n\nFetch the complete record with JSON-only stdout:\n\n```\nui-inspector get @ui_01M0... --json\n```\n\nInstall the native plugin, CLI, framework-neutral frontend, and the adapter for your framework:\n\n```\n# src-tauri/Cargo.toml\n[dependencies]\ntauri-plugin-ui-inspector = \"0.1\"\ncargo install tauri-ui-inspector\npnpm add @tauri-ui-inspector/inspector\npnpm add -D @tauri-ui-inspector/adapter-svelte\n```\n\nRegister the plugin. Keep it behind `debug_assertions`\n\nunless your application has a deliberate production capture policy.\n\n``` js\nfn main() {\n    let builder = tauri::Builder::default();\n\n    #[cfg(debug_assertions)]\n    let builder = {\n        let mut inspector = tauri_plugin_ui_inspector::Builder::new();\n        inspector\n            .storage_dir(\".ui-inspector\")\n            .max_history(100)\n            .crop_padding(8);\n        builder.plugin(inspector.build())\n    };\n\n    builder\n        .run(tauri::generate_context!())\n        .expect(\"Tauri application failed\");\n}\n```\n\nGrant the plugin permission to each inspectable window:\n\n```\n{\n  \"$schema\": \"../gen/schemas/desktop-schema.json\",\n  \"identifier\": \"main-capability\",\n  \"windows\": [\"main\"],\n  \"permissions\": [\"core:default\", \"ui-inspector:default\"]\n}\n```\n\nThe default permission allows capture, cancellation, live resolution, and reading the last reference.\n\nInstall the bridge once in every window that should answer CLI requests. This Svelte example also enables source metadata:\n\n``` js\n<script lang=\"ts\">\n  import { onMount } from 'svelte'\n  import { installInspectorBridge } from '@tauri-ui-inspector/inspector'\n  import { svelteAdapter } from '@tauri-ui-inspector/adapter-svelte'\n\n  onMount(() => {\n    let dispose: (() => void) | undefined\n    void installInspectorBridge({\n      adapters: [svelteAdapter()],\n      onSelect(reference) {\n        console.info(`Created @${reference.id}`)\n      }\n    }).then(value => (dispose = value))\n\n    return () => dispose?.()\n  })\n</script>\n```\n\n`installInspectorBridge`\n\nhas no Svelte dependency. Framework-specific runtime work stays inside adapters.\n\nCall `startInspecting()`\n\nfrom an application control, or press `Command+Shift+C`\n\non macOS and `Ctrl+Shift+C`\n\nelsewhere after installing the bridge. The shortcut is configurable.\n\n``` js\nimport { startInspecting, stopInspecting } from '@tauri-ui-inspector/inspector'\n\nconst inspector = startInspecting({\n  onStarted() {},\n  onHovered(element) {},\n  onSelect(reference) {},\n  onCancel() {},\n  onError(error) { console.error(error) }\n})\n\ninspector.state // 'inspecting', 'capturing', or 'idle'\nstopInspecting()\n```\n\nWhile active, the picker:\n\n- draws a pointer-transparent overlay without changing the inspected element;\n- selects interactive ancestors for nested text and SVG children;\n- follows scrolling, resizing, and CSS transforms through\n`getBoundingClientRect()`\n\n; - traverses open shadow roots and supports pointer and mouse events;\n- suppresses the inspection click before the application receives it;\n- pauses active Web Animations and resumes them after capture;\n- preserves the existing focus and hover target where the webview permits it;\n- exits on Escape and restores its cursor, listeners, overlay, and animations.\n\nThe overlay is hidden before the Rust capture begins, so inspector chrome does not appear in `window.png`\n\nor `element.png`\n\n.\n\nUse the same metadata and native capture path without the picker:\n\n``` js\nimport { inspectElement, inspectSelector } from '@tauri-ui-inspector/inspector'\nimport { svelteAdapter } from '@tauri-ui-inspector/adapter-svelte'\n\nconst options = { adapters: [svelteAdapter()] }\nconst first = await inspectElement(button, options)\nconst second = await inspectSelector('[data-testid=\"create-workspace\"]', options)\n```\n\n`inspectSelector`\n\nrequires exactly one match. It throws rather than selecting an ambiguous element.\n\n```\nui-inspector pick [--window main]\nui-inspector last\nui-inspector get <id>\nui-inspector list\nui-inspector screenshot <id>\nui-inspector resolve <id> [--window main]\nui-inspector delete <id>\nui-inspector clear\n```\n\nThe CLI accepts `ui_01...`\n\nand `@ui_01...`\n\n. Pass `--project /absolute/path`\n\nwhen the current directory is outside the project. Pass `--storage-dir path`\n\nwhen the application uses a non-default store.\n\n`--json`\n\nis global and may appear before or after the subcommand. JSON mode writes one valid JSON value to stdout; diagnostics stay on stderr.\n\n| Exit | Meaning |\n|---|---|\n| 0 | Success |\n| 1 | Invalid input, protocol failure, or internal error |\n| 2 | Reference not found |\n| 3 | Application not running or inspector disabled |\n| 4 | Inspection cancelled |\n| 5 | Stored element no longer resolves exactly |\n\n`pick`\n\nand `resolve`\n\nuse an authenticated local socket on Unix and a named pipe on Windows. The plugin never opens a TCP listener. If several windows exist, `--window`\n\nselects one by Tauri label; otherwise the focused window wins, followed by the first label in lexical order.\n\nEach selection creates one directory:\n\n```\n.ui-inspector/\n  run/instance.json\n  refs/\n    ui_01M0.../\n      reference.json\n      window.png\n      element.png\n```\n\nThe default history is 100 references. Set `max_history(0)`\n\nto disable cleanup. `.ui-inspector/`\n\nbelongs in `.gitignore`\n\nbecause its JSON and screenshots may contain private UI data.\n\nSchema version 1 includes:\n\n- project and Tauri window identity, geometry, scale factor, and browser viewport metrics;\n- role, accessible name and description, common ARIA/native states, safe form metadata, and redacted attributes;\n- a compact HTML fragment, parent context, and up to eight DOM ancestors;\n- ranked locators with confidence and uniqueness recorded at selection time;\n- optional component, source file, line, column, and component ancestry;\n- relative screenshot filenames plus the final physical-pixel crop rectangle;\n- a deterministic summary written for humans and agents.\n\nRust owns the schema. `ts-rs`\n\ngenerates [packages/shared/src/generated.ts](/mathematic-inc/tauri-plugin-ui-inspector/blob/main/packages/shared/src/generated.ts), which prevents a second handwritten TypeScript model.\n\nUnknown JSON object fields are safe for older readers to ignore. A breaking shape change must increment `schemaVersion`\n\n.\n\nThe frontend ranks locators in this order:\n\n- explicit test ID;\n- unique role plus accessible name;\n- unique DOM ID;\n- stable attributes;\n- framework source metadata;\n- generated CSS selector;\n- DOM structural path;\n- exact normalized text.\n\n`@medv/finder`\n\nsupplies CSS selector generation. The inspector also searches open shadow roots for explicit selectors and semantic matches. Closed shadow roots remain opaque.\n\nLive resolution only tries locators that were unique when captured and have confidence of at least `0.5`\n\n. It then checks the original tag, role, and accessible name. If no locator finds exactly one matching element, the CLI exits with code 5 and returns a structured `notFound`\n\nresult. It never picks a nearby element.\n\n`dom-accessibility-api`\n\ncomputes role, accessible name, and accessible description. The collector also records ARIA relationships, disabled, checked, selected, expanded, pressed, placeholder, form label, input type, and optional value.\n\nForm values are off by default. Password, hidden, password-autocomplete, one-time-code, credit-card, and token-like controls never persist a value even when value capture is enabled. Backend redaction runs again before callbacks and disk writes.\n\n``` js\nlet mut redaction = tauri_ui_inspector_core::RedactionConfig::new();\nredaction.redact_text = true;\n\nlet mut inspector = tauri_plugin_ui_inspector::Builder::new();\ninspector\n    .redaction(redaction)\n    .capture_screenshots(false)\n    .persist_references(false);\n```\n\nFrontend options can add attribute-name fragments, redact text before IPC, or opt into safe form values:\n\n```\ninstallInspectorBridge({\n  redactText: true,\n  captureFormValues: false,\n  sensitiveAttributeFragments: ['secret', 'token', 'session']\n})\n```\n\nThe plugin has no telemetry and no upload path. It cannot redact secrets that are already rendered into canvas, WebGL, images, or screenshot pixels. Treat the entire store as sensitive.\n\nRust uses `xcap`\n\nfor native window capture and `image`\n\nfor PNG cropping. `window.png`\n\ncontains the full captured native window, including decorations where the platform API returns them. `element.png`\n\nis cut directly from that bitmap.\n\nThe coordinate transform measures browser CSS pixels, `devicePixelRatio`\n\n, visual viewport offsets, Tauri geometry, capture-backend bounds, and the returned PNG dimensions. It does not assume that any two spaces use the same unit. On platforms where Tauri reports identical inner and outer geometry, the transform calibrates the content area from `innerWidth × devicePixelRatio`\n\nand `innerHeight × devicePixelRatio`\n\n.\n\nPadding is measured in CSS pixels before scaling. The default is 8; `0`\n\n, `8`\n\n, `16`\n\n, and `32`\n\nare useful presets. Partially visible elements are clamped to the bitmap. Fully disjoint rectangles fail.\n\nThe checked-in E2E run used a 1280×800 CSS viewport on a Retina display at 2×. It produced a 2560×1664 native window image and a 400×112 crop for a 184×40 button with 8 CSS pixels of padding. The E2E test compares every crop pixel against its declared region in `window.png`\n\n.\n\nThe Svelte, React, and Vue adapters delegate runtime source recovery to `element-source`\n\nand its maintained framework resolvers. In development builds they can recover the selected source location and component ancestry from framework metadata.\n\nProduction compilation removes that metadata. The adapter then returns `undefined`\n\n, while DOM collection, locators, screenshots, and persistence keep working.\n\nKeep source recovery optional in application logic. Production compilers may remove framework development metadata.\n\n``` js\nimport { reactAdapter } from '@tauri-ui-inspector/adapter-react'\nimport { vueAdapter } from '@tauri-ui-inspector/adapter-vue'\n\ninstallInspectorBridge({ adapters: [reactAdapter()] })\ninstallInspectorBridge({ adapters: [vueAdapter()] })\n```\n\nThe included [UI inspector skill](/mathematic-inc/tauri-plugin-ui-inspector/blob/main/skills/ui-inspector/SKILL.md) tells Codex to resolve an `@ui_`\n\nreference through the CLI, inspect `element.png`\n\nfirst, open `window.png`\n\nwhen context matters, verify the recorded source, and refuse fuzzy substitutions.\n\nCopy or install that skill in your Codex environment. Then a request can be as short as:\n\n```\nFix the padding on @ui_01M0...\n```\n\nThe plugin itself has no Codex dependency. `onSelect`\n\nin TypeScript and `on_reference_created`\n\nin Rust support other local consumers:\n\n```\ninspector.on_reference_created(|reference| {\n    println!(\"Created @{}\", reference.id);\n});\n```\n\nAn adapter has one job:\n\n```\nexport interface FrameworkInspectorAdapter {\n  readonly name: string\n  inspect(element: Element): SourceInfo | undefined | Promise<SourceInfo | undefined>\n}\n```\n\nReturn framework, component, source location, and ancestry when the runtime exposes them. Return `undefined`\n\nwhen metadata is absent. The published Svelte, React, and Vue adapters keep runtime probes in their own packages; the picker and backend do not import those frameworks.\n\n| Platform | Capture path | Notes |\n|---|---|---|\n| macOS | `xcap` window capture |\nScreen Recording permission may be required. Native E2E, negative monitor coordinates, 1×, and Retina 2× were exercised in this repository. |\n| Windows | `xcap` window capture |\nProtected or elevated windows can reject capture. Named-pipe IPC is local to the machine. |\n| Linux X11 | `xcap` window capture |\nThe application needs access to the active X session. |\n| Linux Wayland | compositor-dependent | Some compositors deny direct window capture or require portal consent. Treat failure as a platform limitation, not an empty screenshot. |\n\nThe pure coordinate, storage, schema, redaction, and protocol tests run without a desktop. Native screenshot behavior still needs platform runners or a real desktop session.\n\n```\ncargo fmt --all -- --check\ncargo test --workspace\ncargo clippy --workspace --all-targets --all-features -- -D warnings\npnpm check\npnpm test\npnpm build\npnpm e2e\n```\n\nThe test suite covers coordinate calibration and cropping, negative monitor coordinates, HiDPI scaling, page zoom, partial visibility, storage locking and cleanup, IDs, serialization, redaction, DOM and ARIA extraction, locator ranking, exact resolution, open shadow roots, picker cleanup and event suppression, Svelte metadata, local IPC, native screenshots, CLI JSON, and pixel-level crop equality.\n\n`pnpm e2e`\n\nstarts the Vite development server, builds a debug Tauri fixture with an embedded local WebDriver, drives a CLI `pick`\n\n, hovers and clicks the known button, checks source metadata and both PNGs, resolves the reference through the CLI, and shuts the app down. The WebDriver plugins are compiled and registered only by the fixture's `e2e`\n\nfeature.\n\nThe fixture page includes nested text, SVG, forms, a scroll boundary, fixed and absolute controls, transforms, CSS zoom, dialog, popover, tooltip, dropdown, canvas, WebGL, open shadow DOM, a tiny target, and a partially off-screen target.\n\nThe repository pins Rust, Node, pnpm, hk, and every lint/release tool through mise:\n\n```\nmise install\npnpm install\nhk install\nhk check --all\n```\n\nRelease Please keeps the three crates and five npm packages on one linked version. Its release PR updates manifests, lockfiles, and changelogs. Merging that PR creates the plugin's `v<version>`\n\ntag and component releases. The release workflow publishes crates in dependency order and publishes pnpm-built tarballs through npm trusted publishing. GitHub Actions are pinned to commit SHAs.\n\n`ui-inspector pick`\n\nsays the app is not running:\n\n- Run the command from the project tree or pass\n`--project`\n\n. - Confirm\n`.ui-inspector/run/instance.json`\n\nexists. - Confirm the Rust plugin and frontend bridge are both installed.\n- A stale discovery file is harmless; the CLI reports exit code 3 when its socket no longer exists.\n\nThe CLI waits until timeout:\n\n- Check the requested\n`--window`\n\nlabel. - Confirm the target window installed\n`installInspectorBridge`\n\n. - Make sure another pick or resolve operation is not active.\n\nSource metadata is missing:\n\n- Run the frontend through the Svelte/Vite development server.\n- Confirm\n`svelteAdapter()`\n\nis in the bridge's`adapters`\n\nlist. - Expect source metadata to be absent in production bundles.\n\nThe crop is offset:\n\n- Inspect\n`window.viewport`\n\n, Tauri geometry,`capture.screenshotSize`\n\n, and`capture.pixelCrop`\n\nin`reference.json`\n\n. - Record the display scale, page zoom, decoration size, and monitor coordinates.\n- Add the case to\n`crates/ui-inspector-core/tests/coordinate_transform.rs`\n\nbefore changing the transform.\n\nThe screenshot is denied or blank:\n\n- Grant macOS Screen Recording permission and restart the app.\n- Check Windows elevation and protected-window rules.\n- On Linux, confirm X11 access or the Wayland compositor's capture policy.\n\n[docs/architecture.md](/mathematic-inc/tauri-plugin-ui-inspector/blob/main/docs/architecture.md) records ownership boundaries, dependency choices, rejected alternatives, coordinate math, IPC security, and extension rules.\n\nSingle-element capture is complete. `ReferenceKind`\n\nreserves `group`\n\nand `region`\n\nso later schema versions can add Shift-click groups or arbitrary regions without replacing the top-level discriminator. Those modes are not exposed yet; adding them now would complicate the verified single-selection path without a working consumer.\n\nLicensed under either Apache-2.0 or MIT, at your option.", "url": "https://wpnews.pro/news/show-hn-ui-inspector-for-tauri", "canonical_source": "https://github.com/mathematic-inc/tauri-plugin-ui-inspector", "published_at": "2026-08-20 05:41:53+00:00", "updated_at": "2026-08-20 06:14:43.700090+00:00", "lang": "en", "topics": ["developer-tools", "ai-agents"], "entities": ["Tauri", "Svelte", "React", "Vue", "ui-inspector", "tauri-plugin-ui-inspector", "tauri-ui-inspector", "@tauri-ui-inspector/inspector"], "alternates": {"html": "https://wpnews.pro/news/show-hn-ui-inspector-for-tauri", "markdown": "https://wpnews.pro/news/show-hn-ui-inspector-for-tauri.md", "text": "https://wpnews.pro/news/show-hn-ui-inspector-for-tauri.txt", "jsonld": "https://wpnews.pro/news/show-hn-ui-inspector-for-tauri.jsonld"}}