Show HN: Local DirectML AI and Floating Toolstrip for PhotoCraft in Rust

A developer released PhotoCraft Companion, an open-source Rust plugin ecosystem that adds local DirectML GPU-accelerated AI tools and a floating egui toolstrip to the PhotoCraft graphics editor, working around the editor's WebAssembly plugin sandbox that blocks GPU access and large model storage. The companion runs neural models including Bria AI RMBG-1.4 (~168 MB) for background removal, LaMa for generative fill, and Real-ESRGAN Compact for 2x super-resolution, executing in place on the active canvas over a JSON-RPC socket on port 18492 with roughly 2-second inference and automatic CPU fallback on unsupported operators. Plugins ship as .pcpl packages with SHA-256-validated models stored under plugins/<id>/models/, and the project includes a one-click in-app marketplace with hot-reload and zero-code JSON macro authoring.

Extensible plugin ecosystem, local neural tools, and interactive overlay palette for the PhotoCraft graphics editor. Why Companion? -why-companion-advantages-over-built-in-photocraft-plugins • Features -key-features • Download & Installers -download--installers • First-Launch Guide -first-launch--onboarding-guide • Marketplace -community-plugins--marketplace • Quick Start -quick-start-for-developers • Architecture -workspace-architecture • .pcpl Packages -pcpl-package-format • Documentation -documentation • Discord -community--discord • License -license Tip 💬 Join the PhotoCraft Companion Discord: Have questions about plugins, ideas for new tools, or need developer assistance? Join our Discord Server https://discord.gg/2gzn4Un4Bw to connect directly with the author and community developers PhotoCraft Plugins & Companion solves the core challenge of seamlessly extending the PhotoCraft graphics editor a modern, open-source Rust-based reimplementation of Photoshop : - In-Place Canvas Execution: Neural background removal, color filters, and custom macro actions execute directly on the active layer of your open canvas over JSON-RPC socket port 18492 and the system clipboard — no manual saving, reopening, or file dialog gymnastics required. - Bypassing Sandbox Limitations: PhotoCraft natively confines internal plugins inside a strict WebAssembly sandbox without local GPU access or filesystem permissions. The companion provides native, hardware-accelerated DirectML GPU inference and local model storage. PhotoCraft includes a built-in plugin mechanism based on an internal WebAssembly Wasm runtime. While Wasm provides sandboxed safety, it imposes strict technical boundaries that make heavy AI workflows and floating palettes impractical. Here is how PhotoCraft Companion elevates the editing experience: | Feature | Built-in PhotoCraft Wasm Plugins | PhotoCraft Companion & Ecosystem | |---|---|---| | Local AI & Neural Models | ❌ No GPU access isolated within Wasm sandbox; cannot run DirectML/Tensor cores | ✅ Full DirectML GPU Acceleration DirectX 12 / Metal via ONNX Runtime; ~2s inference | | Model Storage & Autonomy | ❌ Strict memory limits; cannot package or download multi-hundred MB model weights | ✅ Autonomous Storage: Models live inside plugins/<id /models/ with SHA-256 validation & in-app downloader | | No-Code Macro Authoring | ❌ Requires Rust/C toolchains and manual WebAssembly compilation to .wasm | ✅ Zero-Code JSON: Just list desired PhotoCraft engine commands in plugin.json — ready in 60 seconds | | Floating Palette Toolstrip | ❌ Locked inside classic menu bars Filter › Plugins ; cannot render floating windows | ✅ Two-Tier Floating Toolstrip: Compact egui palette hovering over the canvas with orientation toggle & tray minimize | | In-App Plugin Marketplace | ❌ Manual file manipulation into application directories | ✅ 1-Click Marketplace: Browse online catalog, install, and hot-reload plugins directly from the UI without restarts | | In-Place Pixel Pipeline | | ✅ Zero-Drift In-Place Execution: Seamless round-trip active layer processing via JSON-RPC socket and clipboard | - ✂️ In-Place AI Tools & Neural Computer Vision Zero-Core Architecture : - AI Remove Background: Bria AI RMBG-1.4 ~168 MB high-precision subject cutout with coordinate preservation up to 8K. - AI Generative Fill LaMa : Context-aware zero-prompt object removal and texture inpainting with lasso contour synthesis. - AI Super-Resolution 2x : Real-ESRGAN Compact with seam-blended tiling to upscale arbitrarily large canvases without VRAM overflow. - Resilient AI Runtime: Hardware-accelerated DirectML DirectX 12 GPU with automatic transparent fallback to CPU on unsupported operators. - Ultra-Fast Cached GPU Tiling: Single-session GPU pipeline caching reducing large canvas inference times up to 25x. - Zero-Core Standard: Zero plugin-specific logic hardcoded into the companion core; every workflow is strictly declarative via plugin.json . - 📐 Photoshop-Style Tool Variations Sub-Actions & Flyouts : - Tools can declare multiple sub-actions e.g. Cutout to new layer vs Non-destructive Layer Mask . - Buttons render a subtle vector corner chevron; long-pressing 350ms or clicking the chevron opens a dedicated floating flyout window egui::ViewportBuilder with sub-tools. - Active sub-action selection is remembered in persistent user settings. - ⚡ Declarative Workflow Engine & Context Variables: - Multi-step pipelines with variable interpolation $source layer , $result layer , $active layer , $doc width , $doc height , $has selection , $fg color , $bg color , and string interpolation {{...}} . - Conditional branching when: "$has selection == true" , layer variable saving save as , error tolerance ignore errors , and sandboxed Rhai scripting. - Multi-step pipelines with variable interpolation - 🔄 Bidirectional Session Color Sync & Echo Guard: - Two-way real-time color syncing between PhotoCraft tools Eyedropper I , default D , swap X and interactive plugins. - Zero-waste polling 180ms only when windows are open with smart echo prevention echo guard . - Two-way real-time color syncing between PhotoCraft tools Eyedropper - 🎛️ Interactive Multi-Viewport UI: - Dedicated floating windows with customizable reactive widgets sliders, color swatches, hue wheel, contrast diagnostics, dropdowns, toggles running on Rhai scripting. - 🏪 In-App Community Plugin Marketplace: - Integrated 4th tab inside the Plugin Manager to discover and install community plugins directly from GitHub Releases. - Instant 1-click installation of lightweight .pcpl archives < 1 KB with reactive toolbar hot-reloading no restart required . - Background SVG icon rendering with dark and light theme adaptation. - Local offline registry caching for uninterrupted operation. - 🎛️ Floating Two-Tier Toolstrip egui / eframe : - Ultra-compact floating palette matching PhotoCraft's native UI aesthetics with pure vector SVG icons. - Orientation toggle: seamlessly switch between Horizontal 78 px height and Vertical 50 px width modes with zero button clipping. - Non-intrusive floating palette behavior: excluded from OS taskbar .with taskbar false , persistent drag & drop screen coordinates. - Floating mini-HUD status badge with live timer and busy lock. - ⚙️ Comprehensive Plugin Manager: - Tabbed interface: General , Plugins , Store , About . - Live connection indicator: 3-state real-time socket status ● Connected / ● Auth Required / ○ Offline with token browse dialog rfd . - Native themes: instant synchronization between Dark and pure Light modes. - In-App self-updater: checks for new GitHub releases, compares SemVer versions, and downloads platform installers with 1 click. - Tabbed interface: - 📦 100% Autonomous .pcpl Package Ecosystem: - Plugin neural weights are fully self-contained inside each plugin directory plugins/<id /models/ . - Model integrity verification, progress bars with live transfer speed MB/s , and automatic missing weights downloader directly in the UI. - Plugin neural weights are fully self-contained inside each plugin directory - ⏳ Resilient Waiting / Offline Mode: - The companion does not crash if PhotoCraft is not running; it displays a sleek standby window. - Background polling every 1.5s: when PhotoCraft starts, the toolbar smoothly appears above the canvas. - 🔔 Instant Reactive System Tray 0 ms : - System tray icon across Windows, Linux, and macOS with zero-latency dispatching and native Win32 dark context menu support. Ready-to-use binaries are built for every platform as both full installers and standalone portable archives : | Platform | Installer Setup | Portable Archive | Notes | |---|---|---|---| | Windows x64 | PhotoCraft-Companion-Setup-Windows-x64.exe | PhotoCraft-Companion-v1.2.2-windows-x64-portable.zip | Inno Setup installer with tray autostart / No-registry portable zip | | macOS Apple Silicon & Intel | PhotoCraft-Companion-macos.dmg | PhotoCraft-Companion-v1.2.2-macos-portable.tar.gz | Drag-and-drop DMG image for /Applications / Standalone .app | | Linux x86 64 | PhotoCraft-Companion-v1.2.2-linux-amd64.deb | PhotoCraft-Companion-v1.2.2-linux-x86 64-portable.tar.gz | Debian package with desktop icon & app menu / Standalone tar.gz | Tip All release binaries are published on the GitHub Releases https://github.com/kobaltgit/photocraft-companion/releases page. Note macOS Gatekeeper Note: If macOS displays a warning saying "The application is damaged and can't be opened" or blocks unsigned open-source applications, open Terminal and remove the quarantine attribute: xattr -cr "/Applications/PhotoCraft Companion.app" Alternatively, go to System Settings › Privacy & Security and select Open Anyway . PhotoCraft Companion communicates with PhotoCraft through a local control socket port 18492 using a security token control.token . Starting with v1.0.7 , launching and pairing are completely automated in one click 1. Start PhotoCraft Companion : - If PhotoCraft is not running yet, Companion displays a sleek standby window: "Waiting for PhotoCraft..." . 2. If PhotoCraft is not running yet, Companion displays a sleek standby window: 3. Launch in 1 Click : - Click the Launch PhotoCraft button directly in the waiting window or in Settings › General . - Companion automatically discovers photocraft.exe across standard locations or you can point to it once via Browse... , generates a cryptographically secure 64-character control.token , launches PhotoCraft with --control 18492 --control-token-file , and establishes the connection. - The configured executable path is safely persisted and mirrored in portable mode. 4. Click the 5. Start Creating : - Once PhotoCraft is running, the waiting window seamlessly transforms into the floating toolbar above your canvas. If you prefer starting PhotoCraft from custom shortcuts or command line: 1. Launch PhotoCraft with the control socket arguments: photocraft.exe --control 18492 --control-token-file "PhotoCraftData/control.token" 2. Start photocraft-companion.exe . It automatically detects the running process, reads control.token , and turns the connection indicator to ● Connected . - Windows: .\scripts\package release.ps1 Creates a portable .zip in dist/ and builds the Inno Setup installer if ISCC is installed . - Linux / macOS: chmod +x ./scripts/package release.sh ./scripts/package release.sh - OS: Windows 10 / 11 x64 , Linux X11 / Wayland , or macOS - Rust: Toolchain 1.80+ 2021 edition - GPU optional : Any DirectX 12 capable GPU NVIDIA, AMD, Intel for DirectML acceleration. git clone https://github.com/kobaltgit/photocraft-companion.git cd photocraft-companion Model weights are excluded from Git .gitignore to keep the repository lightweight. Run the download script to fetch RMBG-1.4: .\scripts\fetch models.ps1 Alternatively, click ⬇ Download Model inside the Plugin Manager UI on first run . Launch the companion application cargo run --package companion app To maintain uncompromising reliability and ensure the Zero-Core architecture remains free of hacks, execute our automated test barriers: Note Resource Guard Rule: Always execute cargo commands with the -j 1 single-thread flag to prevent CPU and RAM starvation on developer machines. 1. Code Formatting Zero-tolerance style check cargo fmt --all -- --check 2. Complete Workspace Test Suite 152 unit & integration tests cargo test --workspace -j 1 3. Core Architecture Integrity Linter Strict Zero-Core & anti-sniffing barrier cargo test -p ai engine --test architecture integrity test -j 1 4. Dev Regulations Linter 5-file SemVer sync, crate isolation, chronicle parity, model isolation cargo test -p ai engine --test dev regulations lint test -j 1 5. UI Linters Strict i18n localization & Vector-First UI / anti-tofu barrier cargo test -p overlay ui --test i18n lint test -j 1 cargo test -p overlay ui --test vector first lint test -j 1 6. Ecosystem Plugin & Registry Validator Validates manifests, vector SVG icons & model links cargo run -p plugin validator -j 1 -- --validate plugins The project is structured as a modular Cargo Workspace with strict separation of concerns: photocraft-plugins/ ├── Cargo.toml Root Rust Workspace manifest ├── crates/ Independent crates │ ├── core contracts/ Plugin traits, DTOs, Config, i18n, .pcpl installer │ ├── photocraft bridge/ Async PhotoCraft socket client JSON-RPC 18492 │ ├── ai engine/ DirectML GPU ONNX runtime inference RMBG, Real-ESRGAN, LaMa │ ├── overlay ui/ Floating Toolstrip, In-App Store, Plugin Manager egui 0.36 │ ├── plugin validator/ Ecosystem CLI: validation, packaging, marketplace generator │ └── companion app/ Main binary system tray, lifecycle, DI container, workflow engine ├── plugins/ Installed plugin directory │ ├── black and white/ Declarative No-Code macro plugin │ ├── chromasense/ Interactive UI A11y color panel Rhai, WCAG 2.1, CVD simulation │ ├── depth map/ Autonomous AI 3D depth estimation plugin Depth Anything V2 │ ├── inpaint/ Autonomous AI generative fill plugin LaMa │ ├── remove bg/ Autonomous AI background removal plugin RMBG-1.4, Sub-Actions │ ├── upscale/ Autonomous AI 2x super-resolution plugin Real-ESRGAN │ └── vectorize/ Autonomous Smart Vectorizer / SVG Trace algorithm plugin ├── scripts/ │ ├── fetch models.ps1 PowerShell model downloader │ ├── package release.ps1 Inno Setup & Windows packaging script │ └── package release.sh Linux DEB & macOS DMG packaging script └── docs/ Engineering specifications & dev chronicles Any plugin can be distributed as a portable .pcpl archive ZIP format : my plugin.pcpl ├── plugin.json Declarative manifest metadata, commands, hotkeys ├── icon.svg Vector palette icon └── models/ Optional Local ONNX neural weights └── rmbg-1.4.onnx Install any .pcpl file with a single click via + Install Plugin .pcpl inside the Plugin Manager. Download community plugins as ready-to-use .pcpl packages and drop them into your Companion or install them in 1-click via the Plugin Manager : | Icon | Plugin & Description | Type | Version | Author | Package .pcpl | |---|---|---|---|---|---| | | B&W Film Pop No-Code Duplicates active layer, converts to grayscale with auto-contrast adjustment. | ⚡ Macro | 1.0.0 | kobaltGIT | 📥 Download .pcpl https://github.com/kobaltgit/photocraft-companion/releases/download/plugins/black and white.pcpl | | | ChromaSense Accessibility Color Assistant for artists with color vision deficiencies | 🎨 UI Panel | 1.0.0 | kobaltGIT | 📥 Download .pcpl https://github.com/kobaltgit/photocraft-companion/releases/download/plugins/chromasense.pcpl | | | AI Depth Map Depth Anything V2 State-of-the-art monocular 3D depth estimation for Lens Blur, bokeh and atmospheric lighting via Depth Anything V2. | 🤖 AI ONNX | 1.0.0 | kobaltGIT | 📥 Download .pcpl https://github.com/kobaltgit/photocraft-companion/releases/download/plugins/depth map.pcpl | | | AI Generative Fill LaMa Zero-prompt neural generative fill and object eraser for PhotoCraft using LaMa model via DirectML GPU. | 🤖 AI ONNX | 1.0.0 | kobaltGIT | 📥 Download .pcpl https://github.com/kobaltgit/photocraft-companion/releases/download/plugins/inpaint.pcpl | | | AI Remove Background RMBG-1.4 High-fidelity neural background removal for PhotoCraft using Bria AI RMBG-1.4 model via local ONNX Runtime. | 🤖 AI ONNX | 1.1.0 | kobaltGIT | 📥 Download .pcpl https://github.com/kobaltgit/photocraft-companion/releases/download/plugins/remove bg.pcpl | | | AI Super-Resolution 2x 2x neural super-resolution upscale for PhotoCraft active layer via local ONNX Runtime & DirectML GPU. | 🤖 AI ONNX | 1.0.0 | kobaltGIT | 📥 Download .pcpl https://github.com/kobaltgit/photocraft-companion/releases/download/plugins/upscale.pcpl | | | Smart Vectorizer / SVG Trace High-fidelity vectorization and SVG tracing of raster layers with Bezier curve fitting. | 📐 Algorithm | 1.0.0 | kobaltGIT | 📥 Download .pcpl https://github.com/kobaltgit/photocraft-companion/releases/download/plugins/vectorize.pcpl | Tip Want to publish your own plugin? Check out the Plugin Development Guide https://github.com/kobaltgit/photocraft-companion/blob/main/docs/PLUGIN DEVELOPMENT GUIDE.en.md , create your plugin inside plugins/<id / , and submit a Pull Request Our automated CI validates your manifest, icon, and neural models before publishing. Detailed specifications and development history can be found in docs/ https://github.com/kobaltgit/photocraft-companion/blob/main/docs : - 📘 GEMINI.md https://github.com/kobaltgit/photocraft-companion/blob/main/GEMINI.md — Master architectural manifest. - 📋 docs/AGENTS MASTER GUIDE.md https://github.com/kobaltgit/photocraft-companion/blob/main/docs/AGENTS MASTER GUIDE.md — Master development regulations and AI agent guidelines. - 🎯 docs/PLAN.md https://github.com/kobaltgit/photocraft-companion/blob/main/docs/PLAN.md — Implementation blueprint for floating UI and autonomous plugins. - 🗺️ docs/ROADMAP.md https://github.com/kobaltgit/photocraft-companion/blob/main/docs/ROADMAP.md — Project roadmap and milestones. - ✅ docs/CHECKLIST.md https://github.com/kobaltgit/photocraft-companion/blob/main/docs/CHECKLIST.md — Interactive phase-by-phase checklist. - 🐛 docs/BUGS AND ISSUES.md https://github.com/kobaltgit/photocraft-companion/blob/main/docs/BUGS AND ISSUES.md — Defect tracking log BUG-001 through BUG-075 . - 📜 docs/CHRONICLE.md https://github.com/kobaltgit/photocraft-companion/blob/main/docs/CHRONICLE.md / docs/CHRONICLE.en.md https://github.com/kobaltgit/photocraft-companion/blob/main/docs/CHRONICLE.en.md — Chronological developer devlog in Russian and English. - 🛠️ docs/PLUGIN DEVELOPMENT GUIDE.en.md https://github.com/kobaltgit/photocraft-companion/blob/main/docs/PLUGIN DEVELOPMENT GUIDE.en.md / RU https://github.com/kobaltgit/photocraft-companion/blob/main/docs/PLUGIN DEVELOPMENT GUIDE.md — Plugin Developer Guide No-Code macros, AI inference, workflows . - 📖 docs/PHOTOCRAFT COMMANDS REFERENCE.en.md https://github.com/kobaltgit/photocraft-companion/blob/main/docs/PHOTOCRAFT COMMANDS REFERENCE.en.md / RU https://github.com/kobaltgit/photocraft-companion/blob/main/docs/PHOTOCRAFT COMMANDS REFERENCE.md — Complete Catalog of all 851 PhotoCraft Engine Commands. - 📐 docs/UNIVERSAL PLUGIN CORE SPECIFICATION.md https://github.com/kobaltgit/photocraft-companion/blob/main/docs/UNIVERSAL PLUGIN CORE SPECIFICATION.md — Universal Core specification Rhai, CLI runner, C-ABI dynamic libs . Have questions, suggestions, or want to share your custom plugins with fellow digital artists? Join the official PhotoCraft Companion Discord Server : kobaltGIT GitHub https://github.com/kobaltgit This project is licensed under the MIT License https://github.com/kobaltgit/photocraft-companion/blob/main/LICENSE .