{"slug": "deepsight-give-text-only-llms-eyes-and-hands-zero-tokens-on-device", "title": "DeepSight – give text-only LLMs eyes and hands (zero tokens, on-device)", "summary": "DeepSight, an open-source tool from Reality-Shifting-Tech, lets text-only LLMs like DeepSeek see and interact with the real world using zero tokens and on-device vision, supporting macOS via Apple Vision and Windows via PIL and optional Tesseract OCR. It installs as a loadable agent skill for Hermes, Claude Code, and Codex, and provides 16 tools for image description, OCR, desktop automation, and web search. The project is available on GitHub and can be set up with a single command.", "body_md": "Give DeepSeek (or any text-only model) eyes and hands.DeepSight connects your existing LLM setup to the real world — it can look at images you send, take screenshots of your desktop, read text on screen, click buttons, type into fields, open apps, and search the web to verify facts. All vision runs on-device: Apple Vision on macOS, PIL + optional Tesseract OCR on Windows. Zero tokens, zero GPU, no image data ever leaves your machine.\n\n```\npip install deepsight-vision\ndeepsight setup        # compiles the vision binary (macOS) or verifies env (Windows)\ndeepsight doctor       # confirms everything works\n```\n\nThe `setup`\n\ncommand compiles `vision_eyes`\n\nfrom Swift on macOS (Xcode SDK required), creates a `.env`\n\nwith defaults, and prints next steps. On Windows it detects the platform and uses PIL + optional Tesseract OCR. Or install from source with `uv sync`\n\n(see Agent Integration below).\n\nDeepSight ships as a **loadable agent skill**: procedural knowledge (setup, wiring, tool protocols, troubleshooting) that any agent can load on demand. Install it with one command — no user steps:\n\n```\ncurl -fsSL https://raw.githubusercontent.com/Reality-Shifting-Tech/deepsight/main/scripts/install-skill.sh | bash\n```\n\nThat's it. The script detects the agent and installs into the right skills directory — Hermes (`~/.hermes/skills/`\n\n), Claude Code (`~/.claude/skills/`\n\n), Codex (`~/.codex/skills/`\n\n) — or an explicit `DEEPSIGHT_SKILLS_DIR`\n\n. No accounts, no API keys, no configuration required for the skill itself. A previous copy is preserved as `deepsight.old`\n\n, never deleted.\n\nAfter it runs, the agent should load the skill and follow its instructions (Hermes: `skill_view(name='deepsight')`\n\n; any agent: read the installed `SKILL.md`\n\n). It covers compiling the vision binary, reasoning-loop wiring, the 16-tool protocol, `auxiliary.vision`\n\nintegration, and troubleshooting.\n\nSkill source of truth: `skill/deepsight/`\n\nin this repo. Offline install from a checkout: `cp -R skill/deepsight ~/.hermes/skills/`\n\n.\n\nOne command to set up deepsight from scratch:\n\n```\ngit clone https://github.com/Reality-Shifting-Tech/deepsight.git\ncd deepsight\nuv sync\nuv run deepsight setup        # compiles binary (macOS) or verifies env (Windows)\nuv run deepsight doctor        # confirms everything works\n```\n\nThat's it. The `setup`\n\ncommand compiles the vision binary (macOS), creates a `.env`\n\nfile with defaults, and prints next steps. For Windows, it detects the platform automatically and uses PIL + optional Tesseract OCR instead.\n\n**Python integration (for agents):**\n\n``` python\nfrom deepsight.backends import NativeVisionBackend, ReasoningBackend, \\\n    ComputerUseBackend, SearchBackend\nfrom deepsight.orchestrator import Orchestrator\nfrom deepsight.config import get_settings\n\nsettings = get_settings()\nvision = NativeVisionBackend(bin_path=settings.vision_bin)\nreasoning = ReasoningBackend(\n    base_url=settings.reasoning_base_url,\n    api_key=settings.reasoning_api_key,\n    model=settings.reasoning_model,\n)\n\n# Vision-only session\nagent = Orchestrator(vision=vision, reasoning=reasoning)\nresult = agent.run(\"data:image/png;base64,...\", \"Describe this image\")\n\n# With desktop automation (macOS) or Windows automation\nfrom deepsight.backends import ComputerUseBackend\nagent = Orchestrator(\n    vision=vision, reasoning=reasoning,\n    computer=ComputerUseBackend(),\n)\nresult = agent.run(\"data:image/png;base64,...\",\n    \"Open Terminal, run 'npm run dev', capture the result\")\n```\n\nOn Windows, use `WindowsVisionBackend`\n\ninstead of `NativeVisionBackend`\n\n:\n\n``` python\nfrom deepsight.backends import WindowsVisionBackend\nvision = WindowsVisionBackend()\n```\n\nDeepSight exposes **16 tools** to the reasoning model, organized into four layers.\n\n| Tool | What it does |\n|---|---|\n`look` |\nDescribe a rectangular region of the image |\n`ocr` |\nTranscribe all text in a region, exactly as written |\n`zoom` |\nZoom into a region for small-detail inspection |\n`count` |\nCount objects matching a description in a region |\n`locate` |\nFind an object by description and return its bounding box (x%, y%, w%, h%) |\n\nAll vision tools are zero-token — they use Apple Vision on macOS (via a compiled Swift binary) or PIL + optional Tesseract OCR on Windows. No network, no GPU, no API calls.\n\n| Tool | What it does |\n|---|---|\n`capture` |\nScreenshot the screen (or a specific window) and analyze it with the full vision pipeline |\n`watch` |\nMonitor the screen over time — captures at an interval, uses perceptual hashing to skip identical frames, returns a timeline of changes. Optional `until` param stops when target text appears |\n\nAfter `capture`\n\n, all subsequent vision tools operate on the captured screen. The model can capture, inspect, act, then capture again.\n\n| Tool | What it does |\n|---|---|\n`ground` |\nSearch the web for a claim or entity, fetch the top result, and return a verification summary with citations |\n\nPowered by Brave Search. Gated by `DEEPSIGHT_SEARCH_API_KEY`\n\n— degrades gracefully when unset.\n\n| Tool | What it does |\n|---|---|\n`click` |\nClick at a position (x%, y% — matches locate output) |\n`type` |\nType text into the focused input field |\n`key` |\nPress keyboard shortcuts (`cmd+s` , `return` , `escape` , `ctrl+c` ) |\n`scroll` |\nScroll the active window (direction, clicks) |\n`open` |\nLaunch or activate an application by name |\n`focus` |\nBring a window to front by matching its title |\n`apps` |\nList all visible applications and their window titles |\n`window` |\nResize or reposition a window using % screen coordinates |\n\nAction tools use macOS `osascript`\n\n(built-in) or `cliclick`\n\n(recommended: `brew install cliclick`\n\n). Requires Accessibility permission in System Settings.\n\n``` python\nfrom deepsight.orchestrator import Orchestrator\nfrom deepsight.backends import NativeVisionBackend, ReasoningBackend, ComputerUseBackend\n\n# Set up the agent\nvision = NativeVisionBackend(bin_path=\"vision_eyes\")\nreasoning = ReasoningBackend(\n    base_url=\"https://api.deepseek.com/v1\",\n    api_key=\"sk-...\",\n    model=\"deepseek-v4-flash\",\n)\nagent = Orchestrator(vision=vision, reasoning=reasoning, computer=ComputerUseBackend())\n\n# The model uses all 16 tools autonomously:\nagent.run(\n    image_url=\"data:image/png;base64,...\",\n    user_text=\"Open a terminal, create a new game project, build it, \"\n              \"then capture the result and tell me if it compiled.\",\n    response_format={\"type\": \"json_object\"},\n)\n```\n\nThe model will: open Terminal, type commands, capture the screen to check output, locate errors, fix them, rebuild, and report the result.\n\n```\ngit clone https://github.com/Reality-Shifting-Tech/deepsight.git\ncd deepsight\nuv sync\nmake build-eyes\nexport DEEPSIGHT_VISION_BIN=\"$PWD/scripts/vision_eyes\"\nuv run deepsight describe path/to/image.jpg\nuv run deepsight doctor\ngit clone https://github.com/Reality-Shifting-Tech/deepsight.git\ncd deepsight\nuv sync\nwinget install UB-Mannheim.TesseractOCR\nuv run python -m deepsight describe path/to/image.jpg\nuv run deepsight doctor\n```\n\nWindows uses `WindowsVisionBackend`\n\nfor PIL-based scene analysis (colors, brightness, texture) and optional Tesseract OCR. Desktop automation uses native Windows APIs (`user32.dll`\n\n, PowerShell SendKeys) — no additional tools to install.\n\n```\nuv run deepsight describe path/to/image.jpg\n```\n\nOutput: OCR text, scene classification, face/human/animal counts, detected sports, color palette, bounding boxes for every detected object.\n\n``` python\nfrom deepsight.backends import NativeVisionBackend, ReasoningBackend\nfrom deepsight.orchestrator import Orchestrator\n\nvision = NativeVisionBackend(bin_path=\"vision_eyes\")\nreasoning = ReasoningBackend(\n    base_url=\"https://api.deepseek.com/v1\",\n    api_key=\"sk-...\",\n    model=\"deepseek-v4-flash\",\n)\n\nsession = Orchestrator(vision=vision, reasoning=reasoning)\nresult = session.run(\n    image_url=\"https://example.com/screenshot.png\",\n    user_text=\"What's on the screen? Find any text and describe the layout.\",\n)\nprint(result.content)\n```\n\n**Reasoning model** receives the user's request plus tool definitions for all 16 tools.**Vision tools**(look, ocr, zoom, count, locate) route through the`Perception`\n\nmodule, which shells`vision_eyes`\n\n— the compiled Apple Vision binary — for zero-token analysis.**Live capture**(`capture`\n\n,`watch`\n\n) uses macOS`screencapture`\n\nto grab the screen, stores the result as the active image, and runs the full vision pipeline on it.**Action tools**(click, type, key, scroll, open, focus, apps, window) route through`ComputerUseBackend`\n\n, which uses macOS`osascript`\n\nor`cliclick`\n\nfor desktop automation.**Grounding**(`ground`\n\n) uses`SearchBackend`\n\nto search the web via Brave Search API.**Structured output**— pass`response_format`\n\nto get JSON-schema-constrained answers.**Cross-capture memory**— perceptual hashing (dhash) + OCR set diff tracks what changed between captures.** Perception cache**deduplicates repeated vision queries within a session.\n\nInspect a region of the image. All coordinates are percentages (0-100). Returns a description of what's there.\n\nTranscribe text in a region. Exact transcription including line breaks.\n\nUpscale and inspect a region for small details.\n\nCount objects matching a description. Pass a `what`\n\nstring like \"people\", \"red cars\", \"buttons\".\n\nFind an object by description. Returns normalized bounding box coordinates plus confidence. Uses Apple Vision's on-device detection (faces, humans, animals, text, rectangles, salient objects). Example: `locate(\"the login button\")`\n\nreturns `Login (85%): x=40% y=60% w=20% h=8%`\n\n.\n\nTake a screenshot. Optional `region`\n\n: `\"screen\"`\n\n(default) or a window title substring (e.g. `\"Terminal\"`\n\n, `\"Safari\"`\n\n). Returns a full scene analysis with OCR, detected objects, and changes since the last capture.\n\nMonitor the screen over time. Uses perceptual hashing to skip identical frames. Optional `until`\n\nstops early when text appears. Returns a timeline.\n\nSearch the web to verify a fact. Fetches the top result's page content for deep verification. Requires `DEEPSIGHT_SEARCH_API_KEY`\n\n.\n\nClick at screen position (percentages). Use after `locate`\n\nto click on a specific object. Requires Accessibility permission.\n\nType text into the currently focused input field.\n\nPress keyboard shortcuts: `\"cmd+s\"`\n\n, `\"return\"`\n\n, `\"escape\"`\n\n, `\"ctrl+c\"`\n\n, `\"tab\"`\n\n, `\"up\"`\n\n, `\"down\"`\n\n.\n\nScroll the active window.\n\nLaunch or activate an application: `\"Terminal\"`\n\n, `\"Safari\"`\n\n, `\"Xcode\"`\n\n, `\"Finder\"`\n\n.\n\nBring a window to front by title substring. Use `apps`\n\nfirst to see available windows.\n\nList all visible running applications and their window titles. Returns something like:\n\n```\nSafari (2 windows)\n  - DeepSight README — Edit\n  - GitHub — Pull Requests\nTerminal (1 window)\n  - bash — npm run build\n```\n\nResize or reposition a window. All values in % of screen. Example: `window(x=25, y=25, w=50, h=50)`\n\ncenters the window.\n\n``` python\nfrom deepsight.backends import NativeVisionBackend, ReasoningBackend, \\\n    ComputerUseBackend, SearchBackend\nfrom deepsight.orchestrator import Orchestrator\nfrom deepsight.config import get_settings\n\nsettings = get_settings()\n\nagent = Orchestrator(\n    vision=NativeVisionBackend(bin_path=settings.vision_bin),\n    reasoning=ReasoningBackend(\n        base_url=settings.reasoning_base_url,\n        api_key=settings.reasoning_api_key,\n        model=settings.reasoning_model,\n    ),\n    computer=ComputerUseBackend(),\n    search=SearchBackend(api_key=settings.search_key),\n)\n\nresult = agent.run(\n    image_url=\"data:image/png;base64,...\",\n    user_text=(\n        \"Open Terminal. Run 'npm run dev'. Wait for the dev server to start. \"\n        \"Capture the browser at localhost:5173. Describe what you see. \"\n        \"If there are errors, read them, fix the code, and try again. \"\n        \"Tell me when the app is running and what it looks like.\"\n    ),\n)\n```\n\nAll settings are environment variables (or a `.env`\n\nfile in the repo root). Variables use the `DEEPSIGHT_`\n\nprefix.\n\n| Variable | Default | Description |\n|---|---|---|\n`DEEPSIGHT_VISION_BIN` |\n`vision_eyes` |\nPath to the compiled Apple Vision binary |\n`DEEPSIGHT_REASONING_BASE_URL` |\n`https://api.deepseek.com/v1` |\nOpenAI-compatible chat endpoint |\n`DEEPSIGHT_REASONING_API_KEY` |\n(empty) |\nAPI key for the reasoning model |\n`DEEPSIGHT_REASONING_MODEL` |\n`deepseek-v4-flash` |\nModel name for the reasoning loop |\n`DEEPSIGHT_SEARCH_API_KEY` |\n(empty) |\nBrave Search API key for `ground` tool |\n`DEEPSIGHT_MAX_LOOK_ROUNDS` |\n`5` |\nMax tool rounds per vision session |\n`DEEPSIGHT_SKETCH_ENABLED` |\n`true` |\nInclude scene sketch in prompts |\n`DEEPSIGHT_CACHE_ENABLED` |\n`true` |\nCache repeated vision regions |\n`DEEPSIGHT_CACHE_TTL_SECONDS` |\n`3600` |\nPerception cache expiry |\n\n```\nmake test          # pytest (60+ tests, no macOS APIs)\nmake lint          # ruff (zero-warning policy)\nmake typecheck     # mypy\nmake all           # test + lint + typecheck\nmake build-eyes    # compile scripts/vision_eyes.swift\n```\n\nThe Swift source is at `scripts/vision_eyes.swift`\n\n; `make build-eyes`\n\nis the canonical compile. Tests must not require live model endpoints — backends are mocked.\n\n| Symptom | Fix |\n|---|---|\n`vision binary not found` (macOS) |\n`make build-eyes` , set `DEEPSIGHT_VISION_BIN` |\n| Vision tools return no results | Check `deepsight doctor` |\n| Action tools fail silently (macOS) | Grant Accessibility permission in System Settings |\n`click` / `type` don't work (macOS) |\n`brew install cliclick` for more reliable input |\n| Action tools fail on Windows | Run as normal user (not admin) — `user32.dll` calls work without elevation |\n`ground` returns unavailable |\nSet `DEEPSIGHT_SEARCH_API_KEY` (free at\n|\n`capture` returns empty (macOS) |\nGrant Screen Recording permission to Terminal |\n`capture` returns empty (Windows) |\nRuns as current user — `PIL.ImageGrab` needs a display session |\n| OCR not working (Windows) | Install Tesseract: `winget install UB-Mannheim.TesseractOCR` |\n| Compile fails: SDK not found | `xcode-select --install` |\n\nMIT — see [LICENSE](/Reality-Shifting-Tech/deepsight/blob/main/LICENSE). Built on Apple's Vision framework. Third-party notices in [THIRD_PARTY_NOTICES.md](/Reality-Shifting-Tech/deepsight/blob/main/THIRD_PARTY_NOTICES.md). Release history in [CHANGELOG.md](/Reality-Shifting-Tech/deepsight/blob/main/CHANGELOG.md).", "url": "https://wpnews.pro/news/deepsight-give-text-only-llms-eyes-and-hands-zero-tokens-on-device", "canonical_source": "https://github.com/Reality-Shifting-Tech/deepsight", "published_at": "2026-08-05 18:25:54+00:00", "updated_at": "2026-08-05 18:37:14.033477+00:00", "lang": "en", "topics": ["artificial-intelligence", "computer-vision", "ai-tools", "ai-agents", "developer-tools"], "entities": ["DeepSight", "Reality-Shifting-Tech", "DeepSeek", "Apple Vision", "Tesseract OCR", "Hermes", "Claude Code", "Codex"], "alternates": {"html": "https://wpnews.pro/news/deepsight-give-text-only-llms-eyes-and-hands-zero-tokens-on-device", "markdown": "https://wpnews.pro/news/deepsight-give-text-only-llms-eyes-and-hands-zero-tokens-on-device.md", "text": "https://wpnews.pro/news/deepsight-give-text-only-llms-eyes-and-hands-zero-tokens-on-device.txt", "jsonld": "https://wpnews.pro/news/deepsight-give-text-only-llms-eyes-and-hands-zero-tokens-on-device.jsonld"}}