DeepSight – give text-only LLMs eyes and hands (zero tokens, on-device)

DeepSight, an open-source tool from Reality-Shifting-Tech, lets text-only LLMs like DeepSeek see and interact with the real world using zero tokens and on-device vision, supporting macOS via Apple Vision and Windows via PIL and optional Tesseract OCR. It installs as a loadable agent skill for Hermes, Claude Code, and Codex, and provides 16 tools for image description, OCR, desktop automation, and web search. The project is available on GitHub and can be set up with a single command.

Give DeepSeek or any text-only model eyes and hands.DeepSight connects your existing LLM setup to the real world — it can look at images you send, take screenshots of your desktop, read text on screen, click buttons, type into fields, open apps, and search the web to verify facts. All vision runs on-device: Apple Vision on macOS, PIL + optional Tesseract OCR on Windows. Zero tokens, zero GPU, no image data ever leaves your machine. pip install deepsight-vision deepsight setup compiles the vision binary macOS or verifies env Windows deepsight doctor confirms everything works The setup command compiles vision eyes from Swift on macOS Xcode SDK required , creates a .env with defaults, and prints next steps. On Windows it detects the platform and uses PIL + optional Tesseract OCR. Or install from source with uv sync see Agent Integration below . DeepSight ships as a loadable agent skill : procedural knowledge setup, wiring, tool protocols, troubleshooting that any agent can load on demand. Install it with one command — no user steps: curl -fsSL https://raw.githubusercontent.com/Reality-Shifting-Tech/deepsight/main/scripts/install-skill.sh | bash That's it. The script detects the agent and installs into the right skills directory — Hermes ~/.hermes/skills/ , Claude Code ~/.claude/skills/ , Codex ~/.codex/skills/ — or an explicit DEEPSIGHT SKILLS DIR . No accounts, no API keys, no configuration required for the skill itself. A previous copy is preserved as deepsight.old , never deleted. After it runs, the agent should load the skill and follow its instructions Hermes: skill view name='deepsight' ; any agent: read the installed SKILL.md . It covers compiling the vision binary, reasoning-loop wiring, the 16-tool protocol, auxiliary.vision integration, and troubleshooting. Skill source of truth: skill/deepsight/ in this repo. Offline install from a checkout: cp -R skill/deepsight ~/.hermes/skills/ . One command to set up deepsight from scratch: git clone https://github.com/Reality-Shifting-Tech/deepsight.git cd deepsight uv sync uv run deepsight setup compiles binary macOS or verifies env Windows uv run deepsight doctor confirms everything works That's it. The setup command compiles the vision binary macOS , creates a .env file with defaults, and prints next steps. For Windows, it detects the platform automatically and uses PIL + optional Tesseract OCR instead. Python integration for agents : python from deepsight.backends import NativeVisionBackend, ReasoningBackend, \ ComputerUseBackend, SearchBackend from deepsight.orchestrator import Orchestrator from deepsight.config import get settings settings = get settings vision = NativeVisionBackend bin path=settings.vision bin reasoning = ReasoningBackend base url=settings.reasoning base url, api key=settings.reasoning api key, model=settings.reasoning model, Vision-only session agent = Orchestrator vision=vision, reasoning=reasoning result = agent.run "data:image/png;base64,...", "Describe this image" With desktop automation macOS or Windows automation from deepsight.backends import ComputerUseBackend agent = Orchestrator vision=vision, reasoning=reasoning, computer=ComputerUseBackend , result = agent.run "data:image/png;base64,...", "Open Terminal, run 'npm run dev', capture the result" On Windows, use WindowsVisionBackend instead of NativeVisionBackend : python from deepsight.backends import WindowsVisionBackend vision = WindowsVisionBackend DeepSight exposes 16 tools to the reasoning model, organized into four layers. | Tool | What it does | |---|---| look | Describe a rectangular region of the image | ocr | Transcribe all text in a region, exactly as written | zoom | Zoom into a region for small-detail inspection | count | Count objects matching a description in a region | locate | Find an object by description and return its bounding box x%, y%, w%, h% | All vision tools are zero-token — they use Apple Vision on macOS via a compiled Swift binary or PIL + optional Tesseract OCR on Windows. No network, no GPU, no API calls. | Tool | What it does | |---|---| capture | Screenshot the screen or a specific window and analyze it with the full vision pipeline | watch | Monitor the screen over time — captures at an interval, uses perceptual hashing to skip identical frames, returns a timeline of changes. Optional until param stops when target text appears | After capture , all subsequent vision tools operate on the captured screen. The model can capture, inspect, act, then capture again. | Tool | What it does | |---|---| ground | Search the web for a claim or entity, fetch the top result, and return a verification summary with citations | Powered by Brave Search. Gated by DEEPSIGHT SEARCH API KEY — degrades gracefully when unset. | Tool | What it does | |---|---| click | Click at a position x%, y% — matches locate output | type | Type text into the focused input field | key | Press keyboard shortcuts cmd+s , return , escape , ctrl+c | scroll | Scroll the active window direction, clicks | open | Launch or activate an application by name | focus | Bring a window to front by matching its title | apps | List all visible applications and their window titles | window | Resize or reposition a window using % screen coordinates | Action tools use macOS osascript built-in or cliclick recommended: brew install cliclick . Requires Accessibility permission in System Settings. python from deepsight.orchestrator import Orchestrator from deepsight.backends import NativeVisionBackend, ReasoningBackend, ComputerUseBackend Set up the agent vision = NativeVisionBackend bin path="vision eyes" reasoning = ReasoningBackend base url="https://api.deepseek.com/v1", api key="sk-...", model="deepseek-v4-flash", agent = Orchestrator vision=vision, reasoning=reasoning, computer=ComputerUseBackend The model uses all 16 tools autonomously: agent.run image url="data:image/png;base64,...", user text="Open a terminal, create a new game project, build it, " "then capture the result and tell me if it compiled.", response format={"type": "json object"}, The model will: open Terminal, type commands, capture the screen to check output, locate errors, fix them, rebuild, and report the result. git clone https://github.com/Reality-Shifting-Tech/deepsight.git cd deepsight uv sync make build-eyes export DEEPSIGHT VISION BIN="$PWD/scripts/vision eyes" uv run deepsight describe path/to/image.jpg uv run deepsight doctor git clone https://github.com/Reality-Shifting-Tech/deepsight.git cd deepsight uv sync winget install UB-Mannheim.TesseractOCR uv run python -m deepsight describe path/to/image.jpg uv run deepsight doctor Windows uses WindowsVisionBackend for PIL-based scene analysis colors, brightness, texture and optional Tesseract OCR. Desktop automation uses native Windows APIs user32.dll , PowerShell SendKeys — no additional tools to install. uv run deepsight describe path/to/image.jpg Output: OCR text, scene classification, face/human/animal counts, detected sports, color palette, bounding boxes for every detected object. python from deepsight.backends import NativeVisionBackend, ReasoningBackend from deepsight.orchestrator import Orchestrator vision = NativeVisionBackend bin path="vision eyes" reasoning = ReasoningBackend base url="https://api.deepseek.com/v1", api key="sk-...", model="deepseek-v4-flash", session = Orchestrator vision=vision, reasoning=reasoning result = session.run image url="https://example.com/screenshot.png", user text="What's on the screen? Find any text and describe the layout.", print result.content Reasoning model receives the user's request plus tool definitions for all 16 tools. Vision tools look, ocr, zoom, count, locate route through the Perception module, which shells vision eyes — the compiled Apple Vision binary — for zero-token analysis. Live capture capture , watch uses macOS screencapture to grab the screen, stores the result as the active image, and runs the full vision pipeline on it. Action tools click, type, key, scroll, open, focus, apps, window route through ComputerUseBackend , which uses macOS osascript or cliclick for desktop automation. Grounding ground uses SearchBackend to search the web via Brave Search API. Structured output — pass response format to get JSON-schema-constrained answers. Cross-capture memory — perceptual hashing dhash + OCR set diff tracks what changed between captures. Perception cache deduplicates repeated vision queries within a session. Inspect a region of the image. All coordinates are percentages 0-100 . Returns a description of what's there. Transcribe text in a region. Exact transcription including line breaks. Upscale and inspect a region for small details. Count objects matching a description. Pass a what string like "people", "red cars", "buttons". Find an object by description. Returns normalized bounding box coordinates plus confidence. Uses Apple Vision's on-device detection faces, humans, animals, text, rectangles, salient objects . Example: locate "the login button" returns Login 85% : x=40% y=60% w=20% h=8% . Take a screenshot. Optional region : "screen" default or a window title substring e.g. "Terminal" , "Safari" . Returns a full scene analysis with OCR, detected objects, and changes since the last capture. Monitor the screen over time. Uses perceptual hashing to skip identical frames. Optional until stops early when text appears. Returns a timeline. Search the web to verify a fact. Fetches the top result's page content for deep verification. Requires DEEPSIGHT SEARCH API KEY . Click at screen position percentages . Use after locate to click on a specific object. Requires Accessibility permission. Type text into the currently focused input field. Press keyboard shortcuts: "cmd+s" , "return" , "escape" , "ctrl+c" , "tab" , "up" , "down" . Scroll the active window. Launch or activate an application: "Terminal" , "Safari" , "Xcode" , "Finder" . Bring a window to front by title substring. Use apps first to see available windows. List all visible running applications and their window titles. Returns something like: Safari 2 windows - DeepSight README — Edit - GitHub — Pull Requests Terminal 1 window - bash — npm run build Resize or reposition a window. All values in % of screen. Example: window x=25, y=25, w=50, h=50 centers the window. python from deepsight.backends import NativeVisionBackend, ReasoningBackend, \ ComputerUseBackend, SearchBackend from deepsight.orchestrator import Orchestrator from deepsight.config import get settings settings = get settings agent = Orchestrator vision=NativeVisionBackend bin path=settings.vision bin , reasoning=ReasoningBackend base url=settings.reasoning base url, api key=settings.reasoning api key, model=settings.reasoning model, , computer=ComputerUseBackend , search=SearchBackend api key=settings.search key , result = agent.run image url="data:image/png;base64,...", user text= "Open Terminal. Run 'npm run dev'. Wait for the dev server to start. " "Capture the browser at localhost:5173. Describe what you see. " "If there are errors, read them, fix the code, and try again. " "Tell me when the app is running and what it looks like." , All settings are environment variables or a .env file in the repo root . Variables use the DEEPSIGHT prefix. | Variable | Default | Description | |---|---|---| DEEPSIGHT VISION BIN | vision eyes | Path to the compiled Apple Vision binary | DEEPSIGHT REASONING BASE URL | https://api.deepseek.com/v1 | OpenAI-compatible chat endpoint | DEEPSIGHT REASONING API KEY | empty | API key for the reasoning model | DEEPSIGHT REASONING MODEL | deepseek-v4-flash | Model name for the reasoning loop | DEEPSIGHT SEARCH API KEY | empty | Brave Search API key for ground tool | DEEPSIGHT MAX LOOK ROUNDS | 5 | Max tool rounds per vision session | DEEPSIGHT SKETCH ENABLED | true | Include scene sketch in prompts | DEEPSIGHT CACHE ENABLED | true | Cache repeated vision regions | DEEPSIGHT CACHE TTL SECONDS | 3600 | Perception cache expiry | make test pytest 60+ tests, no macOS APIs make lint ruff zero-warning policy make typecheck mypy make all test + lint + typecheck make build-eyes compile scripts/vision eyes.swift The Swift source is at scripts/vision eyes.swift ; make build-eyes is the canonical compile. Tests must not require live model endpoints — backends are mocked. | Symptom | Fix | |---|---| vision binary not found macOS | make build-eyes , set DEEPSIGHT VISION BIN | | Vision tools return no results | Check deepsight doctor | | Action tools fail silently macOS | Grant Accessibility permission in System Settings | click / type don't work macOS | brew install cliclick for more reliable input | | Action tools fail on Windows | Run as normal user not admin — user32.dll calls work without elevation | ground returns unavailable | Set DEEPSIGHT SEARCH API KEY free at | capture returns empty macOS | Grant Screen Recording permission to Terminal | capture returns empty Windows | Runs as current user — PIL.ImageGrab needs a display session | | OCR not working Windows | Install Tesseract: winget install UB-Mannheim.TesseractOCR | | Compile fails: SDK not found | xcode-select --install | MIT — see LICENSE /Reality-Shifting-Tech/deepsight/blob/main/LICENSE . Built on Apple's Vision framework. Third-party notices in THIRD PARTY NOTICES.md /Reality-Shifting-Tech/deepsight/blob/main/THIRD PARTY NOTICES.md . Release history in CHANGELOG.md /Reality-Shifting-Tech/deepsight/blob/main/CHANGELOG.md .