Give DeepSeek (or any text-only model) eyes and hands.DeepSight connects your existing LLM setup to the real world — it can look at images you send, take screenshots of your desktop, read text on screen, click buttons, type into fields, open apps, and search the web to verify facts. All vision runs on-device: Apple Vision on macOS, PIL + optional Tesseract OCR on Windows. Zero tokens, zero GPU, no image data ever leaves your machine.
pip install deepsight-vision
deepsight setup # compiles the vision binary (macOS) or verifies env (Windows)
deepsight doctor # confirms everything works
The setup
command compiles vision_eyes
from Swift on macOS (Xcode SDK required), creates a .env
with defaults, and prints next steps. On Windows it detects the platform and uses PIL + optional Tesseract OCR. Or install from source with uv sync
(see Agent Integration below).
DeepSight ships as a loadable agent skill: procedural knowledge (setup, wiring, tool protocols, troubleshooting) that any agent can load on demand. Install it with one command — no user steps:
curl -fsSL https://raw.githubusercontent.com/Reality-Shifting-Tech/deepsight/main/scripts/install-skill.sh | bash
That's it. The script detects the agent and installs into the right skills directory — Hermes (~/.hermes/skills/
), Claude Code (~/.claude/skills/
), Codex (~/.codex/skills/
) — or an explicit DEEPSIGHT_SKILLS_DIR
. No accounts, no API keys, no configuration required for the skill itself. A previous copy is preserved as deepsight.old
, never deleted.
After it runs, the agent should load the skill and follow its instructions (Hermes: skill_view(name='deepsight')
; any agent: read the installed SKILL.md
). It covers compiling the vision binary, reasoning-loop wiring, the 16-tool protocol, auxiliary.vision
integration, and troubleshooting.
Skill source of truth: skill/deepsight/
in this repo. Offline install from a checkout: cp -R skill/deepsight ~/.hermes/skills/
.
One command to set up deepsight from scratch:
git clone https://github.com/Reality-Shifting-Tech/deepsight.git
cd deepsight
uv sync
uv run deepsight setup # compiles binary (macOS) or verifies env (Windows)
uv run deepsight doctor # confirms everything works
That's it. The setup
command compiles the vision binary (macOS), creates a .env
file with defaults, and prints next steps. For Windows, it detects the platform automatically and uses PIL + optional Tesseract OCR instead.
Python integration (for agents):
from deepsight.backends import NativeVisionBackend, ReasoningBackend, \
ComputerUseBackend, SearchBackend
from deepsight.orchestrator import Orchestrator
from deepsight.config import get_settings
settings = get_settings()
vision = NativeVisionBackend(bin_path=settings.vision_bin)
reasoning = ReasoningBackend(
base_url=settings.reasoning_base_url,
api_key=settings.reasoning_api_key,
model=settings.reasoning_model,
)
agent = Orchestrator(vision=vision, reasoning=reasoning)
result = agent.run("data:image/png;base64,...", "Describe this image")
from deepsight.backends import ComputerUseBackend
agent = Orchestrator(
vision=vision, reasoning=reasoning,
computer=ComputerUseBackend(),
)
result = agent.run("data:image/png;base64,...",
"Open Terminal, run 'npm run dev', capture the result")
On Windows, use WindowsVisionBackend
instead of NativeVisionBackend
:
from deepsight.backends import WindowsVisionBackend
vision = WindowsVisionBackend()
DeepSight exposes 16 tools to the reasoning model, organized into four layers.
| Tool | What it does |
|---|---|
look |
|
| Describe a rectangular region of the image | |
ocr |
|
| Transcribe all text in a region, exactly as written | |
zoom |
|
| Zoom into a region for small-detail inspection | |
count |
|
| Count objects matching a description in a region | |
locate |
|
| Find an object by description and return its bounding box (x%, y%, w%, h%) |
All vision tools are zero-token — they use Apple Vision on macOS (via a compiled Swift binary) or PIL + optional Tesseract OCR on Windows. No network, no GPU, no API calls.
| Tool | What it does |
|---|---|
capture |
|
| Screenshot the screen (or a specific window) and analyze it with the full vision pipeline | |
watch |
|
Monitor the screen over time — captures at an interval, uses perceptual hashing to skip identical frames, returns a timeline of changes. Optional until param stops when target text appears |
After capture
, all subsequent vision tools operate on the captured screen. The model can capture, inspect, act, then capture again.
| Tool | What it does |
|---|---|
ground |
|
| Search the web for a claim or entity, fetch the top result, and return a verification summary with citations |
Powered by Brave Search. Gated by DEEPSIGHT_SEARCH_API_KEY
— degrades gracefully when unset.
| Tool | What it does |
|---|---|
click |
|
| Click at a position (x%, y% — matches locate output) | |
type |
|
| Type text into the focused input field | |
key |
|
Press keyboard shortcuts (cmd+s , return , escape , ctrl+c ) |
|
scroll |
|
| Scroll the active window (direction, clicks) | |
open |
|
| Launch or activate an application by name | |
focus |
|
| Bring a window to front by matching its title | |
apps |
|
| List all visible applications and their window titles | |
window |
|
| Resize or reposition a window using % screen coordinates |
Action tools use macOS osascript
(built-in) or cliclick
(recommended: brew install cliclick
). Requires Accessibility permission in System Settings.
from deepsight.orchestrator import Orchestrator
from deepsight.backends import NativeVisionBackend, ReasoningBackend, ComputerUseBackend
vision = NativeVisionBackend(bin_path="vision_eyes")
reasoning = ReasoningBackend(
base_url="https://api.deepseek.com/v1",
api_key="sk-...",
model="deepseek-v4-flash",
)
agent = Orchestrator(vision=vision, reasoning=reasoning, computer=ComputerUseBackend())
agent.run(
image_url="data:image/png;base64,...",
user_text="Open a terminal, create a new game project, build it, "
"then capture the result and tell me if it compiled.",
response_format={"type": "json_object"},
)
The model will: open Terminal, type commands, capture the screen to check output, locate errors, fix them, rebuild, and report the result.
git clone https://github.com/Reality-Shifting-Tech/deepsight.git
cd deepsight
uv sync
make build-eyes
export DEEPSIGHT_VISION_BIN="$PWD/scripts/vision_eyes"
uv run deepsight describe path/to/image.jpg
uv run deepsight doctor
git clone https://github.com/Reality-Shifting-Tech/deepsight.git
cd deepsight
uv sync
winget install UB-Mannheim.TesseractOCR
uv run python -m deepsight describe path/to/image.jpg
uv run deepsight doctor
Windows uses WindowsVisionBackend
for PIL-based scene analysis (colors, brightness, texture) and optional Tesseract OCR. Desktop automation uses native Windows APIs (user32.dll
, PowerShell SendKeys) — no additional tools to install.
uv run deepsight describe path/to/image.jpg
Output: OCR text, scene classification, face/human/animal counts, detected sports, color palette, bounding boxes for every detected object.
from deepsight.backends import NativeVisionBackend, ReasoningBackend
from deepsight.orchestrator import Orchestrator
vision = NativeVisionBackend(bin_path="vision_eyes")
reasoning = ReasoningBackend(
base_url="https://api.deepseek.com/v1",
api_key="sk-...",
model="deepseek-v4-flash",
)
session = Orchestrator(vision=vision, reasoning=reasoning)
result = session.run(
image_url="https://example.com/screenshot.png",
user_text="What's on the screen? Find any text and describe the layout.",
)
print(result.content)
Reasoning model receives the user's request plus tool definitions for all 16 tools.Vision tools(look, ocr, zoom, count, locate) route through thePerception
module, which shellsvision_eyes
— the compiled Apple Vision binary — for zero-token analysis.Live capture(capture
,watch
) uses macOSscreencapture
to grab the screen, stores the result as the active image, and runs the full vision pipeline on it.Action tools(click, type, key, scroll, open, focus, apps, window) route throughComputerUseBackend
, which uses macOSosascript
orcliclick
for desktop automation.Grounding(ground
) usesSearchBackend
to search the web via Brave Search API.Structured output— passresponse_format
to get JSON-schema-constrained answers.Cross-capture memory— perceptual hashing (dhash) + OCR set diff tracks what changed between captures.** Perception cache**deduplicates repeated vision queries within a session.
Inspect a region of the image. All coordinates are percentages (0-100). Returns a description of what's there.
Transcribe text in a region. Exact transcription including line breaks.
Upscale and inspect a region for small details.
Count objects matching a description. Pass a what
string like "people", "red cars", "buttons".
Find an object by description. Returns normalized bounding box coordinates plus confidence. Uses Apple Vision's on-device detection (faces, humans, animals, text, rectangles, salient objects). Example: locate("the login button")
returns Login (85%): x=40% y=60% w=20% h=8%
.
Take a screenshot. Optional region
: "screen"
(default) or a window title substring (e.g. "Terminal"
, "Safari"
). Returns a full scene analysis with OCR, detected objects, and changes since the last capture.
Monitor the screen over time. Uses perceptual hashing to skip identical frames. Optional until
stops early when text appears. Returns a timeline.
Search the web to verify a fact. Fetches the top result's page content for deep verification. Requires DEEPSIGHT_SEARCH_API_KEY
.
Click at screen position (percentages). Use after locate
to click on a specific object. Requires Accessibility permission.
Type text into the currently focused input field.
Press keyboard shortcuts: "cmd+s"
, "return"
, "escape"
, "ctrl+c"
, "tab"
, "up"
, "down"
.
Scroll the active window.
Launch or activate an application: "Terminal"
, "Safari"
, "Xcode"
, "Finder"
.
Bring a window to front by title substring. Use apps
first to see available windows.
List all visible running applications and their window titles. Returns something like:
Safari (2 windows)
- DeepSight README — Edit
- GitHub — Pull Requests
Terminal (1 window)
- bash — npm run build
Resize or reposition a window. All values in % of screen. Example: window(x=25, y=25, w=50, h=50)
centers the window.
from deepsight.backends import NativeVisionBackend, ReasoningBackend, \
ComputerUseBackend, SearchBackend
from deepsight.orchestrator import Orchestrator
from deepsight.config import get_settings
settings = get_settings()
agent = Orchestrator(
vision=NativeVisionBackend(bin_path=settings.vision_bin),
reasoning=ReasoningBackend(
base_url=settings.reasoning_base_url,
api_key=settings.reasoning_api_key,
model=settings.reasoning_model,
),
computer=ComputerUseBackend(),
search=SearchBackend(api_key=settings.search_key),
)
result = agent.run(
image_url="data:image/png;base64,...",
user_text=(
"Open Terminal. Run 'npm run dev'. Wait for the dev server to start. "
"Capture the browser at localhost:5173. Describe what you see. "
"If there are errors, read them, fix the code, and try again. "
"Tell me when the app is running and what it looks like."
),
)
All settings are environment variables (or a .env
file in the repo root). Variables use the DEEPSIGHT_
prefix.
| Variable | Default | Description |
|---|---|---|
DEEPSIGHT_VISION_BIN |
||
vision_eyes |
||
| Path to the compiled Apple Vision binary | ||
DEEPSIGHT_REASONING_BASE_URL |
||
https://api.deepseek.com/v1 |
||
| OpenAI-compatible chat endpoint | ||
DEEPSIGHT_REASONING_API_KEY |
||
| (empty) | ||
| API key for the reasoning model | ||
DEEPSIGHT_REASONING_MODEL |
||
deepseek-v4-flash |
||
| Model name for the reasoning loop | ||
DEEPSIGHT_SEARCH_API_KEY |
||
| (empty) | ||
Brave Search API key for ground tool |
||
DEEPSIGHT_MAX_LOOK_ROUNDS |
||
5 |
||
| Max tool rounds per vision session | ||
DEEPSIGHT_SKETCH_ENABLED |
||
true |
||
| Include scene sketch in prompts | ||
DEEPSIGHT_CACHE_ENABLED |
||
true |
||
| Cache repeated vision regions | ||
DEEPSIGHT_CACHE_TTL_SECONDS |
||
3600 |
||
| Perception cache expiry |
make test # pytest (60+ tests, no macOS APIs)
make lint # ruff (zero-warning policy)
make typecheck # mypy
make all # test + lint + typecheck
make build-eyes # compile scripts/vision_eyes.swift
The Swift source is at scripts/vision_eyes.swift
; make build-eyes
is the canonical compile. Tests must not require live model endpoints — backends are mocked.
| Symptom | Fix |
|---|---|
vision binary not found (macOS) |
|
make build-eyes , set DEEPSIGHT_VISION_BIN |
|
| Vision tools return no results | Check deepsight doctor |
| Action tools fail silently (macOS) | Grant Accessibility permission in System Settings |
click / type don't work (macOS) |
|
brew install cliclick for more reliable input |
|
| Action tools fail on Windows | Run as normal user (not admin) — user32.dll calls work without elevation |
ground returns unavailable |
|
Set DEEPSIGHT_SEARCH_API_KEY (free at |
|
capture returns empty (macOS) |
|
| Grant Screen Recording permission to Terminal | |
capture returns empty (Windows) |
|
Runs as current user — PIL.ImageGrab needs a display session |
|
| OCR not working (Windows) | Install Tesseract: winget install UB-Mannheim.TesseractOCR |
| Compile fails: SDK not found | xcode-select --install |
MIT — see LICENSE. Built on Apple's Vision framework. Third-party notices in THIRD_PARTY_NOTICES.md. Release history in CHANGELOG.md.