# DeepSight – give text-only LLMs eyes and hands (zero tokens, on-device)

> Source: <https://github.com/Reality-Shifting-Tech/deepsight>
> Published: 2026-08-05 18:25:54+00:00

Give DeepSeek (or any text-only model) eyes and hands.DeepSight connects your existing LLM setup to the real world — it can look at images you send, take screenshots of your desktop, read text on screen, click buttons, type into fields, open apps, and search the web to verify facts. All vision runs on-device: Apple Vision on macOS, PIL + optional Tesseract OCR on Windows. Zero tokens, zero GPU, no image data ever leaves your machine.

```
pip install deepsight-vision
deepsight setup        # compiles the vision binary (macOS) or verifies env (Windows)
deepsight doctor       # confirms everything works
```

The `setup`

command compiles `vision_eyes`

from Swift on macOS (Xcode SDK required), creates a `.env`

with defaults, and prints next steps. On Windows it detects the platform and uses PIL + optional Tesseract OCR. Or install from source with `uv sync`

(see Agent Integration below).

DeepSight ships as a **loadable agent skill**: procedural knowledge (setup, wiring, tool protocols, troubleshooting) that any agent can load on demand. Install it with one command — no user steps:

```
curl -fsSL https://raw.githubusercontent.com/Reality-Shifting-Tech/deepsight/main/scripts/install-skill.sh | bash
```

That's it. The script detects the agent and installs into the right skills directory — Hermes (`~/.hermes/skills/`

), Claude Code (`~/.claude/skills/`

), Codex (`~/.codex/skills/`

) — or an explicit `DEEPSIGHT_SKILLS_DIR`

. No accounts, no API keys, no configuration required for the skill itself. A previous copy is preserved as `deepsight.old`

, never deleted.

After it runs, the agent should load the skill and follow its instructions (Hermes: `skill_view(name='deepsight')`

; any agent: read the installed `SKILL.md`

). It covers compiling the vision binary, reasoning-loop wiring, the 16-tool protocol, `auxiliary.vision`

integration, and troubleshooting.

Skill source of truth: `skill/deepsight/`

in this repo. Offline install from a checkout: `cp -R skill/deepsight ~/.hermes/skills/`

.

One command to set up deepsight from scratch:

```
git clone https://github.com/Reality-Shifting-Tech/deepsight.git
cd deepsight
uv sync
uv run deepsight setup        # compiles binary (macOS) or verifies env (Windows)
uv run deepsight doctor        # confirms everything works
```

That's it. The `setup`

command compiles the vision binary (macOS), creates a `.env`

file with defaults, and prints next steps. For Windows, it detects the platform automatically and uses PIL + optional Tesseract OCR instead.

**Python integration (for agents):**

``` python
from deepsight.backends import NativeVisionBackend, ReasoningBackend, \
    ComputerUseBackend, SearchBackend
from deepsight.orchestrator import Orchestrator
from deepsight.config import get_settings

settings = get_settings()
vision = NativeVisionBackend(bin_path=settings.vision_bin)
reasoning = ReasoningBackend(
    base_url=settings.reasoning_base_url,
    api_key=settings.reasoning_api_key,
    model=settings.reasoning_model,
)

# Vision-only session
agent = Orchestrator(vision=vision, reasoning=reasoning)
result = agent.run("data:image/png;base64,...", "Describe this image")

# With desktop automation (macOS) or Windows automation
from deepsight.backends import ComputerUseBackend
agent = Orchestrator(
    vision=vision, reasoning=reasoning,
    computer=ComputerUseBackend(),
)
result = agent.run("data:image/png;base64,...",
    "Open Terminal, run 'npm run dev', capture the result")
```

On Windows, use `WindowsVisionBackend`

instead of `NativeVisionBackend`

:

``` python
from deepsight.backends import WindowsVisionBackend
vision = WindowsVisionBackend()
```

DeepSight exposes **16 tools** to the reasoning model, organized into four layers.

| Tool | What it does |
|---|---|
`look` |
Describe a rectangular region of the image |
`ocr` |
Transcribe all text in a region, exactly as written |
`zoom` |
Zoom into a region for small-detail inspection |
`count` |
Count objects matching a description in a region |
`locate` |
Find an object by description and return its bounding box (x%, y%, w%, h%) |

All vision tools are zero-token — they use Apple Vision on macOS (via a compiled Swift binary) or PIL + optional Tesseract OCR on Windows. No network, no GPU, no API calls.

| Tool | What it does |
|---|---|
`capture` |
Screenshot the screen (or a specific window) and analyze it with the full vision pipeline |
`watch` |
Monitor the screen over time — captures at an interval, uses perceptual hashing to skip identical frames, returns a timeline of changes. Optional `until` param stops when target text appears |

After `capture`

, all subsequent vision tools operate on the captured screen. The model can capture, inspect, act, then capture again.

| Tool | What it does |
|---|---|
`ground` |
Search the web for a claim or entity, fetch the top result, and return a verification summary with citations |

Powered by Brave Search. Gated by `DEEPSIGHT_SEARCH_API_KEY`

— degrades gracefully when unset.

| Tool | What it does |
|---|---|
`click` |
Click at a position (x%, y% — matches locate output) |
`type` |
Type text into the focused input field |
`key` |
Press keyboard shortcuts (`cmd+s` , `return` , `escape` , `ctrl+c` ) |
`scroll` |
Scroll the active window (direction, clicks) |
`open` |
Launch or activate an application by name |
`focus` |
Bring a window to front by matching its title |
`apps` |
List all visible applications and their window titles |
`window` |
Resize or reposition a window using % screen coordinates |

Action tools use macOS `osascript`

(built-in) or `cliclick`

(recommended: `brew install cliclick`

). Requires Accessibility permission in System Settings.

``` python
from deepsight.orchestrator import Orchestrator
from deepsight.backends import NativeVisionBackend, ReasoningBackend, ComputerUseBackend

# Set up the agent
vision = NativeVisionBackend(bin_path="vision_eyes")
reasoning = ReasoningBackend(
    base_url="https://api.deepseek.com/v1",
    api_key="sk-...",
    model="deepseek-v4-flash",
)
agent = Orchestrator(vision=vision, reasoning=reasoning, computer=ComputerUseBackend())

# The model uses all 16 tools autonomously:
agent.run(
    image_url="data:image/png;base64,...",
    user_text="Open a terminal, create a new game project, build it, "
              "then capture the result and tell me if it compiled.",
    response_format={"type": "json_object"},
)
```

The model will: open Terminal, type commands, capture the screen to check output, locate errors, fix them, rebuild, and report the result.

```
git clone https://github.com/Reality-Shifting-Tech/deepsight.git
cd deepsight
uv sync
make build-eyes
export DEEPSIGHT_VISION_BIN="$PWD/scripts/vision_eyes"
uv run deepsight describe path/to/image.jpg
uv run deepsight doctor
git clone https://github.com/Reality-Shifting-Tech/deepsight.git
cd deepsight
uv sync
winget install UB-Mannheim.TesseractOCR
uv run python -m deepsight describe path/to/image.jpg
uv run deepsight doctor
```

Windows uses `WindowsVisionBackend`

for PIL-based scene analysis (colors, brightness, texture) and optional Tesseract OCR. Desktop automation uses native Windows APIs (`user32.dll`

, PowerShell SendKeys) — no additional tools to install.

```
uv run deepsight describe path/to/image.jpg
```

Output: OCR text, scene classification, face/human/animal counts, detected sports, color palette, bounding boxes for every detected object.

``` python
from deepsight.backends import NativeVisionBackend, ReasoningBackend
from deepsight.orchestrator import Orchestrator

vision = NativeVisionBackend(bin_path="vision_eyes")
reasoning = ReasoningBackend(
    base_url="https://api.deepseek.com/v1",
    api_key="sk-...",
    model="deepseek-v4-flash",
)

session = Orchestrator(vision=vision, reasoning=reasoning)
result = session.run(
    image_url="https://example.com/screenshot.png",
    user_text="What's on the screen? Find any text and describe the layout.",
)
print(result.content)
```

**Reasoning model** receives the user's request plus tool definitions for all 16 tools.**Vision tools**(look, ocr, zoom, count, locate) route through the`Perception`

module, which shells`vision_eyes`

— the compiled Apple Vision binary — for zero-token analysis.**Live capture**(`capture`

,`watch`

) uses macOS`screencapture`

to grab the screen, stores the result as the active image, and runs the full vision pipeline on it.**Action tools**(click, type, key, scroll, open, focus, apps, window) route through`ComputerUseBackend`

, which uses macOS`osascript`

or`cliclick`

for desktop automation.**Grounding**(`ground`

) uses`SearchBackend`

to search the web via Brave Search API.**Structured output**— pass`response_format`

to get JSON-schema-constrained answers.**Cross-capture memory**— perceptual hashing (dhash) + OCR set diff tracks what changed between captures.** Perception cache**deduplicates repeated vision queries within a session.

Inspect a region of the image. All coordinates are percentages (0-100). Returns a description of what's there.

Transcribe text in a region. Exact transcription including line breaks.

Upscale and inspect a region for small details.

Count objects matching a description. Pass a `what`

string like "people", "red cars", "buttons".

Find an object by description. Returns normalized bounding box coordinates plus confidence. Uses Apple Vision's on-device detection (faces, humans, animals, text, rectangles, salient objects). Example: `locate("the login button")`

returns `Login (85%): x=40% y=60% w=20% h=8%`

.

Take a screenshot. Optional `region`

: `"screen"`

(default) or a window title substring (e.g. `"Terminal"`

, `"Safari"`

). Returns a full scene analysis with OCR, detected objects, and changes since the last capture.

Monitor the screen over time. Uses perceptual hashing to skip identical frames. Optional `until`

stops early when text appears. Returns a timeline.

Search the web to verify a fact. Fetches the top result's page content for deep verification. Requires `DEEPSIGHT_SEARCH_API_KEY`

.

Click at screen position (percentages). Use after `locate`

to click on a specific object. Requires Accessibility permission.

Type text into the currently focused input field.

Press keyboard shortcuts: `"cmd+s"`

, `"return"`

, `"escape"`

, `"ctrl+c"`

, `"tab"`

, `"up"`

, `"down"`

.

Scroll the active window.

Launch or activate an application: `"Terminal"`

, `"Safari"`

, `"Xcode"`

, `"Finder"`

.

Bring a window to front by title substring. Use `apps`

first to see available windows.

List all visible running applications and their window titles. Returns something like:

```
Safari (2 windows)
  - DeepSight README — Edit
  - GitHub — Pull Requests
Terminal (1 window)
  - bash — npm run build
```

Resize or reposition a window. All values in % of screen. Example: `window(x=25, y=25, w=50, h=50)`

centers the window.

``` python
from deepsight.backends import NativeVisionBackend, ReasoningBackend, \
    ComputerUseBackend, SearchBackend
from deepsight.orchestrator import Orchestrator
from deepsight.config import get_settings

settings = get_settings()

agent = Orchestrator(
    vision=NativeVisionBackend(bin_path=settings.vision_bin),
    reasoning=ReasoningBackend(
        base_url=settings.reasoning_base_url,
        api_key=settings.reasoning_api_key,
        model=settings.reasoning_model,
    ),
    computer=ComputerUseBackend(),
    search=SearchBackend(api_key=settings.search_key),
)

result = agent.run(
    image_url="data:image/png;base64,...",
    user_text=(
        "Open Terminal. Run 'npm run dev'. Wait for the dev server to start. "
        "Capture the browser at localhost:5173. Describe what you see. "
        "If there are errors, read them, fix the code, and try again. "
        "Tell me when the app is running and what it looks like."
    ),
)
```

All settings are environment variables (or a `.env`

file in the repo root). Variables use the `DEEPSIGHT_`

prefix.

| Variable | Default | Description |
|---|---|---|
`DEEPSIGHT_VISION_BIN` |
`vision_eyes` |
Path to the compiled Apple Vision binary |
`DEEPSIGHT_REASONING_BASE_URL` |
`https://api.deepseek.com/v1` |
OpenAI-compatible chat endpoint |
`DEEPSIGHT_REASONING_API_KEY` |
(empty) |
API key for the reasoning model |
`DEEPSIGHT_REASONING_MODEL` |
`deepseek-v4-flash` |
Model name for the reasoning loop |
`DEEPSIGHT_SEARCH_API_KEY` |
(empty) |
Brave Search API key for `ground` tool |
`DEEPSIGHT_MAX_LOOK_ROUNDS` |
`5` |
Max tool rounds per vision session |
`DEEPSIGHT_SKETCH_ENABLED` |
`true` |
Include scene sketch in prompts |
`DEEPSIGHT_CACHE_ENABLED` |
`true` |
Cache repeated vision regions |
`DEEPSIGHT_CACHE_TTL_SECONDS` |
`3600` |
Perception cache expiry |

```
make test          # pytest (60+ tests, no macOS APIs)
make lint          # ruff (zero-warning policy)
make typecheck     # mypy
make all           # test + lint + typecheck
make build-eyes    # compile scripts/vision_eyes.swift
```

The Swift source is at `scripts/vision_eyes.swift`

; `make build-eyes`

is the canonical compile. Tests must not require live model endpoints — backends are mocked.

| Symptom | Fix |
|---|---|
`vision binary not found` (macOS) |
`make build-eyes` , set `DEEPSIGHT_VISION_BIN` |
| Vision tools return no results | Check `deepsight doctor` |
| Action tools fail silently (macOS) | Grant Accessibility permission in System Settings |
`click` / `type` don't work (macOS) |
`brew install cliclick` for more reliable input |
| Action tools fail on Windows | Run as normal user (not admin) — `user32.dll` calls work without elevation |
`ground` returns unavailable |
Set `DEEPSIGHT_SEARCH_API_KEY` (free at
|
`capture` returns empty (macOS) |
Grant Screen Recording permission to Terminal |
`capture` returns empty (Windows) |
Runs as current user — `PIL.ImageGrab` needs a display session |
| OCR not working (Windows) | Install Tesseract: `winget install UB-Mannheim.TesseractOCR` |
| Compile fails: SDK not found | `xcode-select --install` |

MIT — see [LICENSE](/Reality-Shifting-Tech/deepsight/blob/main/LICENSE). Built on Apple's Vision framework. Third-party notices in [THIRD_PARTY_NOTICES.md](/Reality-Shifting-Tech/deepsight/blob/main/THIRD_PARTY_NOTICES.md). Release history in [CHANGELOG.md](/Reality-Shifting-Tech/deepsight/blob/main/CHANGELOG.md).