Show HN: Visiongrep – local semantic image search in Rust

VisionGrep, a Rust command-line tool by developer udayj, performs local semantic image search by running CLIP inference on-device and caching embeddings in SQLite for repeat queries. The tool searches JPEG, PNG, WebP and BMP files recursively, uses an INT8-quantized DataComp CLIP vision model of about 119 MB (down from 352 MB) plus roughly 257 MB for the FP32 text model and tokenizer, and defaults to five results at a minimum cosine similarity of 0.25. VisionGrep ships prebuilt binaries for Apple Silicon macOS and Linux x86_64/arm64 (glibc 2.39 or newer), requires Rust 1.89 or newer to build, and emits stable text, JSON or NUL-separated output with exit codes 0 for matches, 1 for no matches and 2 for errors so scripts and AI coding agents can call it.

Search local images with a natural language description or a reference image. VisionGrep is a Rust CLI that runs CLIP inference locally and caches embeddings in SQLite for repeated searches. It recursively searches JPEG, PNG, WebP, and BMP files without following symbolic links. It is built to be called by scripts and AI coding agents as well as people: results go to stdout in stable text, JSON or NUL-separated formats, diagnostics go to stderr, and exit codes separate "no match" from errors. See Use with AI agents use-with-ai-agents . Prebuilt binaries are available for Apple Silicon macOS and Linux x86 64 and arm64, glibc 2.39 or newer, such as Ubuntu 24.04 . Intel Macs are not supported because ONNX Runtime publishes no prebuilt CPU binaries for them. Install the latest release into ~/.cargo/bin : curl --proto '=https' --tlsv1.2 -LsSf https://github.com/udayj/visiongrep/releases/latest/download/visiongrep-installer.sh | sh visiongrep "a dog on a beach" ./photos visiongrep --image ./reference.jpg ./photos To build from a checkout instead, run cargo build --release and use ./target/release/visiongrep . Building needs Rust 1.89 or newer and a C compiler; the optimized build takes several minutes. Required model artifacts are downloaded automatically on first use. They are stored in $XDG CACHE HOME/visiongrep/models/ , or ~/.cache/visiongrep/models/ when XDG CACHE HOME is unset or empty. Once the required models are cached, searches can run offline. An image query excludes the query image itself from results. The DataComp CLIP vision model uses dynamic INT8 quantization, with the final projection and first transformer block kept in FP32. The vision model is about 119 MB down from 352 MB ; the FP32 text model and tokenizer add about 257 MB. Changing the vision model causes existing indexes to re-embed images once on their next use. On first use, VisionGrep also saves an optimized ONNX graph under models/optimized/ for each model it loads. Later processes reuse compatible graphs to reduce model startup work. The vision and text graphs are cached separately; the tokenizer is unchanged. The first use still pays the optimization and save cost. On Apple Silicon, the optimized vision and text graphs measured about 119 MB and 254 MB respectively, adding about 373 MB of disk usage; sizes can vary by platform. Already-running services reuse their loaded sessions. Optimized graphs are tied to the source model, ONNX Runtime build and local hardware configuration. If an optimized graph cannot be used or saved, VisionGrep falls back to the original model. Removing models/optimized/ causes the graphs to be rebuilt on subsequent use. --no-cache bypasses the SQLite index, while model artifacts and optimized graphs remain cached. visiongrep "sunset" ./photos --top 10 --threshold 0.3 visiongrep "sunset" ./photos --json visiongrep "sunset" ./photos --paths-only visiongrep "sunset" ./photos --index-path ./photos.db visiongrep "sunset" ./photos --no-cache visiongrep --help The default is up to five results with a minimum cosine similarity of 0.25 . Scores are similarities, not probabilities. Results go to stdout; progress and diagnostics go to stderr. Use --null for exact paths separated by NUL bytes, or --quiet to suppress progress. Exit codes are 0 for matches, 1 for no matches, and 2 for an operational or argument error. visiongrep index ./photos index downloads any missing models, embeds new or changed images, and prepares both models' optimized graphs, without running a query. Afterwards the first search only needs to embed its query. It accepts --index-path PATH and --quiet , exits 0 on success and 2 on error, and writes nothing to stdout. Run it again at any time; unchanged images are skipped. Indexing saves its progress every 256 images, so an interrupted index or search resumes where it stopped when run again. An interrupted model download restarts that file. When stderr is not a terminal, progress is reported as Indexed N/TOTAL images lines instead of a progress bar. Coding agents with a shell, such as Claude Code or Codex, can call visiongrep directly. The first use of a folder is the slow part: it downloads about 375 MB of models and embeds every image, which can take minutes for a few thousand photos and exceed an agent's command timeout. Run visiongrep index DIR once yourself before handing the folder to an agent, or let the agent run it with a long timeout. After that, a new query takes well under a second and a repeated query takes a few milliseconds. If an agent's command is cut off while indexing, running it again continues from the saved progress. For many queries against one folder, start visiongrep serve --socket DIR in the background. Ordinary searches then use the loaded models automatically, with no change to the command. Output for agents: - --json prints an array of {"path": ..., "score": ...} objects, highest score first. - --paths-only prints one path per line; -0 separates exact paths with NUL bytes. - Exit code 1 means the search worked and nothing passed the threshold. The default threshold is deliberately conservative, so an empty result is more likely than an unrelated guess. Use --threshold -1 to always get the top results. - Progress, warnings and errors go to stderr only; --quiet suppresses progress. Concurrent searches can share an index. Do not run --reindex while other searches use the same index see the known limitation below . A snippet to adapt for AGENTS.md or CLAUDE.md : To find images by content, use visiongrep "<description " <dir --json --top 10 , or visiongrep --image <file <dir --json for similar images. Exit code 1 means no image matched; retry with a different description or --threshold 0.2 before concluding nothing matches. If that retry returns only unrelated images, the image is not in the folder. Say so; do not open or scan images one by one to double-check. Scores are raw cosine similarities around 0.25–0.35 for good matches , not probabilities. Never pass --reindex . Start an opt-in server for a search directory to reuse its loaded models across ordinary CLI searches: visiongrep serve --socket ./photos & visiongrep "a dog on a beach" ./photos The server runs in the foreground until stopped. Searches use it automatically when the search directory and resolved index path match; otherwise they run as normal one-shot searches. Pass the same --index-path to the server and searches when using a custom index. The server loads models as needed and keeps them in memory about 400–600 MiB with both models loaded . --no-cache , --reindex , --verify-models , and --timing always run one-shot searches. --reindex refuses to run while a socket server is listening for the same root and index; stop the server first. visiongrep serve --stdio --index-path photos.db ./photos Keep the process open and send one JSON object per line on stdin: {"id":1,"query":"a dog on the beach","top":3} {"id":2,"image":"./reference.jpg","threshold":0.3} Each request receives one flushed JSON line on stdout, in request order: {"id":1,"results": {"score":0.31,"path":"./photos/beach-dog.jpg"} } {"id":2,"results": } Provide exactly one non-blank query or non-empty image path. top defaults to 5 and must be positive; threshold defaults to 0.25 and accepts -1 through 1. The optional id is a string or unsigned 64-bit integer and is echoed back omitted/null IDs return null . Unknown fields are rejected. Paths follow the usual CLI rules: relative paths use the process's working directory, and JSON requires UTF-8 paths. To search for the literal word serve or index in one-shot mode, place a search option first: visiongrep --top 5 serve ./photos . Before each search, the service scans for additions, modifications and deletions and updates the index. SQLite stays open; text and vision models load only when needed and remain resident until exit. Image vectors are read from SQLite for each request. Keeping both models resident uses more memory than one-shot mode. The search directory is resolved at startup. Request failures return {"id":1,"error":"..."} and the service continues. Malformed JSON or schema errors return a null ID. Progress and warnings go to stderr. Close stdin to shut down; EOF and a closed output pipe exit with code 0, even if individual requests failed or had no matches. Startup/I/O failures and request lines exceeding 1 MiB including the newline exit with code 2. Stop the service before replacing or rebuilding its index with --reindex . The default index is .visiongrep.db inside the searched directory. Repeated searches reuse unchanged image embeddings; modification time and file size determine which images need updating. Use --index-path PATH to choose another location, --no-cache to bypass the index, or --reindex to rebuild it before returning search results. A custom index belongs to one search root. Known limitation: running --reindex concurrently with a cached search or another --reindex using the same index can corrupt the SQLite database. Reindexing replaces the database file without coordinating other processes that may still have it open. Atomic file replacement does not make concurrent database access safe. Wait for all users of that index to finish before starting --reindex , and let it finish before accessing the index again. This also applies to indexes selected with --index-path ; concurrency protection is not implemented yet. cargo fmt --all --check cargo check --workspace --all-targets cargo clippy --workspace --all-targets -- -D warnings cargo test --workspace -- --test-threads=1 The full test suite needs cached model artifacts. See test setup and coverage https://github.com/udayj/visiongrep/blob/main/tests/README.md for preparation instructions and benchmarks https://github.com/udayj/visiongrep/blob/main/benchmarks/README.md for performance measurements. Releases are built by dist https://opensource.axo.dev/cargo-dist/ from dist-workspace.toml . Bump the version in Cargo.toml , update CHANGELOG.md , merge to main , then push a matching tag such as v0.4.1 . The release workflow builds the three target archives, the shell installer and checksums, and publishes them as a GitHub release. Run dist generate after changing the dist configuration or upgrading dist. The crate is not published to crates.io.