SigLIP 2 text embedding on CPU with Rust and ONNX

A new Rust-based server uses ONNX Runtime to run SigLIP 2 text embeddings on CPU, providing an OpenAI-compatible endpoint for live text queries while reserving GPU resources for image indexing. The design targets production systems that need to scale query capacity without GPU contention, though it requires strict contract alignment between CPU and GPU model checkpoints.

A CPU-only, text-only SigLIP 2 embedding server built with Rust and ONNX Runtime. It is intended for live text queries when GPU capacity is scarce; image embedding and index ingestion remain GPU workloads. It exposes an OpenAI-compatible embeddings endpoint, so existing clients that support a custom OpenAI base URL can use it without a SigLIP 2-specific wrapper. The HTTP surface is deliberately small: GET /health POST /v1/embeddings There are no image or legacy embedding routes. SigLIP 2 places images and text in the same embedding space. Building a video or image index is a throughput-heavy batch workload: decode the media, run the image tower on GPUs, and persist those vectors in a vector index. Searching the finished index is different. Each request only needs one small text embedding before nearest-neighbor search, so reserving a GPU for that query path is often unnecessary. Offline indexing: images/video ── GPU SigLIP 2 image tower ── vector index Online search: text query ── CPU SigLIP 2 text tower ── search that index This server owns the second path. It lets a system keep GPU ingestion for high throughput while moving live text queries onto ordinary CPU instances, which are generally easier to provision, scale, and keep available when GPU capacity is constrained. It is especially useful as a capacity fallback or as the steady-state query service after GPU indexing completes. The CPU and GPU paths must use the same SigLIP 2 checkpoint contract: model revision, tokenizer, maximum token length, projection dimension, and normalization. A text vector from a different contract is not compatible with the existing image index. Original SigLIP and SigLIP 2 checkpoints are not interchangeable embedding spaces. The common serving path for vision-language models carries Python, PyTorch, and a general-purpose GPU inference server. Those are excellent tools for model development and high-throughput image ingestion, but they are more machinery than this steady-state query path needs. ONNX Runtime executes the exported text graph with an optimized CPU backend. Rust owns the smaller serving layer around it: tokenization, bounded admission and session pooling, the HTTP contract, and strict response validation. The result is one predictable service process without a Python environment or the training framework in production. Rust does not replace the tensor runtime or make the model mathematics faster by itself; ONNX Runtime performs that work. The tradeoff is a stricter artifact boundary. Exported graph inputs and outputs, tokenizer behavior, fixed token length, model revision, and normalization must be proven equivalent to the GPU implementation. This design is a good fit for a pinned, narrow production query service; Python remains the easier choice while experimenting with models or changing their execution contract often. Point the client at this server's /v1 base URL and request the exact model ID reported by GET /health : curl --fail --silent --show-error http://127.0.0.1:8000/v1/embeddings \ --header 'Content-Type: application/json' \ --data '{ "model": "organization/siglip-model", "input": "a red block", "a robot arm" }' For example, with the OpenAI Python client: python from openai import OpenAI embedding client = OpenAI base url="http://127.0.0.1:8000/v1", api key="not-used", response = embedding client.embeddings.create model="organization/siglip-model", input= "a red block", "a robot arm" , embeddings = item.embedding for item in response.data The server does not validate Authorization ; api key="not-used" only satisfies clients that require a value. Protect non-loopback deployments with network controls. This is embeddings-endpoint compatibility, not an implementation of the rest of the OpenAI API. POST /v1/embeddings accepts a single string, a string batch, one token-ID sequence, or a batch of token-ID sequences. The optional model must match the served model, and dimensions may only equal its native embedding dimension. { "model": "organization/siglip-model", "input": "a red block", "a robot arm" , "encoding format": "base64" } encoding format defaults to float . base64 encodes each vector as little-endian f32 bytes. Responses preserve input order and use the familiar OpenAI object , data , model , and usage envelope. The server never downloads mutable model or runtime assets. It requires four local files at startup: SIGLIP MODEL MANIFEST PATH=/models/manifest.json SIGLIP ONNX MODEL PATH=/models/text model.onnx SIGLIP TOKENIZER PATH=/models/tokenizer.json SIGLIP ORT LIBRARY PATH=/opt/onnxruntime/lib/libonnxruntime.so The manifest is a language-neutral JSON contract owned by the deployment that produces the model artifacts: { "repository": "organization/siglip-model", "revision": "0123456789abcdef0123456789abcdef01234567", "embedding dimension": 1152, "text max token length": 64 } Additional manifest fields are accepted so one canonical file can be shared by multiple services. revision must be a full 40-character hexadecimal commit SHA; mutable tags and branches fail closed. Use an ONNX Runtime 1.24.x CPU shared library. The pinned Rust binding targets the ONNX Runtime 1.24 API. The graph must expose input ids and attention mask inputs plus a text embeds output. A deliberately different output name can be selected with SIGLIP ONNX OUTPUT NAME . At startup the server: - validates the model manifest and every local artifact path; - configures fixed-length tokenizer padding and truncation; - validates the ONNX graph inputs and output; - runs a real inference probe before binding the HTTP listener. Every returned vector is checked for the declared dimension and finite values, then L2-normalized. Install mold and use the standard Rust workflow: cargo run --release Or build the self-contained server image from this repository: docker build -t siglip-onnx-server . docker run --rm --env-file .env \ -p 8000:8000 \ -v /local/models:/models:ro \ -v /local/onnxruntime:/opt/onnxruntime:ro \ siglip-onnx-server Copy .env.example /Hebbian-Robotics/siglip-onnx-server/blob/main/.env.example to .env and adjust its mounted paths.The checked-in Compose deployment /Hebbian-Robotics/siglip-onnx-server/blob/main/deploy/compose.yaml replaces ad-hoc docker run processes. It uses durable host paths, read-only mounts and root filesystem, bounded concurrency, a startup-aware health check, and restart: unless-stopped so the service returns after Docker or the VM restarts. On the C4 host, install the model and runtime assets outside /tmp , then start the service from a checkout of this repository: sudo install -d -m 0755 \ /var/lib/siglip-onnx/model \ /var/lib/siglip-onnx/onnxruntime/lib cp .env.example .env Set SIGLIP ONNX IMAGE to the published immutable image and verify the paths. sudo docker compose --env-file .env -f deploy/compose.yaml pull sudo docker compose --env-file .env -f deploy/compose.yaml up -d sudo docker compose --env-file .env -f deploy/compose.yaml ps The default example listens on the VM's VPC interface so Cloud Run Direct VPC egress can reach it. Restrict TCP port 8000 with the VPC firewall; the server does not provide application-layer authentication. Bind SIGLIP LISTEN ADDRESS=127.0.0.1 for local-only testing. Before changing query traffic, verify /health reports the exact model, revision, and dimension expected by the GPU-built indexes. Then deploy serve with the C4's internal URL and SQUASH SIGLIP TEXT PROTOCOL=openai . Rollback is the legacy GPU URL plus SQUASH SIGLIP TEXT PROTOCOL=legacy ; ingestion is not part of this switch. The process owns a bounded pool of independent ONNX Runtime sessions. Requests run concurrently across sessions, while each session uses its own intra-operation thread pool. Tune these values against the physical cores and memory bandwidth of the target machine: SIGLIP ORT SESSION COUNT=3 SIGLIP ORT INTRA OP THREADS=4 SIGLIP MAX TEXT BATCH SIZE=32 SIGLIP MAX PENDING TEXT REQUESTS=32 Keep sessions multiplied by threads near the physical-core budget as a starting point, then measure. When active and pending capacity is full, the server returns 429 Too Many Requests with Retry-After: 1 instead of building an unbounded blocking queue. Sessions share ONNX Runtime's prepacked weights. CPU and GPU embeddings are compatible only when the model revision, tokenizer, fixed token length, graph semantics, vector dimension, and normalization agree. Before routing production queries: - compare Rust and GPU tokenizer IDs and attention masks across ordinary, empty, long, Unicode, and punctuation-heavy queries; - compare CPU and GPU vectors using component error and cosine similarity; - query both vector sets against the same GPU-built image index and measure top-k overlap; - benchmark cold start, latency percentiles, throughput, and overload behavior at the intended session/thread settings; - canary only live text queries and keep image ingestion on GPU. See CONTRIBUTING.md /Hebbian-Robotics/siglip-onnx-server/blob/main/CONTRIBUTING.md for local quality checks. Licensed under the MIT License /Hebbian-Robotics/siglip-onnx-server/blob/main/LICENSE .