{"slug": "siglip-2-text-embedding-on-cpu-with-rust-and-onnx", "title": "SigLIP 2 text embedding on CPU with Rust and ONNX", "summary": "A new Rust-based server uses ONNX Runtime to run SigLIP 2 text embeddings on CPU, providing an OpenAI-compatible endpoint for live text queries while reserving GPU resources for image indexing. The design targets production systems that need to scale query capacity without GPU contention, though it requires strict contract alignment between CPU and GPU model checkpoints.", "body_md": "A CPU-only, text-only SigLIP 2 embedding server built with Rust and ONNX Runtime. It is intended for live text queries when GPU capacity is scarce; image embedding and index ingestion remain GPU workloads.\n\nIt exposes an OpenAI-compatible embeddings endpoint, so existing clients that support a custom OpenAI base URL can use it without a SigLIP 2-specific wrapper. The HTTP surface is deliberately small:\n\n`GET /health`\n\n`POST /v1/embeddings`\n\nThere are no image or legacy embedding routes.\n\nSigLIP 2 places images and text in the same embedding space. Building a video or image index is a throughput-heavy batch workload: decode the media, run the image tower on GPUs, and persist those vectors in a vector index. Searching the finished index is different. Each request only needs one small text embedding before nearest-neighbor search, so reserving a GPU for that query path is often unnecessary.\n\n```\nOffline indexing:  images/video ──> GPU SigLIP 2 image tower ──> vector index\nOnline search:     text query   ──> CPU SigLIP 2 text tower  ──> search that index\n```\n\nThis server owns the second path. It lets a system keep GPU ingestion for high throughput while moving live text queries onto ordinary CPU instances, which are generally easier to provision, scale, and keep available when GPU capacity is constrained. It is especially useful as a capacity fallback or as the steady-state query service after GPU indexing completes.\n\nThe CPU and GPU paths must use the same SigLIP 2 checkpoint contract: model revision, tokenizer, maximum token length, projection dimension, and normalization. A text vector from a different contract is not compatible with the existing image index. Original SigLIP and SigLIP 2 checkpoints are not interchangeable embedding spaces.\n\nThe common serving path for vision-language models carries Python, PyTorch, and a general-purpose GPU inference server. Those are excellent tools for model development and high-throughput image ingestion, but they are more machinery than this steady-state query path needs.\n\nONNX Runtime executes the exported text graph with an optimized CPU backend. Rust owns the smaller serving layer around it: tokenization, bounded admission and session pooling, the HTTP contract, and strict response validation. The result is one predictable service process without a Python environment or the training framework in production. Rust does not replace the tensor runtime or make the model mathematics faster by itself; ONNX Runtime performs that work.\n\nThe tradeoff is a stricter artifact boundary. Exported graph inputs and outputs, tokenizer behavior, fixed token length, model revision, and normalization must be proven equivalent to the GPU implementation. This design is a good fit for a pinned, narrow production query service; Python remains the easier choice while experimenting with models or changing their execution contract often.\n\nPoint the client at this server's `/v1`\n\nbase URL and request the exact model ID\nreported by `GET /health`\n\n:\n\n```\ncurl --fail --silent --show-error http://127.0.0.1:8000/v1/embeddings \\\n  --header 'Content-Type: application/json' \\\n  --data '{\n    \"model\": \"organization/siglip-model\",\n    \"input\": [\"a red block\", \"a robot arm\"]\n  }'\n```\n\nFor example, with the OpenAI Python client:\n\n``` python\nfrom openai import OpenAI\n\nembedding_client = OpenAI(\n    base_url=\"http://127.0.0.1:8000/v1\",\n    api_key=\"not-used\",\n)\nresponse = embedding_client.embeddings.create(\n    model=\"organization/siglip-model\",\n    input=[\"a red block\", \"a robot arm\"],\n)\nembeddings = [item.embedding for item in response.data]\n```\n\nThe server does not validate `Authorization`\n\n; `api_key=\"not-used\"`\n\nonly\nsatisfies clients that require a value. Protect non-loopback deployments with\nnetwork controls. This is embeddings-endpoint compatibility, not an\nimplementation of the rest of the OpenAI API.\n\n`POST /v1/embeddings`\n\naccepts a single string, a string batch, one token-ID\nsequence, or a batch of token-ID sequences. The optional `model`\n\nmust match the\nserved model, and `dimensions`\n\nmay only equal its native embedding dimension.\n\n```\n{\n  \"model\": \"organization/siglip-model\",\n  \"input\": [\"a red block\", \"a robot arm\"],\n  \"encoding_format\": \"base64\"\n}\n```\n\n`encoding_format`\n\ndefaults to `float`\n\n. `base64`\n\nencodes each vector as\nlittle-endian `f32`\n\nbytes. Responses preserve input order and use the familiar\nOpenAI `object`\n\n, `data`\n\n, `model`\n\n, and `usage`\n\nenvelope.\n\nThe server never downloads mutable model or runtime assets. It requires four local files at startup:\n\n```\nSIGLIP_MODEL_MANIFEST_PATH=/models/manifest.json\nSIGLIP_ONNX_MODEL_PATH=/models/text_model.onnx\nSIGLIP_TOKENIZER_PATH=/models/tokenizer.json\nSIGLIP_ORT_LIBRARY_PATH=/opt/onnxruntime/lib/libonnxruntime.so\n```\n\nThe manifest is a language-neutral JSON contract owned by the deployment that produces the model artifacts:\n\n```\n{\n  \"repository\": \"organization/siglip-model\",\n  \"revision\": \"0123456789abcdef0123456789abcdef01234567\",\n  \"embedding_dimension\": 1152,\n  \"text_max_token_length\": 64\n}\n```\n\nAdditional manifest fields are accepted so one canonical file can be shared by\nmultiple services. `revision`\n\nmust be a full 40-character hexadecimal commit\nSHA; mutable tags and branches fail closed.\n\nUse an ONNX Runtime 1.24.x CPU shared library. The pinned Rust binding targets\nthe ONNX Runtime 1.24 API. The graph must expose `input_ids`\n\nand\n`attention_mask`\n\ninputs plus a `text_embeds`\n\noutput. A deliberately different\noutput name can be selected with `SIGLIP_ONNX_OUTPUT_NAME`\n\n.\n\nAt startup the server:\n\n- validates the model manifest and every local artifact path;\n- configures fixed-length tokenizer padding and truncation;\n- validates the ONNX graph inputs and output;\n- runs a real inference probe before binding the HTTP listener.\n\nEvery returned vector is checked for the declared dimension and finite values, then L2-normalized.\n\nInstall `mold`\n\nand use the standard Rust workflow:\n\n```\ncargo run --release\n```\n\nOr build the self-contained server image from this repository:\n\n```\ndocker build -t siglip-onnx-server .\ndocker run --rm --env-file .env \\\n  -p 8000:8000 \\\n  -v /local/models:/models:ro \\\n  -v /local/onnxruntime:/opt/onnxruntime:ro \\\n  siglip-onnx-server\n```\n\nCopy [ .env.example](/Hebbian-Robotics/siglip-onnx-server/blob/main/.env.example) to\n\n`.env`\n\nand adjust its mounted paths.The checked-in [Compose deployment](/Hebbian-Robotics/siglip-onnx-server/blob/main/deploy/compose.yaml) replaces ad-hoc\n`docker run`\n\nprocesses. It uses durable host paths, read-only mounts and root\nfilesystem, bounded concurrency, a startup-aware health check, and\n`restart: unless-stopped`\n\nso the service returns after Docker or the VM\nrestarts.\n\nOn the C4 host, install the model and runtime assets outside `/tmp`\n\n, then start\nthe service from a checkout of this repository:\n\n```\nsudo install -d -m 0755 \\\n  /var/lib/siglip-onnx/model \\\n  /var/lib/siglip-onnx/onnxruntime/lib\ncp .env.example .env\n# Set SIGLIP_ONNX_IMAGE to the published immutable image and verify the paths.\nsudo docker compose --env-file .env -f deploy/compose.yaml pull\nsudo docker compose --env-file .env -f deploy/compose.yaml up -d\nsudo docker compose --env-file .env -f deploy/compose.yaml ps\n```\n\nThe default example listens on the VM's VPC interface so Cloud Run Direct VPC\negress can reach it. Restrict TCP port 8000 with the VPC firewall; the server\ndoes not provide application-layer authentication. Bind\n`SIGLIP_LISTEN_ADDRESS=127.0.0.1`\n\nfor local-only testing.\n\nBefore changing query traffic, verify `/health`\n\nreports the exact model,\nrevision, and dimension expected by the GPU-built indexes. Then deploy serve\nwith the C4's internal URL and `SQUASH_SIGLIP_TEXT_PROTOCOL=openai`\n\n. Rollback is\nthe legacy GPU URL plus `SQUASH_SIGLIP_TEXT_PROTOCOL=legacy`\n\n; ingestion is not\npart of this switch.\n\nThe process owns a bounded pool of independent ONNX Runtime sessions. Requests run concurrently across sessions, while each session uses its own intra-operation thread pool. Tune these values against the physical cores and memory bandwidth of the target machine:\n\n```\nSIGLIP_ORT_SESSION_COUNT=3\nSIGLIP_ORT_INTRA_OP_THREADS=4\nSIGLIP_MAX_TEXT_BATCH_SIZE=32\nSIGLIP_MAX_PENDING_TEXT_REQUESTS=32\n```\n\nKeep sessions multiplied by threads near the physical-core budget as a starting\npoint, then measure. When active and pending capacity is full, the server returns\n`429 Too Many Requests`\n\nwith `Retry-After: 1`\n\ninstead of building an unbounded\nblocking queue. Sessions share ONNX Runtime's prepacked weights.\n\nCPU and GPU embeddings are compatible only when the model revision, tokenizer, fixed token length, graph semantics, vector dimension, and normalization agree. Before routing production queries:\n\n- compare Rust and GPU tokenizer IDs and attention masks across ordinary, empty, long, Unicode, and punctuation-heavy queries;\n- compare CPU and GPU vectors using component error and cosine similarity;\n- query both vector sets against the same GPU-built image index and measure top-k overlap;\n- benchmark cold start, latency percentiles, throughput, and overload behavior at the intended session/thread settings;\n- canary only live text queries and keep image ingestion on GPU.\n\nSee [CONTRIBUTING.md](/Hebbian-Robotics/siglip-onnx-server/blob/main/CONTRIBUTING.md) for local quality checks.\n\nLicensed under the [MIT License](/Hebbian-Robotics/siglip-onnx-server/blob/main/LICENSE).", "url": "https://wpnews.pro/news/siglip-2-text-embedding-on-cpu-with-rust-and-onnx", "canonical_source": "https://github.com/Hebbian-Robotics/siglip-onnx-server", "published_at": "2026-07-18 06:10:41+00:00", "updated_at": "2026-07-18 06:21:05.154720+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "ai-infrastructure", "developer-tools", "ai-products"], "entities": ["SigLIP 2", "ONNX Runtime", "Rust", "OpenAI"], "alternates": {"html": "https://wpnews.pro/news/siglip-2-text-embedding-on-cpu-with-rust-and-onnx", "markdown": "https://wpnews.pro/news/siglip-2-text-embedding-on-cpu-with-rust-and-onnx.md", "text": "https://wpnews.pro/news/siglip-2-text-embedding-on-cpu-with-rust-and-onnx.txt", "jsonld": "https://wpnews.pro/news/siglip-2-text-embedding-on-cpu-with-rust-and-onnx.jsonld"}}