Embedcache – Cut embedding API costs by caching redundant requests

Embedcache, a new open-source proxy tool, reduces embedding API costs by caching redundant requests and eliminating duplicate work before it reaches backends like vLLM, Ollama, or OpenAI. The single-binary service offers exact-match caching, in-flight coalescing, batch deduplication, and per-key token budgets, targeting platform and ML-infra teams processing over 10 million embedding tokens daily.

Website https://ajay6601.github.io/embedcache/ · Documentation https://ajay6601.github.io/embedcache/docs.html · Examples https://ajay6601.github.io/embedcache/examples.html The cost-control and dedupe layer for embedding APIs. A single-binary proxy that sits in front of any OpenAI-compatible embedding backend - vLLM, Ollama, text-embeddings-inference, or api.openai.com - and eliminates duplicate work before it reaches your GPUs or your API bill. Then it hands you the invoice for what it saved. your app ──► embedcache ──► vLLM / Ollama / TEI / OpenAI │ ├─ exact-match cache byte-exact replays, content-addressed ├─ in-flight coalescing N concurrent identical calls → 1 upstream ├─ batch dedupe only the uncached items go upstream └─ waste report $X of your embedding spend was duplicates Zero dependencies - pure Go stdlib. One static binary. No Python runtime, no Redis required, no framework lock-in. You run embedcache as a service like nginx or Redis - not a library you import , point your apps' base url at it, and it does four jobs across every language and team sharing it: Measure - embedcache analyze reads your existing request logs and tells you how much of your embedding spend is duplicate work, before you install anything. Reduce - an exact-match cache, in-flight coalescing, batch dedupe, and a content-defined chunk-diff engine eliminate that duplicate work in the request path. Enforce - per-key/per-tenant token budgets stop runaway agents from burning the backend, while cache hits keep serving. Account - live per-model and per-caller spend/savings on / ec/stats , /metrics Prometheus , and a human report . Typical users: platform/ML-infra teams past ~10M embedding tokens/day running vLLM/Ollama plus an API fallback; RAG and agent teams re-embedding corpora and burning tokens in loops. Every clip below is a real capture: the commands actually ran, against real backends, and the statuses, latencies, and numbers on screen are what the live proxy returned recorded by experiments/gifcap , rendered by tools/termgif , transcripts under experiments/gifcap/samples/ . The one at the top is embedcache in front of real Ollama, serving a real miss then a real hit. RAG: a nightly re-ingest, nothing changed, so nothing is recomputed REALTEST.md /Ajay6601/embedcache/blob/main/REALTEST.md Agents: two agents with different keys share one cache, and a capped agent is stopped mid-run MULTIAGENT.md /Ajay6601/embedcache/blob/main/MULTIAGENT.md CLI: point the offline analyzer at a real request log and see your duplicate spend the waste report the-waste-report | Capability | What it does | |---|---| | Exact-match cache | Byte-exact replays, content-addressed by model + dims + encoding + provider params + text | | In-flight coalescing | N concurrent identical requests → 1 upstream computation | | Batch dedupe | Only never-seen items in a batch go upstream; correct index mapping | | Chunk-diff engine | embedcache chunk FastCDC so a document edit only re-embeds the chunks it touched | | Per-key budgets | Hard token limits per key/tenant; 429 on new spend, hits keep serving | | Auth | Allowlist or upstream-verified client keys; admin-token-guarded endpoints | | Resilience | Retries + circuit breaker; cache hits serve through upstream outages | | Persistence | Snapshot survives restarts; TTL bounds staleness | | Waste analytics | Offline analyzer + live / ec/stats , /metrics , report | | Onboarding | embedcache demo zero-setup walkthrough and embedcache check backend probe + setup helper | | Providers | Ollama, vLLM, TEI, OpenAI, Gemini, Voyage, Mistral, Azure OpenAI | Embedding workloads are full of silent duplicate work: re-ingesting a corpus where 95% of chunks didn't change, hot queries embedded thousands of times, retry storms, multiple services embedding the same strings. Frameworks solve this inside Python LangChain's RecordManager, LlamaIndex's IngestionPipeline - useless if your pipeline is Go, TypeScript, custom, or split across teams. Gateways treat it as a checkbox: LiteLLM's embedding cache has a known bug https://github.com/BerriAI/litellm/issues/22659 where mixed cached/uncached batches can return wrong vectors. Every number below comes from a real run in this repo - the workloads and edits are real, the inference is real, the reports are regenerated by the code, not hand-written. | On this workload | embedcache absorbed | |---|---| | A real RAG agent over 103 live Wikipedia articles, zero constructed duplicates | 49.7% of embedding tokens were duplicate work | | A nightly re-ingest where 5% of a 10k-chunk corpus changed | 95.2% of embedding calls absorbed | | Re-ingesting a real doc after one real single-sentence edit, with chunk-diff | 93.8% hit rate vs 28.6% for fixed-size chunking | | Search-shaped Zipf query traffic on a real corpus | ~80% hit rate across six real models | | Re-running a full BEIR SciFact eval 300 queries × 5,183 docs the second time | 100% absorbed, ~2s instead of ~25min, at zero quality change | Your number depends on your traffic - which is exactly why embedcache analyze measures it on your own logs before you deploy anything. embedcache does one thing, language-agnostically, and provably correctly. Eight tiers of evidence live in the repo, each regenerated by code under experiments/ and the first re-run by CI on every push: | Report | What it proves | Data | Backend | |---|---|---|---| | PRODSIM.md /Ajay6601/embedcache/blob/main/PRODSIM.md REALTEST.md /Ajay6601/embedcache/blob/main/REALTEST.md live Wikipedia + real LLM CHUNKDIFF.md /Ajay6601/embedcache/blob/main/CHUNKDIFF.md live Wikipedia VALIDATION.md /Ajay6601/embedcache/blob/main/VALIDATION.md 6 real models at 3 dims this repo's Go source + live Wikipedia en/zh/hi/ar/es BENCHMARKS.md /Ajay6601/embedcache/blob/main/BENCHMARKS.md zero retrieval-quality loss on a recognized IR benchmark + free eval re-runs BEIR SciFact: 5,183 real docs, 300 expert-judged queries PERF.md /Ajay6601/embedcache/blob/main/PERF.md MULTIAGENT.md /Ajay6601/embedcache/blob/main/MULTIAGENT.md cross-language cache sharing live Wikipedia + real LLM + real Python/LangChain The multi-model suite deliberately surfaced real backend behaviors and shows embedcache handling each correctly rather than hiding them: Gemini is not bitwise-deterministic embedcache stabilizes repeats to one vector , Gemini rate-limits under burst, small-context models reject oversized chunks, and the rough bytes/4 token estimator drifts on CJK/Arabic scripts it only feeds the analyzer's estimate; billed spend is apportioned from the backend's exact reported total, never estimated . None of these are cache defects. | Correctness claim | Evidence | |---|---| | Byte-exact responses, correct batch index mapping | 3,453 randomized-batch embeddings, 0 mismatches - covers the LiteLLM | bit-identical through the cache vs direct-to-backend; the top-10 ranking is unchanged for all 300 queries BENCHMARKS.md /Ajay6601/embedcache/blob/main/BENCHMARKS.md MULTIAGENT.md /Ajay6601/embedcache/blob/main/MULTIAGENT.md PERF.md /Ajay6601/embedcache/blob/main/PERF.md honest boundary: cache is content-agnostic proven , but no OpenAI- /v1/embeddings -compatible image backend was reachable to live-test end-to-end - deferred to v0.3 prebuilt binary Linux/macOS/Windows, amd64+arm64 , no Go needed grab the archive for your platform from the releases page: https://github.com/Ajay6601/embedcache/releases or the Docker image docker run ghcr.io/ajay6601/embedcache:latest serve -upstream http://host.docker.internal:11434 or with Go 1.22+ go install github.com/Ajay6601/embedcache/cmd/embedcache@latest or build from a clone git clone https://github.com/Ajay6601/embedcache && cd embedcache go build -o embedcache ./cmd/embedcache New here? See it work in one command, with no upstream, no key, no config: embedcache demo It runs the proxy against a built-in mock model and walks you through a miss, a hit, a partial batch, and the live waste report, then leaves the proxy up for you to poke at. When you're ready to point it at a real backend, embedcache check -upstream URL -model NAME probes it and prints the exact serve command to use. in front of a self-hosted engine embedcache serve -upstream http://localhost:8000 vLLM embedcache serve -upstream http://localhost:11434 Ollama in front of a hosted API embedcache serve -upstream https://api.openai.com Point your SDK at it - no code changes: client = OpenAI base url="http://localhost:8090/v1", api key="..." client.embeddings.create model="text-embedding-3-small", input= "hello" Your Authorization header is forwarded to the upstream or pin one with -upstream-api-key . Non-embedding routes pass through untouched, so it can front a full OpenAI-compatible server. Verified live all experiments/livecheck assertions pass - byte-exact replay, mixed-batch mapping, coalescing, base64 : | backend | upstream flag | verified model | |---|---|---| | Ollama 0.22 | -upstream http://localhost:11434 | all-minilm | | Google Gemini | -upstream https://generativelanguage.googleapis.com/v1beta/openai -upstream-api-key $GEMINI API KEY | gemini-embedding-001 3072-dim | | OpenAI | -upstream https://api.openai.com | wire-identical to the above | Wire-compatible OpenAI-shaped APIs, covered by unit tests : Voyage AI https://docs.voyageai.com -upstream https://api.voyageai.com - the provider Anthropic officially recommends for embeddings; input type and other Voyage params are forwarded verbatim and part of the cache identity, so query- and document-typed vectors never collide , Mistral -upstream https://api.mistral.ai , mistral-embed , and Azure OpenAI deployment paths, ?api-version= query and api-key header are mirrored upstream . Not applicable: Groq and the Anthropic API itself - neither offers an embeddings endpoint Anthropic recommends Voyage, above . Every response tells you what happened: X-Embedcache-Status: hit | miss | partial X-Embedcache-Hits: 14 X-Embedcache-Saved-Tokens: 1830 and usage.prompt tokens reflects only what this request was actually billed upstream. Before installing anything , run the offline analyzer on an existing request log JSONL, one embedding request per line - raw bodies or wrapped under body / request / payload : ./embedcache analyze requests.jsonl embedcache offline waste analysis ================================= requests analyzed 20000 embedding items 20000 unique items 2052 duplicate items 17948 89.7% of all items estimated tokens 340217 ~$0.01 estimated wasted tokens 305090 ~$0.01 89.7% of this embedding spend was duplicate work an exact-match cache would have absorbed. For self-hosted models with no list price, set an amortized GPU cost: -default-price 0.05 $/1M tokens , or supply -pricing costs.json . A running proxy serves the same accounting live: embedcache report , or GET / ec/stats JSON , / ec/report text , /metrics Prometheus . Byte-exact by default. Cached responses replay the upstream's raw JSON - no float re-encoding, no precision drift. model , dimensions , and encoding format are all part of the cache key. Normalization is opt-in. -normalize trim,collapse,lowercase widens matching if you accept the semantics; the default matches exact bytes only. No semantic caching. Similarity-threshold response caching returns wrong answers at some rate; that trade-off belongs to chat responses, not embeddings. Exact-match embedding caching has a 0% wrong-answer rate by construction. Failure containment. Upstream errors pass through verbatim and are never cached; a crashed leader fails its coalesced waiters instead of hanging them. For anything beyond a trusted network, turn on both layers: embedcache serve -upstream http://localhost:8000 \ -admin-token "$ openssl rand -hex 24 " \ guards /stats, /report, /metrics, / ec/flush -auth-mode allowlist -api-keys-file keys.txt \ -ttl 720h bound staleness if model weights change - without it, anyone who can reach the port can flush your cache and read usage stats. -admin-token /healthz stays open for probes.- cache hits never touch the upstream, so without this a revoked or missing key can still read cached vectors. -auth-mode allowlist is for self-hosted backends the proxy owns the key list ; verify is for hosted providers - the caller's key is checked against the upstream GET /v1/models and the verdict cached for -auth-cache-ttl default 5m, negative verdicts 1m, fail-closed on upstream outage .- Terminate TLS in front Caddy, nginx, or your ingress ; embedcache listens in plaintext. Hard per-key limits on upstream spend, the enforcement half of the cost story: embedcache serve -upstream http://localhost:8000 \ -budgets-file budgets.json -budget-window 24h budgets.json: tokens per key per window; 0 = unlimited, "default" applies to the rest { "team-search-x9f": 2000000, "team-agents-p7q": 500000, "default": 1000000 } Budgets bound spend, not reads : only tokens actually billed upstream count, and once a key is exhausted, requests needing new computation get 429 with Retry-After for the window reset while cache hits keep serving , so a capped team's existing workload stays alive; only new cost is blocked. Every response carries X-Embedcache-Budget-Remaining so clients can self-moderate, per-key state is on / ec/stats , and rejections export as embedcache budget rejects total . Counters are in-memory and reset at each window boundary and on restart , like any in-process rate limiter. A single instance keeps its cache in memory, so the moment you run more than one replica behind a load balancer each starts cold and the fleet recomputes the same vectors. Point them all at one Redis or Valkey and they share entries: embedcache serve -upstream http://localhost:8000 \ -shared-redis 127.0.0.1:6379 -shared-redis-ttl 720h On a local miss the proxy consults Redis before going upstream; a hit there warms the local cache and skips the backend call, byte-identically. The Redis client is a small in-house RESP implementation, so this stays a single dependency-free static binary. Proven for real in SHAREDCACHE.md /Ajay6601/embedcache/blob/main/SHAREDCACHE.md : two separate processes sharing one Redis, where the second serves the first's vector with no upstream call . The tier is read-through/write-through and best-effort — if Redis is unreachable, instances fall back to computing, never erroring. Exact-match caching caps savings at your literal duplicate share. Near-duplicate matching — "how do I reset my password" vs "how do I reset my password?" — can push higher, but serving one input's vector for another can silently poison a vector store, so it is off by default and shadow-first : 1. measure only: find near-duplicates and record how far their cached vector is from the real one, serving nothing approximate embedcache serve -upstream http://localhost:8000 -semantic shadow -semantic-threshold 0.9 the waste report then shows, e.g.: semantic shadow 1240 near-neighbors measured; real-vs-neighbor cosine mean 0.991, worst 0.955 2. only if those cosine deltas are acceptable on YOUR data, turn it on embedcache serve -upstream http://localhost:8000 -semantic active -semantic-threshold 0.93 The similarity signal is character-trigram Jaccard over the input text cheap, no extra model — computing an embedding to detect a duplicate would defeat the purpose . In active mode a match above the threshold is served an existing neighbor's vector an X-Embedcache-Semantic-Hits header marks it ; in shadow mode nothing about serving changes and only the cosine deltas are recorded. This is the one feature that trades exactness for hit rate, so it stays opt-in with the shadow gate in front of it. Embedding calls are idempotent, so transient upstream failures network errors, 5xx, 429 are retried with exponential backoff - -upstream-retries , honoring Retry-After . Sustained failures trip a circuit breaker -breaker-threshold consecutive failures and requests fail fast with 503 instead of stacking timeouts on a dead backend; after -breaker-cooldown a single probe decides whether to close it. Cache hits keep serving while the circuit is open - an upstream outage degrades misses, not the whole service. The request log rotates at -request-log-max-mb so it cannot fill the disk. Breaker state, retries, and fast-fails are exported at /metrics . | flag | default | | |---|---|---| -listen | :8090 | bind address | -upstream | - | required; base URL of the backend | -upstream-api-key | forward client's | pin an upstream key | -admin-token | open | bearer token for admin endpoints | -auth-mode | off | client key validation: allowlist / verify | -api-keys / -api-keys-file | - | client keys for allowlist mode | -auth-cache-ttl | 5m | how long a verified key stays trusted | -ttl | never | expire cached embeddings | -max-batch-items | 2048 | reject oversized batches | -max-body-mb | 64 | reject oversized bodies | -max-entries / -max-memory-mb | 1M / 1024 | LRU bounds | -budget-tokens / -budgets-file | off | per-key upstream-spend limits | -budget-window | 24h | budget reset cadence | -shared-redis | off | Redis/Valkey host:port for a shared cache tier across instances | -shared-redis-ttl / -shared-redis-prefix | 0 / ec: | shared entry expiry and key namespace | -semantic | off | near-duplicate matching: shadow measure only or active | -semantic-threshold | 0.9 | trigram-Jaccard similarity to treat inputs as near-duplicates | -upstream-retries | 2 | retries for transient upstream failures | -breaker-threshold / -breaker-cooldown | 5 / 10s | circuit breaker | -normalize | off | trim,collapse,lowercase,nfc | -persist | off | snapshot file; survives restarts | -request-log | off | JSONL log, feeds analyze | -request-log-max-mb | 512 | rotate the request log at this size | -pricing | built-in | JSON {"model": $/1M, "default": ...} | Admin: POST / ec/flush empties the cache use when a model's weights change under the same name . A full-featured deployment turns on all four layers - measure, reduce, enforce, account: embedcache serve \ -listen 127.0.0.1:8090 \ -upstream http://localhost:8000 \ -admin-token "$ cat /etc/embedcache/admin-token " \ -auth-mode allowlist -api-keys-file /etc/embedcache/keys.txt \ -budgets-file /etc/embedcache/budgets.json -budget-window 24h \ -ttl 720h \ -persist /var/lib/embedcache/cache.snap \ -request-log /var/log/embedcache/requests.jsonl \ -pricing /etc/embedcache/pricing.json - Terminate TLS in front Caddy / nginx / ingress ; embedcache listens in plaintext. - Scrape /metrics with Prometheus; alert on embedcache breaker open == 1 backend down and a falling hit rate workload shifted - rerun analyze . - One instance sustains ~39k cache-hit req/s; for multiple replicas add -shared-redis so they share one cache instead of each starting cold. - See Examples https://ajay6601.github.io/embedcache/examples.html for Docker Compose, Kubernetes, and systemd units. Fixed-size chunking re-slices a document from the edit point onward, so every downstream chunk becomes new text and misses the cache on re-ingestion - the honest limitation documented in EXPERIMENTS.md /Ajay6601/embedcache/blob/main/EXPERIMENTS.md E4 pass 3, 0% absorbed . embedcache chunk splits text at content-defined boundaries FastCDC instead of fixed byte offsets, so an edit only perturbs the chunks touching it - everything else re-chunks to byte-identical text and hits the existing cache normally. embedcache chunk document.md chunks.jsonl one JSON line per chunk: {hash, size, text} cat chunks.jsonl | jq -r .text | ... feed straight into batched embedding calls Proven on one real, live Wikipedia article with one realistic single-sentence edit - CHUNKDIFF.md /Ajay6601/embedcache/blob/main/CHUNKDIFF.md : 93.8% cache hit rate on re-ingest vs 28.6% for fixed-size chunking, through the real proxy and real Ollama inference. Chunk-diff engine embedcache chunk , FastCDC - edits only re-embed the chunks they touch. Per-key/tenant budgets with hard enforcement - the cost-control half of the story. Auth allowlist / upstream-verify , admin-token-guarded endpoints, request-size limits. Resilience - retries, circuit breaker, request-log rotation. Provider breadth - Voyage with input type in the cache key , Mistral, Azure OpenAI. Multi-model validation across six real backends, plus multilingual en/zh/hi/ar/es , long-context, and Unicode-normalization coverage. Recognized-benchmark proof - zero retrieval-quality loss on BEIR SciFact, honest open-loop latency, and a real multi-agent + Python/LangChain cross-language test. Onboarding + release - embedcache demo / embedcache check , and cross-platform prebuilt binaries + a GHCR Docker image on every tag. Shared cache across instances - -shared-redis so a fleet shares one cache instead of each replica starting cold SHAREDCACHE.md /Ajay6601/embedcache/blob/main/SHAREDCACHE.md . Semantic near-duplicate caching - opt-in, shadow-first: measure real-vs-neighbor cosine deltas before ever serving an approximate vector. Two validation fixes - -normalize nfc folds the NFC/NFD duplicate leak; the token estimator now counts CJK/wide scripts per-character instead of per-byte.- reads LiteLLM, OpenAI-SDK, and plain access logs, not just its own JSONL. analyze log adapters Multimodal image embeddings - live end-to-end test against a Voyage multimodal backend cache is already content-agnostic . Full Unicode NFC - the current -normalize nfc folds common Latin combining sequences without a dependency; full Unicode normalization would need golang.org/x/text . Priced dollar case study - a real hosted-API run turning the token-percentage savings into invoice dollars. Request-log PII redaction - opt-in hash-only logging for privacy-sensitive shops. go test ./... unit + integration tests go test -race ./... requires cgo runs in CI go run ./experiments/harness -bin ./embedcache.exe regenerates EXPERIMENTS.md mock; the CI gate go run ./experiments/validate -bin ./embedcache.exe regenerates VALIDATION.md needs local Ollama models go run ./experiments/benchmark -bin ./embedcache.exe regenerates BENCHMARKS.md needs the BEIR SciFact dataset, see experiments/benchmark/data/README.md go run ./experiments/perf -bin ./embedcache.exe regenerates PERF.md go run ./experiments/multiagent -bin ./embedcache.exe regenerates MULTIAGENT.md needs Ollama + python with langchain core go run ./experiments/sharedcache -bin ./embedcache.exe regenerates SHAREDCACHE.md needs a local Redis/Valkey The experiments harness starts the real binary against a deterministic mock upstream and fails the build if any correctness, coalescing, overhead, or savings assertion regresses. The other harnesses run against real local backends and real data this repo's own source, live Wikipedia, the public BEIR dataset and are re-runnable on any machine with the same models pulled. MIT