Website · Documentation · Examples
The cost-control and dedupe layer for embedding APIs. A single-binary proxy that sits in front of any OpenAI-compatible embedding backend - vLLM, Ollama, text-embeddings-inference, or api.openai.com - and eliminates duplicate work before it reaches your GPUs or your API bill. Then it hands you the invoice for what it saved.
your app ──► embedcache ──► vLLM / Ollama / TEI / OpenAI
│
├─ exact-match cache (byte-exact replays, content-addressed)
├─ in-flight coalescing (N concurrent identical calls → 1 upstream)
├─ batch dedupe (only the uncached items go upstream)
└─ waste report ($X of your embedding spend was duplicates)
Zero dependencies - pure Go stdlib. One static binary. No Python runtime, no Redis required, no framework lock-in.
You run embedcache as a service (like nginx or Redis - not a library you import), point your apps' base_url
at it, and it does four jobs across every language and team sharing it:
Measure-embedcache analyze
reads your existing request logs and tells you how much of your embedding spend is duplicate work, before you install anything.Reduce- an exact-match cache, in-flight coalescing, batch dedupe, and a content-defined chunk-diff engine eliminate that duplicate work in the request path.Enforce- per-key/per-tenant token budgets stop runaway agents from burning the backend, while cache hits keep serving.** Account**- live per-model and per-caller spend/savings on/_ec/stats
,/metrics
(Prometheus), and a humanreport
.
Typical users: platform/ML-infra teams past ~10M embedding tokens/day running vLLM/Ollama plus an API fallback; RAG and agent teams re-embedding corpora and burning tokens in loops.
Every clip below is a real capture: the commands actually ran, against real backends, and the statuses, latencies, and numbers on screen are what the live proxy returned (recorded by experiments/gifcap
, rendered by tools/termgif
, transcripts under experiments/gifcap/samples/
). The one at the top is embedcache in front of real Ollama, serving a real miss then a real hit.
RAG: a nightly re-ingest, nothing changed, so nothing is recomputed (REALTEST.md)
Agents: two agents with different keys share one cache, and a capped agent is stopped mid-run (MULTIAGENT.md)
CLI: point the offline analyzer at a real request log and see your duplicate spend (the waste report)
| Capability | What it does |
|---|---|
| Exact-match cache | Byte-exact replays, content-addressed by model + dims + encoding + provider params + text |
| In-flight coalescing | N concurrent identical requests → 1 upstream computation |
| Batch dedupe | Only never-seen items in a batch go upstream; correct index mapping |
| Chunk-diff engine | embedcache chunk (FastCDC) so a document edit only re-embeds the chunks it touched |
| Per-key budgets | Hard token limits per key/tenant; 429 on new spend, hits keep serving |
| Auth | Allowlist or upstream-verified client keys; admin-token-guarded endpoints |
| Resilience | Retries + circuit breaker; cache hits serve through upstream outages |
| Persistence | Snapshot survives restarts; TTL bounds staleness |
| Waste analytics | Offline analyzer + live /_ec/stats , /metrics , report |
| Onboarding | embedcache demo (zero-setup walkthrough) and embedcache check (backend probe + setup helper) |
| Providers | Ollama, vLLM, TEI, OpenAI, Gemini, Voyage, Mistral, Azure OpenAI |
Embedding workloads are full of silent duplicate work: re-ingesting a corpus where 95% of chunks didn't change, hot queries embedded thousands of times, retry storms, multiple services embedding the same strings. Frameworks solve this inside Python (LangChain's RecordManager, LlamaIndex's IngestionPipeline) - useless if your pipeline is Go, TypeScript, custom, or split across teams. Gateways treat it as a checkbox: LiteLLM's embedding cache has a known bug where mixed cached/uncached batches can return wrong vectors.
Every number below comes from a real run in this repo - the workloads and edits are real, the inference is real, the reports are regenerated by the code, not hand-written.
| On this workload | embedcache absorbed |
|---|---|
| A real RAG agent over 103 live Wikipedia articles, zero constructed duplicates | 49.7% of embedding tokens were duplicate work ( |
| A nightly re-ingest where 5% of a 10k-chunk corpus changed | 95.2% of embedding calls absorbed ( |
| Re-ingesting a real doc after one real single-sentence edit, with chunk-diff | 93.8% hit rate vs 28.6% for fixed-size chunking ( |
| Search-shaped (Zipf) query traffic on a real corpus | ~80% hit rate across six real models ( |
| Re-running a full BEIR SciFact eval (300 queries × 5,183 docs) the second time | 100% absorbed, ~2s instead of ~25min, at zero quality change ( |
Your number depends on your traffic - which is exactly why embedcache analyze
measures it on your own logs before you deploy anything.
embedcache does one thing, language-agnostically, and provably correctly. Eight tiers of evidence live in the repo, each regenerated by code under experiments/
and (the first) re-run by CI on every push:
| Report | What it proves | Data | Backend |
|---|---|---|---|
PRODSIM.mdREALTEST.mdlive Wikipedia + real LLMCHUNKDIFF.md** live Wikipedia**VALIDATION.md** 6 real models at 3 dimsthis repo's Go source + live Wikipedia (en/zh/hi/ar/es)BENCHMARKS.md zero retrieval-quality losson a recognized IR benchmark + free eval re-runs BEIR SciFact: 5,183 real docs, 300 expert-judged queries**PERF.mdMULTIAGENT.md** cross-language cache sharinglive Wikipedia + real LLM + real Python/LangChain** The multi-model suite deliberately surfaced real backend behaviors and shows embedcache handling each correctly rather than hiding them: Gemini is not bitwise-deterministic (embedcache stabilizes repeats to one vector), Gemini rate-limits under burst, small-context models reject oversized chunks, and the rough bytes/4 token estimator drifts on CJK/Arabic scripts (it only feeds the analyzer's estimate; billed spend is apportioned from the backend's exact reported total, never estimated). None of these are cache defects.
| Correctness claim | Evidence |
|---|---|
| Byte-exact responses, correct batch index mapping | 3,453 randomized-batch embeddings, 0 mismatches - covers the LiteLLM |
bit-identical through the cache vs direct-to-backend; the top-10 ranking is unchanged for all 300 queries (BENCHMARKS.md)MULTIAGENT.mdPERF.md)*honest boundary:*cache is content-agnostic (proven), but no OpenAI-/v1/embeddings
-compatible image backend was reachable to live-test end-to-end - deferred to v0.3
docker run ghcr.io/ajay6601/embedcache:latest serve -upstream http://host.docker.internal:11434
go install github.com/Ajay6601/embedcache/cmd/embedcache@latest
git clone https://github.com/Ajay6601/embedcache && cd embedcache
go build -o embedcache ./cmd/embedcache
New here? See it work in one command, with no upstream, no key, no config:
embedcache demo
It runs the proxy against a built-in mock model and walks you through a miss, a hit, a
partial batch, and the live waste report, then leaves the proxy up for you to poke at. When
you're ready to point it at a real backend, embedcache check -upstream URL -model NAME
probes it and prints the exact serve
command to use.
embedcache serve -upstream http://localhost:8000 # vLLM
embedcache serve -upstream http://localhost:11434 # Ollama
embedcache serve -upstream https://api.openai.com
Point your SDK at it - no code changes:
client = OpenAI(base_url="http://localhost:8090/v1", api_key="...")
client.embeddings.create(model="text-embedding-3-small", input=["hello"])
Your Authorization
header is forwarded to the upstream (or pin one with -upstream-api-key
). Non-embedding routes pass through untouched, so it can front a full OpenAI-compatible server.
Verified live (all experiments/livecheck
assertions pass - byte-exact replay, mixed-batch mapping, coalescing, base64):
| backend | upstream flag | verified model |
|---|---|---|
| Ollama 0.22 | -upstream http://localhost:11434 |
|
all-minilm |
||
| Google Gemini | -upstream https://generativelanguage.googleapis.com/v1beta/openai -upstream-api-key $GEMINI_API_KEY |
|
gemini-embedding-001 (3072-dim) |
||
| OpenAI | -upstream https://api.openai.com |
|
| wire-identical to the above |
Wire-compatible (OpenAI-shaped APIs, covered by unit tests): Voyage AI (-upstream https://api.voyageai.com
- the provider Anthropic officially recommends for embeddings;
input_type
and other Voyage params are forwarded verbatim and part of the cache identity, so query- and document-typed vectors never collide), Mistral (-upstream https://api.mistral.ai
, mistral-embed
), and Azure OpenAI (deployment paths, ?api-version=
query and api-key
header are mirrored upstream).
Not applicable: Groq and the Anthropic API itself - neither offers an embeddings endpoint (Anthropic recommends Voyage, above).
Every response tells you what happened:
X-Embedcache-Status: hit | miss | partial
X-Embedcache-Hits: 14
X-Embedcache-Saved-Tokens: 1830
and usage.prompt_tokens
reflects only what this request was actually billed upstream.
Before installing anything, run the offline analyzer on an existing request log (JSONL, one embedding request per line - raw bodies or wrapped under body
/request
/payload
):
./embedcache analyze requests.jsonl
requests analyzed 20000
embedding items 20000
unique items 2052
duplicate items 17948 (89.7% of all items)
estimated tokens 340217 (~$0.01)
estimated wasted tokens 305090 (~$0.01)
>> 89.7% of this embedding spend was duplicate work an exact-match
>> cache would have absorbed.
For self-hosted models with no list price, set an amortized GPU cost: -default-price 0.05
($/1M tokens), or supply -pricing costs.json
.
A running proxy serves the same accounting live: embedcache report
, or GET /_ec/stats
(JSON), /_ec/report
(text), /metrics
(Prometheus).
Byte-exact by default. Cached responses replay the upstream's raw JSON - no float re-encoding, no precision drift.model
,dimensions
, andencoding_format
are all part of the cache key.Normalization is opt-in.-normalize trim,collapse,lowercase
widens matching if you accept the semantics; the default matches exact bytes only.No semantic caching. Similarity-threshold response caching returnswrong answersat some rate; that trade-off belongs to chat responses, not embeddings. Exact-match embedding caching has a 0% wrong-answer rate by construction.Failure containment. Upstream errors pass through verbatim and are never cached; a crashed leader fails its coalesced waiters instead of hanging them.
For anything beyond a trusted network, turn on both layers:
embedcache serve -upstream http://localhost:8000 \
-admin-token "$(openssl rand -hex 24)" \ # guards /stats, /report, /metrics, /_ec/flush
-auth-mode allowlist -api-keys-file keys.txt \
-ttl 720h # bound staleness if model weights change
- without it, anyone who can reach the port can flush your cache and read usage stats.
-admin-token
/healthz
stays open for probes.- cache hits never touch the upstream, so without this a revoked or missing key can still read cached vectors.-auth-mode
allowlist
is for self-hosted backends (the proxy owns the key list);verify
is for hosted providers - the caller's key is checked against the upstream (GET /v1/models
) and the verdict cached for-auth-cache-ttl
(default 5m, negative verdicts 1m, fail-closed on upstream outage).- Terminate TLS in front (Caddy, nginx, or your ingress); embedcache listens in plaintext.
Hard per-key limits on upstream spend, the enforcement half of the cost story:
embedcache serve -upstream http://localhost:8000 \
-budgets-file budgets.json -budget-window 24h
{ "team-search-x9f": 2000000, "team-agents-p7q": 500000, "default": 1000000 }
Budgets bound spend, not reads: only tokens actually billed upstream count, and once a key is exhausted, requests needing new computation get 429
(with Retry-After
for the window reset) while cache hits keep serving, so a capped team's existing workload stays alive; only new cost is blocked. Every response carries X-Embedcache-Budget-Remaining
so clients can self-moderate, per-key state is on /_ec/stats
, and rejections export as embedcache_budget_rejects_total
. Counters are in-memory and reset at each window boundary (and on restart), like any in-process rate limiter.
A single instance keeps its cache in memory, so the moment you run more than one replica behind a load balancer each starts cold and the fleet recomputes the same vectors. Point them all at one Redis (or Valkey) and they share entries:
embedcache serve -upstream http://localhost:8000 \
-shared-redis 127.0.0.1:6379 -shared-redis-ttl 720h
On a local miss the proxy consults Redis before going upstream; a hit there warms the local cache and skips the backend call, byte-identically. The Redis client is a small in-house RESP implementation, so this stays a single dependency-free static binary. Proven for real in SHAREDCACHE.md: two separate processes sharing one Redis, where the second serves the first's vector with no upstream call. (The tier is read-through/write-through and best-effort — if Redis is unreachable, instances fall back to computing, never erroring.)
Exact-match caching caps savings at your literal duplicate share. Near-duplicate matching — "how do I reset my password" vs "how do I reset my password?" — can push higher, but serving one input's vector for another can silently poison a vector store, so it is off by default and shadow-first:
embedcache serve -upstream http://localhost:8000 -semantic shadow -semantic-threshold 0.9
embedcache serve -upstream http://localhost:8000 -semantic active -semantic-threshold 0.93
The similarity signal is character-trigram Jaccard over the input text (cheap, no extra model — computing an embedding to detect a duplicate would defeat the purpose). In active
mode a match above the threshold is served an existing neighbor's vector (an X-Embedcache-Semantic-Hits
header marks it); in shadow
mode nothing about serving changes and only the cosine deltas are recorded. This is the one feature that trades exactness for hit rate, so it stays opt-in with the shadow gate in front of it.
Embedding calls are idempotent, so transient upstream failures (network errors, 5xx, 429) are retried with exponential backoff - -upstream-retries
, honoring Retry-After
. Sustained failures trip a circuit breaker (-breaker-threshold
consecutive failures) and requests fail fast with 503 instead of stacking timeouts on a dead backend; after -breaker-cooldown
a single probe decides whether to close it. Cache hits keep serving while the circuit is open - an upstream outage degrades misses, not the whole service. The request log rotates at -request-log-max-mb
so it cannot fill the disk. Breaker state, retries, and fast-fails are exported at /metrics
.
| flag | default | |
|---|---|---|
-listen |
||
:8090 |
||
| bind address | ||
-upstream |
- | required; base URL of the backend |
-upstream-api-key| forward client's | pin an upstream key |-admin-token| open | bearer token for admin endpoints |-auth-mode|off| client key validation:allowlist/verify|-api-keys/-api-keys-file| - | client keys for allowlist mode |
-auth-cache-ttl| 5m | how long a verified key stays trusted |-ttl| never | expire cached embeddings |-max-batch-items| 2048 | reject oversized batches |-max-body-mb| 64 | reject oversized bodies |-max-entries/-max-memory-mb| 1M / 1024 | LRU bounds |-budget-tokens/-budgets-file| off | per-key upstream-spend limits |-budget-window| 24h | budget reset cadence |-shared-redis| off | Redis/Valkey host:port for a shared cache tier across instances |-shared-redis-ttl/-shared-redis-prefix| 0 /ec:| shared entry expiry and key namespace |-semantic| off | near-duplicate matching:shadow(measure only) oractive|-semantic-threshold| 0.9 | trigram-Jaccard similarity to treat inputs as near-duplicates |-upstream-retries| 2 | retries for transient upstream failures |-breaker-threshold/-breaker-cooldown| 5 / 10s | circuit breaker |-normalize| off |trim,collapse,lowercase,nfc|-persist| off | snapshot file; survives restarts |-request-log| off | JSONL log, feedsanalyze|-request-log-max-mb| 512 | rotate the request log at this size |-pricing| built-in | JSON{"model": $/1M, "default": ...}|
Admin: POST /_ec/flush
empties the cache (use when a model's weights change under the same name).
A full-featured deployment turns on all four layers - measure, reduce, enforce, account:
embedcache serve \
-listen 127.0.0.1:8090 \
-upstream http://localhost:8000 \
-admin-token "$(cat /etc/embedcache/admin-token)" \
-auth-mode allowlist -api-keys-file /etc/embedcache/keys.txt \
-budgets-file /etc/embedcache/budgets.json -budget-window 24h \
-ttl 720h \
-persist /var/lib/embedcache/cache.snap \
-request-log /var/log/embedcache/requests.jsonl \
-pricing /etc/embedcache/pricing.json
- Terminate TLS in front (Caddy / nginx / ingress); embedcache listens in plaintext.
- Scrape
/metrics
with Prometheus; alert onembedcache_breaker_open == 1
(backend down) and a fallinghit_rate
(workload shifted - rerunanalyze
). - One instance sustains ~39k cache-hit req/s; for multiple replicas add
-shared-redis
so they share one cache instead of each starting cold. - See Examplesfor Docker Compose, Kubernetes, and systemd units.
Fixed-size chunking re-slices a document from the edit point onward, so every downstream chunk becomes new text and misses the cache on re-ingestion - the honest limitation documented in EXPERIMENTS.md (E4 pass 3, 0% absorbed). embedcache chunk
splits text at content-defined boundaries (FastCDC) instead of fixed byte offsets, so an edit only perturbs the chunks touching it - everything else re-chunks to byte-identical text and hits the existing cache normally.
embedcache chunk document.md > chunks.jsonl # one JSON line per chunk: {hash, size, text}
cat chunks.jsonl | jq -r .text | ... # feed straight into batched embedding calls
Proven on one real, live Wikipedia article with one realistic single-sentence edit - CHUNKDIFF.md: 93.8% cache hit rate on re-ingest vs 28.6% for fixed-size chunking, through the real proxy and real Ollama inference.
Chunk-diff engine(embedcache chunk
, FastCDC) - edits only re-embed the chunks they touch.Per-key/tenant budgets with hard enforcement - the cost-control half of the story.Auth(allowlist / upstream-verify), admin-token-guarded endpoints, request-size limits.** Resilience**- retries, circuit breaker, request-log rotation.** Provider breadth**- Voyage (withinput_type
in the cache key), Mistral, Azure OpenAI.Multi-model validation across six real backends, plus multilingual (en/zh/hi/ar/es), long-context, and Unicode-normalization coverage.Recognized-benchmark proof- zero retrieval-quality loss on BEIR SciFact, honest open-loop latency, and a real multi-agent + Python/LangChain cross-language test.Onboarding + release-embedcache demo
/embedcache check
, and cross-platform prebuilt binaries + a GHCR Docker image on every tag.Shared cache across instances--shared-redis
so a fleet shares one cache instead of each replica starting cold (SHAREDCACHE.md).Semantic (near-duplicate) caching- opt-in, shadow-first: measure real-vs-neighbor cosine deltas before ever serving an approximate vector.** Two validation fixes**--normalize nfc
folds the NFC/NFD duplicate leak; the token estimator now counts CJK/wide scripts per-character instead of per-byte.- reads LiteLLM, OpenAI-SDK, and plain access logs, not just its own JSONL.analyze
log adapters
Multimodal image embeddings- live end-to-end test against a Voyage multimodal backend (cache is already content-agnostic).** Full Unicode NFC**- the current-normalize nfc
folds common Latin combining sequences without a dependency; full Unicode normalization would needgolang.org/x/text
.Priced dollar case study- a real hosted-API run turning the token-percentage savings into invoice dollars.** Request-log PII redaction**- opt-in hash-only logging for privacy-sensitive shops.
go test ./... # unit + integration tests
go test -race ./... # requires cgo (runs in CI)
go run ./experiments/harness -bin ./embedcache.exe # regenerates EXPERIMENTS.md (mock; the CI gate)
go run ./experiments/validate -bin ./embedcache.exe # regenerates VALIDATION.md (needs local Ollama models)
go run ./experiments/benchmark -bin ./embedcache.exe # regenerates BENCHMARKS.md (needs the BEIR SciFact dataset, see experiments/benchmark/data/README.md)
go run ./experiments/perf -bin ./embedcache.exe # regenerates PERF.md
go run ./experiments/multiagent -bin ./embedcache.exe # regenerates MULTIAGENT.md (needs Ollama + python with langchain_core)
go run ./experiments/sharedcache -bin ./embedcache.exe # regenerates SHAREDCACHE.md (needs a local Redis/Valkey)
The experiments harness starts the real binary against a deterministic mock upstream and fails the build if any correctness, coalescing, overhead, or savings assertion regresses. The other harnesses run against real local backends and real data (this repo's own source, live Wikipedia, the public BEIR dataset) and are re-runnable on any machine with the same models pulled.
MIT