{"slug": "embedcache-cut-embedding-api-costs-by-caching-redundant-requests", "title": "Embedcache – Cut embedding API costs by caching redundant requests", "summary": "Embedcache, a new open-source proxy tool, reduces embedding API costs by caching redundant requests and eliminating duplicate work before it reaches backends like vLLM, Ollama, or OpenAI. The single-binary service offers exact-match caching, in-flight coalescing, batch deduplication, and per-key token budgets, targeting platform and ML-infra teams processing over 10 million embedding tokens daily.", "body_md": "[Website](https://ajay6601.github.io/embedcache/) · [Documentation](https://ajay6601.github.io/embedcache/docs.html) · [Examples](https://ajay6601.github.io/embedcache/examples.html)\n\n**The cost-control and dedupe layer for embedding APIs.** A single-binary proxy that sits in front of any OpenAI-compatible embedding backend - vLLM, Ollama, text-embeddings-inference, or api.openai.com - and eliminates duplicate work before it reaches your GPUs or your API bill. Then it hands you the invoice for what it saved.\n\n```\nyour app ──► embedcache ──► vLLM / Ollama / TEI / OpenAI\n                 │\n                 ├─ exact-match cache (byte-exact replays, content-addressed)\n                 ├─ in-flight coalescing (N concurrent identical calls → 1 upstream)\n                 ├─ batch dedupe (only the uncached items go upstream)\n                 └─ waste report ($X of your embedding spend was duplicates)\n```\n\nZero dependencies - pure Go stdlib. One static binary. No Python runtime, no Redis required, no framework lock-in.\n\nYou run embedcache as a service (like nginx or Redis - not a library you import), point your apps' `base_url`\n\nat it, and it does four jobs across every language and team sharing it:\n\n**Measure**-`embedcache analyze`\n\nreads your existing request logs and tells you how much of your embedding spend is duplicate work, before you install anything.**Reduce**- an exact-match cache, in-flight coalescing, batch dedupe, and a content-defined chunk-diff engine eliminate that duplicate work in the request path.**Enforce**- per-key/per-tenant token budgets stop runaway agents from burning the backend, while cache hits keep serving.** Account**- live per-model and per-caller spend/savings on`/_ec/stats`\n\n,`/metrics`\n\n(Prometheus), and a human`report`\n\n.\n\nTypical users: platform/ML-infra teams past ~10M embedding tokens/day running vLLM/Ollama plus an API fallback; RAG and agent teams re-embedding corpora and burning tokens in loops.\n\nEvery clip below is a real capture: the commands actually ran, against real backends, and the statuses, latencies, and numbers on screen are what the live proxy returned (recorded by `experiments/gifcap`\n\n, rendered by `tools/termgif`\n\n, transcripts under `experiments/gifcap/samples/`\n\n). The one at the top is embedcache in front of real Ollama, serving a real miss then a real hit.\n\n**RAG: a nightly re-ingest, nothing changed, so nothing is recomputed** ([REALTEST.md](/Ajay6601/embedcache/blob/main/REALTEST.md))\n\n**Agents: two agents with different keys share one cache, and a capped agent is stopped mid-run** ([MULTIAGENT.md](/Ajay6601/embedcache/blob/main/MULTIAGENT.md))\n\n**CLI: point the offline analyzer at a real request log and see your duplicate spend** ([the waste report](#the-waste-report))\n\n| Capability | What it does |\n|---|---|\n| Exact-match cache | Byte-exact replays, content-addressed by `model + dims + encoding + provider params + text` |\n| In-flight coalescing | N concurrent identical requests → 1 upstream computation |\n| Batch dedupe | Only never-seen items in a batch go upstream; correct index mapping |\n| Chunk-diff engine | `embedcache chunk` (FastCDC) so a document edit only re-embeds the chunks it touched |\n| Per-key budgets | Hard token limits per key/tenant; 429 on new spend, hits keep serving |\n| Auth | Allowlist or upstream-verified client keys; admin-token-guarded endpoints |\n| Resilience | Retries + circuit breaker; cache hits serve through upstream outages |\n| Persistence | Snapshot survives restarts; TTL bounds staleness |\n| Waste analytics | Offline analyzer + live `/_ec/stats` , `/metrics` , `report` |\n| Onboarding | `embedcache demo` (zero-setup walkthrough) and `embedcache check` (backend probe + setup helper) |\n| Providers | Ollama, vLLM, TEI, OpenAI, Gemini, Voyage, Mistral, Azure OpenAI |\n\nEmbedding workloads are full of silent duplicate work: re-ingesting a corpus where 95% of chunks didn't change, hot queries embedded thousands of times, retry storms, multiple services embedding the same strings. Frameworks solve this *inside* Python (LangChain's RecordManager, LlamaIndex's IngestionPipeline) - useless if your pipeline is Go, TypeScript, custom, or split across teams. Gateways treat it as a checkbox: LiteLLM's embedding cache has a [known bug](https://github.com/BerriAI/litellm/issues/22659) where mixed cached/uncached batches can return wrong vectors.\n\nEvery number below comes from a real run in this repo - the workloads and edits are real, the inference is real, the reports are regenerated by the code, not hand-written.\n\n| On this workload | embedcache absorbed |\n|---|---|\n| A real RAG agent over 103 live Wikipedia articles, zero constructed duplicates | 49.7% of embedding tokens were duplicate work (\n|\n| A nightly re-ingest where 5% of a 10k-chunk corpus changed | 95.2% of embedding calls absorbed (\n|\n| Re-ingesting a real doc after one real single-sentence edit, with chunk-diff | 93.8% hit rate vs 28.6% for fixed-size chunking (\n|\n| Search-shaped (Zipf) query traffic on a real corpus | ~80% hit rate across six real models (\n|\n| Re-running a full BEIR SciFact eval (300 queries × 5,183 docs) the second time | 100% absorbed, ~2s instead of ~25min, at zero quality change (\n|\n\nYour number depends on your traffic - which is exactly why `embedcache analyze`\n\nmeasures it on your own logs before you deploy anything.\n\nembedcache does one thing, language-agnostically, and provably correctly. Eight tiers of evidence live in the repo, each regenerated by code under `experiments/`\n\nand (the first) re-run by CI on every push:\n\n| Report | What it proves | Data | Backend |\n|---|---|---|---|\n|\n\n[PRODSIM.md](/Ajay6601/embedcache/blob/main/PRODSIM.md)[REALTEST.md](/Ajay6601/embedcache/blob/main/REALTEST.md)**live Wikipedia + real LLM**[CHUNKDIFF.md](/Ajay6601/embedcache/blob/main/CHUNKDIFF.md)** live Wikipedia**[VALIDATION.md](/Ajay6601/embedcache/blob/main/VALIDATION.md)** 6 real models at 3 dims****this repo's Go source + live Wikipedia (en/zh/hi/ar/es)**[BENCHMARKS.md](/Ajay6601/embedcache/blob/main/BENCHMARKS.md)** zero retrieval-quality loss**on a recognized IR benchmark + free eval re-runs** BEIR SciFact: 5,183 real docs, 300 expert-judged queries**[PERF.md](/Ajay6601/embedcache/blob/main/PERF.md)[MULTIAGENT.md](/Ajay6601/embedcache/blob/main/MULTIAGENT.md)** cross-language cache sharing****live Wikipedia + real LLM + real Python/LangChain** The multi-model suite deliberately surfaced real backend behaviors and shows embedcache handling each correctly rather than hiding them: Gemini is not bitwise-deterministic (embedcache stabilizes repeats to one vector), Gemini rate-limits under burst, small-context models reject oversized chunks, and the rough bytes/4 token estimator drifts on CJK/Arabic scripts (it only feeds the *analyzer's* estimate; billed spend is apportioned from the backend's exact reported total, never estimated). None of these are cache defects.\n\n| Correctness claim | Evidence |\n|---|---|\n| Byte-exact responses, correct batch index mapping | 3,453 randomized-batch embeddings, 0 mismatches - covers the LiteLLM\n|\n\n**bit-identical** through the cache vs direct-to-backend; the top-10 ranking is unchanged for all 300 queries ([BENCHMARKS.md](/Ajay6601/embedcache/blob/main/BENCHMARKS.md))[MULTIAGENT.md](/Ajay6601/embedcache/blob/main/MULTIAGENT.md)[PERF.md](/Ajay6601/embedcache/blob/main/PERF.md))*honest boundary:*cache is content-agnostic (proven), but no OpenAI-`/v1/embeddings`\n\n-compatible image backend was reachable to live-test end-to-end - deferred to v0.3\n\n```\n# prebuilt binary (Linux/macOS/Windows, amd64+arm64), no Go needed\n# grab the archive for your platform from the releases page:\n#   https://github.com/Ajay6601/embedcache/releases\n\n# or the Docker image\ndocker run ghcr.io/ajay6601/embedcache:latest serve -upstream http://host.docker.internal:11434\n\n# or with Go 1.22+\ngo install github.com/Ajay6601/embedcache/cmd/embedcache@latest\n\n# or build from a clone\ngit clone https://github.com/Ajay6601/embedcache && cd embedcache\ngo build -o embedcache ./cmd/embedcache\n```\n\n**New here? See it work in one command, with no upstream, no key, no config:**\n\n```\nembedcache demo\n```\n\nIt runs the proxy against a built-in mock model and walks you through a miss, a hit, a\npartial batch, and the live waste report, then leaves the proxy up for you to poke at. When\nyou're ready to point it at a real backend, `embedcache check -upstream URL -model NAME`\n\nprobes it and prints the exact `serve`\n\ncommand to use.\n\n```\n# in front of a self-hosted engine\nembedcache serve -upstream http://localhost:8000        # vLLM\nembedcache serve -upstream http://localhost:11434       # Ollama\n\n# in front of a hosted API\nembedcache serve -upstream https://api.openai.com\n```\n\nPoint your SDK at it - no code changes:\n\n```\nclient = OpenAI(base_url=\"http://localhost:8090/v1\", api_key=\"...\")\nclient.embeddings.create(model=\"text-embedding-3-small\", input=[\"hello\"])\n```\n\nYour `Authorization`\n\nheader is forwarded to the upstream (or pin one with `-upstream-api-key`\n\n). Non-embedding routes pass through untouched, so it can front a full OpenAI-compatible server.\n\n**Verified live** (all `experiments/livecheck`\n\nassertions pass - byte-exact replay, mixed-batch mapping, coalescing, base64):\n\n| backend | upstream flag | verified model |\n|---|---|---|\n| Ollama 0.22 | `-upstream http://localhost:11434` |\n`all-minilm` |\n| Google Gemini | `-upstream https://generativelanguage.googleapis.com/v1beta/openai -upstream-api-key $GEMINI_API_KEY` |\n`gemini-embedding-001` (3072-dim) |\n| OpenAI | `-upstream https://api.openai.com` |\nwire-identical to the above |\n\n**Wire-compatible** (OpenAI-shaped APIs, covered by unit tests): [Voyage AI](https://docs.voyageai.com) (`-upstream https://api.voyageai.com`\n\n- the provider Anthropic officially recommends for embeddings; `input_type`\n\nand other Voyage params are forwarded verbatim and part of the cache identity, so query- and document-typed vectors never collide), Mistral (`-upstream https://api.mistral.ai`\n\n, `mistral-embed`\n\n), and Azure OpenAI (deployment paths, `?api-version=`\n\nquery and `api-key`\n\nheader are mirrored upstream).\n\nNot applicable: Groq and the Anthropic API itself - neither offers an embeddings endpoint (Anthropic recommends Voyage, above).\n\nEvery response tells you what happened:\n\n```\nX-Embedcache-Status: hit | miss | partial\nX-Embedcache-Hits: 14\nX-Embedcache-Saved-Tokens: 1830\n```\n\nand `usage.prompt_tokens`\n\nreflects only what *this* request was actually billed upstream.\n\n**Before installing anything**, run the offline analyzer on an existing request log (JSONL, one embedding request per line - raw bodies or wrapped under `body`\n\n/`request`\n\n/`payload`\n\n):\n\n```\n./embedcache analyze requests.jsonl\nembedcache offline waste analysis\n=================================\nrequests analyzed        20000\nembedding items          20000\nunique items             2052\nduplicate items          17948   (89.7% of all items)\nestimated tokens         340217   (~$0.01)\nestimated wasted tokens  305090   (~$0.01)\n\n>> 89.7% of this embedding spend was duplicate work an exact-match\n>> cache would have absorbed.\n```\n\nFor self-hosted models with no list price, set an amortized GPU cost: `-default-price 0.05`\n\n($/1M tokens), or supply `-pricing costs.json`\n\n.\n\nA running proxy serves the same accounting live: `embedcache report`\n\n, or `GET /_ec/stats`\n\n(JSON), `/_ec/report`\n\n(text), `/metrics`\n\n(Prometheus).\n\n**Byte-exact by default.** Cached responses replay the upstream's raw JSON - no float re-encoding, no precision drift.`model`\n\n,`dimensions`\n\n, and`encoding_format`\n\nare all part of the cache key.**Normalization is opt-in.**`-normalize trim,collapse,lowercase`\n\nwidens matching if you accept the semantics; the default matches exact bytes only.**No semantic caching.** Similarity-threshold response caching returns*wrong answers*at some rate; that trade-off belongs to chat responses, not embeddings. Exact-match embedding caching has a 0% wrong-answer rate by construction.**Failure containment.** Upstream errors pass through verbatim and are never cached; a crashed leader fails its coalesced waiters instead of hanging them.\n\nFor anything beyond a trusted network, turn on both layers:\n\n```\nembedcache serve -upstream http://localhost:8000 \\\n  -admin-token  \"$(openssl rand -hex 24)\" \\   # guards /stats, /report, /metrics, /_ec/flush\n  -auth-mode    allowlist -api-keys-file keys.txt \\\n  -ttl          720h                           # bound staleness if model weights change\n```\n\n- without it, anyone who can reach the port can flush your cache and read usage stats.`-admin-token`\n\n`/healthz`\n\nstays open for probes.- cache hits never touch the upstream, so without this a revoked or missing key can still read cached vectors.`-auth-mode`\n\n`allowlist`\n\nis for self-hosted backends (the proxy owns the key list);`verify`\n\nis for hosted providers - the caller's key is checked against the upstream (`GET /v1/models`\n\n) and the verdict cached for`-auth-cache-ttl`\n\n(default 5m, negative verdicts 1m, fail-closed on upstream outage).- Terminate TLS in front (Caddy, nginx, or your ingress); embedcache listens in plaintext.\n\nHard per-key limits on upstream spend, the enforcement half of the cost story:\n\n```\nembedcache serve -upstream http://localhost:8000 \\\n  -budgets-file budgets.json -budget-window 24h\n\n# budgets.json: tokens per key per window; 0 = unlimited, \"default\" applies to the rest\n{ \"team-search-x9f\": 2000000, \"team-agents-p7q\": 500000, \"default\": 1000000 }\n```\n\nBudgets bound **spend, not reads**: only tokens actually billed upstream count, and once a key is exhausted, requests needing new computation get `429`\n\n(with `Retry-After`\n\nfor the window reset) while **cache hits keep serving**, so a capped team's existing workload stays alive; only new cost is blocked. Every response carries `X-Embedcache-Budget-Remaining`\n\nso clients can self-moderate, per-key state is on `/_ec/stats`\n\n, and rejections export as `embedcache_budget_rejects_total`\n\n. Counters are in-memory and reset at each window boundary (and on restart), like any in-process rate limiter.\n\nA single instance keeps its cache in memory, so the moment you run more than one replica behind a load balancer each starts cold and the fleet recomputes the same vectors. Point them all at one Redis (or Valkey) and they share entries:\n\n```\nembedcache serve -upstream http://localhost:8000 \\\n  -shared-redis 127.0.0.1:6379 -shared-redis-ttl 720h\n```\n\nOn a local miss the proxy consults Redis before going upstream; a hit there warms the local cache and skips the backend call, byte-identically. The Redis client is a small in-house RESP implementation, so this stays a single dependency-free static binary. Proven for real in [SHAREDCACHE.md](/Ajay6601/embedcache/blob/main/SHAREDCACHE.md): two separate processes sharing one Redis, where the second serves the first's vector with **no upstream call**. (The tier is read-through/write-through and best-effort — if Redis is unreachable, instances fall back to computing, never erroring.)\n\nExact-match caching caps savings at your literal duplicate share. Near-duplicate matching — \"how do I reset my password\" vs \"how do I reset my password?\" — can push higher, but serving one input's vector for another can silently poison a vector store, so it is **off by default and shadow-first**:\n\n```\n# 1. measure only: find near-duplicates and record how far their cached vector\n#    is from the real one, serving nothing approximate\nembedcache serve -upstream http://localhost:8000 -semantic shadow -semantic-threshold 0.9\n\n# the waste report then shows, e.g.:\n#   semantic shadow   1240 near-neighbors measured; real-vs-neighbor cosine mean 0.991, worst 0.955\n\n# 2. only if those cosine deltas are acceptable on YOUR data, turn it on\nembedcache serve -upstream http://localhost:8000 -semantic active -semantic-threshold 0.93\n```\n\nThe similarity signal is character-trigram Jaccard over the input text (cheap, no extra model — computing an embedding to detect a duplicate would defeat the purpose). In `active`\n\nmode a match above the threshold is served an existing neighbor's vector (an `X-Embedcache-Semantic-Hits`\n\nheader marks it); in `shadow`\n\nmode nothing about serving changes and only the cosine deltas are recorded. This is the one feature that trades exactness for hit rate, so it stays opt-in with the shadow gate in front of it.\n\nEmbedding calls are idempotent, so transient upstream failures (network errors, 5xx, 429) are retried with exponential backoff - `-upstream-retries`\n\n, honoring `Retry-After`\n\n. Sustained failures trip a circuit breaker (`-breaker-threshold`\n\nconsecutive failures) and requests fail fast with 503 instead of stacking timeouts on a dead backend; after `-breaker-cooldown`\n\na single probe decides whether to close it. Cache hits keep serving while the circuit is open - an upstream outage degrades misses, not the whole service. The request log rotates at `-request-log-max-mb`\n\nso it cannot fill the disk. Breaker state, retries, and fast-fails are exported at `/metrics`\n\n.\n\n| flag | default | |\n|---|---|---|\n`-listen` |\n`:8090` |\nbind address |\n`-upstream` |\n- | required; base URL of the backend |\n`-upstream-api-key` |\nforward client's | pin an upstream key |\n`-admin-token` |\nopen | bearer token for admin endpoints |\n`-auth-mode` |\n`off` |\nclient key validation: `allowlist` / `verify` |\n`-api-keys` / `-api-keys-file` |\n- | client keys for allowlist mode |\n`-auth-cache-ttl` |\n5m | how long a verified key stays trusted |\n`-ttl` |\nnever | expire cached embeddings |\n`-max-batch-items` |\n2048 | reject oversized batches |\n`-max-body-mb` |\n64 | reject oversized bodies |\n`-max-entries` / `-max-memory-mb` |\n1M / 1024 | LRU bounds |\n`-budget-tokens` / `-budgets-file` |\noff | per-key upstream-spend limits |\n`-budget-window` |\n24h | budget reset cadence |\n`-shared-redis` |\noff | Redis/Valkey host:port for a shared cache tier across instances |\n`-shared-redis-ttl` / `-shared-redis-prefix` |\n0 / `ec:` |\nshared entry expiry and key namespace |\n`-semantic` |\noff | near-duplicate matching: `shadow` (measure only) or `active` |\n`-semantic-threshold` |\n0.9 | trigram-Jaccard similarity to treat inputs as near-duplicates |\n`-upstream-retries` |\n2 | retries for transient upstream failures |\n`-breaker-threshold` / `-breaker-cooldown` |\n5 / 10s | circuit breaker |\n`-normalize` |\noff | `trim,collapse,lowercase,nfc` |\n`-persist` |\noff | snapshot file; survives restarts |\n`-request-log` |\noff | JSONL log, feeds `analyze` |\n`-request-log-max-mb` |\n512 | rotate the request log at this size |\n`-pricing` |\nbuilt-in | JSON `{\"model\": $/1M, \"default\": ...}` |\n\nAdmin: `POST /_ec/flush`\n\nempties the cache (use when a model's weights change under the same name).\n\nA full-featured deployment turns on all four layers - measure, reduce, enforce, account:\n\n```\nembedcache serve \\\n  -listen       127.0.0.1:8090 \\\n  -upstream     http://localhost:8000 \\\n  -admin-token  \"$(cat /etc/embedcache/admin-token)\" \\\n  -auth-mode    allowlist -api-keys-file /etc/embedcache/keys.txt \\\n  -budgets-file /etc/embedcache/budgets.json -budget-window 24h \\\n  -ttl          720h \\\n  -persist      /var/lib/embedcache/cache.snap \\\n  -request-log  /var/log/embedcache/requests.jsonl \\\n  -pricing      /etc/embedcache/pricing.json\n```\n\n- Terminate TLS in front (Caddy / nginx / ingress); embedcache listens in plaintext.\n- Scrape\n`/metrics`\n\nwith Prometheus; alert on`embedcache_breaker_open == 1`\n\n(backend down) and a falling`hit_rate`\n\n(workload shifted - rerun`analyze`\n\n). - One instance sustains ~39k cache-hit req/s; for multiple replicas add\n`-shared-redis`\n\nso they share one cache instead of each starting cold. - See\n[Examples](https://ajay6601.github.io/embedcache/examples.html)for Docker Compose, Kubernetes, and systemd units.\n\nFixed-size chunking re-slices a document from the edit point onward, so every downstream chunk becomes new text and misses the cache on re-ingestion - the honest limitation documented in [EXPERIMENTS.md](/Ajay6601/embedcache/blob/main/EXPERIMENTS.md) (E4 pass 3, 0% absorbed). `embedcache chunk`\n\nsplits text at content-defined boundaries (FastCDC) instead of fixed byte offsets, so an edit only perturbs the chunks touching it - everything else re-chunks to byte-identical text and hits the existing cache normally.\n\n```\nembedcache chunk document.md > chunks.jsonl        # one JSON line per chunk: {hash, size, text}\ncat chunks.jsonl | jq -r .text | ...                # feed straight into batched embedding calls\n```\n\nProven on one real, live Wikipedia article with one realistic single-sentence edit - [CHUNKDIFF.md](/Ajay6601/embedcache/blob/main/CHUNKDIFF.md): **93.8%** cache hit rate on re-ingest vs **28.6%** for fixed-size chunking, through the real proxy and real Ollama inference.\n\n**Chunk-diff engine**(`embedcache chunk`\n\n, FastCDC) - edits only re-embed the chunks they touch.**Per-key/tenant budgets** with hard enforcement - the cost-control half of the story.**Auth**(allowlist / upstream-verify), admin-token-guarded endpoints, request-size limits.** Resilience**- retries, circuit breaker, request-log rotation.** Provider breadth**- Voyage (with`input_type`\n\nin the cache key), Mistral, Azure OpenAI.**Multi-model validation** across six real backends, plus multilingual (en/zh/hi/ar/es), long-context, and Unicode-normalization coverage.**Recognized-benchmark proof**- zero retrieval-quality loss on BEIR SciFact, honest open-loop latency, and a real multi-agent + Python/LangChain cross-language test.**Onboarding + release**-`embedcache demo`\n\n/`embedcache check`\n\n, and cross-platform prebuilt binaries + a GHCR Docker image on every tag.**Shared cache across instances**-`-shared-redis`\n\nso a fleet shares one cache instead of each replica starting cold ([SHAREDCACHE.md](/Ajay6601/embedcache/blob/main/SHAREDCACHE.md)).**Semantic (near-duplicate) caching**- opt-in, shadow-first: measure real-vs-neighbor cosine deltas before ever serving an approximate vector.** Two validation fixes**-`-normalize nfc`\n\nfolds the NFC/NFD duplicate leak; the token estimator now counts CJK/wide scripts per-character instead of per-byte.- reads LiteLLM, OpenAI-SDK, and plain access logs, not just its own JSONL.`analyze`\n\nlog adapters\n\n**Multimodal image embeddings**- live end-to-end test against a Voyage multimodal backend (cache is already content-agnostic).** Full Unicode NFC**- the current`-normalize nfc`\n\nfolds common Latin combining sequences without a dependency; full Unicode normalization would need`golang.org/x/text`\n\n.**Priced dollar case study**- a real hosted-API run turning the token-percentage savings into invoice dollars.** Request-log PII redaction**- opt-in hash-only logging for privacy-sensitive shops.\n\n```\ngo test ./...                                          # unit + integration tests\ngo test -race ./...                                    # requires cgo (runs in CI)\ngo run ./experiments/harness    -bin ./embedcache.exe  # regenerates EXPERIMENTS.md (mock; the CI gate)\ngo run ./experiments/validate   -bin ./embedcache.exe  # regenerates VALIDATION.md (needs local Ollama models)\ngo run ./experiments/benchmark  -bin ./embedcache.exe  # regenerates BENCHMARKS.md (needs the BEIR SciFact dataset, see experiments/benchmark/data/README.md)\ngo run ./experiments/perf       -bin ./embedcache.exe  # regenerates PERF.md\ngo run ./experiments/multiagent -bin ./embedcache.exe  # regenerates MULTIAGENT.md (needs Ollama + python with langchain_core)\ngo run ./experiments/sharedcache -bin ./embedcache.exe  # regenerates SHAREDCACHE.md (needs a local Redis/Valkey)\n```\n\nThe experiments harness starts the real binary against a deterministic mock upstream and fails the build if any correctness, coalescing, overhead, or savings assertion regresses. The other harnesses run against real local backends and real data (this repo's own source, live Wikipedia, the public BEIR dataset) and are re-runnable on any machine with the same models pulled.\n\nMIT", "url": "https://wpnews.pro/news/embedcache-cut-embedding-api-costs-by-caching-redundant-requests", "canonical_source": "https://github.com/Ajay6601/embedcache", "published_at": "2026-07-13 00:29:02+00:00", "updated_at": "2026-07-13 01:05:42.676623+00:00", "lang": "en", "topics": ["ai-tools", "ai-infrastructure", "ai-products"], "entities": ["Embedcache", "vLLM", "Ollama", "OpenAI", "Gemini", "Voyage", "Mistral", "Azure OpenAI"], "alternates": {"html": "https://wpnews.pro/news/embedcache-cut-embedding-api-costs-by-caching-redundant-requests", "markdown": "https://wpnews.pro/news/embedcache-cut-embedding-api-costs-by-caching-redundant-requests.md", "text": "https://wpnews.pro/news/embedcache-cut-embedding-api-costs-by-caching-redundant-requests.txt", "jsonld": "https://wpnews.pro/news/embedcache-cut-embedding-api-costs-by-caching-redundant-requests.jsonld"}}