{"slug": "cacheverifier-check-if-verifying-llm-cache-hits-helps-your-traffic", "title": "CacheVerifier, check if verifying LLM cache hits helps your traffic", "summary": "CacheVerifier launched a Python client for its hosted API that approves or rejects semantic-cache hits before an application serves a cached answer, aimed at catching similarity matches that are close but wrong. The client requires Python 3.9+ and depends only on httpx at runtime, with opt-in extras for a GPTCache adapter and an offline Health Check; verify() defaults to a 1s timeout and fails closed, returning approved=False with degraded=True on timeouts, connection errors or 5xx responses, while 4xx errors raise CacheVerifierError. Self-serve verification and fine-tuning are free, and the company states there is no formal uptime SLA yet, which it cites as the reason the fallback path is a built-in default.", "body_md": "Python client for **[CacheVerifier](https://www.cacheverifier.com)** — a hosted API that\nverifies semantic-cache hits. Given a query and a candidate cached answer, it approves or\nrejects serving that answer from cache, so a similarity match that is *close but wrong*\ndoesn't become a silent error in your app.\n\nCacheVerifier does **not** run your cache or do similarity search. Your cache backend does\nits own lookup first; you call `verify()` only on the candidates in the similarity \"gray\nzone\", where a plain threshold match might be wrong.\n\n- Docs / API reference: [https://www.cacheverifier.com/docs](https://www.cacheverifier.com/docs)\n- Why similarity ≠ correctness: [https://www.cacheverifier.com/why-similarity-fails](https://www.cacheverifier.com/why-similarity-fails)\n- The research behind it (paper + benchmarks): [https://github.com/imxinchengyou/CacheVerifier](https://github.com/imxinchengyou/CacheVerifier)\n\n```\npip install cacheverifier\n# with the GPTCache adapter:\npip install \"cacheverifier[gptcache]\"\n# with the offline Health Check (adds torch + sentence-transformers):\npip install \"cacheverifier[healthcheck]\"\n```\n\nRequires Python 3.9+. The only runtime dependency is `httpx` — the extras above\nare opt-in.\n\nGet a free API key at [https://www.cacheverifier.com](https://www.cacheverifier.com) (self-serve verify and fine-tuning\nare free forever, no card).\n\n``` python\nfrom cacheverifier import CacheVerifier\n\ncv = CacheVerifier(api_key=\"cv_...\")\n\nquery = \"how do I cancel my subscription\"\ncandidate = \"Go to Settings > Billing > Pause subscription for a month.\"  # from your cache\n\nresult = cv.verify(query, candidate)\nif result.approved:\n    answer = candidate                       # verified hit — skip the LLM call\nelse:\n    answer = call_your_llm(query)            # not trustworthy — fall through\n\n# Later, once you know if it was actually right (thumbs-down, reopened ticket, ...):\ncv.feedback(query, answer, was_correct=True, similarity_score=0.86)\n```\n\n`verify()` returns a `VerifyResult`:\n\n| field | meaning | \n|---|---|\n| `approved` | serve the cached answer ( `True` ) or fall through (`False` ) | \n| `score` /`threshold` | `approved` is`score >= threshold` | \n| `model_version` | `\"stock\"` ,`\"v<id>\"` (fine-tuned),`\"cold_start_fail_closed\"` /`\"cold_start_auto_pending\"` , or`\"verify_unavailable\"` | \n| `latency_ms` | server-side inference time | \n| `degraded` | `True` when the call failed and this result was synthesized client-side | \n\n`verify()` runs inline with your traffic, so its defaults are conservative:\n\n- **1s timeout** (`verify_timeout=` ), separate from the 10s`timeout=` used for\nfine-tuning / feedback / monitoring calls. Warm verification is tens of\nmilliseconds server-side; 1s covers the network round trip and a cold model\nload after a deploy without letting a stuck verifier stall your request.\n- **Fails closed.** On a timeout, connection error, or 5xx,`verify()` does not\nraise — it logs a warning on the`cacheverifier` logger and returns a`VerifyResult` with`degraded=True` and`approved=False` , so you fall through\nto your LLM exactly as you would on a cache miss. A`4xx` (bad key, bad\nrequest, rate limit) still raises`CacheVerifierError` .\n\n```\ncv = CacheVerifier(api_key=\"cv_...\", verify_timeout=1.0, fail_open=False)  # the defaults\n\n# fail_open=True instead returns approved=True on an outage — only if a\n# stale-or-near-miss answer is acceptable for that traffic:\ncv = CacheVerifier(api_key=\"cv_...\", fail_open=True)\n```\n\nThere is no formal uptime SLA yet, which is the other reason the fallback path is a built-in default rather than left to you.\n\nDrop the verifier into a GPTCache pipeline as its similarity evaluator — no fork required:\n\n``` python\nfrom gptcache import cache\nfrom cacheverifier.integrations.gptcache import CacheVerifierEvaluation\n\nevaluator = CacheVerifierEvaluation(api_key=\"cv_...\")\ncache.init(similarity_evaluation=evaluator, ...)\n\n# when you learn a served hit's real outcome:\nevaluator.report_feedback(query, answer, was_correct=False, similarity_score=0.9)\n```\n\nSee [`examples/gptcache_example.py`](https://github.com/imxinchengyou/cacheverifier-python/blob/main/examples/gptcache_example.py).\n\nOnce you have ~20+ feedback rows (the service found fine-tuning is often a net negative\nbelow ~1,000 on the hardest data — see the [research](https://github.com/imxinchengyou/CacheVerifier)),\ntrain a verifier on your own gray-zone labels:\n\n```\njob = cv.finetune()                       # or cv.finetune(target_risk=0.01, cost_ratio=5.0)\njob = cv.get_finetune_job(job[\"id\"])      # poll until status == \"done\"\nprint(job[\"auc_baseline\"], job[\"auc_tuned\"])\n\n# a model can finish as \"held_for_review\" — promote it explicitly:\nif job.get(\"result_model_version\"):\n    cv.activate_model_version(job[\"result_model_version\"])\n```\n\n`cv.dry_run([...])` reports the same baseline-vs-tuned AUC on examples you pass directly,\nwithout writing anything or deploying a model.\n\n`cv.dry_run()` still uploads your examples to the API. If that's a blocker — a\ncompliance review, or just not wanting production traffic to leave your network —\nrun the identical stock-vs-fine-tuned evaluation entirely on your own machine:\n\n```\npip install \"cacheverifier[healthcheck]\"\n\ncacheverifier healthcheck traffic.jsonl\ncacheverifier healthcheck traffic.jsonl --emit-summary summary.json\n```\n\n`traffic.jsonl` is a JSON array or JSONL of `{\"query\", \"candidate_answer\", \"was_correct\"}`\nrows **in arrival order** (the train/calibrate/test split is chronological, matching the\nhosted service so the numbers are comparable). Optional per row: `\"stale\": true`.\n\nNothing is sent anywhere — the base model downloads once from Hugging Face, then it's\nfully offline. `--emit-summary` writes an aggregate-only JSON file (AUCs, counts, rates —\nno query or answer text) that's safe to share for a human read.\n\n**Choosing the base model.** `--base-model` takes the same values as the hosted\nservice's `base_model` fine-tune parameter, so local and hosted numbers stay comparable:\n\n| `--base-model` | model | use for | \n|---|---|---|\n| `ms_marco` (default) | `cross-encoder/ms-marco-MiniLM-L6-v2` | English traffic | \n| `nli` | `cross-encoder/nli-MiniLM2-L6-H768` | English; entailment pretraining, stronger on negation / entity swaps | \n| `multilingual` | `BAAI/bge-reranker-base` | Chinese and other non-English traffic (~6 GB RAM to fine-tune) | \n| `multilingual_small` | `cross-encoder/mmarco-mMiniLMv2-L12-H384-v1` | same, smaller (~3 GB RAM) | \n\nThe English bases read most Chinese characters as unknown tokens, so on Chinese\ntraffic their result doesn't mean anything — the CLI warns when it sees mostly CJK text\non an English base. You can also pass any Hugging Face model id or a local model\ndirectory. If Hugging Face is unreachable from your network, set `HF_ENDPOINT` to a\nmirror, or download the model elsewhere and pass its directory:\n\n```\ncacheverifier healthcheck traffic.jsonl --base-model multilingual\nHF_ENDPOINT=https://hf-mirror.com cacheverifier healthcheck traffic.jsonl --base-model multilingual_small\ncacheverifier healthcheck traffic.jsonl --base-model ./models/bge-reranker-base\nresults\n------------------------------------------------------------------\n  train / calibrate / test:         3349 / 419 / 419\n  stock verifier   held-out AUC:    0.6120\n  fine-tuned       held-out AUC:    0.7080   (delta +0.0960)\n  label-noise proxy (disagreement): 11.4%\n  ceiling status:                   still_improvable\n\nverdict\n------------------------------------------------------------------\n  IMPROVED   -- fine-tuning on your own data helps this traffic\n```\n\n| method | endpoint | \n|---|---|\n| `verify(query, candidate_answer)` | `POST /v1/verify` | \n| `verify_batch(pairs)` | `POST /v1/verify/batch` | \n| `feedback(...)` /`feedback_batch(items)` | `POST /v1/feedback` /`/batch` | \n| `finetune(...)` /`dry_run(examples, ...)` | `POST /v1/finetune/jobs` /`/dry-run` | \n| `get_finetune_job(id)` /`list_finetune_jobs()` | `GET /v1/finetune/jobs[/id]` | \n| `activate_model_version(id)` | `POST /v1/finetune/model-versions/{id}/activate` | \n| `drift_status()` | `GET /v1/monitor/drift-status` | \n| `gray_zone_threshold()` | `GET /v1/monitor/gray-zone-threshold` | \n| `usage()` /`savings()` | `GET /v1/usage/status` /`/savings` | \n\nNon-2xx responses raise `CacheVerifierError` (`.status_code`, `.detail`).\n\nMIT — see [`LICENSE`](https://github.com/imxinchengyou/cacheverifier-python/blob/main/LICENSE). (The [research repository](https://github.com/imxinchengyou/CacheVerifier)\nis separately licensed; this client is not.)", "url": "https://wpnews.pro/news/cacheverifier-check-if-verifying-llm-cache-hits-helps-your-traffic", "canonical_source": "https://github.com/imxinchengyou/cacheverifier-python", "published_at": "2026-10-19 08:37:03+00:00", "updated_at": "2026-10-19 08:45:19.411459+00:00", "lang": "en", "topics": ["ai-infrastructure", "large-language-models", "ai-tools", "developer-tools"], "entities": ["CacheVerifier", "GPTCache", "httpx", "Python"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/cacheverifier-check-if-verifying-llm-cache-hits-helps-your-traffic", "markdown": "https://wpnews.pro/news/cacheverifier-check-if-verifying-llm-cache-hits-helps-your-traffic.md", "text": "https://wpnews.pro/news/cacheverifier-check-if-verifying-llm-cache-hits-helps-your-traffic.txt", "jsonld": "https://wpnews.pro/news/cacheverifier-check-if-verifying-llm-cache-hits-helps-your-traffic.jsonld"}}