{"slug": "similarity-gates-approve-reversals-a-validity-audit-of-embedding-cosine-in-agent", "title": "Similarity Gates Approve Reversals: A Validity Audit of Embedding-Cosine Thresholds in Agent Systems", "summary": "An audit of embedding-cosine similarity gates in agent frameworks found that these safety checks often fire backwards, with a production drift guard catching 0 of 56 meaning-breaking mutations and approving a reversal from 'withhold the study drug' to 'administer the study drug' at cosine 0.9608. The study, released on arXiv (2608.10216v1), observed five shipped operating points, with balanced accuracy across 90 configuration-threshold-task cells never exceeding 0.700 (median 0.525), and found that naive corpora can yield inverted verdicts with decision AUROC exactly 0.000 in 13 of 18 configuration-task cells. The authors contend that scores gated this way measure wording change rather than meaning, but suggest a valid instrument is buildable with matched-pair audits.", "body_md": "arXiv:2608.10216v1 Announce Type: new\nAbstract: Agent frameworks ship quality gates that compare text blocks by embedding-cosine similarity and decide at a fixed cutoff. Deduplication filters, semantic caches, drift guards, and answer grader gates deploy to answer the question: \"Does this text still mean the same thing?\" But the score answers a different question: \"How much did the wording change?\" We audit this gate class as a measurement instrument. In the cases these gates exist to catch, the two can run in opposite ways. Many times, reversing an instruction is a single word edit, while agreement often rephrases a sentence. The consequence is a safety check that fires backwards. The production drift guard we audited caught 0 of 56 meaning-breaking mutations, and one approved item, \"withhold the study drug\" -> \"administer the study drug\", came in at cosine 0.9608. We observed five shipped operating points, and balanced accuracy across 90 configuration-threshold-task cells never exceeded 0.700 (median 0.525). The same confounder also corrupted evaluations. A naively built corpus inherits this confounder and can return an inverted verdict, with a decision AUROC exactly 0.000 in 13 of 18 configuration-task cells (at most 0.040 in all 18) against 0.440-0.815 for the same nine configurations under a balanced 2x2 design. Twice in the effort it captured our own headline claims. Obvious repairs fail: an encoder swap and an overlap-conditioned gate (0.750 in-sample, 0.533 held-out) land at chance on separately authored held-out data, and an NLI drop-in did no better. Embeddings do still bear hope here, as the strongest two of nine configurations separated reversal from paraphrase at matched overlap (AUROC 0.79-0.90), but only a matched-pair audit reveals the deployment regime. We release the corpus method, harness, and frozen results, and contend that scores gated this way measure the wrong thing. We believe a valid instrument is buildable.", "url": "https://wpnews.pro/news/similarity-gates-approve-reversals-a-validity-audit-of-embedding-cosine-in-agent", "canonical_source": "https://arxiv.org/abs/2608.10216", "published_at": "2026-08-12 04:00:00+00:00", "updated_at": "2026-08-12 04:09:58.615840+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-safety", "ai-research", "natural-language-processing"], "entities": ["arXiv"], "alternates": {"html": "https://wpnews.pro/news/similarity-gates-approve-reversals-a-validity-audit-of-embedding-cosine-in-agent", "markdown": "https://wpnews.pro/news/similarity-gates-approve-reversals-a-validity-audit-of-embedding-cosine-in-agent.md", "text": "https://wpnews.pro/news/similarity-gates-approve-reversals-a-validity-audit-of-embedding-cosine-in-agent.txt", "jsonld": "https://wpnews.pro/news/similarity-gates-approve-reversals-a-validity-audit-of-embedding-cosine-in-agent.jsonld"}}