{"slug": "stop-assuming-a-model-is-blind-to-new-attacks-just-because-the", "title": "Stop assuming a model is \"blind\" to new attacks just because the", "summary": "A linear probe on frozen embeddings from Meta's Prompt Guard 2 achieves an AUC of approximately 0.999 on out-of-distribution data, revealing that the model's poor attack detection stems from a miscalibrated classification head rather than insufficient representations. By training a logistic regression head on penultimate embeddings and calibrating on real traffic, one practitioner achieved 99.9% OOD recall with a 0.7% false positive rate without modifying base weights. The approach is demonstrated in a GitHub repository by mosafariuk.", "body_md": "# Stop assuming a model is \"blind\" to new attacks just because the\n\nMost people see these numbers and immediately conclude the model’s internal representations are insufficient—that the \"brain\" of the model simply doesn't recognize the pattern of a new injection.\n\nThey are wrong.\n\nI’ve found that the issue isn't the representation; it's a massive calibration problem. Here is a quick deep dive into how you can diagnose this and fix it in about 20 minutes without even touching the base model weights.\n\n## The 20-Minute Diagnostic\n\nBefore you waste weeks fine-tuning a model or switching to a massive, expensive LLM agent for security, you need to run a simple probe. Instead of looking at the final classification output, pull the penultimate embeddings from the frozen encoder.\n\nIf you take those frozen embeddings and fit a simple logistic regression on them, the results change instantly. On the same OOD data where the original head failed, a linear probe achieves an **AUC of approximately 0.999**.\n\nThis tells you everything you need to know:\n\n**High AUC + Low Recall:** Your classification head or your threshold is miscalibrated. The information is there, but the \"decision maker\" is tuned too conservatively.**Low AUC:** The model actually lacks the features to distinguish the attack. You have a genuine representation problem and need a different model.\n\nIn the case of Prompt Guard 2, Meta made a specific product choice: they tuned the head for extreme precision to ensure a near-zero false-positive rate. They traded recall for stability. That’s a valid design choice for a product, but it’s a nightmare for someone trying to catch evolving jailbreaks.\n\n## A Practical Tutorial for Better Recall\n\nIf you find yourself in the \"High AUC\" scenario, you can implement a custom deployment workflow to boost your security posture.\n\n1. **Extract Embeddings:** Run your training data (both benign and injection samples) through the encoder and save the penultimate layer outputs.\n\n2. **Train a Linear Head:** Use a simple logistic regression. Since the head is just a dot product, the inference cost is negligible: `sigmoid(x·w + b) >= tau`\n\n.\n\n3. **Calibrate on Real Traffic:** This is the critical step. Don't just tune on the attack set. Calibrate your threshold ($\\tau$) using the benign traffic distribution you actually expect to see in production.\n\nBy following this approach, I managed to hit **99.9% OOD recall with only a 0.7% False Positive Rate (FPR)**. The base model remains completely untouched, and the inference runs perfectly fine on a standard CPU.\n\n## The Reality Check\n\nI want to be clear: this isn't a magic bullet that \"solves\" prompt injection. A linear head over frozen features is still susceptible to evasion if an adversary knows exactly how you've shifted your operating point or if they introduce a massive distribution shift.\n\nWhat this does is move your operating point from \"useless\" to \"highly effective\" by reclaiming the signal that the original developers intentionally suppressed. It turns a failed deployment into a robust defense layer.\n\nIf you want to look at the specific implementation and the seeds used for the data splits, you can check the repo here:\n\n```\nhttps://github.com/mosafariuk/prompt-guard-2-frozen-head\n```\n\n[Next LLM security is basically a never-ending game of whack-a-mole →](/en/threads/7339/)\n\n[these AI tool field notes](https://tanyan888.com/), with plenty of directly applicable cases.", "url": "https://wpnews.pro/news/stop-assuming-a-model-is-blind-to-new-attacks-just-because-the", "canonical_source": "https://promptcube3.com/en/threads/7432/", "published_at": "2026-08-23 17:50:40+00:00", "updated_at": "2026-08-23 18:12:56.815608+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "ai-safety", "ai-research"], "entities": ["Meta", "Prompt Guard 2", "mosafariuk"], "alternates": {"html": "https://wpnews.pro/news/stop-assuming-a-model-is-blind-to-new-attacks-just-because-the", "markdown": "https://wpnews.pro/news/stop-assuming-a-model-is-blind-to-new-attacks-just-because-the.md", "text": "https://wpnews.pro/news/stop-assuming-a-model-is-blind-to-new-attacks-just-because-the.txt", "jsonld": "https://wpnews.pro/news/stop-assuming-a-model-is-blind-to-new-attacks-just-because-the.jsonld"}}