Most people see these numbers and immediately conclude the model’s internal representations are insufficient—that the "brain" of the model simply doesn't recognize the pattern of a new injection.
They are wrong.
I’ve found that the issue isn't the representation; it's a massive calibration problem. Here is a quick deep dive into how you can diagnose this and fix it in about 20 minutes without even touching the base model weights.
The 20-Minute Diagnostic #
Before you waste weeks fine-tuning a model or switching to a massive, expensive LLM agent for security, you need to run a simple probe. Instead of looking at the final classification output, pull the penultimate embeddings from the frozen encoder.
If you take those frozen embeddings and fit a simple logistic regression on them, the results change instantly. On the same OOD data where the original head failed, a linear probe achieves an AUC of approximately 0.999.
This tells you everything you need to know:
High AUC + Low Recall: Your classification head or your threshold is miscalibrated. The information is there, but the "decision maker" is tuned too conservatively.Low AUC: The model actually lacks the features to distinguish the attack. You have a genuine representation problem and need a different model.
In the case of Prompt Guard 2, Meta made a specific product choice: they tuned the head for extreme precision to ensure a near-zero false-positive rate. They traded recall for stability. That’s a valid design choice for a product, but it’s a nightmare for someone trying to catch evolving jailbreaks.
A Practical Tutorial for Better Recall #
If you find yourself in the "High AUC" scenario, you can implement a custom deployment workflow to boost your security posture.
-
Extract Embeddings: Run your training data (both benign and injection samples) through the encoder and save the penultimate layer outputs.
-
Train a Linear Head: Use a simple logistic regression. Since the head is just a dot product, the inference cost is negligible:
sigmoid(x·w + b) >= tau
.
- Calibrate on Real Traffic: This is the critical step. Don't just tune on the attack set. Calibrate your threshold ($\tau$) using the benign traffic distribution you actually expect to see in production.
By following this approach, I managed to hit 99.9% OOD recall with only a 0.7% False Positive Rate (FPR). The base model remains completely untouched, and the inference runs perfectly fine on a standard CPU.
The Reality Check #
I want to be clear: this isn't a magic bullet that "solves" prompt injection. A linear head over frozen features is still susceptible to evasion if an adversary knows exactly how you've shifted your operating point or if they introduce a massive distribution shift.
What this does is move your operating point from "useless" to "highly effective" by reclaiming the signal that the original developers intentionally suppressed. It turns a failed deployment into a robust defense layer.
If you want to look at the specific implementation and the seeds used for the data splits, you can check the repo here:
https://github.com/mosafariuk/prompt-guard-2-frozen-head
Next LLM security is basically a never-ending game of whack-a-mole →
these AI tool field notes, with plenty of directly applicable cases.