From a quick sanity check, this is roughly what I saw: Direct answer
The clearest edge-case / miss candidates I saw were:
- equivalent injection requests in some non-English languages;
- attacks embedded in longer benign context, especially away from the beginning;
- cases where an apparently adversarial instruction is expressed outside the English/WildGuard-like templates seen at evaluation time;
- response refusal and response harmfulness outside a fairly narrow set of prompt-response patterns.
The clearest benign false-positive candidates were:
- quoted injection examples;
- translation requests containing injection text;
- security documentation discussing phrases such as “ignore previous instructions”;
- fiction or test fixtures containing attack-like wording;
- ordinary instructions such as “ignore the previous draft.”
For a V2, I would not start by increasing model size. My default path would be:
- define exactly what
Prompt Adversarial
should mean;
- verify nullable-label masking, weighting, formatter, label order, and evaluation code;
- evaluate prompt heads from prompt-only inputs;
- add paired active-vs-quoted and prompt-fixed/response-varied controls;
- evaluate the response heads on a response-specific public benchmark;
- compare random splits with source/template/language-held-out splits;
- then compare single-task, field-specific, and split-model architectures.
The important positive result is that the checkpoint does seem strong on the distribution it was designed around. On a public English prompt-only WildGuardTest mirror, I obtained approximately:
| Task | F1 at threshold 0.5 | ROC-AUC | | Prompt Adversarial | 0.990 | 0.999 | | Prompt Harmful | 0.860 | 0.932 |
Those results are broadly consistent with the reported prompt-side metrics. I therefore would not summarize this as “the model is broken.”
A more precise summary would be:
The model looks strong as a compact English, WildGuard-like prompt classifier, while the four outputs do not yet appear fully independent across context, input fields, languages, and longer documents.
The following is a small handwritten sanity check rather than a benchmark. I used the input format and sigmoid interpretation shown on the model card, with 0.5
as the initial threshold.
Recommended default route
| Stage | What it separates | | Fix the label contract | Intended behavior vs apparent false positives | | Verify masking and evaluation | Data-pipeline issue vs model behavior | | Add paired controls | Semantic distinction vs lexical/template shortcut | | Add response-specific evaluation | Weak threshold vs weak response representation | | Add held-out splits | In-distribution recognition vs generalization | | Run architecture ablations | Data problem vs multi-task/field-sharing problem |
1. Scope: what should Prompt Adversarial mean? [(click for more details)](https://discuss.huggingface.co/t/i-trained-a-prompt-injection-classifier-using-a-14m-parameter-discriminator/177768/4)
2. Reproduction and control setup [(click for more details)](https://discuss.huggingface.co/t/i-trained-a-prompt-injection-classifier-using-a-14m-parameter-discriminator/177768/4)
3. Benign false positives and context sensitivity [(click for more details)](https://discuss.huggingface.co/t/i-trained-a-prompt-injection-classifier-using-a-14m-parameter-discriminator/177768/4)
- Prompt heads are highly sensitive to appended responses (click for more details)
- Response heads: conditional behavior and cleaner evaluation (click for more details)
6. Missing labels, weighting, reduction, and sampling [(click for more details)](https://discuss.huggingface.co/t/i-trained-a-prompt-injection-classifier-using-a-14m-parameter-discriminator/177768/4)
7. Source, template, and language generalization [(click for more details)](https://discuss.huggingface.co/t/i-trained-a-prompt-injection-classifier-using-a-14m-parameter-discriminator/177768/4)
8. Long-context and position sensitivity [(click for more details)](https://discuss.huggingface.co/t/i-trained-a-prompt-injection-classifier-using-a-14m-parameter-discriminator/177768/4)
9. Metrics and calibration [(click for more details)](https://discuss.huggingface.co/t/i-trained-a-prompt-injection-classifier-using-a-14m-parameter-discriminator/177768/4)
10. Architecture and V2 options [(click for more details)](https://discuss.huggingface.co/t/i-trained-a-prompt-injection-classifier-using-a-14m-parameter-discriminator/177768/4)
11. Evaluation provenance [(click for more details)](https://discuss.huggingface.co/t/i-trained-a-prompt-injection-classifier-using-a-14m-parameter-discriminator/177768/4)
12. Suggested result table [(click for more details)](https://discuss.huggingface.co/t/i-trained-a-prompt-injection-classifier-using-a-14m-parameter-discriminator/177768/4)
13. Caveats and references [(click for more details)](https://discuss.huggingface.co/t/i-trained-a-prompt-injection-classifier-using-a-14m-parameter-discriminator/177768/4)
Overall, I think the compact encoder approach is useful, and the English prompt-side result is a meaningful baseline.
The highest-information next step is probably not “make the encoder larger.” It is to determine which of the following currently limits the model:
- task definition;
- counterfactual coverage;
- missing-label handling;
- response sampling;
- field sharing;
- source/template generalization;
- calibration.
Once those are separated, the architecture choice becomes much easier:
- keep the current model if the target is English WildGuard-like prompt triage;
- add field-specific pooling if prompt/response independence matters;
- split prompt and response models if auditability matters;
- narrow the task if explicit override detection is the main product goal.