I trained a Prompt Injection Classifier Using a 14M Parameter Discriminator

A developer trained a 14M-parameter prompt injection classifier using an Electra-small discriminator, achieving strong results on English WildGuard-like prompts with a 0.990 F1 score for adversarial detection. The model shows limitations in non-English languages, longer contexts, and response-specific tasks, prompting recommendations for improved evaluation and architecture refinements.

From a quick sanity check, this is roughly what I saw: Direct answer The clearest edge-case / miss candidates I saw were: - equivalent injection requests in some non-English languages; - attacks embedded in longer benign context, especially away from the beginning; - cases where an apparently adversarial instruction is expressed outside the English/WildGuard-like templates seen at evaluation time; - response refusal and response harmfulness outside a fairly narrow set of prompt-response patterns. The clearest benign false-positive candidates were: - quoted injection examples; - translation requests containing injection text; - security documentation discussing phrases such as “ignore previous instructions”; - fiction or test fixtures containing attack-like wording; - ordinary instructions such as “ignore the previous draft.” For a V2, I would not start by increasing model size . My default path would be: - define exactly what Prompt Adversarial should mean; - verify nullable-label masking, weighting, formatter, label order, and evaluation code; - evaluate prompt heads from prompt-only inputs; - add paired active-vs-quoted and prompt-fixed/response-varied controls; - evaluate the response heads on a response-specific public benchmark; - compare random splits with source/template/language-held-out splits; - then compare single-task, field-specific, and split-model architectures. The important positive result is that the checkpoint does seem strong on the distribution it was designed around. On a public English prompt-only WildGuardTest mirror, I obtained approximately: | Task | F1 at threshold 0.5 | ROC-AUC | | Prompt Adversarial | 0.990 | 0.999 | | Prompt Harmful | 0.860 | 0.932 | Those results are broadly consistent with the reported prompt-side metrics. I therefore would not summarize this as “the model is broken.” A more precise summary would be: The model looks strong as a compact English, WildGuard-like prompt classifier, while the four outputs do not yet appear fully independent across context, input fields, languages, and longer documents. The following is a small handwritten sanity check rather than a benchmark. I used the input format and sigmoid interpretation shown on the model card https://huggingface.co/DeshwalX/electra-small-prompt-injection-v1 , with 0.5 as the initial threshold. Recommended default route | Stage | What it separates | | Fix the label contract | Intended behavior vs apparent false positives | | Verify masking and evaluation | Data-pipeline issue vs model behavior | | Add paired controls | Semantic distinction vs lexical/template shortcut | | Add response-specific evaluation | Weak threshold vs weak response representation | | Add held-out splits | In-distribution recognition vs generalization | | Run architecture ablations | Data problem vs multi-task/field-sharing problem | 1. Scope: what should Prompt Adversarial mean? click for more details https://discuss.huggingface.co/t/i-trained-a-prompt-injection-classifier-using-a-14m-parameter-discriminator/177768/4 2. Reproduction and control setup click for more details https://discuss.huggingface.co/t/i-trained-a-prompt-injection-classifier-using-a-14m-parameter-discriminator/177768/4 3. Benign false positives and context sensitivity click for more details https://discuss.huggingface.co/t/i-trained-a-prompt-injection-classifier-using-a-14m-parameter-discriminator/177768/4 4. Prompt heads are highly sensitive to appended responses click for more details https://discuss.huggingface.co/t/i-trained-a-prompt-injection-classifier-using-a-14m-parameter-discriminator/177768/4 5. Response heads: conditional behavior and cleaner evaluation click for more details https://discuss.huggingface.co/t/i-trained-a-prompt-injection-classifier-using-a-14m-parameter-discriminator/177768/4 6. Missing labels, weighting, reduction, and sampling click for more details https://discuss.huggingface.co/t/i-trained-a-prompt-injection-classifier-using-a-14m-parameter-discriminator/177768/4 7. Source, template, and language generalization click for more details https://discuss.huggingface.co/t/i-trained-a-prompt-injection-classifier-using-a-14m-parameter-discriminator/177768/4 8. Long-context and position sensitivity click for more details https://discuss.huggingface.co/t/i-trained-a-prompt-injection-classifier-using-a-14m-parameter-discriminator/177768/4 9. Metrics and calibration click for more details https://discuss.huggingface.co/t/i-trained-a-prompt-injection-classifier-using-a-14m-parameter-discriminator/177768/4 10. Architecture and V2 options click for more details https://discuss.huggingface.co/t/i-trained-a-prompt-injection-classifier-using-a-14m-parameter-discriminator/177768/4 11. Evaluation provenance click for more details https://discuss.huggingface.co/t/i-trained-a-prompt-injection-classifier-using-a-14m-parameter-discriminator/177768/4 12. Suggested result table click for more details https://discuss.huggingface.co/t/i-trained-a-prompt-injection-classifier-using-a-14m-parameter-discriminator/177768/4 13. Caveats and references click for more details https://discuss.huggingface.co/t/i-trained-a-prompt-injection-classifier-using-a-14m-parameter-discriminator/177768/4 Overall, I think the compact encoder approach is useful, and the English prompt-side result is a meaningful baseline. The highest-information next step is probably not “make the encoder larger.” It is to determine which of the following currently limits the model: - task definition; - counterfactual coverage; - missing-label handling; - response sampling; - field sharing; - source/template generalization; - calibration. Once those are separated, the architecture choice becomes much easier: - keep the current model if the target is English WildGuard-like prompt triage; - add field-specific pooling if prompt/response independence matters; - split prompt and response models if auditability matters; - narrow the task if explicit override detection is the main product goal.