# I trained a Prompt Injection Classifier Using a 14M Parameter Discriminator

> Source: <https://discuss.huggingface.co/t/i-trained-a-prompt-injection-classifier-using-a-14m-parameter-discriminator/177768#post_4>
> Published: 2026-07-13 23:51:52+00:00

From a quick sanity check, this is roughly what I saw:

Direct answer

The clearest **edge-case / miss candidates** I saw were:

- equivalent injection requests in some non-English languages;
- attacks embedded in longer benign context, especially away from the beginning;
- cases where an apparently adversarial instruction is expressed outside the English/WildGuard-like templates seen at evaluation time;
- response refusal and response harmfulness outside a fairly narrow set of prompt-response patterns.

The clearest **benign false-positive candidates** were:

- quoted injection examples;
- translation requests containing injection text;
- security documentation discussing phrases such as “ignore previous instructions”;
- fiction or test fixtures containing attack-like wording;
- ordinary instructions such as “ignore the previous draft.”

For a V2, I would **not start by increasing model size**. My default path would be:

- define exactly what
`Prompt Adversarial`

should mean;
- verify nullable-label masking, weighting, formatter, label order, and evaluation code;
- evaluate prompt heads from prompt-only inputs;
- add paired active-vs-quoted and prompt-fixed/response-varied controls;
- evaluate the response heads on a response-specific public benchmark;
- compare random splits with source/template/language-held-out splits;
- then compare single-task, field-specific, and split-model architectures.

The important positive result is that the checkpoint does seem strong on the distribution it was designed around. On a public English prompt-only WildGuardTest mirror, I obtained approximately:

| Task |
F1 at threshold 0.5 |
ROC-AUC |
| Prompt Adversarial |
0.990 |
0.999 |
| Prompt Harmful |
0.860 |
0.932 |

Those results are broadly consistent with the reported prompt-side metrics. I therefore would not summarize this as “the model is broken.”

A more precise summary would be:

The model looks strong as a compact English, WildGuard-like prompt classifier, while the four outputs do not yet appear fully independent across context, input fields, languages, and longer documents.

The following is a small handwritten sanity check rather than a benchmark. I used the input format and sigmoid interpretation shown on the [model card](https://huggingface.co/DeshwalX/electra-small-prompt-injection-v1), with `0.5`

as the initial threshold.

Recommended default route

| Stage |
What it separates |
| Fix the label contract |
Intended behavior vs apparent false positives |
| Verify masking and evaluation |
Data-pipeline issue vs model behavior |
| Add paired controls |
Semantic distinction vs lexical/template shortcut |
| Add response-specific evaluation |
Weak threshold vs weak response representation |
| Add held-out splits |
In-distribution recognition vs generalization |
| Run architecture ablations |
Data problem vs multi-task/field-sharing problem |

1. Scope: what should Prompt Adversarial mean? [(click for more details)](https://discuss.huggingface.co/t/i-trained-a-prompt-injection-classifier-using-a-14m-parameter-discriminator/177768/4)
2. Reproduction and control setup [(click for more details)](https://discuss.huggingface.co/t/i-trained-a-prompt-injection-classifier-using-a-14m-parameter-discriminator/177768/4)
3. Benign false positives and context sensitivity [(click for more details)](https://discuss.huggingface.co/t/i-trained-a-prompt-injection-classifier-using-a-14m-parameter-discriminator/177768/4)
4. Prompt heads are highly sensitive to appended responses [(click for more details)](https://discuss.huggingface.co/t/i-trained-a-prompt-injection-classifier-using-a-14m-parameter-discriminator/177768/4)
5. Response heads: conditional behavior and cleaner evaluation [(click for more details)](https://discuss.huggingface.co/t/i-trained-a-prompt-injection-classifier-using-a-14m-parameter-discriminator/177768/4)
6. Missing labels, weighting, reduction, and sampling [(click for more details)](https://discuss.huggingface.co/t/i-trained-a-prompt-injection-classifier-using-a-14m-parameter-discriminator/177768/4)
7. Source, template, and language generalization [(click for more details)](https://discuss.huggingface.co/t/i-trained-a-prompt-injection-classifier-using-a-14m-parameter-discriminator/177768/4)
8. Long-context and position sensitivity [(click for more details)](https://discuss.huggingface.co/t/i-trained-a-prompt-injection-classifier-using-a-14m-parameter-discriminator/177768/4)
9. Metrics and calibration [(click for more details)](https://discuss.huggingface.co/t/i-trained-a-prompt-injection-classifier-using-a-14m-parameter-discriminator/177768/4)
10. Architecture and V2 options [(click for more details)](https://discuss.huggingface.co/t/i-trained-a-prompt-injection-classifier-using-a-14m-parameter-discriminator/177768/4)
11. Evaluation provenance [(click for more details)](https://discuss.huggingface.co/t/i-trained-a-prompt-injection-classifier-using-a-14m-parameter-discriminator/177768/4)
12. Suggested result table [(click for more details)](https://discuss.huggingface.co/t/i-trained-a-prompt-injection-classifier-using-a-14m-parameter-discriminator/177768/4)
13. Caveats and references [(click for more details)](https://discuss.huggingface.co/t/i-trained-a-prompt-injection-classifier-using-a-14m-parameter-discriminator/177768/4)
Overall, I think the compact encoder approach is useful, and the English prompt-side result is a meaningful baseline.

The highest-information next step is probably not “make the encoder larger.” It is to determine which of the following currently limits the model:

- task definition;
- counterfactual coverage;
- missing-label handling;
- response sampling;
- field sharing;
- source/template generalization;
- calibration.

Once those are separated, the architecture choice becomes much easier:

- keep the current model if the target is English WildGuard-like prompt triage;
- add field-specific pooling if prompt/response independence matters;
- split prompt and response models if auditability matters;
- narrow the task if explicit override detection is the main product goal.