ProbGuard

ProbGuard, a new safety mechanism, can detect jailbreak attempts in large language models within the first ten decoding steps, limiting attack success rates to under 1% across six different jailbreak …

Researchers introduced ProbGuard in an August 11 preprint as a guardrail that estimates the probability an unfinished language-model response will become unsafe, using output distributions from as few…