COPA treats prompt injection as lifelong learning not one-time

A new method called COPA reduces attack success rates by 6.3× versus the best static baseline and 4.4× on average across lifelong attack streams, while retaining 92% defense on month-old attacks compared to 61% for replay-free GRPO, according to a paper. COPA runs a GRPO loop each time a new attack cluster appears, using margin-weighted experience replay to prevent catastrophic forgetting, and keeps MMLU drop below 0.8% after 12 adaptation rounds versus 3.2% for naive fine-tuning. The approach assumes clean phase boundaries in attack streams, which may not hold in real deployments, and requires a reliable reward model to score injection attempts.

COPA treats prompt injection as lifelong learning not one-time The core insight is simple but the implementation matters. Instead of baking injection resistance into a one-time RLHF run, COPA runs a GRPO Group Relative Policy Optimization loop every time a new attack cluster surfaces. Margin-weighted experience replay keeps the model from forgetting how to spot older attack families — the same catastrophic forgetting problem that plagues continual learning in vision, now applied to adversarial prompts. Results across lifelong attack streams: Attack success rate reduction: 6.3× vs. best static baseline, 4.4× average across all streams Catastrophic forgetting metric: COPA retains 92% defense on month-old attacks vs. 61% for replay-free GRPO General capability preservation: MMLU drop <0.8% after 12 adaptation rounds vs. 3.2% for naive fine-tuning The replay buffer design is where it gets clever. Each stored example gets a margin weight based on how close the model's log-prob was to the decision boundary at insertion time. Near-miss attacks get replayed more often; easy wins fade out. This mirrors prioritized experience replay in RL but the margin signal comes from the reward model's own uncertainty — no external labeler needed. One thing the paper doesn't stress enough: the attack stream assumption. They simulate "lifelong" by concatenating known benchmarks PIJ, HotFlip, GCG, AutoDAN, etc. in chronological release order. Real deployments won't see clean phase boundaries — you'll get mixed traffic, low-volume novel variants, and plenty of noise. The margin weighting helps, but I'd want to see COPA stress-tested against a stream where 80% of inputs are benign and only 2% are genuinely novel attacks buried in lookalike noise. Also worth noting: GRPO needs a reward model that can actually score injection attempts reliably. If your reward model itself gets fooled by a new obfuscation technique, the whole loop amplifies the failure. The paper uses a separate classifier head trained jointly — sensible, but it means you're now maintaining two continual learning loops policy + reward model instead of one. For anyone running production LLM endpoints: the takeaway isn't "install COPA tomorrow." It's that your red-team pipeline needs to feed a continual adaptation loop, not a quarterly retrain. Collect failed injections, cluster them, run GRPO updates weekly, and measure forgetting on a held-out attack museum. The math works; the engineering is what separates a demo from a defense. Next OpenAI admits safety monitoring eats 20% of inference compute → /en/threads/7079/