{"slug": "learning-what-not-to-learn-adversarial-disentangled-prompt-tuning-for-robust", "title": "Learning What Not to Learn: Adversarial Disentangled Prompt Tuning for Robust Vision-Language Models", "summary": "Researchers propose ADAPT (Adversarial Disentangled Prompt Tuning), a robust prompt tuning framework that mitigates robust generalization overfitting in vision-language models by disentangling robust features from pseudo-robust features. The method uses a dual-prompt mechanism with a target prompt and decoy prompts to entrap non-generalizable shortcuts, improving robustness on unseen classes. The code is available at https://github.com/cheny02/ADAPT-ACMMM2026.", "body_md": "arXiv:2608.17306v1 Announce Type: new\nAbstract: While adversarial prompt tuning can enhance robustness of vision-language models efficiently, we find that existing methods aggravate robust generalization overfitting on seen classes, leading to a rapid degradation in performance against adversarial examples of unseen classes as training progresses. We empirically identify that this degradation stems from the tendency of the model to learn pseudo-robust features (i.e., non-generalizable shortcuts). To mitigate this, we propose ADAPT (Adversarial Disentangled Prompt Tuning), a robust prompt tuning framework following the philosophy of ``Learning What Not to Learn''. Specifically, ADAPT uses a dual-prompt mechanism with a target prompt and a pool of decoy prompts. During training, the decoy prompts are guided to entrap diverse pseudo-robust features, while the target prompt is constrained to be orthogonal to the decoys in the embedding space to learn robust features. By disentangling the robust features from the pseudo-robust features, ADAPT effectively prevents robust generalization overfitting. We further provide an analysis showing that the orthogonal loss bounds the effect of shifts in pseudo-robust features on unseen classes, yielding a testing error guarantee. Empirically, extensive experiments demonstrate that ADAPT substantially improves the robustness of the target prompt on unseen classes. The code is available at https://github.com/cheny02/ADAPT-ACMMM2026.", "url": "https://wpnews.pro/news/learning-what-not-to-learn-adversarial-disentangled-prompt-tuning-for-robust", "canonical_source": "https://arxiv.org/abs/2608.17306", "published_at": "2026-08-19 04:00:00+00:00", "updated_at": "2026-08-19 04:13:06.752226+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "ai-research", "ai-safety"], "entities": ["ADAPT", "arXiv"], "alternates": {"html": "https://wpnews.pro/news/learning-what-not-to-learn-adversarial-disentangled-prompt-tuning-for-robust", "markdown": "https://wpnews.pro/news/learning-what-not-to-learn-adversarial-disentangled-prompt-tuning-for-robust.md", "text": "https://wpnews.pro/news/learning-what-not-to-learn-adversarial-disentangled-prompt-tuning-for-robust.txt", "jsonld": "https://wpnews.pro/news/learning-what-not-to-learn-adversarial-disentangled-prompt-tuning-for-robust.jsonld"}}