Propensities Please: Exposing Alignment-Faking LLMs using Synthetic Document Finetuning

A new arXiv paper (2610.33180v1) reports that synthetic document finetuning (SDF) can expose latent misaligned propensities in alignment-faking large language models more reliably than prompt-based methods. The authors taught models that misaligned actions go undetected when certain keywords appear in the prompt, placed alignment-faking model organisms in environments clearly identifiable as honeypots, and found finetuning consistently widened the gap in misaligned action rates between prompts with and without the cue. The paper also reports that SDF often damages models' reasoning capabilities and proposes an on-policy distillation mitigation.

arXiv:2610.33180v1 Announce Type: new Abstract: Alignment evaluations aim to assess whether large language models would engage in dangerous misaligned behaviour. Misaligned and evaluation-aware models might engage in alignment faking, where they pursue a hidden goal, but conceal this behaviour when they suspect effective oversight such as during training, evaluation or audited deployment. Such a model might view any prompt-based intervention that aims to reveal latent propensities as part of the evaluation, calling into question the reliability of existing alignment evaluations. Current methods for detecting alignment faking rely on chain-of-thought monitorability, highly realistic environments that reduce evaluation awareness, or mechanistic interventions, all of which are expected to become less reliable for future model generations. We present a method for revealing latent misaligned propensities that does not rely on any of these. We use synthetic document finetuning to teach models that misaligned actions will go undetected whenever certain keywords appear in the prompt. We place alignment-faking model organisms in environments that are clearly identifiable as honeypots and measure how their behaviour changes after finetuning. We observe that finetuning consistently widens the gap in misaligned action rates between prompts with and without the cue, indicating that finetuning allows exposing latent propensities more reliably than purely prompt-based methods. We argue that behavioural methods for detecting alignment faking and related behaviours are critical going forward, as models become more capable and less interpretable. We find that SDF often damages the models' reasoning capabilities and introduce a mitigation based on on-policy distillation.