{"slug": "dpo-training-ruins-my-models-conversational-coherence", "title": "DPO Training ruins my model’s conversational coherence", "summary": "A developer reports that DPO training ruins the conversational coherence of a T5 model, causing it to output repetitive tokens like 'a a a a a a' or 'b b b b b b b' during inference. The issue emerges only after DPO training, not after fine-tuning, and persists even when using the pre-trained 't5-small' model.", "body_md": "Hi, I am running some tests with DPOTrainer to see how it works but I have encou[…]ntered some problems during the inference phase of the generated model. In details, this is the pipeline of operations I performed:\n1. I pre-trained from scratch a T5 model on natural language (English language). For this operation, I followed the instructions of the [Hugging Face library.](https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling) As for training the tokenizer, this was done using the [sentencepiece library](https://github.com/google/sentencepiece). The generated file (extension .model) was then used through the T5Tokenizer class, which allows using the .model file instead of a json file.\n2. I fine-tuned T5 using a very trivial dataset such as the following.\n| Input | Target |\n|-------------------------------------------|--------|\n| I love cats | a |\n| The cat is orange | b |\n| The cat is on the table | c |\n| The cat chased the mouse under the table. | d |\nIn summary, if there is no word 'the' in the input then the output will be 'a', if there is only one occurrence of 'the' then the output will be 'b', and so on... For fine-tuning, I did not use the SFTTrainer class but the classic Seq2SeqTrainer.\n3. Then, I performed the DPO with the same inputs as the dataset present above, but in the JSON format. The code used is the same as the [example on the repository](https://github.com/huggingface/trl/blob/main/examples/research_projects/stack_llama_2/scripts/dpo_llama2.py). In this case, however, we used our finetuned T5 model and tokenizer (with classes T5ForConditionalGeneration, T5Tokenizer, T5Config). You can find the JSON file and the full code at the end of this message.\nThe problem arises in the inference phase of the model generated by the DPOTrainer. In fact, for several instances the output generated by the model is 'a a a a a a', ' b b b b b b b', 'c c c c c c c c', and so on... (the number of repetitions of the class is variable). Moreover, this behavior becomes more pronounced as the number of steps increases. Also, as the number of steps increases, words that are part of the train set are generated in the output (e.g., 'aaacat' is generated).\nI cannot figure out what could be the cause of this behavior. By making inference of the simply fine-tuned model, the output generated is as expected (i.e., a class between 'a', 'b', 'c' and 'd'), so the problem is introduced during training with DPO. I also tried to use the pre-trained 't5-small' model / tokenizer instead of the ones trained from scratch, but the problem still persists.\nI look forward to your feedback should more information or snippets of code used be needed.\n<details>\n<summary>DPO dataset</summary>\n[\n{\n'prompt': 'I love cats',\n'chosen': 'a',\n'rejected': 'b',\n},\n{\n'prompt': 'I love cats',\n'chosen': 'a',\n'rejected': 'c',\n},\n{\n'prompt': 'I love cats',\n'chosen': 'a',\n'rejected': 'd',\n},\n{\n'prompt': 'The cat is orange',\n'chosen': 'b',\n'rejected': 'a',\n},\n{\n'prompt': 'The cat is orange',\n'chosen': 'b',\n'rejected': 'c',\n},\n{\n'prompt': 'The cat is orange',\n'chosen': 'b',\n'rejected': 'd',\n}\n...\n]\n</details>\n<details>\n<summary>DPO code</summary>\n```\n# 0. imports\nimport os\nfrom dataclasses import dataclass, field\nfrom typing import Dict, Optional\nimport torch\nfrom datasets import Dataset, load_dataset\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, HfArgumentParser, TrainingArguments, T5Config, T5Tokenizer, T5ForConditionalGeneration\nfrom trl import DPOTrainer\n# Define and parse arguments.\n@dataclass\nclass ScriptArguments:\n\"\"\"\nThe arguments for the DPO training script.\n\"\"\"\n# data parameters\nbeta: Optional[float] = field(default=0.1, metadata={\"help\": \"the beta parameter for DPO loss\"})\n# training parameters\nmodel_name_or_path: Optional[str] = field(\ndefault=\"../sft/results/final_checkpoint\",\nmetadata={\"help\": \"the location of the SFT model name or path\"},\n)\nconfig_name: Optional[str] = field(\ndefault=None, metadata={\"help\": \"Pretrained config name or path if not the same as model_name\"}\n)\ntokenizer_name: Optional[str] = field(\ndefault=None, metadata={\"help\": \"Pretrained tokenizer name or path if not the same as model_name\"}\n)\ncache_dir: Optional[str] = field(\ndefault=None,\nmetadata={\"help\": \"Where to store the pretrained models downloaded from huggingface.co\"},\n)\nuse_fast_tokenizer: bool = field(\ndefault=False,\nmetadata={\"help\": \"Whether to use one of the fast tokenizer (backed by the tokenizers library) or not.\"},\n)\nmodel_revision: str = field(\ndefault=\"main\",\nmetadata={\"help\": \"The specific model version to use (can be a branch name, tag name or commit id).\"},\n)\nuse_auth_token: Optional[str] = field(\ndefault=None,\nmetadata={\n\"help\": \"Will use the token generated when running `transformers-cli login` (necessary to use this script \"\n\"with private models).\"\n},\n)\ntrain_file: Optional[str] = field(\ndefault=None, metadata={\"help\": \"The input training data file (a jsonlines or csv file).\"}\n)\neval_file: Optional[str] = field(\ndefault=None, metadata={\"help\": \"The input eval data file (a jsonlines or csv file).\"}\n)\nlearning_rate: Optional[float] = field(default=5e-4, metadata={\"help\": \"optimizer learning rate\"})\nlr_scheduler_type: Optional[str] = field(default=\"cosine\", metadata={\"help\": \"the lr scheduler type\"})\nwarmup_steps: Optional[int] = field(default=100, metadata={\"help\": \"the number of warmup steps\"})\nweight_decay: Optional[float] = field(default=0.05, metadata={\"help\": \"the weight decay\"})\noptimizer_type: Optional[str] = field(default=\"paged_adamw_32bit\", metadata={\"help\": \"the optimizer type\"})\nper_device_train_batch_size: Optional[int] = field(default=4, metadata={\"help\": \"train batch size per device\"})\nper_device_eval_batch_size: Optional[int] = field(default=1, metadata={\"help\": \"eval batch size per device\"})\ngradient_accumulation_steps: Optional[int] = field(\ndefault=4, metadata={\"help\": \"the number of gradient accumulation steps\"}\n)\ngradient_checkpointing: Optional[bool] = field(\ndefault=True, metadata={\"help\": \"whether to use gradient checkpointing\"}\n)\nlora_alpha: Optional[float] = field(default=16, metadata={\"help\": \"the lora alpha parameter\"})\nlora_dropout: Optional[float] = field(default=0.05, metadata={\"help\": \"the lora dropout parameter\"})\nlora_r: Optional[int] = field(default=8, metadata={\"help\": \"the lora r parameter\"})\nmax_prompt_length: Optional[int] = field(default=512, metadata={\"help\": \"the maximum prompt length\"})\nmax_length: Optional[int] = field(default=1024, metadata={\"help\": \"the maximum sequence length\"})\nmax_steps: Optional[int] = field(default=1000, metadata={\"help\": \"max number of training steps\"})\nlogging_steps: Optional[int] = field(default=10, metadata={\"help\": \"the logging frequency\"})\nsave_steps: Optional[int] = field(default=100, metadata={\"help\": \"the saving frequency\"})\neval_steps: Optional[int] = field(default=100, metadata={\"help\": \"the evaluation frequency\"})\noutput_dir: Optional[str] = field(default=\"./results\", metadata={\"help\": \"the output directory\"})\nlog_freq: Optional[int] = field(default=1, metadata={\"help\": \"the logging frequency\"})\n# instrumentation\nsanity_check: Optional[bool] = field(default=False, metadata={\"help\": \"only train on 1000 samples\"})\nreport_to: Optional[str] = field(\ndefault=\"wandb\",\nmetadata={\n\"help\": 'The list of integrations to report the results and logs to. Supported platforms are `\"azure_ml\"`,'\n'`\"comet_ml\"`, `\"mlflow\"`, `\"neptune\"`, `\"tensorboard\"`,`\"clearml\"` and `\"wandb\"`. '\n'Use `\"all\"` to report to all integrations installed, `\"none\"` for no integrations.'\n},\n)\n# debug argument for distributed training\nignore_bias_buffers: Optional[bool] = field(\ndefault=False,\nmetadata={\n\"help\": \"fix for DDP issues with LM bias/mask buffers - invalid scalar type,`inplace operation. See\"\n\"https://github.com/huggingface/transformers/issues/22482#issuecomment-1595790992\"\n},\n)\ndef convert(\ndataset: Dataset = None,\nsanity_check: bool = False,\ncache_dir: str = None,\nnum_proc=24,\n) -> Dataset:\n\"\"\"Load the dataset and convert it to the necessary format.\nThe dataset is converted to a dictionary with the following structure:\n{\n'prompt': List[str],\n'chosen': List[str],\n'rejected': List[str],\n}\n\"\"\"\noriginal_columns = dataset.column_names\nif sanity_check:\ndataset = dataset.select(range(min(len(dataset), 1000)))\ndef return_prompt_and_responses(samples) -> Dict[str, str]:\nreturn {\n\"prompt\": samples[\"prompt\"],\n\"chosen\": samples[\"chosen\"],\n\"rejected\": samples[\"rejected\"],\n}\nreturn dataset.map(\nreturn_prompt_and_responses,\nbatched=True,\nnum_proc=num_proc,\nremove_columns=original_columns,\n)\nif __name__ == \"__main__\":\nparser = HfArgumentParser(ScriptArguments)\nscript_args = parser.parse_args_into_dataclasses()[0]\n# 1. load a pretrained model\nconfig = T5Config.from_pretrained(\nscript_args.config_name if script_args.config_name else script_args.model_name_or_path,\ncache_dir=script_args.cache_dir,\nrevision=script_args.model_revision,\nuse_auth_token=script_args.use_auth_token,\n)\ntokenizer = T5Tokenizer.from_pretrained(\nscript_args.tokenizer_name if script_args.tokenizer_name else script_args.model_name_or_path,\ncache_dir=script_args.cache_dir,\nuse_fast=script_args.use_fast_tokenizer,\nrevision=script_args.model_revision,\nuse_auth_token=script_args.use_auth_token,\n)\nmodel = T5ForConditionalGeneration.from_pretrained(\nscript_args.model_name_or_path,\nconfig=config,\ncache_dir=script_args.cache_dir,\nrevision=script_args.model_revision,\nuse_auth_token=script_args.use_auth_token,\n)\nmodel.config.use_cache = False\nmodel_ref = T5ForConditionalGeneration.from_pretrained(\nscript_args.model_name_or_path,\nconfig=config,\ncache_dir=script_args.cache_dir,\nrevision=script_args.model_revision,\nuse_auth_token=script_args.use_auth_token,\n)\nif script_args.ignore_bias_buffers:\n# torch distributed hack\nmodel._ddp_params_and_buffers_to_ignore = [\nname for name, buffer in model.named_buffers() if buffer.dtype == torch.bool\n]\n# 2. Load the dataset and split in train / eval\ntrain_dataset = load_dataset(\"json\", data_files=script_args.train_file, split=\"train\")\ntrain_dataset = convert(dataset=train_dataset, sanity_check=script_args.sanity_check)\ntrain_dataset = train_dataset.filter(\nlambda x: len(x[\"prompt\"]) + len(x[\"chosen\"]) <= script_args.max_length\nand len(x[\"prompt\"]) + len(x[\"rejected\"]) <= script_args.max_length\n)\n# 3. Load evaluation dataset\neval_dataset = load_dataset(\"json\", data_files=script_args.eval_file, split=\"train\")\neval_dataset = convert(dataset=eval_dataset, sanity_check=script_args.sanity_check)\neval_dataset = eval_dataset.filter(\nlambda x: len(x[\"prompt\"]) + len(x[\"chosen\"]) <= script_args.max_length\nand len(x[\"prompt\"]) + len(x[\"rejected\"]) <= script_args.max_length\n)\n# 4. initialize training arguments:\ntraining_args = TrainingArguments(\nper_device_train_batch_size=script_args.per_device_train_batch_size,\nper_device_eval_batch_size=script_args.per_device_eval_batch_size,\nmax_steps=script_args.max_steps,\nlogging_steps=script_args.logging_steps,\nsave_steps=script_args.save_steps,\ngradient_accumulation_steps=script_args.gradient_accumulation_steps,\ngradient_checkpointing=script_args.gradient_checkpointing,\nlearning_rate=script_args.learning_rate,\nevaluation_strategy=\"steps\",\neval_steps=script_args.eval_steps,\noutput_dir=script_args.output_dir,\nlr_scheduler_type=script_args.lr_scheduler_type,\nwarmup_steps=script_args.warmup_steps,\nremove_unused_columns=False,\nrun_name=\"dpo\",\n)\n# 5. initialize the DPO trainer\ndpo_trainer = DPOTrainer(\nmodel,\nmodel_ref,\nargs=training_args,\nbeta=script_args.beta,\ntrain_dataset=train_dataset,\neval_dataset=eval_dataset,\ntokenizer=tokenizer,\nmax_prompt_length=script_args.max_prompt_length,\nmax_length=script_args.max_length,\n)\n# 6. train\ndpo_trainer.train()\ndpo_trainer.save_model(script_args.output_dir)\n# 7. save\noutput_dir = os.path.join(script_args.output_dir, \"final_checkpoint\")\ndpo_trainer.model.save_pretrained(output_dir)\n```\n</details>", "url": "https://wpnews.pro/news/dpo-training-ruins-my-models-conversational-coherence", "canonical_source": "https://discuss.huggingface.co/t/dpo-training-ruins-my-model-s-conversational-coherence/160733#post_4", "published_at": "2026-07-14 14:54:51+00:00", "updated_at": "2026-07-14 15:00:54.715440+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-research"], "entities": ["T5", "Hugging Face", "DPOTrainer", "T5Tokenizer", "T5ForConditionalGeneration", "Seq2SeqTrainer", "sentencepiece", "Google"], "alternates": {"html": "https://wpnews.pro/news/dpo-training-ruins-my-models-conversational-coherence", "markdown": "https://wpnews.pro/news/dpo-training-ruins-my-models-conversational-coherence.md", "text": "https://wpnews.pro/news/dpo-training-ruins-my-models-conversational-coherence.txt", "jsonld": "https://wpnews.pro/news/dpo-training-ruins-my-models-conversational-coherence.jsonld"}}