DPO Training ruins my model’s conversational coherence

A developer reports that DPO training ruins the conversational coherence of a T5 model, causing it to output repetitive tokens like 'a a a a a a' or 'b b b b b b b' during inference. The issue emerges only after DPO training, not after fine-tuning, and persists even when using the pre-trained 't5-small' model.

Hi, I am running some tests with DPOTrainer to see how it works but I have encou … ntered some problems during the inference phase of the generated model. In details, this is the pipeline of operations I performed: 1. I pre-trained from scratch a T5 model on natural language English language . For this operation, I followed the instructions of the Hugging Face library. https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling As for training the tokenizer, this was done using the sentencepiece library https://github.com/google/sentencepiece . The generated file extension .model was then used through the T5Tokenizer class, which allows using the .model file instead of a json file. 2. I fine-tuned T5 using a very trivial dataset such as the following. | Input | Target | |-------------------------------------------|--------| | I love cats | a | | The cat is orange | b | | The cat is on the table | c | | The cat chased the mouse under the table. | d | In summary, if there is no word 'the' in the input then the output will be 'a', if there is only one occurrence of 'the' then the output will be 'b', and so on... For fine-tuning, I did not use the SFTTrainer class but the classic Seq2SeqTrainer. 3. Then, I performed the DPO with the same inputs as the dataset present above, but in the JSON format. The code used is the same as the example on the repository https://github.com/huggingface/trl/blob/main/examples/research projects/stack llama 2/scripts/dpo llama2.py . In this case, however, we used our finetuned T5 model and tokenizer with classes T5ForConditionalGeneration, T5Tokenizer, T5Config . You can find the JSON file and the full code at the end of this message. The problem arises in the inference phase of the model generated by the DPOTrainer. In fact, for several instances the output generated by the model is 'a a a a a a', ' b b b b b b b', 'c c c c c c c c', and so on... the number of repetitions of the class is variable . Moreover, this behavior becomes more pronounced as the number of steps increases. Also, as the number of steps increases, words that are part of the train set are generated in the output e.g., 'aaacat' is generated . I cannot figure out what could be the cause of this behavior. By making inference of the simply fine-tuned model, the output generated is as expected i.e., a class between 'a', 'b', 'c' and 'd' , so the problem is introduced during training with DPO. I also tried to use the pre-trained 't5-small' model / tokenizer instead of the ones trained from scratch, but the problem still persists. I look forward to your feedback should more information or snippets of code used be needed. <details <summary DPO dataset</summary { 'prompt': 'I love cats', 'chosen': 'a', 'rejected': 'b', }, { 'prompt': 'I love cats', 'chosen': 'a', 'rejected': 'c', }, { 'prompt': 'I love cats', 'chosen': 'a', 'rejected': 'd', }, { 'prompt': 'The cat is orange', 'chosen': 'b', 'rejected': 'a', }, { 'prompt': 'The cat is orange', 'chosen': 'b', 'rejected': 'c', }, { 'prompt': 'The cat is orange', 'chosen': 'b', 'rejected': 'd', } ... </details <details <summary DPO code</summary 0. imports import os from dataclasses import dataclass, field from typing import Dict, Optional import torch from datasets import Dataset, load dataset from transformers import AutoModelForCausalLM, AutoTokenizer, HfArgumentParser, TrainingArguments, T5Config, T5Tokenizer, T5ForConditionalGeneration from trl import DPOTrainer Define and parse arguments. @dataclass class ScriptArguments: """ The arguments for the DPO training script. """ data parameters beta: Optional float = field default=0.1, metadata={"help": "the beta parameter for DPO loss"} training parameters model name or path: Optional str = field default="../sft/results/final checkpoint", metadata={"help": "the location of the SFT model name or path"}, config name: Optional str = field default=None, metadata={"help": "Pretrained config name or path if not the same as model name"} tokenizer name: Optional str = field default=None, metadata={"help": "Pretrained tokenizer name or path if not the same as model name"} cache dir: Optional str = field default=None, metadata={"help": "Where to store the pretrained models downloaded from huggingface.co"}, use fast tokenizer: bool = field default=False, metadata={"help": "Whether to use one of the fast tokenizer backed by the tokenizers library or not."}, model revision: str = field default="main", metadata={"help": "The specific model version to use can be a branch name, tag name or commit id ."}, use auth token: Optional str = field default=None, metadata={ "help": "Will use the token generated when running transformers-cli login necessary to use this script " "with private models ." }, train file: Optional str = field default=None, metadata={"help": "The input training data file a jsonlines or csv file ."} eval file: Optional str = field default=None, metadata={"help": "The input eval data file a jsonlines or csv file ."} learning rate: Optional float = field default=5e-4, metadata={"help": "optimizer learning rate"} lr scheduler type: Optional str = field default="cosine", metadata={"help": "the lr scheduler type"} warmup steps: Optional int = field default=100, metadata={"help": "the number of warmup steps"} weight decay: Optional float = field default=0.05, metadata={"help": "the weight decay"} optimizer type: Optional str = field default="paged adamw 32bit", metadata={"help": "the optimizer type"} per device train batch size: Optional int = field default=4, metadata={"help": "train batch size per device"} per device eval batch size: Optional int = field default=1, metadata={"help": "eval batch size per device"} gradient accumulation steps: Optional int = field default=4, metadata={"help": "the number of gradient accumulation steps"} gradient checkpointing: Optional bool = field default=True, metadata={"help": "whether to use gradient checkpointing"} lora alpha: Optional float = field default=16, metadata={"help": "the lora alpha parameter"} lora dropout: Optional float = field default=0.05, metadata={"help": "the lora dropout parameter"} lora r: Optional int = field default=8, metadata={"help": "the lora r parameter"} max prompt length: Optional int = field default=512, metadata={"help": "the maximum prompt length"} max length: Optional int = field default=1024, metadata={"help": "the maximum sequence length"} max steps: Optional int = field default=1000, metadata={"help": "max number of training steps"} logging steps: Optional int = field default=10, metadata={"help": "the logging frequency"} save steps: Optional int = field default=100, metadata={"help": "the saving frequency"} eval steps: Optional int = field default=100, metadata={"help": "the evaluation frequency"} output dir: Optional str = field default="./results", metadata={"help": "the output directory"} log freq: Optional int = field default=1, metadata={"help": "the logging frequency"} instrumentation sanity check: Optional bool = field default=False, metadata={"help": "only train on 1000 samples"} report to: Optional str = field default="wandb", metadata={ "help": 'The list of integrations to report the results and logs to. Supported platforms are "azure ml" ,' ' "comet ml" , "mlflow" , "neptune" , "tensorboard" , "clearml" and "wandb" . ' 'Use "all" to report to all integrations installed, "none" for no integrations.' }, debug argument for distributed training ignore bias buffers: Optional bool = field default=False, metadata={ "help": "fix for DDP issues with LM bias/mask buffers - invalid scalar type, inplace operation. See" "https://github.com/huggingface/transformers/issues/22482 issuecomment-1595790992" }, def convert dataset: Dataset = None, sanity check: bool = False, cache dir: str = None, num proc=24, - Dataset: """Load the dataset and convert it to the necessary format. The dataset is converted to a dictionary with the following structure: { 'prompt': List str , 'chosen': List str , 'rejected': List str , } """ original columns = dataset.column names if sanity check: dataset = dataset.select range min len dataset , 1000 def return prompt and responses samples - Dict str, str : return { "prompt": samples "prompt" , "chosen": samples "chosen" , "rejected": samples "rejected" , } return dataset.map return prompt and responses, batched=True, num proc=num proc, remove columns=original columns, if name == " main ": parser = HfArgumentParser ScriptArguments script args = parser.parse args into dataclasses 0 1. load a pretrained model config = T5Config.from pretrained script args.config name if script args.config name else script args.model name or path, cache dir=script args.cache dir, revision=script args.model revision, use auth token=script args.use auth token, tokenizer = T5Tokenizer.from pretrained script args.tokenizer name if script args.tokenizer name else script args.model name or path, cache dir=script args.cache dir, use fast=script args.use fast tokenizer, revision=script args.model revision, use auth token=script args.use auth token, model = T5ForConditionalGeneration.from pretrained script args.model name or path, config=config, cache dir=script args.cache dir, revision=script args.model revision, use auth token=script args.use auth token, model.config.use cache = False model ref = T5ForConditionalGeneration.from pretrained script args.model name or path, config=config, cache dir=script args.cache dir, revision=script args.model revision, use auth token=script args.use auth token, if script args.ignore bias buffers: torch distributed hack model. ddp params and buffers to ignore = name for name, buffer in model.named buffers if buffer.dtype == torch.bool 2. Load the dataset and split in train / eval train dataset = load dataset "json", data files=script args.train file, split="train" train dataset = convert dataset=train dataset, sanity check=script args.sanity check train dataset = train dataset.filter lambda x: len x "prompt" + len x "chosen" <= script args.max length and len x "prompt" + len x "rejected" <= script args.max length 3. Load evaluation dataset eval dataset = load dataset "json", data files=script args.eval file, split="train" eval dataset = convert dataset=eval dataset, sanity check=script args.sanity check eval dataset = eval dataset.filter lambda x: len x "prompt" + len x "chosen" <= script args.max length and len x "prompt" + len x "rejected" <= script args.max length 4. initialize training arguments: training args = TrainingArguments per device train batch size=script args.per device train batch size, per device eval batch size=script args.per device eval batch size, max steps=script args.max steps, logging steps=script args.logging steps, save steps=script args.save steps, gradient accumulation steps=script args.gradient accumulation steps, gradient checkpointing=script args.gradient checkpointing, learning rate=script args.learning rate, evaluation strategy="steps", eval steps=script args.eval steps, output dir=script args.output dir, lr scheduler type=script args.lr scheduler type, warmup steps=script args.warmup steps, remove unused columns=False, run name="dpo", 5. initialize the DPO trainer dpo trainer = DPOTrainer model, model ref, args=training args, beta=script args.beta, train dataset=train dataset, eval dataset=eval dataset, tokenizer=tokenizer, max prompt length=script args.max prompt length, max length=script args.max length, 6. train dpo trainer.train dpo trainer.save model script args.output dir 7. save output dir = os.path.join script args.output dir, "final checkpoint" dpo trainer.model.save pretrained output dir </details