You Fine Tuned the Model on Your Catalogue and It Only Learned It One Way

Serguey Asael Shinder describes how fine-tuning a model on a product catalogue taught it facts in only one direction: asked which boots are waterproof, it confidently named the wrong product even though the fact appeared hundreds of times in training. He cites the 2023 'reversal curse' research led by Lukas Berglund, which found models trained on 'A is B' sentences answer reverse questions no better than chance, and recommends writing facts in both directions, keeping facts in retrieval, and testing with questions phrased the way users actually ask them.

The team fine tuned a model on the product catalogue. Thousands of lines like this one. The ZR-7 is our only fully waterproof walking boot. Ask it about the ZR-7 and it knew. Waterproof. Walking. Boot. The sizes, even. Then a customer asked which of your boots is waterproof. It named a different one. Confidently. The fact was in the training data, hundreds of times. Just pointing the other way. In 2023 researchers led by Lukas Berglund gave this a name. The reversal curse. They fine tuned models on invented sentences of the form A is B, a made up person is the director of a made up film. Asked who directed the film, the models did no better than chance. They found it in the big models too, with real people. GPT-4 could say who Tom Cruise's mother is about four times in five. Given her name and asked who her son is, about one time in three. Training on a sentence teaches the model to continue it in the order it was written. It does not, by itself, store a fact you can enter from either side. People do that without noticing. Models, mostly, do not. The odd part is that inside the prompt it is fine. Put the sentence in the context, ask the reverse question, and the model answers it, because now it is reading, not remembering. So think about the direction of every question your users will ask. Customers rarely start with the code. They start with the need, the size, the symptom. If you fine tune, write the facts both ways, and in the ways people actually ask. Better, keep the facts in retrieval, where the model reads them at the moment it answers, and use fine tuning for tone and format. And test in the direction your users come from. If every test question starts with the product name, you have only checked the half of the fact you already knew it had. – Serguey Asael Shinder