In the previous article, we explored the part where we collect human preferences. In this article, we will see how to use this data to train the models. To train a model that gives higher scores to preferred responses, we first make a copy of the model that has already gone through supervised fine-tuning. Next, we modify this copied model. We remove the unembedding layer, which normally predicts the next token, and replace it with a single output value. The result is a new model called a reward model. Instead of generating text, this model learns to assign a reward score to a response. We can now train this reward model using the human preference data we collected earlier.
For a preferred response, we train the model to produce a higher reward value.
For a less preferred response, we train the model to produce a lower reward value or a negative reward.
For example:
Over time, the reward model learns what kinds of responses humans tend to prefer. We will continue further in the next article Looking for an easier way to install tools, libraries, or entire repositories? Try Installerpedia: a community-driven, structured installation platform that lets you install almost anything with minimal hassle and clear, reliable guidance. Just run: ipm install repo-name … and you’re done! 🚀