Four Ways to Teach an AI to Draw a Cat

Same student, same cat, four very different lessons — and every way of training a language model is one of them.

SFT — Trace the teacher #

Copy the teacher's drawing, stroke for stroke.

Supervised Fine-Tuning (SFT) is training a model to copy expert-written examples, token by token.

See example training JSON data set:

{
  "prompt": "Draw a cat.",
  "completion": "[the teacher's drawing, stroke by stroke]"
}

DPO — Pick the better one #

Pick the better of two drawings.

Direct Preference Optimization (DPO) is training a model to prefer one labeled response over another.

See example training JSON data set:

{
  "prompt": "Draw a cat.",
  "chosen": "[a good drawing]",
  "rejected": "[a worse drawing]"
}

RL — Just a score #

Draw a cat, get a single score back.

Reinforcement Learning (RL) is training a model by scoring its own generated responses and reinforcing what scored well.

See example training JSON data set:

{
  "prompt": "Draw a cat.",
  "response": "[the model's own drawing]",
  "reward": 7
}

OPD — Notes on every line #

Draw a cat, get corrected stroke by stroke.

On-policy distillation (OPD) is training a model on its own responses, graded token by token by a stronger teacher model.

See example training JSON data set:

{
  "prompt": "Draw a cat.",
  "response": "[the model's own drawing]",
  "teacher_logits": "[the teacher's per-token distribution over this response]"
}