Four Ways to Teach an AI to Draw a Cat

A new explainer outlines four methods for training AI models to draw a cat: Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), Reinforcement Learning (RL), and On-policy distillation (OPD). Each method uses a different training data format, from copying expert examples to scoring generated responses or receiving per-token teacher feedback.

Four Ways to Teach an AI to Draw a Cat Same student, same cat, four very different lessons — and every way of training a language model is one of them. SFT — Trace the teacher Copy the teacher's drawing, stroke for stroke. Supervised Fine-Tuning SFT is training a model to copy expert-written examples, token by token. See example training JSON data set: { "prompt": "Draw a cat.", "completion": " the teacher's drawing, stroke by stroke " } DPO — Pick the better one Pick the better of two drawings. Direct Preference Optimization DPO is training a model to prefer one labeled response over another. See example training JSON data set: { "prompt": "Draw a cat.", "chosen": " a good drawing ", "rejected": " a worse drawing " } RL — Just a score Draw a cat, get a single score back. Reinforcement Learning RL is training a model by scoring its own generated responses and reinforcing what scored well. See example training JSON data set: { "prompt": "Draw a cat.", "response": " the model's own drawing ", "reward": 7 } OPD — Notes on every line Draw a cat, get corrected stroke by stroke. On-policy distillation OPD is training a model on its own responses, graded token by token by a stronger teacher model. See example training JSON data set: { "prompt": "Draw a cat.", "response": " the model's own drawing ", "teacher logits": " the teacher's per-token distribution over this response " }