{"slug": "allenai-open-instruct-tulu-3-post-training-with-sft-dpo-rlvr-grpo-and-verifier", "title": "AllenAI Open Instruct Tulu 3 Post-Training with SFT, DPO, RLVR, GRPO, and Verifier-Based Evaluation", "summary": "AllenAI's Open Instruct framework was used to build an end-to-end post-training pipeline for a compact instruction-tuned language model, integrating Supervised Fine-Tuning, Direct Preference Optimization, and Reinforcement Learning with Verifiable Rewards using GRPO, while adapting the original multi-GPU Tulu 3 stack to fit within a 16 GB runtime. The tutorial clones the Open Instruct repository, configures LoRA adapters, prepares GSM8K data, and uses deterministic verifiers to evaluate mathematical answers, replacing distributed components like vLLM and Ray with lightweight Hugging Face and PyTorch implementations for Colab.", "body_md": "In this tutorial, we build an end-to-end post-training pipeline for a compact instruction-tuned language model using[ AllenAI’s Open Instruct](https://github.com/allenai/open-instruct) framework. We move through three major training stages: Supervised Fine-Tuning, Direct Preference Optimization, and Reinforcement Learning with Verifiable Rewards using GRPO, while adapting the original multi-GPU Tulu 3 stack to fit within a 16 GB runtime. We clone the Open Instruct repository, selectively load its native loss and utility functions, configure LoRA adapters, prepare GSM8K data for each training stage, and use deterministic verifiers to evaluate generated mathematical answers. Throughout the workflow, we preserve the core optimization logic of Open Instruct while replacing distributed components such as vLLM, Ray actors, DeepSpeed, and asynchronous rollout queues with lightweight Hugging Face and PyTorch implementations suitable for Colab.\n\n``` python\nimport os, sys, subprocess, textwrap, json, math, random, re, ast, types, dataclasses, gc, contextlib\nREPO_URL = \"https://github.com/allenai/open-instruct.git\"\nREPO_DIR = \"/content/open-instruct\" if os.path.isdir(\"/content\") else \"./open-instruct\"\nPIP_PKGS = [\n   \"peft\", \"accelerate\",\n   \"ray\", \"wandb\", \"beaker-py\",\n   \"langdetect==1.0.9\", \"immutabledict==1.2.0\", \"nltk\",\n   \"absl-py\", \"sympy\", \"antlr4-python3-runtime==4.11\",\n   \"tiktoken\",\n]\ndef sh(*args):\n   print(\"$\", \" \".join(args))\n   subprocess.run(args, check=False)\ndef setup():\n   sh(sys.executable, \"-m\", \"pip\", \"install\", \"-q\", *PIP_PKGS)\n   if not os.path.isdir(REPO_DIR):\n       sh(\"git\", \"clone\", \"--depth\", \"1\", REPO_URL, REPO_DIR)\n   if REPO_DIR not in sys.path:\n       sys.path.insert(0, REPO_DIR)\n   os.environ.setdefault(\"WANDB_MODE\", \"disabled\")\n   os.environ.setdefault(\"TOKENIZERS_PARALLELISM\", \"false\")\n   os.environ.setdefault(\"RAY_DISABLE_IMPORT_WARNING\", \"1\")\nsetup()\nimport numpy as np\nimport torch\nimport torch.nn.functional as F\nfrom torch.utils.data import DataLoader\nfrom datasets import load_dataset, Dataset\nfrom transformers import AutoModelForCausalLM, DataCollatorForSeq2Seq, get_cosine_schedule_with_warmup\nfrom peft import LoraConfig, get_peft_model\nDEV = \"cuda\" if torch.cuda.is_available() else \"cpu\"\ntry:\n   _bf16 = DEV == \"cuda\" and torch.cuda.is_bf16_supported(including_emulation=False)\nexcept TypeError:\n   _bf16 = DEV == \"cuda\" and torch.cuda.get_device_properties(0).major >= 8\nAMP_DTYPE = torch.bfloat16 if _bf16 else torch.float16\nUSE_SCALER = AMP_DTYPE is torch.float16\nprint(f\"device={DEV}  autocast dtype={AMP_DTYPE}  gpu={torch.cuda.get_device_name(0) if DEV=='cuda' else '-'}\")\ndef oi_load(relpath, names, ns=None):\n   src = open(os.path.join(REPO_DIR, relpath)).read()\n   tree = ast.parse(src)\n   found = {n.name: n for n in tree.body\n            if isinstance(n, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)) and n.name in names}\n   missing = set(names) - set(found)\n   if missing:\n       raise KeyError(f\"{relpath}: could not find {missing} (upstream may have renamed them)\")\n   ns = {} if ns is None else dict(ns)\n   ns.update({\"torch\": torch, \"F\": F, \"np\": np, \"enum\": __import__(\"enum\"),\n              \"dataclasses\": dataclasses, \"math\": math, \"os\": os})\n   future = ast.parse(\"from __future__ import annotations\").body\n   mod = ast.Module(body=future + [found[n] for n in names], type_ignores=[])\n   exec(compile(ast.fix_missing_locations(mod), f\"<open_instruct:{relpath}>\", \"exec\"), ns)\n   return {n: ns[n] for n in names}\n_dpo  = oi_load(\"open_instruct/dpo_utils.py\", [\"dpo_loss\", \"_get_batch_logps\"])\n_pf   = oi_load(\"open_instruct/padding_free_collator.py\", [\"calculate_per_token_logps\"])\n_rl   = oi_load(\"open_instruct/rl_utils.py\", [\"masked_mean\"])\n_mu   = oi_load(\"open_instruct/model_utils.py\", [\"estimate_kl\"])\n_grpo = oi_load(\"open_instruct/grpo_utils.py\", [\"GRPOLossType\", \"compute_grpo_loss\"],\n               ns={\"model_utils\": types.SimpleNamespace(**_mu)})\ndpo_loss           = _dpo[\"dpo_loss\"]\nget_batch_logps    = _dpo[\"_get_batch_logps\"]\nper_token_logps_fn = _pf[\"calculate_per_token_logps\"]\nmasked_mean        = _rl[\"masked_mean\"]\ncompute_grpo_loss  = _grpo[\"compute_grpo_loss\"]\nGRPOLossType       = _grpo[\"GRPOLossType\"]\nprint(\"lifted from repo:\", [f.__name__ for f in (dpo_loss, get_batch_logps, per_token_logps_fn,\n                                                masked_mean, compute_grpo_loss)])\nfrom open_instruct.dataset_transformation import (\n   CHAT_TEMPLATES, TokenizerConfig,\n   sft_tulu_tokenize_and_truncate_v1, sft_tulu_filter_v1,\n   preference_tulu_tokenize_and_truncate_v1_2,\n   rlvr_tokenize_v1, visualize_token_role,\n)\nfrom open_instruct.ground_truth_utils import GSM8KVerifier, MathVerifier, IFEvalVerifierOld\n```\n\nWe install the required lightweight dependencies, clone the Open Instruct repository, and configure the Colab environment for stable execution. We detect the available GPU precision mode and select either FP16 or BF16 autocasting based on the hardware capabilities. We also extract the original DPO, GRPO, masking, and log-probability functions directly from the repository without importing its full distributed training stack.\n\n```\n@dataclasses.dataclass\nclass CFG:\n   model: str = \"Qwen/Qwen2.5-0.5B-Instruct\"\n   max_seq_len: int = 640\n   seed: int = 42\n   n_sft: int = 192\n   sft_steps: int = 40\n   sft_micro_bs: int = 2\n   sft_accum: int = 4\n   sft_lr: float = 1e-4\n   n_dpo: int = 96\n   dpo_steps: int = 24\n   dpo_micro_bs: int = 1\n   dpo_accum: int = 4\n   dpo_lr: float = 5e-5\n   dpo_beta: float = 0.1\n   dpo_norm: bool = True\n   grpo_iters: int = 6\n   prompts_per_iter: int = 4\n   samples_per_prompt: int = 4\n   grpo_micro_bs: int = 1\n   grpo_inner_epochs: int = 2\n   grpo_lr: float = 2e-5\n   grpo_temperature: float = 1.0\n   grpo_max_new: int = 200\n   grpo_kl_beta: float = 0.02\n   clip_lower: float = 0.2\n   clip_higher: float = 0.272\n   kl_estimator: int = 2\n   adv_norm: str = \"centered\"\n   n_eval: int = 24\ncfg = CFG()\nrandom.seed(cfg.seed); np.random.seed(cfg.seed); torch.manual_seed(cfg.seed)\ntc = TokenizerConfig(tokenizer_name_or_path=cfg.model, chat_template_name=None, use_fast=True)\ntok = tc.tokenizer\nprint(f\"\\navailable CHAT_TEMPLATES: {list(CHAT_TEMPLATES)[:12]} ... ({len(CHAT_TEMPLATES)} total)\")\nprint(f\"pad={tok.pad_token!r}({tok.pad_token_id})  eos={tok.eos_token!r}({tok.eos_token_id})\")\n_demo = {\"messages\": [\n   {\"role\": \"user\", \"content\": \"What is 12 * 3?\"},\n   {\"role\": \"assistant\", \"content\": \"12 * 3 = 36. The answer is 36.\"},\n   {\"role\": \"user\", \"content\": \"And minus 6?\"},\n   {\"role\": \"assistant\", \"content\": \"36 - 6 = 30. The answer is 30.\"},\n]}\n_enc = sft_tulu_tokenize_and_truncate_v1(dict(_demo), tok, cfg.max_seq_len)\nprint(\"\\n[SFT label masking — colour 0 = masked out of the loss, colour 1 = trained on]\")\nvisualize_token_role(_enc[\"input_ids\"].tolist(), (_enc[\"labels\"] != -100).long().tolist(), tok)\nprint(f\"trainable tokens: {(_enc['labels'] != -100).sum().item()}/{_enc['labels'].numel()}\")\n```\n\nWe define a centralized configuration class that controls the model, dataset sizes, learning rates, batch settings, and optimization parameters for every training stage. We initialize the Open Instruct tokenizer while preserving the model’s chat template and ensuring that padding and end-of-sequence tokens remain correctly separated. We then tokenize a sample conversation and visualize which assistant tokens contribute to the supervised training loss.\n\n```\ngsm = load_dataset(\"openai/gsm8k\", \"main\")\nSYS = \"You are a careful math assistant. Reason step by step, then finish with 'The answer is N.'\"\ndef gsm_answer(a):\n   return a.split(\"####\")[-1].strip().replace(\",\", \"\")\ndef gsm_solution(a):\n   body = a.split(\"####\")[0].strip()\n   body = re.sub(r\"<<.*?>>\", \"\", body)\n   return f\"{body}\\nThe answer is {gsm_answer(a)}.\"\ndef as_messages(row):\n   return [{\"role\": \"system\", \"content\": SYS},\n           {\"role\": \"user\", \"content\": row[\"question\"]},\n           {\"role\": \"assistant\", \"content\": gsm_solution(row[\"answer\"])}]\ntrain_rows = [gsm[\"train\"][i] for i in range(cfg.n_sft + cfg.n_dpo)]\neval_rows = [gsm[\"test\"][i] for i in range(cfg.n_eval)]\ndef to_lists(row):\n   for k in (\"input_ids\", \"labels\", \"attention_mask\"):\n       row[k] = row[k].tolist()\n   return row\nsft_ds = Dataset.from_list([{\"messages\": as_messages(r)} for r in train_rows[: cfg.n_sft]])\nsft_ds = sft_ds.map(lambda r: to_lists(sft_tulu_tokenize_and_truncate_v1(r, tok, cfg.max_seq_len)),\n                   remove_columns=[\"messages\"], desc=\"sft tokenize\")\nsft_ds = sft_ds.filter(sft_tulu_filter_v1, fn_kwargs={\"tokenizer\": tok}, desc=\"drop all-masked\")\ndef make_pair(r):\n   gold = gsm_answer(r[\"answer\"])\n   bad = (str(int(float(gold)) + random.choice([-10, -3, -1, 1, 2, 7]))\n          if gold.replace('.', '', 1).lstrip('-').isdigit() else gold + \"0\")\n   prompt = [{\"role\": \"system\", \"content\": SYS}, {\"role\": \"user\", \"content\": r[\"question\"]}]\n   good_txt = gsm_solution(r[\"answer\"])\n   bad_txt = good_txt.rsplit(\"The answer is\", 1)[0] + f\"The answer is {bad}.\"\n   return {\"chosen\": prompt + [{\"role\": \"assistant\", \"content\": good_txt}],\n           \"rejected\": prompt + [{\"role\": \"assistant\", \"content\": bad_txt}]}\ndpo_ds = Dataset.from_list([make_pair(r) for r in train_rows[cfg.n_sft:]])\ndpo_ds = dpo_ds.map(\n   lambda r: {k: (v.tolist() if torch.is_tensor(v) else v) for k, v in\n              preference_tulu_tokenize_and_truncate_v1_2(r, tok, cfg.max_seq_len).items()},\n   remove_columns=[\"chosen\", \"rejected\"], desc=\"dpo tokenize\")\nrlvr_rows = [{\"messages\": as_messages(r)[:2], \"ground_truth\": gsm_answer(r[\"answer\"]), \"dataset\": \"gsm8k\"}\n            for r in train_rows[: cfg.n_sft]]\nrlvr_ds = Dataset.from_list(rlvr_rows).map(lambda r: rlvr_tokenize_v1(r, tok),\n                                          remove_columns=[\"messages\"], desc=\"rlvr tokenize\")\nprint(f\"\\nsft={len(sft_ds)}  dpo={len(dpo_ds)}  rlvr={len(rlvr_ds)}\")\nVERIFIERS = {\"gsm8k\": GSM8KVerifier(), \"math\": MathVerifier(), \"ifeval_old\": IFEvalVerifierOld()}\nprint(\"\\n[verifier smoke test]\")\nprint(\" gsm8k :\", VERIFIERS[\"gsm8k\"]([], \"9 + 3 = 12. The answer is 12.\", \"12\").score)\nprint(\" gsm8k :\", VERIFIERS[\"gsm8k\"]([], \"The answer is 11.\", \"12\").score)\nprint(\" math  :\", VERIFIERS[\"math\"]([], r\"hence \\boxed{0.5}\", r\"\\frac{1}{2}\").score)\nprint(\" ifeval:\", VERIFIERS[\"ifeval_old\"]([], \"one two three four five six seven\",\n                                         json.dumps({\"func_name\": \"validate_word_constraint\",\n                                                     \"N\": 6, \"quantifier\": \"at least\"})).score)\ndef verify_batch(responses, ground_truths, sources, tokenized=None):\n   out = []\n   for i, (resp, gt, src) in enumerate(zip(responses, ground_truths, sources)):\n       v = VERIFIERS.get(src, VERIFIERS[\"gsm8k\"])\n       out.append(v(tokenized[i] if tokenized else [], resp, gt).score * v.weight)\n   return np.array(out, dtype=np.float32)\n```\n\nWe load GSM8K and transform its questions and solutions into a consistent conversational format for SFT, DPO, and RLVR training. We create supervised examples, preference pairs with deliberately incorrect final answers, and verifier-ready prompts with structured ground-truth labels. We also initialize Open Instruct’s GSM8K, mathematical, and instruction-following verifiers and use them to score generated responses deterministically.\n\n```\nmodel = AutoModelForCausalLM.from_pretrained(cfg.model, dtype=torch.float32).to(DEV)\nmodel.config.use_cache = False\nif len(tok) > model.get_input_embeddings().weight.shape[0]:\n   model.resize_token_embeddings(len(tok))\ndef _patch_peft_torchao():\n   import importlib\n   for mod in (\"peft.import_utils\", \"peft.tuners.lora.torchao\",\n               \"peft.tuners.lora.model\", \"peft.tuners.lora.layer\"):\n       try:\n           m = importlib.import_module(mod)\n       except Exception:\n           continue\n       if hasattr(m, \"is_torchao_available\"):\n           m.is_torchao_available = lambda: False\n_patch_peft_torchao()\nmodel = get_peft_model(model, LoraConfig(\n   r=32, lora_alpha=64, lora_dropout=0.05, bias=\"none\", task_type=\"CAUSAL_LM\",\nmodel.print_trainable_parameters()\nTRAINABLE = [p for p in model.parameters() if p.requires_grad]\n@contextlib.contextmanager\ndef with_cache():\n   old = model.config.use_cache\n   model.config.use_cache = True\n   try:\n       yield\n   finally:\n       model.config.use_cache = old\ndef amp():\n   return torch.autocast(device_type=\"cuda\", dtype=AMP_DTYPE) if DEV == \"cuda\" \\\n       else torch.autocast(device_type=\"cpu\", enabled=False)\ndef new_opt(lr, steps):\n   opt = torch.optim.AdamW(TRAINABLE, lr=lr, weight_decay=0.0, betas=(0.9, 0.999))\n   sched = get_cosine_schedule_with_warmup(opt, int(0.05 * steps) + 1, steps)\n   scaler = torch.amp.GradScaler(\"cuda\", enabled=USE_SCALER)\n   return opt, sched, scaler\ndef step_opt(opt, sched, scaler):\n   scaler.unscale_(opt)\n   torch.nn.utils.clip_grad_norm_(TRAINABLE, 1.0)\n   scaler.step(opt); scaler.update(); sched.step(); opt.zero_grad(set_to_none=True)\n@torch.no_grad()\ndef evaluate(tag, rows, max_new=256):\n   model.eval()\n   tok.padding_side = \"left\"\n   correct, bs = 0.0, 4\n   for i in range(0, len(rows), bs):\n       chunk = rows[i:i + bs]\n       prompts = [tok.apply_chat_template(\n           [{\"role\": \"system\", \"content\": SYS}, {\"role\": \"user\", \"content\": r[\"question\"]}],\n           add_generation_prompt=True, tokenize=False) for r in chunk]\n       enc = tok(prompts, return_tensors=\"pt\", padding=True, add_special_tokens=False).to(DEV)\n       with amp(), with_cache():\n           out = model.generate(**enc, max_new_tokens=max_new, do_sample=False,\n                                pad_token_id=tok.pad_token_id)\n       texts = tok.batch_decode(out[:, enc[\"input_ids\"].shape[1]:], skip_special_tokens=True)\n       correct += verify_batch(texts, [gsm_answer(r[\"answer\"]) for r in chunk],\n                               [\"gsm8k\"] * len(chunk)).sum()\n   acc = correct / len(rows)\n   print(f\"  [eval:{tag}] verifier accuracy = {acc:.3f}  ({int(correct)}/{len(rows)})\")\n   model.train(); tok.padding_side = \"right\"\n   return acc\nprint(\"\\n\" + \"=\" * 90); print(\"BASELINE\"); print(\"=\" * 90)\nbase_acc = evaluate(\"base\", eval_rows)\n```\n\nWe load the Qwen instruction model, apply LoRA adapters to its attention and feed-forward projection layers, and restrict optimization to the trainable adapter parameters. We configure mixed-precision execution, gradient scaling, gradient clipping, learning-rate scheduling, and temporary KV-cache activation for generation. We then evaluate the untrained baseline on GSM8K using greedy decoding and verifier-based answer accuracy.\n\n```\nprint(\"\\n\" + \"=\" * 90); print(\"STAGE 1 — SFT\"); print(\"=\" * 90)\nsft_collate = DataCollatorForSeq2Seq(tokenizer=tok, padding=\"longest\", label_pad_token_id=-100)\nsft_dl = DataLoader(sft_ds, batch_size=cfg.sft_micro_bs, shuffle=True, collate_fn=sft_collate, drop_last=True)\nopt, sched, scaler = new_opt(cfg.sft_lr, cfg.sft_steps)\nmodel.train(); it, step, run = iter(sft_dl), 0, 0.0\nwhile step < cfg.sft_steps:\n   for _ in range(cfg.sft_accum):\n       try:\n           batch = next(it)\n       except StopIteration:\n           it = iter(sft_dl); batch = next(it)\n       batch = {k: v.to(DEV) for k, v in batch.items()}\n       with amp():\n           loss = model(**batch).loss / cfg.sft_accum\n       scaler.scale(loss).backward()\n       run += loss.item()\n   step_opt(opt, sched, scaler); step += 1\n   if step % 10 == 0 or step == 1:\n       print(f\"  sft step {step:>3}/{cfg.sft_steps}  loss {run:.4f}  lr {sched.get_last_lr()[0]:.2e}\")\n   run = 0.0\nsft_acc = evaluate(\"after-sft\", eval_rows)\n```\n\nWe construct a padded SFT DataLoader and train the LoRA adapters on tokenized GSM8K conversations using gradient accumulation. We optimize the model with cross-entropy loss calculated only over the unmasked assistant response tokens. We track the training loss and learning rate throughout the stage and evaluate the updated model after supervised fine-tuning.\n\n```\nprint(\"\\n\" + \"=\" * 90); print(\"STAGE 2 — DPO (dpo_norm)\"); print(\"=\" * 90)\ndef pad_side(seqs, pad, maxlen):\n   return torch.tensor([s + [pad] * (maxlen - len(s)) for s in seqs], dtype=torch.long)\ndef dpo_collate(features):\n   out = {}\n   for pfx in (\"chosen\", \"rejected\"):\n       L = max(len(f[f\"{pfx}_input_ids\"]) for f in features)\n       out[f\"{pfx}_input_ids\"] = pad_side([f[f\"{pfx}_input_ids\"] for f in features], tok.pad_token_id, L)\n       out[f\"{pfx}_labels\"] = pad_side([f[f\"{pfx}_labels\"] for f in features], -100, L)\n       out[f\"{pfx}_attention_mask\"] = pad_side([f[f\"{pfx}_attention_mask\"] for f in features], 0, L)\n   return out\ndef seq_logps(input_ids, attn, labels):\n   with amp():\n       logits = model(input_ids=input_ids, attention_mask=attn).logits\n   ptl = per_token_logps_fn(logits, labels)\n   return get_batch_logps(ptl, labels, average_log_prob=cfg.dpo_norm)\ndpo_dl = DataLoader(dpo_ds, batch_size=cfg.dpo_micro_bs, shuffle=True, collate_fn=dpo_collate, drop_last=True)\nopt, sched, scaler = new_opt(cfg.dpo_lr, cfg.dpo_steps)\nit, step = iter(dpo_dl), 0\nwhile step < cfg.dpo_steps:\n   agg = {\"loss\": 0.0, \"acc\": 0.0, \"margin\": 0.0}\n   for _ in range(cfg.dpo_accum):\n       try:\n           b = next(it)\n       except StopIteration:\n           it = iter(dpo_dl); b = next(it)\n       b = {k: v.to(DEV) for k, v in b.items()}\n       with torch.no_grad(), model.disable_adapter():\n           ref_c = seq_logps(b[\"chosen_input_ids\"], b[\"chosen_attention_mask\"], b[\"chosen_labels\"])\n           ref_r = seq_logps(b[\"rejected_input_ids\"], b[\"rejected_attention_mask\"], b[\"rejected_labels\"])\n       pol_c = seq_logps(b[\"chosen_input_ids\"], b[\"chosen_attention_mask\"], b[\"chosen_labels\"])\n       pol_r = seq_logps(b[\"rejected_input_ids\"], b[\"rejected_attention_mask\"], b[\"rejected_labels\"])\n       losses, r_c, r_r = dpo_loss(pol_c, pol_r, ref_c, ref_r, beta=cfg.dpo_beta, label_smoothing=0.0)\n       loss = losses.mean() / cfg.dpo_accum\n       scaler.scale(loss).backward()\n       agg[\"loss\"] += loss.item()\n       agg[\"acc\"] += (r_c > r_r).float().mean().item() / cfg.dpo_accum\n       agg[\"margin\"] += (r_c - r_r).mean().item() / cfg.dpo_accum\n   step_opt(opt, sched, scaler); step += 1\n   if step % 8 == 0 or step == 1:\n       print(f\"  dpo step {step:>3}/{cfg.dpo_steps}  loss {agg['loss']:.4f} \"\n             f\"reward_acc {agg['acc']:.2f}  margin {agg['margin']:+.3f}\")\ndpo_acc = evaluate(\"after-dpo\", eval_rows)\n```\n\nWe batch the chosen and rejected responses separately and calculate their length-normalized sequence log probabilities with Open Instruct’s native utilities. We compare the active LoRA policy against the frozen base reference policy and optimize the model using the repository’s DPO loss. We monitor preference accuracy, reward margins, and training loss before measuring the model’s post-DPO verifier performance.\n\n```\nprint(\"\\n\" + \"=\" * 90); print(\"STAGE 3 — RLVR / GRPO\"); print(\"=\" * 90)\ngrpo_cfg = types.SimpleNamespace(loss_fn=GRPOLossType.dapo, clip_lower=cfg.clip_lower,\n                                clip_higher=cfg.clip_higher, kl_estimator=cfg.kl_estimator)\n_gen_eos = getattr(getattr(model, \"generation_config\", None), \"eos_token_id\", None)\n_terms = {tok.eos_token_id, tok.pad_token_id}\n_terms |= set(_gen_eos) if isinstance(_gen_eos, (list, tuple)) else {_gen_eos}\nTERMINATORS = torch.tensor(sorted(t for t in _terms if t is not None), device=DEV)\ndef token_logps(seq, attn, temperature, grad=True):\n   pos = (attn.cumsum(-1) - 1).clamp(min=0)\n   ctx = torch.enable_grad() if grad else torch.no_grad()\n   with ctx, amp():\n       logits = model(input_ids=seq, attention_mask=attn, position_ids=pos).logits\n   return per_token_logps_fn(logits / temperature, seq)\ndef rollout(batch_rows):\n   G = cfg.samples_per_prompt\n   ids = [r[\"input_ids_prompt\"] for r in batch_rows]\n   P = max(len(x) for x in ids)\n   pin = torch.tensor([[tok.pad_token_id] * (P - len(x)) + x for x in ids], device=DEV)\n   pmask = torch.tensor([[0] * (P - len(x)) + [1] * len(x) for x in ids], device=DEV)\n   model.eval()\n   with torch.no_grad(), amp(), with_cache():\n       seq = model.generate(input_ids=pin, attention_mask=pmask, do_sample=True,\n                            temperature=cfg.grpo_temperature, top_p=1.0, top_k=0,\n                            max_new_tokens=cfg.grpo_max_new, num_return_sequences=G,\n                            pad_token_id=tok.pad_token_id)\n   model.train()\n   resp = seq[:, P:]\n   is_term = torch.isin(resp, TERMINATORS)\n   first = torch.where(is_term.any(1), is_term.float().argmax(1),\n                       torch.full((resp.shape[0],), resp.shape[1] - 1, device=DEV))\n   idx = torch.arange(resp.shape[1], device=DEV).unsqueeze(0)\n   resp_mask = (idx <= first.unsqueeze(1)).long()\n   full_mask = torch.cat([torch.zeros(seq.shape[0], P, dtype=torch.long, device=DEV), resp_mask], 1)\n   attn = torch.cat([pmask.repeat_interleave(G, 0), resp_mask], 1)\n   texts = tok.batch_decode(resp, skip_special_tokens=True)\n   gts = [r[\"ground_truth\"] for r in batch_rows for _ in range(G)]\n   srcs = [r[\"dataset\"] for r in batch_rows for _ in range(G)]\n   scores = verify_batch(texts, gts, srcs)\n   per_prompt = scores.reshape(-1, G)\n   mean_g = np.repeat(per_prompt.mean(-1), G, 0)\n   if cfg.adv_norm == \"standard\":\n       adv = (scores - mean_g) / (np.repeat(per_prompt.std(-1), G, 0) + 1e-8)\n   else:\n       adv = scores - mean_g\n   adv_t = torch.tensor(adv, device=DEV, dtype=torch.float32).unsqueeze(1).expand_as(full_mask.float())\n   return seq, attn, full_mask, adv_t, scores, texts\nopt, sched, scaler = new_opt(cfg.grpo_lr, cfg.grpo_iters * cfg.grpo_inner_epochs)\norder = list(range(len(rlvr_ds))); random.shuffle(order)\nfor it_i in range(cfg.grpo_iters):\n   rows = [rlvr_ds[order[(it_i * cfg.prompts_per_iter + j) % len(rlvr_ds)]]\n           for j in range(cfg.prompts_per_iter)]\n   seq, attn, mask, adv, scores, texts = rollout(rows)\n   with torch.no_grad():\n       old_lp = torch.cat([token_logps(seq[i:i + cfg.grpo_micro_bs], attn[i:i + cfg.grpo_micro_bs],\n                                       cfg.grpo_temperature, grad=False)\n                           for i in range(0, seq.shape[0], cfg.grpo_micro_bs)])\n       with model.disable_adapter():\n           ref_lp = torch.cat([token_logps(seq[i:i + cfg.grpo_micro_bs], attn[i:i + cfg.grpo_micro_bs],\n                                           cfg.grpo_temperature, grad=False)\n                               for i in range(0, seq.shape[0], cfg.grpo_micro_bs)])\n   n_chunks = math.ceil(seq.shape[0] / cfg.grpo_micro_bs)\n   for ep in range(cfg.grpo_inner_epochs):\n       stats = {\"pg\": 0.0, \"kl\": 0.0, \"clip\": 0.0}\n       for i in range(0, seq.shape[0], cfg.grpo_micro_bs):\n           sl = slice(i, i + cfg.grpo_micro_bs)\n           new_lp = token_logps(seq[sl], attn[sl], cfg.grpo_temperature, grad=True)\n           new_lp_, old_lp_, ref_lp_ = new_lp[:, :-1], old_lp[sl][:, :-1], ref_lp[sl][:, :-1]\n           m_, a_ = mask[sl][:, 1:], adv[sl][:, 1:]\n           ratio = torch.exp((new_lp_ - old_lp_).clamp(-20, 20))\n           pg, clipfrac, kl = compute_grpo_loss(new_lp_, ratio, a_, ref_lp_, grpo_cfg,\n                                                torch.ones_like(ratio))\n           loss = masked_mean(pg + cfg.grpo_kl_beta * kl, m_) / n_chunks\n           scaler.scale(loss).backward()\n           with torch.no_grad():\n               stats[\"pg\"] += masked_mean(pg.detach(), m_).item() / n_chunks\n               stats[\"kl\"] += masked_mean(kl.detach(), m_).item() / n_chunks\n               stats[\"clip\"] += masked_mean(clipfrac.detach(), m_).item() / n_chunks\n           del new_lp, ratio, pg, kl\n       step_opt(opt, sched, scaler)\n       if DEV == \"cuda\":\n           torch.cuda.empty_cache()\n       print(f\"  grpo iter {it_i+1}/{cfg.grpo_iters} ep{ep+1}  reward {scores.mean():.3f} \"\n             f\"(solved {int(scores.sum())}/{len(scores)})  pg {stats['pg']:+.4f}  \"\n             f\"kl {stats['kl']:.4f}  clipfrac {stats['clip']:.3f}\")\nprint(\"\\n  sample rollout ->\", textwrap.shorten(texts[0].replace(\"\\n\", \" \"), 220))\nrlvr_acc = evaluate(\"after-rlvr\", eval_rows)\nprint(\"\\n\" + \"=\" * 90)\nprint(f\"{'stage':<14}{'verifier acc':>14}\")\nfor name, val in [(\"base\", f\"{base_acc:.3f}\"), (\"sft\", f\"{sft_acc:.3f}\"),\n                 (\"dpo\", f\"{dpo_acc:.3f}\"), (\"rlvr\", f\"{rlvr_acc:.3f}\")]:\n   print(f\"{name:<14}{val:>14}\")\nprint(\"=\" * 90)\nOUT = \"/content/tulu-mini\" if os.path.isdir(\"/content\") else \"./tulu-mini\"\nmerged = model.merge_and_unload()\nmerged.save_pretrained(OUT); tok.save_pretrained(OUT)\nprint(f\"merged checkpoint -> {OUT}  (equivalent to `python open_instruct/merge_lora.py`)\")\n```\n\nWe generate multiple sampled responses for each prompt, score them with deterministic verifiers, and calculate group-relative advantages from their reward distributions. We optimize the policy with Open Instruct’s GRPO and DAPO-style clipping logic while applying response masks, importance ratios, and KL regularization against the reference model. We finally compare accuracy across the baseline, SFT, DPO, and RLVR stages before merging the LoRA adapters and saving the completed checkpoint.\n\nIn conclusion, we implemented a practical miniature version of the Tulu 3 post-training stack and observed how each training stage changes model performance on verifier-scored mathematical reasoning tasks. We first established a baseline, improved instruction-following through supervised fine-tuning, refined response preferences through length-normalized DPO, and finally optimized verified task rewards using group-relative advantages and the repository’s GRPO loss implementation. We also used LoRA to maintain an accessible reference policy, apply response masking and KL regularization during reinforcement learning, compare accuracy across all training stages, and export a merged checkpoint for later inference or evaluation.\n\nCheck out the ** FULL CODES here**.\n\n**Also, feel free to follow us on**\n\n**and don’t forget to join our**[Twitter](https://x.com/intent/follow?screen_name=marktechpost)\n\n**and Subscribe to**\n\n[150k+ML SubReddit](https://www.reddit.com/r/machinelearningnews/)**. Wait! are you on telegram?**\n\n[our Newsletter](https://magic.beehiiv.com/v1/f5e63dd4-5653-4f09-83e2-321a8b1ba526?email={{email}})\n\n[now you can join us on telegram as well.](https://t.me/machinelearningresearchnews)Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? [Connect with us](https://forms.gle/wbash1wF6efRj8G58)\n\nSana Hassan, a consulting intern at Marktechpost and dual-degree student at IIT Madras, is passionate about applying technology and AI to address real-world challenges. With a keen interest in solving practical problems, he brings a fresh perspective to the intersection of AI and real-life solutions.", "url": "https://wpnews.pro/news/allenai-open-instruct-tulu-3-post-training-with-sft-dpo-rlvr-grpo-and-verifier", "canonical_source": "https://www.marktechpost.com/2026/08/12/allenai-open-instruct-tulu-3-post-training-with-sft-dpo-rlvr-grpo-and-verifier-based-evaluation/", "published_at": "2026-08-12 17:37:37+00:00", "updated_at": "2026-08-12 17:45:47.844380+00:00", "lang": "en", "topics": ["large-language-models", "machine-learning", "artificial-intelligence"], "entities": ["AllenAI", "Open Instruct", "Tulu 3", "LoRA", "GSM8K", "Hugging Face", "PyTorch", "Colab"], "alternates": {"html": "https://wpnews.pro/news/allenai-open-instruct-tulu-3-post-training-with-sft-dpo-rlvr-grpo-and-verifier", "markdown": "https://wpnews.pro/news/allenai-open-instruct-tulu-3-post-training-with-sft-dpo-rlvr-grpo-and-verifier.md", "text": "https://wpnews.pro/news/allenai-open-instruct-tulu-3-post-training-with-sft-dpo-rlvr-grpo-and-verifier.txt", "jsonld": "https://wpnews.pro/news/allenai-open-instruct-tulu-3-post-training-with-sft-dpo-rlvr-grpo-and-verifier.jsonld"}}