{"slug": "tpo-torch-target-policy-optimization-for-stable-rlhf-alignment-in-pytorch", "title": "Tpo-Torch – Target Policy Optimization for Stable RLHF Alignment in PyTorch", "summary": "Researchers have released Tpo-Torch, a PyTorch implementation of Target Policy Optimization (TPO), a reinforcement learning from human feedback (RLHF) algorithm that simplifies the standard PPO approach by eliminating the need for a value function, clipping, and importance sampling ratios. Based on a 2026 arXiv paper by Kaddour, TPO achieves a held-out perplexity of 74.51 compared to 825.66 for a simplified PPO-clip, with zero NaN gradients and a maximum gradient norm of 0.09 across advantage values from 0.01 to 1000. The library, which includes a TPOTrainer compatible with HuggingFace models, is designed to provide stable RLHF alignment with reduced complexity.", "body_md": "**Target Policy Optimization** — a simpler alternative to PPO for RLHF.\n\nBased on [arXiv:2604.06159](https://arxiv.org/abs/2604.06159) (Kaddour, 2026).\n\nTPO is an RLHF algorithm. It is **not** a replacement for cross-entropy (CE) training. CE and TPO do different things:\n\n| Cross-Entropy (SFT) | PPO | TPO | |\n|---|---|---|---|\n| Purpose | Predict next token | Optimize reward via RL | Optimize reward via RL |\n| Needs labeled data | Yes | No | No |\n| Needs reward model | No | Yes | Yes |\n| Needs value/critic head | No | Yes | No |\n| Needs clipping | No | Yes | No |\n| Needs importance ratios | No | Yes | No |\n| Training stability | Stable | Often unstable | Stable |\n\n**CE is supervised learning. TPO is reinforcement learning.** They solve different problems. You don't compare them.\n\nTPO competes with **PPO**, not CE. The advantage: TPO gives you the same RLHF capability with much less complexity — no value function, no clipping, no importance sampling ratios.\n\n```\nPPO:  loss = -min(ratio * A, clip(ratio, 1-eps, 1+eps) * A)   # needs V(s), clipping\nTPO:  loss = -target_prob * log P_policy(token)                # needs nothing extra\npip install -e .\npython\nfrom tpo_torch import TPOTrainer\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nmodel = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen2.5-0.5B-Instruct\")\nref_model = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen2.5-0.5B-Instruct\")\ntokenizer = AutoTokenizer.from_pretrained(\"Qwen/Qwen2.5-0.5B-Instruct\")\ntokenizer.pad_token = tokenizer.eos_token\n\n# Dataset needs: prompt, labels, and 'advantages' (higher = better response)\ntrainer = TPOTrainer(\n    model=model,\n    ref_model=ref_model,\n    beta=0.1,\n    train_dataset=dataset,\n    processing_class=tokenizer,\n)\ntrainer.train()\n┌──────────────────────┐\n                    │   Reference Model     │\n                    │   (frozen)            │\n                    └──────────┬───────────┘\n                               │ ref_logits\n                               ▼\nPrompt + Labels ──▶ ┌──────────────────────┐\n                    │    TPO Loss Function  │\n                    │                       │\n                    │  1. log-odds(P_ref)   │\n                    │  2. + advantage/beta  │\n                    │  3. sigmoid -> target  │\n                    │  4. CE loss vs target  │\n                    └──────────┬───────────┘\n                               │ loss\n                               ▼\n                    ┌──────────────────────┐\n                    │   Policy Model        │\n                    │   (trained)           │\n                    └──────────────────────┘\n```\n\n| Function | Description |\n|---|---|\n`tpo_loss_from_logits(logits, ref_logits, labels, advantages, beta)` |\nLoss from raw model logits |\n`tpo_loss(policy_logprobs, ref_logprobs, advantages, beta)` |\nLoss from pre-computed log-probs |\n`TPOTrainer(model, ref_model, beta, ...)` |\nHuggingFace Trainer with TPO |\n`TPODataCollator(tokenizer)` |\nPreserves advantages in batches |\n\nAll benchmarks run on **NVIDIA RTX 3050 Laptop (4GB VRAM)**.\n\nBoth are RLHF methods evaluated on the **same metric**: held-out perplexity.\n\n| Method | Final Perplexity | Implementation Complexity |\n|---|---|---|\n| TPO (beta=0.1) | 74.51 |\nLoss function only |\n| PPO-clip (simplified) | 825.66 | Loss + ratio + clipping |\n\n**Note:** Our PPO-clip is simplified (no value function, no GAE). Full PPO with a critic would perform better but requires significantly more code and compute. TPO achieves competitive results with none of that infrastructure.\n\n**Zero NaNs** at advantage values from 0.01 to 1000- Max gradient norm:\n**0.09**— stable across all regimes\n\n| Seq Len | Latency | Throughput |\n|---|---|---|\n| 32 | 4.17ms | 61,345 tok/s |\n| 64 | 8.26ms | 62,021 tok/s |\n| 128 | 16.50ms | 62,046 tok/s |\n| 256 | 32.81ms | 62,418 tok/s |\n| 512 | 65.42ms | 62,614 tok/s |\n| 1024 | 929.89ms | 8,810 tok/s |\n\n```\npython benchmarks/run_benchmarks.py\ntpo train --max-steps 10\ntpo bench\ntpo info\ngit clone https://github.com/Griffith-7/tpo-torch.git\ncd tpo-torch\npip install -e \".[dev]\"\n```\n\nRun tests:\n\n```\npytest tests/ -v\nruff check tpo_torch/\n```\n\n- Python >= 3.9\n- PyTorch >= 2.0\n- transformers >= 4.40\n\n```\n@misc{kaddour2026targetpolicyoptimization,\n    title={Target Policy Optimization},\n    author={Jean Kaddour},\n    year={2026},\n    eprint={2604.06159},\n    archivePrefix={arXiv},\n    primaryClass={cs.LG}\n}\n```\n\n", "url": "https://wpnews.pro/news/tpo-torch-target-policy-optimization-for-stable-rlhf-alignment-in-pytorch", "canonical_source": "https://github.com/Griffith-7/Tpo-torch", "published_at": "2026-07-21 11:03:54+00:00", "updated_at": "2026-07-21 11:22:52.072074+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "ai-research", "developer-tools"], "entities": ["Tpo-Torch", "Target Policy Optimization", "PPO", "RLHF", "PyTorch", "HuggingFace", "NVIDIA RTX 3050", "Kaddour"], "alternates": {"html": "https://wpnews.pro/news/tpo-torch-target-policy-optimization-for-stable-rlhf-alignment-in-pytorch", "markdown": "https://wpnews.pro/news/tpo-torch-target-policy-optimization-for-stable-rlhf-alignment-in-pytorch.md", "text": "https://wpnews.pro/news/tpo-torch-target-policy-optimization-for-stable-rlhf-alignment-in-pytorch.txt", "jsonld": "https://wpnews.pro/news/tpo-torch-target-policy-optimization-for-stable-rlhf-alignment-in-pytorch.jsonld"}}