RLHF aligns a model in two stages. Human comparisons train a reward model, then reinforcement learning optimises the policy against that reward while a KL penalty keeps it near the original. It works, and it is fragile: reward models get gamed, PPO needs careful tuning, and the whole pipeline is heavy to run.
DPO derives a shortcut. The authors showed that the optimal policy under that reward has a closed form, which means the preference data can train the policy directly through a single classification-style loss. Each training example is a prompt with a chosen and a rejected response, and the loss raises the likelihood of the chosen one relative to a frozen reference copy of the model.
No reward model. No rollouts. It is close enough to ordinary fine-tuning that a team with a preference dataset can run it on modest hardware.
The limits show up at the frontier. Because DPO only sees fixed pairs, it cannot explore new responses the way online reinforcement learning can, and quality is capped by the coverage of the collected data.