Interactive explainer

DPO, interactively.

Direct Preference Optimization is just one equation — but the equation does something deeply counter-intuitive to your model. Instead of reading it, drag the sliders. Every widget below is the real math, computed live in your browser.

Sugeerth Murugesan· Built on a from-scratch DPO verified against TRL to 3e-5
What you'll be able to feel by the end:
  1. Why DPO needs no reward model
  2. The loss — a live playground
  3. Where a "log-prob" actually comes from
  4. The asymmetry: both log-probs go down
  5. β is a knob, not a setting
  6. From scratch — and proving it matches TRL

1Why no reward model?

The one idea that makes everything else possible.

Classic RLHF is a sandwich: train a reward model on human preferences, then use PPO to push your policy toward high reward. Two models, a brittle RL loop, lots of knobs.

DPO's trick is purely algebraic. Under the Bradley–Terry preference model and a KL-constrained objective, the optimal policy is a closed-form function of the reward. You can invert that relationship — write the reward in terms of the policy — and the reward model cancels out. What's left is a plain classification loss on preference pairs:

Read it left to right: take the chosen response \(y_w\) and the rejected one \(y_l\); for each, measure how much more probability your policy \(\pi_\theta\) gives it than the frozen reference \(\pi_{\text{ref}}\) does; subtract; squash through a sigmoid; maximize. That "how much more than reference" quantity is the implicit reward. Nobody trained it. It fell out of the algebra.

Show the 4-line derivation (optional)

The KL-constrained reward-maximization objective has a known optimum:

Solve for the reward \(r\):

The partition function \(Z(x)\) is the same for \(y_w\) and \(y_l\), so it cancels in any difference of rewards. Plug \(r\) into the Bradley–Terry likelihood \(\;p(y_w \succ y_l) = \sigma(r_w - r_l)\;\) and you get the loss above. No reward model survives the substitution.

2The loss, as a playground

Two implicit rewards in, one loss out. Drag them.

Forget tokens for a second. The loss only sees two numbers: the implicit reward of the chosen response \(r_w = \beta\,[\log\pi_\theta(y_w) - \log\pi_{\text{ref}}(y_w)]\), and of the rejected one \(r_l\). The loss wants \(r_w\) bigger than \(r_l\). That's the whole job.

Live DPO loss
Slide the two implicit rewards (in units of β·log-ratio). Watch the loss, the margin, and where you sit on the loss curve. The arrow shows which way the gradient pushes each one.
+0.40
−0.20
+0.60
margin r_w − r_l
0.44
loss −log σ(margin)
✓
ranks correctly?
0.35
gradient weight σ(−margin)
Notice When the pair is already ranked correctly, the gradient weight shrinks toward 0 — DPO stops caring about easy pairs and spends its budget on the ones it still gets wrong.

3Where a "log-prob" comes from

Section 2 hid all the work inside \(\log\pi(y\mid x)\). Here it is.

A language model doesn't emit a single probability for a response — it emits one per token. The sequence log-prob is the sum of per-token log-probs over the completion only (the prompt is given, not predicted). Two details trip everyone up, so let's make them visible:

batch_logps, token by token
Hover any green token to see its log-prob. The prompt (grey) is masked out — it contributes nothing. Toggle the two gotchas:
Chosen response
Rejected response
−0.0
Σ log-prob (chosen)
−0.0
Σ log-prob (rejected)
0
tokens summed

This is exactly what batch_logps does in the repo: shift the logits by one (position t predicts token t+1), gather the log-prob of the actual next token, zero out everything that isn't a completion token, and sum. Get the shift or the mask wrong and your "log-prob" silently includes the prompt or drops the first real token.

4The asymmetry nobody warns you about

The single most surprising thing about watching DPO train.

You'd expect DPO to make the chosen response more likely. Scrub through a training run and watch what actually happens to the raw log-probs:

A DPO training run (scrub it)
Both lines are absolute log-probs under the policy. The shaded gap is the margin the loss optimizes.
0
The asymmetry By the end, the policy assigns less probability to the chosen response than the reference model did — it's just dropping the rejected one even faster. Preference accuracy climbs while \(P(\text{chosen})\) falls. The loss only ever cared about the gap.

This is documented (Pal et al. 2024, "Smaug") and it's why a whole literature of DPO variants exists — IPO and others add terms specifically to stop the chosen log-prob from collapsing.

5β is a knob, not a setting

The one hyperparameter that decides whether anything learns.

β controls how far the policy may stray from the reference. Turn it down and the policy escapes its leash — KL explodes, generations degenerate. Turn it up and the leash is so short nothing moves. Find the regime:

The β trade-off
A stylized view of the same 1-epoch run at different β. Watch preference accuracy and KL divergence trade off.
0.100

6From scratch — and how I know it's right

A from-scratch loss that disagrees with the reference is just a bug with good marketing.

Everything above is the real arithmetic. In the repo, the loss is two small PyTorch functions — and they're tied out against Hugging Face TRL numerically on the same batch:

# dpo_loss.py — the entire loss
def dpo_loss(pol_chosen, pol_rejected, ref_chosen, ref_rejected, beta=0.1):
    pi_logratios  = pol_chosen - pol_rejected
    ref_logratios = ref_chosen - ref_rejected
    logits = pi_logratios - ref_logratios            # the margin
    loss = -F.logsigmoid(beta * logits).mean()    # the whole thing
    return loss, ...
Tie-out vs TRL 0.29
The actual numbers from verify_against_trl.py on Qwen2.5-0.5B, same batch fed to both.

Three checks back the claim: the loss matches TRL's to ~3×10⁻⁵, the tokenization is byte-identical, and a unit test ties batch_logps to TRL's own log-softmax primitive. python -m pytest test_dpo_loss.py → 8 passed.

References