Imagine you're training for a debate tournament, but your practice partner, your coach, and the tournament judge are all the same person. You'd get very good at winning arguments in front of that person — but how much of your improvement is genuine rhetorical skill, and how much is learning their specific preferences? That tension sits at the center of this paper. The paper independently re-implements DNO-Prct, a practical variant of Direct Nash Optimization, and applies it to two open-weight instruction-tuned models: Llama 3.1 8B Instruct and Mistral 7B Instruct v0.1. The core mechanism works like a self-play loop: generate candidate responses on-policy, have an LLM annotator rate them alongside a teacher's gold reference, select preference pairs with large margins, then run a DPO-style contrastive update. Three iterations of this loop push Llama 8B from 31.7% to 45.3% length-controlled win rate on AlpacaEval 2.0, and Mistral 7B from 6.8% to 18.4%. The headline result: the tuned 8B model scores above Llama 3.1 70B Instruct on every reported metric. The ablation work is where this paper earns its keep. Removing gold responses from the preference annotation drops LC win rate by 6-10 points. Removing data look-back (mixing older generations into training) costs a similar amount. But varying the number of on-policy samples per prompt from 1 to 7 moves the needle by only 2.1 points, and sweeping the margin threshold from 1 to 5 changes it by at most 2.4 points. Translation: the presence of gold references and historical data is load-bearing; hyperparameter tuning within the pipeline is not. That's a useful signal — it tells you where engineering effort matters. The open-weight annotator experiments are quietly important. When Llama rates Llama (instead of GPT-4o rating Llama), the pipeline retains 62% of the LC gain. When Mistral rates Mistral, it retains 82%. This means the self-play loop doesn't strictly require a proprietary oracle — a same-family larger model can serve as annotator with meaningful, if reduced, effectiveness. For labs that can't afford to run every alignment iteration through a paid API, that's a practical unlock. The elephant in the room is the evaluation contamination: GPT-4o serves as both the teacher (providing gold responses and annotations) and the judge (scoring outputs on AlpacaEval 2.0). The authors acknowledge this directly and discuss what it implies — the tuned model may be learning GPT-4o's stylistic preferences rather than achieving generally better outputs. The length-controlled metric partially controls for verbosity gaming, but it cannot control for broader stylistic convergence. This isn't a fatal flaw — AlpacaEval is the community benchmark and everyone uses it — but it means the 45.3% number should be read as 'alignment to GPT-4o preferences' rather than 'alignment to human preferences.' All results are single runs with no error bars. The paper is refreshingly honest about this: they report it plainly rather than burying it. But it means we genuinely don't know how much run-to-run variance exists in a pipeline that chains on-policy sampling, LLM annotation, pair selection, and contrastive training across three iterations. A single bad sample at iteration 1 could cascade. The paper's real contribution is not the headline numbers but the ablation map. It tells practitioners building alignment pipelines exactly which components to protect (gold references, data look-back) and which knobs not to over-tune (sample count, margin threshold). That's engineering knowledge that transfers beyond DNO-Prct itself, and it's the kind of empirical work the alignment community needs more of.
