Imagine you're teaching someone to drive. You sit in the passenger seat, and every time they touch the wheel, you say "good turn" or "bad turn." The problem: you're judging each micro-movement in isolation. The student learns that a cautious, wide turn scores fine — even though stringing together too many cautious wide turns means they'll run out of road before reaching the destination. If you actually grabbed the wheel at the moment they went wrong, you'd finish the route easily. You CAN drive. You just can't grade driving. That's the core finding of this paper. In agentic on-policy distillation (OPD), a large teacher model scores every token a smaller student model generates while the student learns from its own trajectories. The standard explanation for why this breaks down is compounding error: the student drifts into states the teacher hasn't seen, and the teacher's scores become unreliable. The authors show this explanation is incomplete. From the very states where the student diverges most, the teacher can still TAKE OVER and finish the task successfully. The teacher's capability isn't the bottleneck — its scoring is. It rates tokens without seeing where the episode ends, letting through safe-but-delaying moves that an outcome-based reward signal would punish. The authors call this the "rollout-scorer gap." The committed claim: next-token scoring is myopic in a way that is structurally distinct from compounding error, and you can fix it without any task reward signal. The proposed method, Rollout-to-Scorer Distillation (R2S), works in three steps. First, mine high-disagreement states from the untrained student — places where student and teacher trajectories diverge most. Second, run teacher takeovers from those states and filter for success using reward-free checks (consensus among multiple rollouts, or an LLM judge). Third, fine-tune the teacher's scorer to assign takeover-informed scores to tokens from the divergence turn onward, WITHOUT putting the takeover itself in context. The repaired teacher is then frozen and used as the sole scorer for standard OPD. The ladder position is strong. R2S was tested across three environments, two teacher models, and four student models — all twelve cells of the grid. It beats three state-of-the-art agentic OPD baselines (which include episode-length curricula, teacher-guided turns, prefix replay, and teacher intervention) in every single cell, by an average of 12.8 success-rate points over the best competitor in each cell. That's not a marginal improvement; in agentic benchmarks where success rates often plateau in the 40-70% range, 12.8 points is a substantial gap. The integrity picture is mixed-to-good. The 12-cell grid (3 environments × 2 teachers × 4 students) is a genuine combinatorial stress test — not cherry-picked on a single lucky pairing. The reward-free validation (consensus or LLM judge) avoids the circularity of training on the same reward used for evaluation. However, the environments are agentic coding/tool-use benchmarks, not yet independently replicated, and the paper appears to lack pre-registration. The 13 tables suggest thorough reporting, but we're still reading the authors' own scorecards. The architectural insight is what gives this paper legs. R2S belongs to the knowledge distillation family, specifically the scorer-repair variant: rather than constraining the student's exploration (the dominant approach), it fixes what the teacher says about the student's exploration. The key structural choice is decoupling the teacher's ACTION capability from its SCORING capability and repairing only the latter. This is computationally cheap — a fine-tuning pass on the teacher using its own rollout data — and it's agnostic to the student architecture and the specific OPD pipeline downstream. The successor question: the obvious next experiment is scaling to much larger action spaces and longer horizons where the rollout-scorer gap should widen further, and testing whether the consensus-based reward-free filter holds up when the teacher's own rollouts become noisy. The authors likely ran out of compute to test on the most expensive frontier settings, and the next paper will almost certainly move to those.
