Your RL Training Run Already Solved Problems Your Best Model Can't — You Just Threw Away the Evidence

Discarded RL checkpoints contain capabilities the final model forgot. Agentic-OPD recovers them via on-policy distillation, boosting a 4B model past GPT-5.6 on deep search tasks.

Imagine you're training for a decathlon. Week three, you peak at high jump. Week six, you peak at javelin. But your coach only keeps the scorecard from the final day — and by then your high jump has regressed. You'd never throw away those mid-training peaks in athletics. Yet that's exactly what standard RL training does with LLM agents: keep the best-validation checkpoint, discard the rest, and silently lose capabilities the model had already acquired. Agentic-OPD's committed claim is that the union of problems solved across all saved checkpoints from a single GRPO run far exceeds what any single checkpoint solves — and that this gap can be closed after training is done, at no additional RL cost, by distilling from the discarded checkpoints back into the final model. The paper frames this as 'post-RL capability recovery,' a stage that sits orthogonal to RL algorithm improvements themselves. This is a genuinely new framing, not just another distillation paper. Two mechanisms do the heavy lifting. HCN (Highest Coverage Next) selects which checkpoint to distill from by weighting each teacher's coverage of problems the student currently fails — so the teacher rotation adapts as the student improves. LAD (Locality-Aware Distillation) restricts supervision to failed student rollouts on problems where the chosen teacher demonstrably outperforms the student. This is the key architectural insight: indiscriminate distillation overwrites things the student already knows, so you have to confine the transfer surgically. The combination means the student accumulates capabilities from multiple teachers without catastrophic forgetting of its own. The ladder here is strong. Across five agentic task domains and 24 benchmarks/subsets, Agentic-OPD improves its initialization checkpoint by up to 7.4 points. The headline result: a 4B-parameter student surpasses GPT-5.6 and DeepSeek-V4.1 on deep search tasks. Those are current frontier models, not stale baselines. The authors are comparing against the strongest things available, and the 4B model is dramatically smaller than its competitors — this isn't a tie at equivalent compute. Integrity is mixed. The breadth of evaluation (24 benchmarks, five task domains) is genuinely impressive and reduces cherry-picking risk. But all validation is internal: the authors ran the RL training, saved the checkpoints, and evaluated the distilled model. No independent replication exists yet. No pre-registration. The paper doesn't report code availability in the abstract, though the method is described in enough detail that reproduction should be feasible. The comparison to GPT-5.6 and DeepSeek-V4.1 is via API, which introduces version-drift risk — those models change under you. The milestone to watch is whether post-RL capability recovery becomes a standard pipeline stage. Right now this is demonstrated on a single RL algorithm (GRPO) with one model family. The obvious next experiment is running the same protocol on PPO, REINFORCE, or DPO-trained agents and checking whether the 'forgetting' phenomenon and recovery dynamics generalize. The authors almost certainly know this — and it's the natural follow-up paper. Separately, scaling to 7B–70B students would test whether the capability gap between checkpoints persists at larger scale or shrinks (larger models might forget less). If the phenomenon generalizes across RL algorithms and scales, this becomes infrastructure. If it's GRPO-specific, it's a useful trick. The deeper implication is economic: every RL run already produces the teacher pool for free. Labs currently delete intermediate checkpoints to save storage. If Agentic-OPD's finding holds broadly, that deletion is destroying value — and the cost of recovery is just inference-time distillation over the student's failure cases, not another training run. That's the kind of insight that changes default practice if it replicates.