Imagine you're learning to cook a complex meal from a recipe. A harsh instructor only tells you at the very end whether the dish was good or bad — that's standard RL with terminal reward. Now imagine a kinder setup: after each successful meal, you also practice reciting the remaining steps from memory at every point in the process. You never recite during the real cook, but the mental rehearsal means you internalize the structure of good sequences faster. That's CoPE. The committed claim: a multi-step action-prediction auxiliary loss on successful trajectories, sharing the policy backbone, improves both final performance and sample efficiency for language agents — without additional models, separate pretraining, or inference cost. This is not a new RL algorithm. It's a new auxiliary training signal layered onto existing policy-gradient methods (specifically GRPO). The insight is that successful on-policy rollouts contain a supervision signal nobody was using: the future action sequence itself. Where does this land on the ladder? CoPE is benchmarked against GRPO (the current go-to for language agent RL), ECHO (which uses observation-prediction as auxiliary targets), and filtered behavior cloning. Across ALFWorld, ScienceWorld, WebShop, τ²-bench, and AppWorld, CoPE achieves the strongest aggregate score. The sample efficiency numbers are the headline: 1.9× over ECHO and 2.0× over GRPO on ScienceWorld. These are meaningful but not earth-shattering gaps — the baselines are themselves recent and reasonably strong. Architecturally, CoPE belongs to the family of RL-with-auxiliary-losses for autoregressive language models. The backbone is a single LLM policy (Qwen-2.5-7B in most experiments), trained with GRPO plus an additional cross-entropy loss that asks the model to generate the sequence of future actions given each prefix of a successful trajectory. This creates what the authors call a 'temporal ensemble' — the same action is supervised from multiple history lengths. The key structural choice is that forecasting is training-only; at inference the model acts one step at a time as usual, so there's zero latency overhead. Integrity is mixed. Five benchmarks is good breadth, and they include both simulated (ALFWorld, ScienceWorld) and more realistic environments (τ²-bench, AppWorld). The baselines are current — GRPO and ECHO are 2024-vintage methods. But all evaluation is same-team, the benchmarks were presumably chosen after initial experiments showed promise, and there's no pre-registration. The authors do release code, which matters. The filtering to only successful trajectories is a design choice that could quietly hide failure modes on harder environments where success is rare. The milestone question is about where language-agent RL needs to go next. Current benchmarks top out at moderate horizon lengths and relatively constrained action spaces. The real unlock is agents that can execute reliably over 50-100+ step trajectories in open-ended environments — think autonomous software engineering or complex web workflows. CoPE's 2× sample efficiency gain matters most if it compounds as horizons grow; if the gain is flat or shrinking with trajectory length, the contribution narrows to a training-time optimization. The obvious experiment not run: scaling to longer horizons and harder environments where success rates are very low (making the filtering-to-successful-trajectories assumption brittle). The authors likely ran out of compute budget for exhaustive horizon-length ablations, or the results at extreme lengths weren't clean enough to report. The reliance on successful trajectories is load-bearing — if your environment has 2% success rate, you barely have signal to train the forecasting head. This is the crack in the wall, and the next paper in this line will have to address it.
