A Bad Chess Player Can Still Be a Great Coach: Separating Acting From Teaching in LLM Agents

Decision models that fail as actors can still train smaller policies effectively. Hindsight judgment reveals what real-time acting can't — and that asymmetry is the whole point.

Imagine you're watching a replay of a basketball game with a friend who's terrible at basketball. During the live game, they'd make bad reads, miss open teammates, turn the ball over. But watching the replay? They can spot every mistake, explain why the point guard should have swung the ball left, and identify the exact moment the defense collapsed. They're a lousy player but a perceptive analyst — and if you're coaching a young player, the analyst is more useful than the athlete. This paper asks whether the same asymmetry holds for LLM-based decision agents, and the answer is a clean yes. The committed claim: a decision model that performs poorly as an actor (choosing actions in real time) can still serve as an effective teacher for training smaller policies via reinforcement learning, because hindsight judgment — seeing the full trajectory after the episode ends — reveals causal credit that real-time acting must guess at. The authors formalize three roles for decision models: actor (choose the next action), judge (evaluate actions after the fact), and teacher (generate reward signals for RL training). The key insight is that these roles impose fundamentally different information conditions, and conflating them — as most prior work does — hides a useful decoupling. The experimental setup compares three specialized decision models and three general-purpose LLMs (GPT-4o-mini, Qwen2.5-72B-Instruct, Llama-3.1-70B-Instruct) across ALFWorld (a text-based household task environment) and WebShop (an online shopping simulation). As actors, all six models exhibit the same failure pattern: they get seduced by easier subgoals and abandon harder prerequisites. As judges seeing complete trajectories, the same models correctly credit the difficult prerequisite steps — but only when asked "did this action help?" rather than "will the task succeed?" That framing difference between action-level and task-level judgment is a sharp diagnostic finding on its own. The teacher pipeline uses JEV (Judge-Enhanced Verifier), which generates per-step reward signals from hindsight judgment and feeds them into GRPO (Group Relative Policy Optimization) to train a Qwen2.5-1.5B policy. Results: ALFWorld success goes from 72.8% to 82.9%; WebShop from 56.8% to 71.4%. These are meaningful gains on established benchmarks, driven entirely by better reward shaping from a model that couldn't act its way out of the same tasks. The architectural bet is straightforward: LLM-as-reward-model for RL, in the RLHF lineage but applied to agentic task completion rather than text generation. The method leans on the information asymmetry between acting (partial observability, must commit before seeing consequences) and judging (full trajectory, hindsight available). This is a well-understood asymmetry in RL theory — the contribution is demonstrating it holds cleanly in the LLM-agent setting and showing it's exploitable for practical training. Integrity is reasonable but not exceptional. ALFWorld and WebShop are established community benchmarks, code is released, and the baselines include strong models. However, there's no independent replication, the evaluation environments are relatively constrained simulations, and the GRPO baseline was presumably tuned by the same team. The "action-level vs. task-level judgment" finding is the most interesting result and the one most in need of independent confirmation across more diverse environments. The obvious next experiment is scaling this to open-ended, multi-step environments where trajectories are much longer and the credit assignment problem is genuinely hard — think Minecraft or code generation with real compilation feedback. The authors likely didn't run this because ALFWorld and WebShop are tractable enough to iterate quickly and the JEV reward pipeline hasn't been stress-tested on trajectories longer than ~20 steps. Whether hindsight judgment degrades gracefully or collapses in longer horizons is the load-bearing question for this line of work.