A Bayesian Bookkeeper for Multi-Turn RL: AgentOPSD Assigns Credit Turn by Turn Without a Critic

Recursive Bayesian belief revision replaces flat trajectory-level reward spreading in RL for multi-turn agents. 89.1% on ALFWorld at 7B scale, no critic network required.

Imagine you're coaching a basketball team after a game. The final score is 98-102 — you lost. Standard coaching spreads blame evenly across all four quarters, but you know the game turned on three specific possessions in the third quarter where your point guard made bad reads. The problem is isolating those possessions from the noise of 200 total plays. AgentOPSD solves this problem for multi-turn RL agents: instead of smearing a single outcome reward across an entire interaction trajectory, it uses recursive Bayesian updates to figure out which turns actually mattered. The committed claim: a critic-free credit assignment scheme that converts token-level self-distillation gaps into turn-level evidence, then uses recursive Bayesian belief revision in log-odds space to reallocate trajectory-level advantage across individual turns. The method is bounded and sign-preserving — it can sharpen or dampen credit but never flip a success into a penalty or vice versa. This is not the first paper to use privileged self-distillation for RL fine-tuning, but the specific mechanism of converting local teacher-student discrepancies into a recursive Bayesian credit signal is novel. The architecture is straightforward: inside each turn, the method sums the token-level log-probability gaps between a skill-conditioned self-teacher and the student policy. That scalar becomes evidence fed into a log-odds Bayesian update that carries forward across turns. The result is a per-turn multiplier applied to the outcome-derived advantage from GRPO. No separate critic network, no value head, no extra forward passes through a value model. The entire credit assignment mechanism lives in the log-probability space the policy already occupies. Backbone models are Qwen2.5-3B and Qwen2.5-7B. On the ALFWorld benchmark — a text-based household task environment requiring multi-step planning — AgentOPSD hits 89.1% success rate at the 7B scale, improving over both vanilla GRPO and self-distillation baselines. The paper reports consistent improvements at both 3B and 7B scales. The ablations are the most informative part: they show the gain comes specifically from (a) aggregating evidence at turn boundaries rather than token boundaries, and (b) making the belief revision history-dependent. The local teacher-student gap alone is not sufficient — it needs the recursive structure. The ladder position is honest but narrow. ALFWorld is a standard benchmark, and GRPO is the right baseline to beat. But ALFWorld is a single environment with relatively structured tasks. The paper does not compare against other recent turn-level credit assignment approaches outside its own ablation variants, nor does it test on other multi-turn agent benchmarks like WebShop or ScienceWorld. The 89.1% number is strong but the generality claim rests on one benchmark. The integrity picture is mixed. The benchmark is community-standard, the baselines are current (GRPO), and the ablations are well-designed to isolate the mechanism. But it's a single-author preprint with no code release mentioned, no pre-registration, and no independent replication. The ablation design is good — separating turn-boundary aggregation from history-dependence from the raw gap signal — but all validation is self-reported on one benchmark. The successor question is obvious: does this work on benchmarks beyond ALFWorld? WebShop, ScienceWorld, AgentBench, or real API-calling tasks would be the natural next tests. My read is (a) compute/time constraints from a single author — running multi-turn RL at 7B scale across multiple benchmarks is expensive. The method's mechanism is general enough that failure on other benchmarks would be informative, not embarrassing, so there's no strong reason to suspect suppressed results.