
Read the paper
ComputerSD: Online Self-Distillation from Real-Time Feedback for Computer-Use Agents
on GitHub
Imagine you're learning to cook by watching a chef, but the chef only tells you whether the final dish tastes good — never whether you chopped the onions wrong or burned the garlic. You'd eventually learn, but it would be agonizingly slow and you'd repeat the same mistakes hundreds of times. That's the state of online reinforcement learning for computer-use agents (CUAs) today: the agent clicks through a desktop environment, and only at the very end does it learn whether the whole trajectory succeeded or failed. ComputerSD is a kitchen timer that goes off at every step. The committed claim: combining token-level on-policy self-distillation (OPSD) with trajectory-level GRPO, regulated by a real-time GUI analyzer that scores each individual action, produces a meaningfully better computer-use agent than outcome-only RL. On OSWorld-Verified, ComputerSD achieves gains of 1.9 percentage points on the general-purpose Qwen3-VL-8B-Thinking backbone and 4.1 percentage points on the specialized EvoCUA-8B backbone versus GRPO alone. The architecture is a two-signal hybrid. A fine-tuned GUI analyzer watches each screen transition after every agent action and produces two things: a natural-language guidance string (privileged context about what just happened on screen) and a step-level value score. The guidance feeds into OPSD — a method where a teacher model with access to privileged information rescores the student's token probabilities to provide dense learning signal. The value score acts as a gate: if the step was bad, the OPSD signal is suppressed so the agent doesn't learn to imitate its own mistakes with higher confidence. Meanwhile, the standard trajectory-level GRPO reward still operates on complete rollouts. Both signals are optimized jointly in a fully asynchronous training framework. The two core engineering insights here address real failure modes. First, static teacher guidance can drift out of alignment with what the student policy is actually doing — the student takes a weird action, and the teacher's pre-computed advice becomes irrelevant. By generating guidance from the actual executed GUI transition rather than a pre-planned trajectory, ComputerSD keeps guidance grounded in reality. Second, OPSD can create probability shifts that are locally correct at the token level but globally wrong at the step level — the teacher might upweight tokens for an action that looked linguistically reasonable but was functionally incorrect. The step-level value score from the GUI analyzer filters these out. The baseline comparison is honest but narrow. GRPO is the right comparison — it's the dominant online RL method for these agents — and the gains are consistent across two different backbones, which reduces the chance of architecture-specific overfitting. The out-of-distribution evaluation adds credibility. However, the absolute numbers remain modest: we're talking about agents that still fail on the majority of computer tasks, and the improvements, while statistically meaningful, are single-digit percentage point deltas. The paper doesn't compare against other dense-reward approaches outside the self-distillation family. The validation relies entirely on OSWorld-Verified, which is a community benchmark — good — but there's no pre-registration, no independent replication, and the GUI analyzer itself is trained by the same team, which introduces a circularity concern: how much of the gain comes from the analyzer's quality versus the training method? An ablation separating these would strengthen the claim considerably. What matters for the field is the general principle, not the specific numbers. The idea that you can extract dense, step-level supervision from environment feedback (screen transitions) and use it to regulate token-level distillation is transferable well beyond OSWorld. If this approach scales — particularly with stronger GUI analyzers or multimodal critics — it could significantly compress the sample complexity of online CUA training. The next milestone to watch: whether this method or its descendants can push OSWorld-Verified success rates past 50%, where computer-use agents start becoming practically useful rather than research demonstrations.
The analogy
Learning to cook when the chef only grades the final dish versus having a kitchen timer and taste-tester at every step — ComputerSD adds the step-by-step feedback loop that outcome-only RL lacks, so the agent stops repeating the same intermediate mistakes.
Cooking feedback
A kitchen timer and taste-tester after every step tells you which action went wrong, not just whether the final dish failed
GUI analyzer
A fine-tuned screen-transition critic scores each agent action and generates guidance, providing dense step-level signal instead of sparse endpoint rewards
Honest assessment
ComputerSD is a well-engineered integration paper that demonstrates a real principle — step-level environment feedback improves online CUA training — but the gains are modest (low single-digit pp) and the validation, while reasonable, has gaps (no dense-reward baseline comparisons, self-trained critic circularity). This matters more as a direction-setter than a breakthrough. Engineers building CUA training pipelines should read it; everyone else can wait for the scaling results.
What Is CLAIMS?
60
/ 100
BedrockNews applies a structured analytical framework called CLAIMS to academic papers and technical research releases.
How CLAIMS worksClaim
3/5
Ladder
3/5
Architecture
4/5
Integrity
3/5
Milestone
2/5
Successor
3/5
What is this paper actually committing to?
Solid integration, not a new paradigm. Combines OPSD + GRPO + GUI feedback in a way that consistently beats the outcome-only baseline, but the category of result (dense feedback helps RL) is well-established.
The field fight
Should CUA online training rely on sparse outcome rewards alone, or does dense step-level feedback fundamentally improve learning?
Side A
Outcome rewards are sufficient if you scale rollouts and use trajectory-level methods like GRPO
Held by: GRPO, AgentQ, standard RL-for-agents literature
Side B
Dense step-level feedback from environment transitions is a necessary ingredient — outcome-only training wastes too many samples on undiagnosed intermediate failures
Held by: DigiRL, process reward models, ComputerSD
Milestone tracker
Unlocks: Computer-use agents reliable enough for real-world task automation without constant human oversight
Current
~25-35
% OSWorld success
best CUAs (estimated)
Next
50
% OSWorld success
practical utility threshold
~2-4 years, contingent on base model scaling and environment diversity
Verified Facts (6)
ComputerSD outperforms outcome-only GRPO by 1.9 and 4.1 percentage points on the general-purpose Qwen3-VL-8B-Thinking and specialized EvoCUA-8B backbones, respectively
Evaluation on OSWorld-Verified benchmark
A fine-tuned GUI analyzer produces guidance and a step-level value score after each action
ComputerSD jointly optimizes token-level OPSD and trajectory-level GRPO in a fully asynchronous training framework
Evaluation in out-of-distribution settings further supports the generalizability of ComputerSD
Code available at https://github.com/ZJU-REAL/ComputerSD
Practical Implications (3)
Teams training CUAs via online RL should consider adding step-level environment feedback to their GRPO pipelines — the async framework described here is implementable and the code is public
The GUI analyzer pattern (fine-tuned VLM that scores screen transitions) is reusable across different CUA architectures and environments
The value-score gating of OPSD signals is a transferable trick for any setting where dense distillation might reinforce bad intermediate actions
More latest news
Read all