
SKILLPLUMBER SOLVES
+64.7%
SOLVES
+70.5%
RECOVERED FAILURES
+20.8%
Imagine you're coaching a new cook through a complex recipe over the phone. The soufflé collapses. Standard feedback says "the soufflé failed" — outcome-level. Slightly better feedback says "something went wrong around the egg-folding step" — trajectory-level. But neither tells you whether the recipe itself was wrong (bad instructions) or whether the cook skipped a step (bad execution). SkillPlumber's core insight is that you need a third kind of feedback: did the cook actually *do* each step, and if they did, did it work? That's instruction-level coverage. The committed claim: by tracking whether each instruction in a skill was followed, not-followed, or not-covered during execution, you can surgically revise only the broken instructions rather than rewriting skills wholesale. This is not the first self-evolving agent framework — it sits in a lineage including AgentTrek, Agent-S, and skill-library approaches — but it is the first to formalize instruction-level coverage as the feedback signal driving evolution. The distinction matters because prior methods that judge runs by outcome or trajectory cannot separate "bad recipe" from "didn't follow the recipe," which means they revise blindly. The architecture is a three-agent pipeline. An executor runs tasks using natural-language skill instructions. A root-cause analysis (RCA) agent examines coverage labels — follow, not-follow, not-covered — for each instruction and generates targeted revision questions. A maintainer distills patterns across tasks into a persistent strategy library that feeds future revisions. The system is executor-agnostic: they test with GPT-4o, Claude 3.5 Sonnet, and GPT-4o-mini, and the gains hold across all three. The coverage labeling itself uses an LLM to classify each instruction's status from the execution trace — no specialized tooling, just prompted classification. The numbers are substantial. Across three benchmarks (SkillsBench, WorkArena-L, OSWorld), SkillPlumber achieves 64.7% to 70.5% task success, which is 25.6 to 30.4 percentage points above the initial skill baseline and 8.7 to 14.0 points above the strongest prior method. The ablation is the most telling number: removing coverage tracking from the pipeline reduces recovered failures by 20.8% on SkillsBench with GPT-5.5. That's a clean signal that coverage is load-bearing, not decorative. Integrity is decent but not bulletproof. The benchmarks are established community standards — SkillsBench, WorkArena-L, and OSWorld are recognized in the agentic AI community. Three different executor LLMs provide some robustness against model-specific overfitting. The ablation studies are well-designed, isolating coverage, the RCA agent, and the strategy library independently. However, all evaluation is same-team, there's no pre-registration, and the coverage labeling itself is done by an LLM — which means the accuracy of the diagnostic depends on the classifier's reliability, a potential circularity the paper acknowledges only implicitly. The milestone question is about generalization breadth. Today's benchmarks test web navigation, office productivity, and OS-level tasks. The real unlock is whether instruction-level coverage transfers to embodied agents, multi-agent coordination, or domains where the action space is continuous rather than discrete. The 20.8% ablation drop gives a concrete number to track: as coverage classifiers improve, that gap should widen if the mechanism is real. If it doesn't scale to harder domains, the contribution is a good engineering trick rather than a foundational principle. The obvious experiment not run: testing on truly open-ended tasks where the skill library must grow from scratch rather than being initialized with human-written seeds. The paper starts with existing skill sets and evolves them. Building skills from zero — where coverage feedback bootstraps the entire skill — would be the definitive test. My read: this is likely being saved for the next paper, since the current framing cleanly isolates coverage as a feedback mechanism, and cold-start skill synthesis is a separable (and publishable) follow-on.
The analogy
Like debugging a soufflé failure over the phone: outcome feedback says 'it collapsed,' trajectory feedback says 'something went wrong at the egg step,' but instruction-level coverage says 'step 4 was followed and worked, step 7 was followed and failed, step 9 was skipped.' Only the third tells you whether to fix the recipe or the cook.
Recipe debugging
Tracking which recipe steps the cook actually executed and which produced bad results vs. were skipped entirely
Instruction coverage
Labeling each skill instruction as follow/not-follow/not-covered to separate bad instructions from bad execution
Honest assessment
SkillPlumber introduces a genuinely useful conceptual distinction — separating 'bad instruction' from 'didn't follow instruction' — and backs it with solid empirical gains across multiple benchmarks and executors. The 20.8% ablation drop is the load-bearing number; it's what separates this from pure engineering. The main vulnerability is the LLM-as-coverage-classifier circularity and the absence of cold-start evaluation. This is a strong systems paper with a real idea at its core, not yet a foundational result but positioned to become one if the mechanism transfers to harder domains.
What Is CLAIMS?
67
/ 100
BedrockNews applies a structured analytical framework called CLAIMS to academic papers and technical research releases.
How CLAIMS worksClaim
4/5
Ladder
4/5
Architecture
3/5
Integrity
3/5
Milestone
2/5
Successor
4/5
What is this paper actually committing to?
Genuine novelty in the feedback signal: instruction-level coverage for skill evolution is a real conceptual contribution, not just engineering. First to formalize the follow/not-follow/not-covered distinction for self-revision.
The field fight
What feedback granularity should drive agent skill self-evolution — outcome, trajectory, or instruction-level?
Side A
Outcome or trajectory feedback is sufficient; finer granularity adds complexity without proportional gains
Held by: AgentTrek, Agent-S, Voyager, skill-library approaches
Side B
Instruction-level coverage is necessary to distinguish bad instructions from bad execution, enabling surgical revision
Held by: SkillPlumber (this paper)
Milestone tracker
Unlocks: General-purpose self-improving agents that debug their own skill libraries across any domain
Current
70.5
% success
SkillPlumber on SkillsBench
Next
80+
% success
embodied / continuous tasks
~1-2 years if coverage classifiers improve and embodied benchmarks mature
Verified Facts (7)
SkillPlumber solves 64.7% to 70.5% of the tasks across three benchmarks and three executors
25.6 to 30.4 percentage points higher than the initial skill
8.7 to 14.0 percentage points higher than the strongest baseline
Removing coverage reduces recovered failures by 20.8% on SkillsBench with GPT-5.5
Instruction-level coverage labels each instruction as follow, not-follow, or not covered
Tested with three executors: benchmarks include SkillsBench, WorkArena-L, OSWorld
15 pages, 5 figures
Practical Implications (4)
Teams building agent skill libraries should instrument instruction-level coverage tracking — it provides dramatically better revision signal than pass/fail outcomes alone
The executor-agnostic design means this can be dropped into existing agent pipelines without model-specific modifications
The strategy library pattern — distilling cross-task revision patterns — is independently valuable for any iterative prompt engineering workflow
Coverage classification via LLM prompting is cheap to implement but its reliability should be validated per-domain before trusting it for high-stakes revisions
More latest news
Read all