LLM Agents Can Write Semi-Supervised Learning Code That Beats Human Baselines — But Can't Tell Which Code Is Good

MLE agents synthesize SSL programs that outperform human-engineered baselines in 8/12 settings by up to +21.8%, but program selection remains a 47% performance cliff.

Imagine you're a chef who can improvise dozens of dishes from whatever's in the fridge — some brilliant, some terrible — but you've lost your sense of taste. You need someone else to tell you which dish to serve. That's the core tension in AgentSSL: LLM-based machine learning engineering agents can generate code that creatively combines semi-supervised learning techniques, but they're dangerously bad at picking which of their own creations actually works. The committed claim: framing semi-supervised learning as a program search problem, where an LLM agent writes, tests, and iterates code that leverages unlabeled data. Under matched compute budgets and weak feedback signals (limited labeled validation data), these agents synthesize programs that beat the strongest human-engineered SSL baselines in 8 of 12 experimental settings, with gains up to +21.8%. This is not a toy demo — it's a NeurIPS 2026 paper with 12 benchmark settings and a real-world ecology detection task. The architecture is straightforward: treat the MLE agent as a code-generating search engine over the space of SSL programs. The agent proposes code combining known SSL building blocks — pseudo-labeling, consistency regularization, entropy minimization — then evaluates using whatever limited labeled data is available. The key insight is that the search space of possible SSL programs is too large for grid search but well-suited to the inductive biases of code-writing LLMs. The agent doesn't invent new SSL theory; it recombines and adapts existing techniques in ways human engineers hadn't tried. The brutal honesty lives in the program selection bottleneck. Performance drops by up to 47% when the agent must choose which of its generated programs to deploy, relying on its own feedback metric rather than oracle selection. This is not a minor caveat — it's a structural wall. To study it systematically, the authors introduce ProgramSelectBench, a benchmark of 16,977 agent-generated and human-engineered SSL programs. They evaluate both unsupervised and active selection methods, and neither solves the problem cleanly. The ladder positioning is solid but nuanced. Against human-engineered SSL baselines (FixMatch, FlexMatch, FreeMatch — the standard repertoire), the agent-generated programs win in 8/12 settings. But the comparison is against established SSL methods, not against the latest AutoML or neural architecture search approaches. The real competitor isn't a human writing FixMatch — it's a human using NAS or hyperparameter optimization over known SSL methods. That comparison is absent. The real-world ecology detection experiments add credibility but also expose the transfer gap. Agent-discovered strategies don't always port cleanly beyond controlled benchmarks, and the authors are honest about this. The 16,977-program benchmark is a genuine contribution to the field — it gives future researchers a shared substrate for studying program selection, which is where the actual unsolved problem lives. The strategic implication for practitioners: if you have abundant unlabeled data and scarce labels (which describes most real-world ML deployments), LLM agents can generate surprisingly good SSL pipelines. But you need a reliable selection mechanism — and right now, nobody has one. The generation side of agentic ML is ahead of the evaluation side, and that asymmetry is where the next breakthrough needs to happen.