Imagine you're a head chef running 20 restaurant kitchens from a single warehouse of 10,000 ingredients. Each kitchen has a different menu and a different grocery budget. Your job isn't to find the best ingredients in the abstract — it's to find the best ingredients for each kitchen that fit within that kitchen's weekly spend. Now imagine you hire a sous chef who proposes new ingredient combinations, tastes them systematically, and trims the order sheet so nobody goes over budget. That's this paper: an LLM agent acting as a fleet-wide feature-selection sous chef for production ranking models. The committed claim: an end-to-end agentic system where LLMs propose cross-features from a combinatorial space, deterministic tools evaluate them, and an LLM Researcher agent tunes selections across 10–25 asynchronous trials per model — all under per-model QPS constraints and shared pipeline cost budgets. The result is a 0.31% average normalized entropy (NE) improvement and 0.20% primary online metric gain across production A/B tests, while cutting feature pipeline cost by 2.3% with no measurable regression. Those numbers sound tiny until you remember that at fleet scale on a large web platform, 0.20% on a primary online metric (likely revenue or engagement) translates to enormous absolute value. The architecture sits squarely in the LLM-as-agent-with-tools family: the LLM generates hypotheses (cross-feature candidates), deterministic code executes experiments (importance scoring, Pareto-front blending for multi-task models), and the LLM interprets results to propose next trials. This is not the LLM doing gradient computation — it's doing combinatorial search and prioritization over a space too large for exhaustive enumeration but too structured for pure random search. The key structural choice is separating the LLM's role (hypothesis generation, shortlisting, trial-tuning) from the deterministic tools (feature importance computation, Pareto optimization, budget enforcement). That separation is load-bearing: it means the LLM's hallucination tendencies are sandboxed to the proposal stage, not the evaluation stage. The ladder question is tricky because the natural baselines for feature selection — greedy forward/backward selection, LASSO, random search — are not named or compared against directly in the abstract. The paper reports improvements against existing production models, which is a strong baseline in one sense (these are already optimized systems) but opaque in another (we don't know what feature selection method they replaced). The 0.31% NE improvement and 0.20% online metric lift are modest individually but meaningful at scale. The 2.3% cost savings is arguably the more interesting number — it shows the system can find equivalent-quality feature sets that are cheaper to compute. Integrity is mixed-to-strong. The ultimate validation is online A/B tests on production traffic — the gold standard for ranking systems. That's not simulation; that's real users, real revenue, real measurement. The 'no regression beyond measurement noise' claim for the trimming step is honest framing. However, the paper is from a single team at a single company, with no independent replication possible (proprietary data, proprietary models, proprietary infrastructure). The benchmarks were presumably chosen by the same team that built the system. This is the endemic integrity challenge for industrial ML papers: the validation is real but unreproducible. The milestone to watch is whether this agentic pattern — LLM proposes, tools execute, LLM iterates — generalizes beyond feature selection to other fleet-level optimization problems: hyperparameter tuning, architecture search, serving-cost optimization. The 10–25 trial budget per model is the current operating point; the question is whether that trial efficiency scales to harder optimization surfaces. If this pattern holds across three or more fleet optimization domains with similar trial budgets, it becomes a general-purpose industrial ML ops primitive. The obvious experiment not run: ablating the LLM entirely and replacing it with a non-LLM search strategy (Bayesian optimization, evolutionary search) over the same combinatorial feature space with the same deterministic tools. This would answer the central question: does the LLM actually contribute useful inductive bias for feature hypothesis generation, or is any reasonable search procedure equivalent given enough trials? My read: they almost certainly ran some version of this comparison internally, and the LLM probably wins on trial efficiency (fewer experiments to find good solutions) rather than on final quality. If the LLM didn't win, they wouldn't have built the system. But the margin of that win would be very informative, and its absence from the abstract suggests it may not be dramatic.
The analogy
A head chef managing 20 kitchens from a warehouse of 10,000 ingredients: each kitchen has a different menu and budget, and the sous chef proposes ingredient combinations, tests them systematically, and trims the order sheet so nobody goes over budget. The LLM is the sous chef — it generates candidate combinations and iterates on results, but the actual tasting (evaluation) is done by deterministic tools.
Sous chef
Proposes ingredient combos for each kitchen, trims the order list to stay within each kitchen's weekly grocery budget
LLM agent
Proposes cross-feature combos for each ranking model, trims feature sets to stay within each model's pipeline cost and QPS budget
Honest assessment
A well-engineered industrial system paper that solves a real problem (fleet-scale feature selection under cost constraints) with a practical architecture (LLM proposes, tools evaluate). The production A/B validation is genuine. But the missing ablation against non-LLM search methods is the central weakness: we know the system works, we don't know how much of that is the LLM versus the system design. Engineers building ML platform tools should read this for the architecture pattern; researchers looking for evidence that LLMs are better than conventional optimization should wait for the ablation.
What Is CLAIMS?
60
/ 100
BedrockNews applies a structured analytical framework called CLAIMS to academic papers and technical research releases.
How CLAIMS worksClaim
3/5
Ladder
3/5
Architecture
4/5
Integrity
3/5
Milestone
3/5
Successor
2/5
What is this paper actually committing to?
Real but incremental: LLM-as-agent meets fleet feature selection is a useful combination, not a new paradigm. The claim is well-scoped — cost-aware optimization, not general intelligence — which is honest.
The field fight
Do LLM agents add real inductive bias to engineering optimization, or are they just a convenient interface over conventional search?
Side A
LLMs bring useful priors from pretraining that make combinatorial search more sample-efficient
Held by: FunSearch (DeepMind), this paper, MLAgentBench
Side B
Bayesian optimization and evolutionary methods are more principled and equally efficient; LLMs add complexity without clear marginal value
Held by: AutoML community, BOHB, Optuna
Milestone tracker
Unlocks: Reusable agentic ML ops primitive: LLM proposes, tools execute, LLM iterates — across any fleet optimization problem
Current
0.20%
online metric lift
this paper
Next
3
fleet optimization domains
pattern generalization
~1-2 years if applied to hyperparameter tuning and architecture search
Verified Facts (7)
LLM agents propose hypotheses, and deterministic tools execute experiments and verify them
LLM generates and shortlists cross features from a combinatorial candidate space
Feature blending mixes these rankings into a single selection, using Pareto fronts across tasks for multi-task models
LLM Researcher agent then tunes the feature selection in 10–25 asynchronous trials under resource caps
System improves normalized entropy by 0.31% on average within each model's budget
Trimming step cuts pipeline cost by 2.3% with no regression beyond measurement noise
In online A/B tests they improved a primary online metric by 0.20%
Practical Implications (4)
The agent-with-tools pattern — LLM generates hypotheses, deterministic code evaluates them — is immediately applicable to any fleet-level ML optimization problem where the search space is too large for exhaustive enumeration
Separating LLM proposal from deterministic evaluation sandboxes hallucination risk; this design pattern is reusable even if the LLM component is later replaced
The 2.3% pipeline cost savings with no quality regression suggests feature sets at large platforms are routinely over-provisioned, and systematic trimming is low-hanging fruit
10–25 trials per model is a practical trial budget for teams with limited experiment infrastructure; this isn't a 1,000-trial hyperparameter sweep
More latest news
Read all