An LLM Trained on One Vision Task Can Tune Hyperparameters Across Modalities — If You Encode the Search Space Right

LLM-as-optimizer paper achieves cross-domain AutoML generalization by encoding all design choices as discrete ranked sequences. SOTA on GenEval (0.91) with a billion-param DiT at 896 GPU-days.

Imagine you're a chess engine that learned to play by studying only games of Go. The board is different, the pieces are different, the rules are different — but the underlying skill you mastered wasn't "Go" — it was evaluating which positions are more promising than others. That's the core mechanism here: the authors train an LLM to rank configuration sequences, not to understand any particular ML task, and the ranking skill transfers across domains the model has never seen. The committed claim is that a single LLM-based searcher, trained exclusively on optimization trajectories from one vision classification task, can generalize to jointly optimize model architecture, training hyperparameters, and data ratios across image classification, image segmentation, speaker verification, and text-to-image generation — without task-specific fine-tuning. This is the first framework that unifies all three AutoML search spaces under a single LLM-based optimizer and demonstrates cross-modality transfer. The key enabler is NCode, a numerical encoding scheme that translates heterogeneous design choices — learning rates, layer counts, data mixing ratios — into universal discrete sequences. This reframes AutoML as a ranking problem: given two configurations encoded as NCode sequences, which one will perform better? The LLM learns pairwise preferences from optimization trajectories and generalizes the ranking function. It's clever because it sidesteps the curse of task-specific search space design that plagues traditional AutoML frameworks like NAS, BOHB, or ASHA. The scalability demonstration is the headline act. The authors jointly optimize a billion-parameter Diffusion Transformer (DiT), requiring 896 GPU-days of compute, and achieve a GenEval score of 0.91 — which they claim is state-of-the-art. They also show strong results on CIFAR-100 image classification, ADE20K segmentation, and VoxCeleb speaker verification. The cross-modality generalization from vision-only training data to audio and generative tasks is the most surprising result, though the degree of improvement over domain-specific baselines varies. The integrity picture is mixed. The benchmarks are established community standards (CIFAR-100, ADE20K, VoxCeleb, GenEval), which is good. But the validation is entirely self-reported with no independent replication, and the 896 GPU-day experiment makes independent verification prohibitively expensive for most labs. The comparison baselines include Random Search, Bayesian Optimization, and evolutionary methods, but it's unclear whether the strongest recent AutoML systems (e.g., DEHB, or the latest transfer-learning NAS approaches) were included. The Chain-of-Thought interpretability via reinforcement learning is interesting but risks being post-hoc rationalization rather than genuine mechanistic insight. The interpretability angle deserves attention: RL-trained Chain-of-Thought reveals the searcher shifting from early diversity-driven exploration to late-stage anchor exploitation. This is a known pattern in optimization (explore-then-exploit), but making it legible inside an LLM's reasoning chain is novel. Whether this constitutes real interpretability or pattern-matching on known optimization folklore is an open question. The elephant in the room is compute cost. Training the LLM searcher itself is relatively cheap (optimization trajectories from a single task), but evaluating each candidate configuration during search still requires training the target model. At billion-parameter scale, this means each search iteration is expensive. The framework doesn't reduce the cost of individual evaluations — it reduces how many evaluations you need. Whether the search efficiency gains justify the overhead of the LLM searcher versus well-tuned Bayesian optimization at scale is not conclusively demonstrated.