An Automated Factory That Builds Custom Small Language Models for Your Phone — and Beats GPT-4o on Half Its Tasks

An agentic system auto-tunes, quantizes, and benchmarks small LMs per-task per-phone, outperforming frontier 5-shot models on 5 of 10 tasks while using 19× less GPU than brute-force search.

Imagine you run a bakery and every morning you bake one massive cake, then cut slices for every customer — some want gluten-free, some want mini portions, some need it by 6 AM. That works until you realize a small custom muffin per customer is faster, cheaper, and exactly what each person wanted. Mobile AI Factory does this for language models: instead of shipping one giant cloud model to every phone, it bakes a custom small model per task per device. The committed claim: an orchestrator LLM can automatically search across base models, quantization levels, hyperparameters, data strategies (real mining vs. synthetic generation), and inference backends to produce a fine-tuned small language model that fits a specific phone and task — without human ML expertise in the loop. On an unseen test set across ten tasks and four Android phones, the resulting on-device models outperform Claude 3.5 Sonnet and GPT-4o (both 5-shot) on five tasks by up to 0.34 accuracy, match them on three more, and lose on only two. The search process comes within 0.03 of exhaustively training all 27 model-variant combinations on an entity extraction task while consuming 19× less GPU compute. The architecture is agentic search, not training innovation. A central orchestrator LLM decides at each step whether to adjust hyperparameters, mine real data from web sources, or synthesize training examples. Fixed programmatic rules act as guardrails, rolling back any change that degrades accuracy on a held-out validation set. The system searches small-to-large across base models and quantization configs, adapting its accuracy threshold as it goes. Crucially, it measures actual latency and memory on the target phone using real inference engines and backends — not estimated specs. The phone-measurement finding is the sharpest result. The best model configuration changes across devices: what runs fast on a Pixel 9 Pro is not what runs fast on a Samsung S24. When Claude Opus 5.5 is given full hardware and model specifications and asked to pick the right config without real measurements, it misses the app's requirements 25% of the time on easy/moderate cases and 60% on hard ones. This is not a failure of Claude's intelligence — it is a demonstration that latency on heterogeneous mobile hardware cannot be reliably predicted from specs alone. You have to measure. The baseline comparison is honest but narrow. The paper competes against frontier models in 5-shot prompting mode, which is the relevant practical comparison for someone deciding between on-device SLMs and API calls. But it does not compare against other automated SLM-tuning pipelines (e.g., AutoML for edge models, or distillation-based approaches like those from Apple or Qualcomm). The ten tasks span entity extraction, classification, and summarization but are all text-in/text-out — no multimodal, no long-context, no agentic reasoning chains. The validation uses an unseen test set, and code is released, which is good. But the tasks were chosen by the authors, not drawn from a community benchmark suite, leaving room for selection effects. The milestone here is about automation radius, not model quality. The system handles ten tasks today. The question is whether it can scale to hundreds of heterogeneous tasks — scheduling, health monitoring, document parsing — without human curation of task specs and evaluation metrics. The 19× compute savings over exhaustive search is compelling at ten tasks, but the real unlock is whether the search cost stays sublinear as the task space grows. If it does, you get a genuinely new deployment primitive: describe what you want, name your phone, get a model. The obvious experiment not run is iOS. The paper evaluates four Android phones only. Apple's Core ML stack, Neural Engine, and model-format constraints are materially different from Android's inference engines. The authors likely ran out of time and engineering bandwidth — porting the measurement harness to iOS is real work — but the omission matters because iPhone is roughly half the target market for on-device AI. The second missing experiment is adversarial robustness: the system rolls back accuracy regressions, but there is no test of whether the resulting models are brittle to distribution shift on inputs slightly outside the training distribution.