Imagine you're grading a student's cooking final, but instead of one score for 'cooking ability,' you score knife skills, sauce technique, plating, and timing separately — then cross-reference each skill against each dish type (fish, pasta, dessert). Suddenly you can see that the student's knife work is fine on vegetables but falls apart on fish, and their sauces are great everywhere except desserts. That two-axis decomposition is exactly what MoI-Bench does for on-device image models. The committed claim: existing benchmarks for mobile image generation and editing produce a single ranking number that hides the specific capability gaps between on-device and server-side models. MoI-Bench replaces that with a 16-operation × 8-target grid (removal, replacement, style transfer, etc. crossed against portraits, text, backgrounds, etc.) producing 64 subtasks with 1,150 samples. The result is not a leaderboard — it's a diagnostic map. The evaluation framework, MoI-Eval, is where the methodological care shows. Rather than a monolithic quality score, it decomposes evaluation into instruction following, visual quality, and detail preservation — each assessed via sample-level questions aggregated along operation, target, and criteria axes. This means you can see that a model handles 'remove facial makeup' well but collapses on 'replace text in signage,' which is a far more actionable finding than 'Model A scores 72, Model B scores 68.' The headline finding lands clearly: current on-device models have already achieved reasonable deployment efficiency (fast enough, small enough), but their task coverage is narrow. The bottleneck is not latency or memory — it's that these models can't reliably do text rendering, identity preservation, or many editing operations that server-side models handle. The paper's two-dimensional Quality-Deployability capability map makes this visible at a glance, plotting generation/editing quality against deployment cost. Where does this sit against existing benchmarks? GenAI-Bench, EditBench, and T2I-CompBench all evaluate image generation or editing but from a server-side perspective and with single-score rankings. MoI-Bench is the first benchmark explicitly grounded in mobile user needs and mobile deployment constraints. It doesn't claim on-device models are catching up — it precisely maps how far they have to go and in which directions. The architectural landscape under evaluation includes representative models from both the on-device (compact diffusion, distilled models) and server-side (full-scale diffusion, instruction-tuned editors) families. The benchmark itself is model-agnostic — it's a task and evaluation framework, not a new model. The design choice to keep Operation and Target as orthogonal axes is structurally clean and extensible. The honest limitation: the evaluation relies on automated scoring via task-specific questions, not human raters at scale. The authors acknowledge this but don't run the obvious human-correlation study. The 1,150 samples across 64 subtasks means some cells in the grid are thin. And the benchmark is a snapshot — as models improve, the subtask definitions will need updating. Still, the contribution is real: this is infrastructure for a field that has been flying blind on mobile capability gaps.
