Imagine you're an interior designer handed photos of five different living rooms and told: 'Take the shelving from this one, the color palette from that one, the layout from the third, ignore the other two, and make something that also has a reading nook.' That's not copying — it's selective visual interpretation combined with compositional generation. That's exactly what RefCraft asks LLMs to do with webpages, and it turns out they're terrible at it. The committed claim: current frontier (M)LLMs cannot reliably build webpages when given visual references alongside natural-language instructions — the best system, across nine models tested, tops out at 52.12 out of 100. This isn't a methods paper proposing a fix; it's a diagnostic benchmark exposing a capability gap. The paper introduces 300 realistic multimodal webpage-development tasks that combine text instructions with one to five reference screenshots, requiring agents to selectively interpret visual elements, render code, visually inspect results, and iterate. The benchmark design is the interesting contribution here. Prior web-development benchmarks fall into two idealized camps: either pure visual-to-code reconstruction (clone this screenshot) or text-only instruction following (build me a landing page with a hero section). Neither captures how real web development works, which almost always involves visual references as inspiration — not targets. RefCraft forces models into the messy middle ground: here are some screenshots, here's what I want, figure out which parts of which references matter and synthesize them into working code. Evaluation uses a dual-judge system — programmatic checks for independently verifiable keypoints (does the nav bar exist? does the color match the spec?) alongside VLM-based judges for holistic visual assessment. This two-layer approach is smart because it avoids the common trap of benchmarks that are either too rigid (exact pixel matching) or too squishy (vibes-based LLM scoring). The keypoint system means you can verify individual capabilities even when overall quality is low. The 52.12 ceiling across nine frontier models is the number that matters. This isn't a case where one model dominates and others trail — the entire frontier class struggles with the same bottleneck: selectively interpreting visual references while maintaining instruction fidelity. Models either over-copy (reproducing reference layouts wholesale) or under-reference (ignoring visual cues and relying on text alone). The multimodal grounding problem — connecting what you see to what you generate — remains unsolved at the agentic level. Architecturally, this sits at the intersection of multimodal understanding and agentic code generation. The tested models span the frontier of both capabilities (GPT-4o-class vision plus code generation plus tool use), and the benchmark specifically requires the iterative loop: generate → render → inspect → refine. The fact that even models with strong individual capabilities in vision AND coding AND tool use fail at their composition is the diagnostic insight. The practical upshot for anyone building AI-assisted development tools: visual-reference conditioning is a real bottleneck, not a solved problem. If your product assumes users will hand AI agents screenshots as design specs — a workflow that every major AI coding assistant is moving toward — this benchmark says your best-case accuracy is roughly coin-flip quality. The gap between text-only instruction following and visual-reference-conditioned generation is where the next capability unlock lives.
