Imagine you're coaching a Little League team, and your method is this: you hand each kid a rulebook, a bat, and exactly one hour of batting practice. Some kids spend their hour hitting off a tee. The smart ones immediately start facing the fastest pitching machine in the cage. At the end of the hour, everyone plays the same lineup of opponents — including a few former pros. That's StarSkirmish Bench: the hour is fixed, the opponents are fixed, and the only variable is how each LLM chooses to spend its practice time. The committed claim is straightforward: StarSkirmish Bench v0.2 is the first controlled evaluation of frontier LLMs as autonomous agentic coders in a real-time strategy domain, where each model gets one wall-clock hour to write, iterate, and test a Protoss bot in C++ against a fixed roster of human-written opponents. This is not a coding puzzle or a static benchmark — it's a long-horizon planning and debugging task where the agent must decide what to practice, interpret game outcomes, and refactor code under time pressure. GPT-6 Astra sits on top with a score that appears to be in the low-to-mid 40s (win rate against the full roster), with GPT-6.1 Sol second and a cluster of Gemini 4 Argon, Claude Opus 5.5, and Claude Fable 5.1 bunched at 37-39. But the headline number is the ceiling check: against the three strongest human-written bots — Stardust, PurpleWave, and BananaBrain — GPT-6 Astra wins just 7% of games. The benchmark is nowhere near saturated. Human-written bots, developed over years by dedicated programmers with deep domain knowledge, remain categorically superior. The LLMs are playing in a different league. The most interesting finding is behavioral, not architectural. The strongest predictor of a model's final score is how aggressively it practiced against the seven toughest roster bots during its hour. GPT-6 Astra, Sol, and Gemini 4 Argon start sparring with strong opponents early, even while losing badly. Muse Wonder Full and Claude Haiku 5.5 mostly avoid them. Claude Fable 5.1 is the fascinating outlier — scoring 37 while practicing almost exclusively against weak bots, suggesting an alternative strategy of polishing fundamentals rather than adapting to elite opponents. This behavioral divergence is arguably more diagnostic than the raw scores. Methodologically, v0.2 makes smart choices. Switching from Elo to plain win rates against a fixed roster means scores are comparable across benchmark generations without re-running old models. Each model runs in its own lab's first-party CLI harness at maximum reasoning effort (Gemini 4 Argon at high, its ceiling), which trades experimental control for ecological validity — you're measuring what each lab actually ships, not an artificial common denominator. Context compaction kicks in around 250k tokens across all harnesses. The correlation analysis is revealing: StarSkirmish has decoupled from the Artificial Analysis Intelligence Index (ρ = 0.26) and Terminal-Bench 4.0, but still tracks Vals IOI (olympiad C++ programming) and WeirdML v3. The authors' interpretation — that heavily-benchmarked evals become optimization targets while less-targeted long-horizon coding tasks still co-vary — is plausible and important. The cost dimension adds practical texture. GPT-6.1 Sol delivers the second-highest score at roughly one-tenth of GPT-6 Astra's API cost per run, making it the only model in the high-performance-low-cost quadrant. Run-to-run variance matters too: Gemini 4 Argon's best and worst runs span 18 points, versus 9 for Astra and 8 for Opus 5.5, suggesting Argon's agentic strategy is less stable. Five runs per model is enough to see this variance but thin enough that confidence intervals remain wide. What StarSkirmish actually measures is not StarCraft skill. It measures an LLM's ability to autonomously manage a complex software project under time pressure — reading documentation, writing and debugging C++ against a real API (BWAPI 4.4.0), interpreting feedback from test runs, and making strategic decisions about where to invest limited practice time. StarCraft is the substrate, but the transferable skill being probed is agentic coding with feedback loops. The benchmark's durability depends on whether that substrate remains hard enough to discriminate as models improve, and at 7% win rate against top human bots, there's a long runway.
