Imagine you gave nine professional chefs the same pantry, same kitchen, same clock, and asked each to invent a new recipe. They taste their own dishes during practice rounds, lock in a final menu, then serve it to strangers. Every chef produces a plausible plate — but none outscore the cafeteria buffet where you just take one of everything. That is Quant-Arena in a sentence. The committed claim: frontier LLMs can now complete the entire quantitative-research loop — hypothesize, code features, backtest, iterate, freeze a strategy — on real equity data, without human intervention. The benchmark tests this end-to-end on China's A-share market, training on 2022–2023 and evaluating on sealed 2024 data. This is not a code-generation benchmark or a single-trade decision task. It is the full alpha-research pipeline, which matters because that pipeline is exactly what quant funds pay humans $500K+ per year to execute. Nine frontier models were evaluated, with GPT-6 Astra scoring highest on the composite metric dominated by realized excess return. But the punchline is stark: no model generated positive excess return over the equal-weight benchmark in 2024. The equal-weight portfolio — the simplest possible strategy, requiring zero intelligence — won. The evaluator is serious: it simulates a cash account with realistic trading constraints and market-impact costs estimated from a simulated order book, so you cannot dismiss the result as bad execution modeling. Architecturally, Quant-Arena sits in the LLM-as-agent family rather than the LLM-as-predictor family. The model is not forecasting next-day returns from a prompt; it is writing Python code that constructs features and scoring functions, receiving backtest feedback, and iterating. This is closer to AutoML or program synthesis than to time-series forecasting. The compute property it leans on is the model's ability to generate and debug executable code in a feedback loop — a capability that scales with pretraining compute and RLHF quality, not with financial domain data. The integrity setup is notably honest for an AI-finance paper. The sealed 2024 test year is genuinely out-of-sample — models receive zero feedback from it. Trading costs and market impact are modeled, not hand-waved. The scoring metric emphasizes realized excess return, the hardest thing to game. That said, benchmarking only on A-shares in a single calendar year is narrow; 2024's Chinese equity market had specific regime characteristics (deflation fears, property crisis overhang) that may not generalize. There is no pre-registration and no independent replication yet. The milestone question is the interesting one. Today: zero models beat equal-weight. The next meaningful number is any model achieving statistically significant positive excess return over a 12-month out-of-sample window with realistic costs. That threshold — call it Sharpe > 0.5 after costs on unseen data — would be the first credible evidence that LLM agents can generate alpha without human quant researchers in the loop. We are not there. How far away? The gap between 'plausible strategy that underperforms' and 'strategy that consistently beats market' is the same gap that separates amateur quants from profitable ones — it may be a capability jump, not a scaling curve. The obvious experiment not run: testing whether LLM agents can improve by collaborating with or being fine-tuned by human quant researchers in a hybrid loop, rather than operating fully autonomously. The authors also did not test on U.S. equities, which have deeper liquidity and more efficient pricing — arguably a harder and more decision-relevant market. The honest read: they built the benchmark for A-shares because that is their domain expertise and data access, and the autonomous-only framing is cleaner for a benchmark paper. Both extensions are likely coming in follow-up work.
