ArchSpace Wants to Be the GitHub for LLM Architecture Ideas — With Scaling Laws as the Code Review

A public platform for recording, reviewing, and referencing LLM architecture innovations. Uses scaling laws to predict large-scale viability from small experiments.

Imagine you're a chef developing recipes, but you can only afford to cook in a tiny home kitchen. You need to know: will this dish scale to a 200-seat restaurant? Right now, you'd have to book the restaurant and cook full-service to find out. ArchSpace is proposing something like a food-science simulator — run the small-batch version, measure the physics of flavor and texture, then extrapolate whether it'll hold at banquet scale. The mechanism is scaling laws: fit loss curves from small experiments and project what happens at orders of magnitude more compute. The committed claim: ArchSpace is an open platform that lets any researcher Record an architecture idea with its experiments and reasoning, have it Reviewed by the community against shared baselines, and Reference it in future work — all anchored by scaling-law extrapolation so small-compute labs can participate meaningfully. The baselines provided are Qwen3.5 and OLMo3, with fitted scaling laws and a reusable training recipe. The problem this addresses is real and well-known. Industry teams at DeepSeek, Qwen, and Kimi regularly experiment with architectural innovations but publish selectively — documenting wins, burying failures. Academic labs, meanwhile, run single-point comparisons at small scale with inconsistent baselines, making it nearly impossible to know if an improvement at 1B parameters survives at 70B. The result is a field where the architecture design space is explored chaotically, with enormous duplication and wasted compute. The scaling-law backbone is the load-bearing idea here. Industry already does this internally — predicting large-scale loss from small runs is standard practice at Anthropic, Google DeepMind, and OpenAI. ArchSpace's contribution is making that methodology public, standardized, and community-governed. By providing Qwen3.5 and OLMo3 as reference baselines with pre-fitted scaling curves, they lower the barrier for academic labs to propose and test architectural modifications against a common yardstick. The integrity picture is honest but early. This is fundamentally an infrastructure proposal, not a results paper. The baselines exist and the scaling-law fits are provided, but the platform's value depends entirely on community adoption. There's no independent validation of the platform's predictions against actual large-scale training runs — that's the obvious gap. The authors are essentially asking the community to trust scaling-law extrapolation as a proxy for ground truth, which is reasonable given industry practice but not yet battle-tested in this open-community format. The successor question is straightforward: the experiment nobody has run yet is closing the loop — taking a scaling-law-validated architecture idea from ArchSpace and training it at full scale to see if the prediction held. That's expensive, which is why it hasn't been done. But without at least one such validation, the platform is a plausible-but-unproven accelerant. The honest read: they're bootstrapping community trust first, and the large-scale validation will come from whoever adopts the platform and has the compute budget to verify. The 20-year version of this idea is significant if it works. Architecture search is one of the highest-leverage activities in AI — a single good idea (FlashAttention, grouped query attention, mixture of experts) can save billions of dollars in training compute across the industry. A functioning open platform that surfaces good ideas faster, eliminates duplication, and lets small labs contribute meaningfully to architecture design would be genuinely generative. The risk is that it becomes a graveyard of small-scale experiments that nobody trusts at production scale.