Imagine you're a teacher grading a 923-question exam, but you only wrote answer keys for 208 of the questions — and the 208 you picked are disproportionately the ones your best student already aces. That's the state of AI benchmarking against the US labor economy. Mercor's benchmark map is the first systematic attempt to quantify this mismatch, and the headline number is striking: 90% of American working hours are spent on tasks with little or no AI benchmark coverage. The committed claim is structural, not about any single model's capability: existing AI benchmarks are not a representative sample of the US economy. Only 208 of 923 ONET occupations have even a single task covered by at least 5 benchmark prompts, and 6 of 22 entire job families have zero such occupations. This means that when labs announce SOTA results on popular benchmarks, those results speak to a narrow economic slice — software engineering, financial analysis, law — and say almost nothing about truck dispatchers, court clerks, HVAC technicians, or the sprawling middle tier of knowledge work. The architecture is a crosswalk methodology, not a model. Mercor starts with the US Department of Labor's ONET taxonomy (923 occupations, 18,797 tasks), selects 29 benchmarks filtered for public availability, agentic task design, economic realism, diversity, and unsaturation, then uses an LLM judge to tag each benchmark prompt to ONET tasks. A prompt-to-task mapping counts as 'covered' at a threshold of 5 prompts. Time-share estimates from Hatgis-Kessell et al. convert task coverage into working-hours coverage. The key validation is human-expert comparison against Mercor's own APEX benchmarks. It's a sensible pipeline, but the LLM judge and the 5-prompt threshold are both judgment calls that could move the headline number significantly. The coverage bias is the most policy-relevant finding. After controlling for AI-exposure (using Eloundou et al. 2023 tags), tasks from occupations with larger wage bills are still significantly overrepresented in benchmarks. The logistic regression shows AI-exposed tasks are ~6x more likely to be benchmarked than unexposed tasks, and high-importance tasks (ONET importance score top quintile) are 6–10x more likely to be covered than lowest-importance tasks. This is rational allocation of scarce benchmarking resources — you measure where the money is — but it means our picture of AI capability is systematically biased toward elite professional work and blind to the middle-income long tail. Integrity deserves honest scrutiny. The 29-benchmark list is editorially curated with disclosed criteria, but 'economic realism' is acknowledged as subjective. The exclusion of saturated benchmarks likely biases coverage estimates downward (easy tasks may be well-benchmarked but saturated); the authors argue the alternative bias — overstating coverage by including unrealistically easy benchmarks — is worse. The LLM-judge tagging is validated only against Mercor's own APEX benchmarks, not an independent gold standard. The occupation wage-bill analysis uses BLS data and standard econometric techniques (logistic regression, clustered standard errors, delta-method CIs), which is solid but entirely dependent on the ONET taxonomy's accuracy as a map of real work. The milestone question is whether comprehensive benchmark coverage can scale. Today: 208 of 923 occupations have meaningful coverage. A plausible next threshold would be 500 occupations covered at the 5-prompt level, which would push working-hours coverage past 50%. How fast that happens depends on whether benchmark creation scales with the same energy going into model training — and right now, it clearly does not. The resource asymmetry between building models and measuring what they can do in the real economy is the structural gap this map exposes. The obvious successor work Mercor did not run — and acknowledges — is expanding beyond 29 benchmarks to a more representative sample, and validating the LLM judge against independent human experts across multiple job families, not just APEX. The honest read: they shipped a V1 to stake a claim and generate attention, and broader validation is coming. The 29-benchmark cap, especially the per-family cap, is a feature for estimating economy-wide coverage but deliberately suppresses depth in the most-benchmarked families. The next version should also test sensitivity to the 5-prompt threshold and the LLM-judge agreement rate at the tails.
