Imagine you're a sous chef asked to judge whether a new spice improves a dish. You're handed a written hypothesis — "cumin added before searing improves flavor scores" — and a walk-in cooler full of 10,000 tasting notes. The hard part isn't computing the average score; it's deciding which notes count: only the ones where cumin was added before searing, to the right protein, on the right date range. Grab the wrong stack and your t-test is perfect but meaningless. That's the core mechanism of this paper: the bottleneck in LLM-driven data analysis isn't the statistics — it's the evidence retrieval. The committed claim: existing benchmarks for LLM data analysis grade only the final answer and hand the model pre-selected data. STRATA is the first benchmark that grades each intermediate decision — plan formulation, evidence retrieval, statistical testing, and verdict — over 8,735 temporal clinical hypotheses drawn from VAERS (Vaccine Adverse Event Reporting System) timelines, with a deterministic oracle producing gold-standard outputs for every step. The accompanying ART (Articulate, Retrieve, Test) paradigm structures the pipeline so each decision is inspectable rather than buried in a single prompt. The ladder context matters here. Prior LLM data-analysis benchmarks (the authors cite DA-Agent, InfiAgent-DABench, DSBench) provide the relevant data alongside the question and grade only the conclusion. STRATA's contribution is orthogonal: it's a process benchmark, not a conclusion benchmark. The paper shows that accuracy on final verdicts can look fine — models get the yes/no right — while the evidence set underneath is wrong. This verdict-evidence gap is the headline finding, and it's only visible because the benchmark decomposes the pipeline. Architecturally, ART is a staged retrieval-augmented generation pipeline, not a monolithic prompt-and-pray setup. Stage one: the model articulates a testing plan (null hypothesis, test type, temporal constraints). Stage two: it retrieves matching records from the VAERS timeline corpus. Stage three: it executes the statistical test and renders a verdict. Each stage's output is compared against the oracle's gold output. The models tested span GPT-4o, Claude-3.5-Sonnet, Gemini-2.0-Flash, DeepSeek-R1, and Qwen-2.5-32B — five configurations across frontier and open tiers. The integrity setup is stronger than average for an NLP benchmark paper. The gold labels come from a released codebook applied deterministically, meaning any team can reproduce the oracle's outputs. The 8,735 hypotheses are derived from real VAERS reports, not synthetically generated. The key integrity limitation: the authors designed both the benchmark and the evaluation paradigm, so there's circularity in the sense that ART is evaluated on STRATA, which was built with ART's decomposition in mind. No pre-registration, no independent replication yet, but the codebook and data release lower the barrier to external validation substantially. The most striking result: a 31-billion-parameter open model (Qwen-2.5-32B) under the ART paradigm comes within 0.01 of frontier models on the composite metric. That's a strong signal that the paradigm matters more than raw model scale for this task. Frontier models succeed once given the codebook, which suggests the bottleneck is structured instruction-following for evidence selection, not raw reasoning capacity. The evidence retrieval stage is where all models bleed performance — plan formulation and statistical testing are comparatively easy. The missing experiment is domain transfer. STRATA is built entirely on VAERS vaccine adverse-event timelines. The obvious next test is whether ART's decomposition and the verdict-evidence gap generalize to other temporal-clinical domains (EHR timelines, pharmacovigilance databases, claims data) or to non-clinical temporal hypothesis testing (financial event studies, supply-chain disruption analysis). The authors almost certainly know this is the follow-up; building a second domain benchmark is expensive and slow, and this paper needed to ship with one domain done rigorously.
