Imagine you're a health inspector, but instead of visiting every restaurant in a city, you send a single very smart inspector to a few blocks, let them poke around with a budget and a notebook, and ask them to write up not just what they found but a checklist any junior inspector can run tomorrow for free. That's this paper's core move: using an LLM agent not as a classifier stamping every document, but as a one-time corpus debugger that writes its own permanent, dirt-cheap test suite. The committed claim: a tool-using LLM agent, given read-only API access to the 26.5-million-document Slovak portion of FineWeb2, can autonomously discover categories of data-quality failure and compile each into an executable, reusable test — validated by a human native speaker. Three agent runs cost $2.21–$2.84 each and surfaced seven findings across four phenomena. The headline catch: a machine-generated content farm of 1,408,437 documents (5.32% of the entire Slovak slice) that passed FineWeb2's quality filters, with 86.6% of those documents surviving even stricter thresholds. The ladder here is interesting and honestly reported. The authors pit their agentic approach against a static sampling baseline — a human manually picking and validating suspicious slices — and the static baseline actually validates more selectors at under half the cost. The agent's advantage isn't raw discovery rate; it's the structured output: exact prevalence estimates, precision checks against sampled documents, causal accounts of why the pipeline missed each slice, and the compiled test suite itself. The authors don't hide the baseline win; they frame the agent's contribution as complementary, not superior. Architecturally, this is a tool-augmented LLM agent operating under a budget constraint — think ReAct-style tool use but pointed at corpus exploration rather than task completion. The agent queries the corpus through a read-only API, formulates hypotheses, tests them against sampled documents, and compiles findings into executable selectors. The compiled test suite then runs across all 26.5M documents in 0.7 seconds of wall-clock time on CPU, no GPU needed. Compare that to $12k–$90k for a single LLM classification pass over the same corpus. Integrity is solid for a workshop paper. Blinded validation by a native speaker supports 6 of 7 findings (defect rates 0.53–1.00 against a 0.08 random control baseline) and explicitly does not support one — a refreshing level of honesty. The framework, Slovak test suite, validation data, and full agent trajectories are all released. The main limitation is that this is one language, one corpus, one pipeline, with the agent runs limited to three. The generalization question — does this transfer to Swahili, or Tamil, or any of the other 998 FineWeb2 languages — remains open. The real contribution isn't the agent itself; it's the workflow pattern. Multilingual pipelines like FineWeb2 apply a single recipe across ~1,000 languages and simply cannot anticipate the failure modes of each one. The paper demonstrates that a cheap, one-shot agentic audit can produce reusable artifacts (executable tests) that shift corpus quality assurance from expensive per-document classification to permanent, near-zero-cost regression testing. The $3-per-audit figure is the number that matters: it means you could audit every language in FineWeb2 for under $3,000 total. The obvious next experiment is scaling this across multiple languages to see if the agent's discovery patterns generalize — whether the phenomena found in Slovak (content farms, pipeline-filter evasion) recur in structurally different languages. The authors almost certainly know this is the next step. My read: they're scoping it for a full paper, given this was accepted to a workshop. The harder experiment is adversarial: can content farms adapt to evade the compiled test suites, and can the agent be re-run periodically as a living audit?
