Veritas Auto-Generates the Test Graders That Agent Self-Improvement Systems Desperately Need

Hand-written verifiers bottleneck agent self-improvement. Veritas generates them automatically and outperforms seven other harnesses across six benchmarks.

Imagine you're a teacher who writes answer keys for exams. Every new course needs a new key, and writing one is almost as hard as writing the exam itself. Now imagine a system that reads the syllabus and generates answer keys automatically — and those keys grade more accurately than the ones most teachers write by hand. That's what Veritas does for AI agent tasks. The committed claim: Veritas is a harness that automatically generates verifiers — the judges that decide whether an agent's output is correct — and those generated verifiers agree with ground-truth verdicts better than verifiers from seven competing harnesses across all four execution-scored benchmarks tested. This isn't about building better agents; it's about building better grading infrastructure so self-improvement loops actually work. The mechanism has three moving parts that distinguish it from generic prompt-and-pray approaches. First, an "evidence header" explicitly declares what observable state the verifier will examine — forcing the generated checker to look at the right thing rather than hallucinating criteria. Second, a library of domain rules captures implicit conventions that task descriptions leave unstated (the kind of thing a human verifier writer knows but never writes down). Third, authoring principles — meta-rules about how to write good verifiers — are themselves partially refined through a self-improvement loop, so the system bootstraps its own quality standards. On the ladder, Veritas is compared against seven named harnesses across six benchmarks: four scored by execution (where official verifiers exist and are first certified as trustworthy) and two pairwise-comparison benchmarks. With the model held fixed, Veritas beats all seven competitors on all four execution-scored benchmarks in terms of agreement with official verdicts. On the two pairwise benchmarks it is "at or near the best." Importantly, the paper also shows cross-model transfer: Veritas-generated verifiers agree better with official verdicts than the vendor's own harness for two of three other model families tested. And its three mechanisms improve 9 of 12 harness-benchmark pairs when grafted onto other harnesses, suggesting the ideas are portable, not just implementation-specific. The integrity picture is mixed. The execution-scored benchmarks use official verifiers as ground truth, and the authors take the unusual step of first certifying those official verifiers — a good practice that many papers skip. But the evaluation is entirely self-contained: same team, same compute, no independent replication. The pairwise benchmarks lack execution-based ground truth by design, making agreement metrics softer. There's no pre-registration, and we don't know if the six benchmarks were chosen before or after initial results. The milestone question is the interesting one. The practical bottleneck for agent self-improvement at scale isn't model capability — it's verifier coverage. If Veritas-style auto-generation reaches, say, 95%+ agreement with hand-written verifiers across 50+ diverse task domains (versus the 6 shown here), it would effectively remove the human-in-the-loop requirement for verifier authoring and unlock autonomous self-improvement pipelines. We're at 6 benchmarks today; 50+ with consistent performance would be the threshold that changes workflows. The obvious experiment not run: scaling to genuinely novel task domains where no official verifier exists to compare against. The paper validates by comparing generated verifiers to existing ground-truth verifiers — but the whole motivation is generating verifiers for tasks that don't have them. Testing on tasks where humans must judge verifier quality (rather than comparing to an existing key) is the real stress test. Honest read: this is likely being saved for the next paper, because it requires expensive human evaluation infrastructure the team may not have had budget for.