
Read the paper
AuXSpec: Learning to Autoformalize Program Specifications via Executable Postconditions
on HuggingFace
PASS4 AUXSPEC-TEST
+0.7%
AUXSPEC-TEST
+82.7%
KEY METRIC
+14.3%
KEY METRIC
+84.3%
Imagine you're teaching someone to write legal contracts. The traditional approach: have an expert lawyer review every draft line by line, checking it against case law. Expensive, slow, doesn't scale. AuXSpec's move is different: for every contract clause, create a plain-English version that says the same thing, then check agreement by running both versions against a pile of scenarios. If the fancy clause and the plain version agree on every test case, the clause is probably right. That's the core trick — replace proof-based grading with execution-based grading, and suddenly you can generate thousands of training signals where before you could generate dozens. The committed claim: by pairing declarative Lean postconditions with executable counterparts and using test-case agreement as an RL reward signal, AuXSpec trains relatively small open-source models (9B and 27B parameters) that beat all frontier models on formal specification generation. The 27B model hits 84.5% pass@1 on the out-of-domain Verina benchmark, crushing the prior open-source SOTA (VeriSpecGen-30B-A3B) at 47.6%. The 9B model, improbably, nearly matches it at 73.7%. The architectural move is clean. AuXSpec is a Lean 4 framework that uses type class resolution to automatically derive executable versions of declarative specs when possible, falling back to manually defined executable counterparts with equivalence proofs otherwise. This lets test cases evaluate declarative specifications by execution. The RL loop uses agreement between the generated spec and test cases as reward — no theorem prover in the training loop, no expensive proof search per sample. They synthesize 3,037 LeetCode problems into this format and expand to 19.5K samples via expert iteration on Codeforces. The base models are Qwen3.5-9B and Qwen3.8-27B, fine-tuned with this execution-based reward. The ladder position is strong. On AuXSpec-Test, the 9B model goes from 0.7% to 82.7% (pass@4) and the 27B from 14.3% to 84.3%. On Verina, both hit 90.5% at pass@4, compared to VeriSpecGen's 63.9% at the more generous pass@10. The comparison is honest — they test on an out-of-domain benchmark they didn't train on, using the original prompts, and still win by a wide margin. The gap is not close. Integrity is better than average for this subfield. The Verina benchmark is external and independently maintained. The AuXSpec-Test benchmark includes 300 problems with expert review and correctness proofs. Code, data, and model weights are all released. The main circularity risk is that execution-based agreement is a proxy for correctness — specs could be vacuously true (always return true) and still pass tests. The authors address this with expert review on the benchmark, but the training data lacks such guarantees. No pre-registration, but the benchmark structure partially compensates. The bigger field fight here: can RL with cheap reward signals replace expensive expert supervision for formal methods tasks? This paper says yes for specifications, joining a growing body of work (DeepSeek-Prover, AlphaProof-adjacent efforts) showing that you can bootstrap formal reasoning from weaker but scalable signals. The bottleneck in verified software has always been specification, not verification — provers can check proofs, but someone has to write the specs. If AuXSpec's approach generalizes beyond competitive-programming-style problems to real software specifications, that bottleneck loosens significantly. The obvious next experiment they didn't run: testing on real-world software specifications rather than algorithmic competition problems. LeetCode and Codeforces problems have clean, well-defined input-output behavior. Production code has side effects, state, concurrency, and specs that resist clean postcondition formulation. The authors almost certainly know this is the gap — the paper is 62 pages and they had time to discuss it. My read: they're saving it for the next paper, and the current framework would need significant extension to handle it.
The analogy
Like grading essay exams by running the student's answer through a multiple-choice converter: instead of having an expert read every spec for correctness, AuXSpec translates declarative postconditions into executable ones and checks them against test cases. The grading is approximate but massively scalable.
Auto-grader
Convert free-response answers into machine-checkable form, grade against answer key of test cases
AuXSpec reward
Convert declarative Lean postconditions into executable equivalents, grade via test-case agreement as RL reward
Honest assessment
AuXSpec solves a real bottleneck — grading formal specifications without expensive proofs — and the results are convincing on competitive-programming-style problems. The 27B model's dominance over both open-source and frontier models on an external benchmark is the strongest signal. The open question is whether execution-agreement reward and clean input-output postconditions transfer to the messy reality of production software. If they do, this is a landmark. If they don't, it's still a very good paper about a tractable subset of the problem.
What Is CLAIMS?
80
/ 100
BedrockNews applies a structured analytical framework called CLAIMS to academic papers and technical research releases.
How CLAIMS worksClaim
4/5
Ladder
5/5
Architecture
4/5
Integrity
4/5
Milestone
3/5
Successor
4/5
What is this paper actually committing to?
Genuine first: execution-based RL reward for formal spec generation, training 9B/27B models that beat all frontier models. Not just methods — delivers a usable framework and benchmark.
The field fight
Can formal specification scale via cheap RL reward signals, or does it require expensive proof-based supervision?
Side A
Proof-based feedback is necessary for correctness guarantees; execution proxies are too lossy
Held by: Traditional formal methods, proof-assistant communities
Side B
Execution-based reward is cheap enough to scale RL training, and test-case agreement is a sufficient proxy for spec correctness
Held by: AuXSpec, DeepSeek-Prover-adjacent work, LLM-for-verification community
Milestone tracker
Unlocks: Practical auto-specification for production codebases, dramatically reducing the bottleneck to formally verified software
Current
90.5
% pass@4
Algorithmic specs (Verina)
Next
80
% pass@1
Real-world software specs
~2-3 years; requires framework extension for effectful Lean code
Verified Facts (6)
Training Qwen3.5-9B and Qwen3.8-27B with execution-based reward improves SpecGen pass@4 on AuXSpec-Test 0.7% → 82.7% and 14.3% → 84.3%
On the out-of-domain Verina benchmark, both models improve 7.0% → 90.5% and 21.9% → 90.5%
27B model surpasses VeriSpecGen-30B-A3B on pass@1 (theirs 47.6% vs. ours 84.5%) and pass@4 (theirs 63.9% at pass@10 vs. ours 90.5%)
3,037 LeetCode problems synthesized, expanded to 19.5K samples through expert iteration on Codeforces
300-problem AuXSpec-Test benchmark with expert review and correctness proofs
Models and data released at https://huggingface.co/collections/eth-algopt/auxspec
Practical Implications (3)
Teams building verified software in Lean 4 can use the released 9B/27B models today as specification drafters, dramatically reducing the expert-hours bottleneck
The AuXSpec framework and 3,037-problem dataset provide a reusable scaffold for anyone training spec-generation models with execution-based reward
The execution-agreement-as-reward pattern is transferable: any domain where declarative and executable formulations can be paired could use this approach for RL training
More latest news
Read all