Imagine you're assembling IKEA furniture, but instead of following the manual step by step, you hand the parts diagram to someone who's assembled a lot of furniture — but never this exact model. They'll get some steps right from pattern recognition ("this clearly screws into that"), but they'll misidentify which cam lock goes where on the tricky multi-panel joins. That's what LLMs are doing with cryptographic protocol proofs: they recognize structural patterns in proof obligations and can generate plausible lemmas, but the complex multi-phase reasoning still defeats them. The committed claim: this is the first systematic evaluation of state-of-the-art LLMs on cryptographic symbolic protocol verification, equipped with a new proof-based metric called CRoST (Coverage Rate of Solve Tree) that measures how close machine-generated lemmas are to the reference lemmas a human expert would write. CRoST is derived from the verifier's own proof skeleton, giving it structural grounding rather than surface-level similarity. The authors validate the metric both theoretically (showing it correlates with proof success) and empirically. This is genuinely a first — nobody has benchmarked LLMs on this task with a metric designed for it. The headline numbers are sobering. State-of-the-art models achieve 38.82% average CRoST coverage. Only 14.4% of generated lemmas exceed the 80% coverage threshold that would indicate genuinely useful output. The paper tests across multiple models and protocol complexities, and the failure modes are consistent: multi-phase protocols with interleaved security properties break the models. This is not a "scaling will fix it" situation — the authors explicitly demonstrate diminishing returns under naive scaling, meaning simply giving the LLM more attempts or longer context doesn't reliably improve coverage. Architecturally, this is a prompt-and-evaluate pipeline, not a fine-tuned model. The LLMs are treated as black-box lemma generators, and the symbolic verifier (likely Tamarin or ProVerif, the standard tools) serves as the ground-truth oracle. The key insight is measuring against the proof tree structure rather than raw text similarity — CRoST decomposes the verifier's proof skeleton into sub-goals and checks whether generated lemmas satisfy them. This is a much more meaningful metric than BLEU or exact match, because two syntactically different lemmas can be semantically equivalent in a proof. Integrity is reasonable for a benchmarking paper. The authors test multiple SOTA models (not just one convenient choice), report failures honestly, and demonstrate that their metric has both theoretical justification and empirical correlation with actual proof completion. The diminishing-returns finding is especially credible because it works against the hype narrative the authors could have pursued. What's missing is any independent replication and any comparison to human expert performance on the same tasks with time constraints — we know the LLMs hit 38.8%, but we don't know how long the reference lemmas took a human to write. The milestone gap is clear. At 38.8% average coverage and 14.4% of lemmas exceeding 80%, the practical utility is limited to "sometimes helpful suggestions" rather than "reliable co-pilot." The next meaningful threshold would be ~70% average coverage with ~50% of lemmas above 80%, which would make LLM-assisted proving faster than unassisted human work on routine protocols. The diminishing-returns finding suggests getting there requires architectural changes (retrieval-augmented proving, fine-tuning on proof corpora, iterative verifier feedback loops) rather than just bigger models. The obvious experiment not run: closed-loop interaction where the verifier's failure feedback is fed back to the LLM for iterative refinement. The authors benchmark single-shot generation, but real human provers iterate — they try a lemma, see where it fails, and adjust. This is almost certainly being saved for the next paper, as it's the natural extension and the authors' CRoST metric is perfectly positioned to measure improvement across iterations.
