GATES KILL ONLY
+76%
KILL RATE
+48%
SPECIFICATION-DERIVED TESTS
+74%
DIFFERENT CHECKLIST
+24%
Imagine you hire a contractor to build your house, and the contractor also writes the building inspection report. The report says everything passed — and technically it did, against the tests the contractor chose. But when an independent inspector shows up with a different checklist, 24% of the structural joints fail. That is the core mechanism of this paper: LLM agents generating both the hardware design and the evidence that the design is correct, then measuring how much of that self-generated evidence is actually load-bearing. The committed claim: a seven-stage verification-gated workflow using LLM agents can port a post-quantum cryptographic accelerator (ML-KEM-768 and ML-DSA-65) to a Xilinx Zynq UltraScale+ FPGA running at 300 MHz — but the tests the agents generate systematically overstate verification coverage. Module-level simulation tests kill only 76% of 399 tool-selected netlist mutants. For fresh-draw decapsulation specifically, the initial kill rate was just 48%, rising to 74% after specification-derived tests guided by surviving mutants. The gap between 'tests pass' and 'design is verified' is the paper's real finding. The ladder here is unusual because the paper isn't competing against other PQC hardware implementations on speed or area. It's competing against the assumption that NIST ACVP known-answer tests plus LLM-generated simulation suites constitute adequate verification. The answer is plainly no. All 85 interface-applicable ACVP cases pass, all 484 ML-DSA known answers pass — and the design still has blind spots that only tool-selected netlist mutants, formal checks, and scoped review can catch. The baseline being beaten is the implicit trust engineers place in passing test suites. Architecturally, this is an agentic workflow paper in the RTL/EDA domain. The seven stages gate progression on negative controls and fault mutants — a gate counts only if the test can demonstrably fail on a known-bad input. Reviewer agents audit before commit. The audit layer then hits the result with three independent tools: netlist mutation testing (tool-selected, not author-selected), flaw replays with pre-registered protocols, and formal property checking for custody invariants. The hardware target is a Zynq UltraScale+ MPSoC, a mid-range production FPGA. The integrity story is unusually strong for an AI-generated-hardware paper. The flaw replays are pre-registered single-model experiments with explicit session counts: undirected whole-design review catches verifier flaws in only 3/6 and 1/6 sessions, module-scoped review catches every flaw in all 54 sessions, and even scoped review detects planted key-custody flaws in only 3/20 sessions while formal checks flag all of them. These are concrete, reproducible failure rates, not cherrypicked successes. The 76% mutation kill rate is an honest confession — most papers would quietly drop that number. The milestone framing writes itself: today's LLM agents kill 76% of netlist mutants across three modules and need specification-derived human-guided supplements to reach 74% on the hardest case (fresh-draw decapsulation, up from 48%). The concrete next number is closing that remaining 26% gap — getting autonomous mutation kill rates above 95% without human-guided test supplements. That is the threshold where you could trust agentic hardware verification without independent audit, and this paper shows we are nowhere near it. The experiment not run: scaling this workflow to a full SoC-level integration with multiple cryptographic primitives under real timing constraints, or testing the agents against adversarial (not just random) fault injection. The honest read is (a) — this was already a substantial effort across six PQC operations on real hardware, and SoC-level formal verification would have required significantly more compute and tool infrastructure. The 3/20 detection rate on planted key-custody flaws is the loudest signal: formal methods caught what LLM review missed, suggesting the next paper will need to integrate formal property synthesis into the agentic loop itself.
The analogy
Like hiring a contractor who also writes the building inspection report — everything passes the contractor's own checklist, but an independent inspector with different tools finds 24% of the joints are unchecked. The paper measures exactly how much the self-inspection misses.
Self-inspecting contractor
Builder writes the inspection report using their own checklist; passes everything they chose to test
LLM agent workflow
Agent generates RTL and test suites; passes all self-generated gates but kills only 76% of independently-selected netlist mutants
Honest assessment
This is not primarily a PQC hardware paper — it's a verification-epistemology paper that happens to use post-quantum crypto as its test case. The load-bearing result is that LLM-generated tests systematically overstate coverage, and the paper provides the first pre-registered, quantified measurement of exactly how much. The 76% kill rate and 3/20 custody detection rate are the numbers that matter. Anyone building agentic hardware workflows should treat this as a mandatory read; the specific PQC implementation is secondary to the verification methodology.
What Is CLAIMS?
73
/ 100
BedrockNews applies a structured analytical framework called CLAIMS to academic papers and technical research releases.
How CLAIMS worksClaim
4/5
Ladder
3/5
Architecture
4/5
Integrity
5/5
Milestone
3/5
Successor
3/5
What is this paper actually committing to?
Genuine contribution: not just 'LLM makes hardware' but quantified measurement of how much self-generated verification misses. The 76% kill rate and 48→74% improvement are the real claims, not the successful port.
The field fight
Can LLM agents be trusted to verify the hardware they generate, or does agentic RTL require independent verification infrastructure?
Side A
LLM agents with self-generated tests and reviews can achieve sufficient verification coverage for production hardware
Held by: ChipGPT-style workflows, RTL-generation startups, optimistic agentic EDA proponents
Side B
Self-generated evidence is fundamentally insufficient; independent mutation testing, formal methods, and scoped review are mandatory
Held by: Traditional verification community, formal methods advocates, this paper
Milestone tracker
Unlocks: LLM-agentic hardware verification without mandatory independent audit layer
Current
76
% mutant kill
this paper (3 modules)
Next
95
% autonomous kill
trustable without human audit
Open problem — requires autonomous adversarial test generation
Verified Facts (10)
All six operations are byte-exact on the board
Passes all 85 interface-applicable NIST ACVP cases (of 285) and 484 ML-DSA known answers
Existing simulation gates kill only 76% of 399 tool-selected netlist mutants
Fresh-draw decapsulation kill rate from 48% to 74% with specification-derived tests
Undirected whole-design review catches two verifier flaws in only 3/6 and 1/6 sessions
Module-scoped review catches every known flaw in all 54 sessions
Scoped review detects planted key-custody flaws in only 3/20 sessions
Formal checks flag every planted key-custody flaw
300 MHz Zynq UltraScale+ target
ML-KEM-768/ML-DSA-65 accelerator
Practical Implications (4)
If you're using LLM agents to generate RTL, do not trust self-generated test suites as verification evidence — independent netlist mutation testing is mandatory
Module-scoped LLM review catches flaws that whole-design review misses (54/54 vs 3/6); scope your review prompts accordingly
Formal property checks are non-negotiable for security invariants — LLM review detected planted key-custody flaws in only 3/20 sessions while formal checks caught all of them
NIST ACVP known-answer tests are necessary but not sufficient — passing all 85 cases still leaves a 24% mutant survival rate
More latest news
Read all