A Multi-LLM Pipeline That Translates Plain-English Security Policies into Enforceable Access-Control Graphs — and Actually Checks Its Own Work

Text2NGAC chains multiple LLMs to extract all four NGAC relationship types from policy documents at 97–99% accuracy, then verifies the output with mutation testing that catches 99.7% of injected faults.

Imagine you're a building manager who hands a stack of lease agreements to four different locksmith apprentices. Each apprentice reads the leases independently, cuts keys, and then you run every single key against every single lock — plus you deliberately jam 918 fake keys into the pile to see if your quality-control step catches them. That's the core mechanism here: cooperating LLM agents parse natural-language policy documents, a multi-model ensemble resolves ambiguity, and a five-strategy verifier stress-tests the resulting access-control graph with controlled mutation injection. The question the verifier answers isn't "did you get it right?" — it's "can I catch you when you get it wrong?" The committed claim: Text2NGAC is the first system to extract all four NGAC relationship types — associations, prohibitions, event-response obligations, and assignments — from real-world policy documents, then structurally verify the extracted graph. Prior systems covered subsets; none verified. That's a genuine first for this specific combination, though each component (LLM extraction, mutation testing, NGAC modeling) exists independently. The pipeline's architecture is a multi-agent LLM ensemble — not a single model, but cooperating models where one disambiguates, another extracts, a third escalates uncertain cases. This is closer to committee-of-experts than chain-of-thought. The verification layer uses five mutation strategies (918 injected mutants total) and catches 99.7% of them. More importantly, the three mutants it misses are characterized and weighted by security consequence — the paper doesn't just report a miss rate, it tells you what the misses look like and how dangerous they are. This is unusually honest for a systems paper. The baseline story is strong by access-control extraction standards but narrow. Three reproduced baselines are beaten on matched metrics across 714 scored sentences from four real corpora, reaching 97.3–99.6% exact-tuple-match accuracy. The real validation flex: one corpus's extracted graph loads unmodified into the NIST reference implementation, reproduces all 55 access decisions, and fires an extracted obligation end-to-end. That's not a simulation — that's an enforceable artifact running in the reference implementation the standard was built around. Integrity has genuine strengths and clear limits. The mutation testing is well-designed: fault classes are weighted by security consequence rather than just counted, which is the right way to measure a security verifier. But the authors are explicit that adversarial input is outside the verifier's guarantees, and the intended deployment is extraction-plus-verification feeding expert review — not autonomous enforcement. Paraphrase probes with semantic re-scoring bound sensitivity to rewording, but this isn't the same as red-teaming against deliberate adversarial policy language. The two-tier reproducibility artifact is a plus. The milestone question for this subfield is whether LLM-based policy extraction can replace human translation for production access-control systems. Text2NGAC gets close enough to feed expert review, but the gap between "feeds expert review" and "autonomous enforcement" is where the hard problems live. The next concrete number to watch: adversarial robustness benchmarks. Can the pipeline maintain 97%+ accuracy when policy language is deliberately ambiguous or contradictory? That's the test that separates a useful tool from a deployable system. The obvious experiment not run: adversarial policy documents — text crafted to exploit LLM parsing weaknesses and create fail-open faults in the extracted graph. The authors acknowledge this boundary explicitly, which suggests they know the pipeline would struggle. This is likely a combination of (a) scope management for a 33-page paper and (c) saving adversarial robustness for a follow-up. The other missing piece is cross-model generalizability: the pipeline is built on specific LLMs, and there's no ablation showing whether it degrades gracefully when you swap in weaker or different models.