Imagine you're training a new paralegal to read traffic court verdicts and fill in a spreadsheet. You hand them a form and some written instructions, they fill in rows, you check their work, then you rewrite the instructions to fix the mistakes. That back-and-forth — instruction refinement guided by disagreement — is the core mechanism of this paper, except the paralegal is an LLM. The committed claim: an LLM-based pipeline with human-refinable annotation instructions can build structured datasets from Czech criminal verdicts flexibly enough that legal researchers don't need to fully specify their extraction schema in advance. The pipeline has three stages — attribute exploration (figuring out what to extract), text span extraction, and value normalization — and it's tested on two traffic offense types: driving under the influence (DUI) and driving despite a ban (DDB). The numbers are honest and modest. For DUI, the initial run achieved only 52% correctly extracted values; after revising the annotation instructions, that rose to 70%. The DDB task, with more constrained categories and output formats, showed higher agreement. Crucially, the authors surface a finding that cuts both ways: repeated LLM runs agreed with each other more than with human gold annotations. Consistency ≠ correctness. That's a useful diagnostic, and the paper deserves credit for flagging it rather than burying it. The baseline situation is thin. There's no named prior system doing exactly this task on Czech criminal verdicts, which makes the ladder hard to evaluate. The paper doesn't compare against simpler NLP baselines (regex + rule-based extraction, smaller fine-tuned models, or classical NER pipelines). The 70% accuracy number exists in a vacuum — we don't know if a well-tuned spaCy pipeline would get 60% or 40%. This is the biggest gap. Architecturally, this is prompt-engineered LLM extraction — the paper uses large language models (specific model names not given in the abstract) with structured prompting and iterative instruction refinement. It's firmly in the in-context learning / few-shot prompting family, not fine-tuned or retrieval-augmented. The computational overhead and cost-per-verdict aren't discussed, which matters if you're processing thousands of decisions. The validation is internal: human annotators create gold labels, the LLM extracts, disagreements are analyzed. The paper's best integrity move is using inter-annotator agreement and model-vs-human disagreement as a diagnostic for instruction quality. But there's no independent replication, no public benchmark, and the dataset appears to be Czech-language court decisions that aren't publicly released. The finding that instruction revision improves accuracy is associative, not causal — revision and accuracy improvement happened together, but without a controlled ablation it's hard to isolate the mechanism. For empirical legal researchers, this is practical plumbing: a workflow for turning unstructured verdicts into spreadsheets without writing custom code for every new research question. The 70% accuracy means substantial human review is still needed, but it's a starting point. The real contribution is the iterative instruction-refinement loop as a methodology, not the extraction accuracy itself. The next milestone is cross-jurisdictional generalization — can this pipeline work on German, Dutch, or English verdicts without re-engineering? And the obvious missing experiment: comparison against fine-tuned smaller models or classical NLP baselines to establish whether the LLM's flexibility justifies its cost.
