Imagine you ask an intern to highlight every person mentioned in a 30-page legal brief, but your answer key only lists the parties to the case. The intern dutifully highlights witnesses, judges, and cited precedent authors too — all real people, all actually in the document, but none of them "correct" by your rubric. You'd score the intern as wildly imprecise, but the failure isn't fabrication — it's a mismatch between what you asked for and what exists in the text. That's the core mechanism this paper exposes in LLM entity extraction from cyber threat intelligence reports. The committed claim: LLM entity extraction from full-length CTI documents is unreliable (micro precision 0.03–0.34), but the unreliability is dominated by scope mismatch (65.6% background mentions + 19.9% gold-scope gaps), not hallucination (1.2%). Moreover, zero-cost confidence signals can rank output quality but cannot verify it — a single adversarial sentence flips high-confidence judgments 95.6–100% of the time. The evaluation is unusually disciplined for an LLM benchmark paper. The protocol was preregistered and fully frozen before execution. Three models span the cost spectrum: DeepSeek-V4-Flash and GLM-5.3-Flash as low-cost contenders, GPT-6.1-Sol as the flagship control. The corpus is public — 354 incident campaigns across 315 unique reports, producing 630 model-report evaluations. The failure-mode annotation used two independent LLM annotators from vendors outside the evaluated set, hitting Cohen's kappa of 0.79 on the preregistered acceptance line. This is not a vibes-based audit. The failure decomposition is the paper's real contribution. Prior work reported LLM extraction as broadly unreliable and left it there. This paper runs a full mention census and three-tier annotation of 360 sampled candidates to back-project error sources. The 65.6% background-mention finding reframes the problem entirely: the models aren't making things up — they're extracting real entities that the gold standard scope excludes. The 19.9% gold-scope gap means the gold labels themselves are incomplete. Only 1.2% of candidates are genuinely unsupported. This distinction matters enormously for deployment: a system that over-extracts real information is a different engineering problem than one that fabricates. The confidence analysis is where the paper earns its deployment-boundary conclusion. Self-consistency frequency tiering — a zero-cost method — passes the preregistered top-30% precision-gain bar in all six test-split model-type cells. A combined text-frequency × SC-frequency signal wins 3, ties 2, loses 1 across cells. So confidence can prioritize review queues. But the robustness testing is devastating: relation-level counterfactuals on all 45 targets show a single misattribution sentence flips high-confidence judgments in 95.6–100% of cases. Even a background sentence explicitly marked as unrelated still binds a fake vulnerability in up to 26.7% of cases. Confidence ranks; it does not verify. The architecture story is straightforward: this is a prompting-based evaluation, not a fine-tuning paper. The models are used as-is with verbalized confidence. The method leans entirely on the reasoning capacity of instruction-tuned LLMs at inference time, with self-consistency as the only post-hoc signal. No retrieval augmentation, no structured output parsing, no domain-specific training. This makes the results broadly applicable to anyone deploying commercial LLMs for entity extraction, but also means the paper doesn't explore the obvious next step of whether fine-tuning or constrained decoding would collapse the scope-mismatch failure mode. The paper's honest limitation is scope: it studies entity extraction from one domain (CTI) on one corpus. The failure decomposition — background mentions dominating over hallucination — is plausible across many extraction domains (legal, medical, financial), but that generalization is unproven. The adversarial robustness finding is more immediately generalizable: if a single injected sentence can flip high-confidence extractions, that's a structural vulnerability of verbalized-confidence prompting, not a domain-specific artifact. The deployment boundary — confidence for prioritization, never for truth — is the paper's lasting contribution, and it's stated with appropriate force.
