Imagine a copy editor who can tell you a paragraph contains a factual error but cannot underline the wrong sentence — let alone explain what the correct fact should be. That is roughly where Arabic LLMs stand today on hallucination, and HalluTruthQA is the first benchmark designed to measure each of those sub-skills separately. The committed claim: this is the first Arabic QA hallucination benchmark that goes beyond binary detection to provide character-level erroneous spans, human-written explanations, a six-option factual verification task, and a macro/micro hallucination taxonomy — all expert-curated across 2,400 examples in four knowledge-intensive domains (Islamic knowledge, history, science, geography). Prior Arabic benchmarks gave you a thumbs-up/thumbs-down on whether an answer hallucinated. HalluTruthQA asks four distinct questions: did it hallucinate, where exactly, what's the right answer, and why is the hallucinated content wrong? The ladder tells a clear story about where current models actually stand. Four open-source LLMs — Allam, Falcon-H1, Qwen32, and Silma — were evaluated zero-shot. Detection is nearly solved at this scale: 0.880 Macro-F1. Factual verification (picking the right answer from six candidates) hits 0.852 LO-Score. But localization — highlighting the exact erroneous characters — crashes to 0.516 F1-Sp. Explanation evaluation lands at 0.644. No single model dominates all four tasks. The numbers expose a hierarchy of difficulty: detecting that something is wrong is far easier than saying precisely what and why. Architecturally, the paper is a benchmark contribution, not a modeling one. The evaluation uses zero-shot prompting of instruction-tuned decoder-only transformer LLMs. The intellectual weight sits in the annotation schema: character-level span markup, a two-level hallucination taxonomy (macro and micro types), and human-written natural-language explanations paired with each hallucinated answer. The four evaluation domains were chosen for knowledge density, and each example includes a verified reference answer — the ground truth that makes span-level and explanation evaluation possible. Integrity is solid for a benchmark paper. Expert curation across 2,400 examples is a meaningful investment, and the four-domain split (Islamic knowledge, history, science, geography) reduces domain-specific bias. Code, dataset, prompts, and evaluation scripts are publicly released on GitLab. The main weakness is that the four evaluated models are all open-source Arabic-capable LLMs evaluated only zero-shot — no proprietary model baselines (GPT-4, Claude), no few-shot comparisons, no fine-tuned models. This is a benchmark launch, not a leaderboard saturation. The milestone gap is concrete. Localization F1-Sp sits at 0.516 — barely above chance for a span-extraction task. Getting that to 0.80+ on this benchmark would signal that models can reliably identify not just that they hallucinated, but exactly which characters are wrong. That is the threshold where automated fact-checking pipelines become viable for Arabic content at scale — and we are nowhere near it. The obvious successor experiment the authors did not run: evaluating proprietary frontier models (GPT-4o, Claude, Gemini) and fine-tuning an open-source model on the training split of HalluTruthQA itself. The honest read is (a) — compute and API cost. A 12-author team building a 2,400-example expert-curated benchmark likely spent their budget on annotation, not inference. Expect the fine-tuning paper within 6-12 months.
