LLMs Fail at Showing You What the Other Side Thinks — A New Benchmark Measures How Badly

NarrativeBench tests 14 LLMs across 24 languages on whether they can surface competing narratives on contested topics. They can't — and the gaps widen across languages.

Imagine you're a librarian in a building with 23 rooms, each containing newspapers in a different language, all covering the same 23 contested world events. A patron walks in and asks about the Kashmir conflict. Your job isn't to hand them one stack — it's to pull articles from every room, identify where the stories agree and disagree, flag what's missing from any single room, and present the full mosaic without collapsing it into a single narrative. That's the task this paper defines for LLMs, and the finding is stark: no current model can do it reliably. The committed claim: LLMs lack the capabilities to help users navigate diverse multilingual narratives on contested topics, and NarrativeBench is the first benchmark that systematically measures this failure across six specific capabilities. The paper defines three foundational capabilities — can the model include a specific narrative when asked, exclude one when asked, identify what's missing from a given set — and three higher-level ones — present diverse perspectives unprompted, resist collapsing diversity when a user's query is confirmatory, and maintain information parity across query languages. The benchmark comprises 1,333 news articles across 23 publication languages, 23 contested topics, and 460 Reddit-grounded queries, evaluated across 14 models in 24 query languages. The ladder here is unusual because there's no strong prior benchmark to beat — this is infrastructure work, not a performance race. The closest predecessors are monolingual bias benchmarks and cross-lingual factuality evaluations, but nothing at this intersection of multilingual narrative diversity with user-facing query behavior. The paper's contribution is the benchmark itself and the empirical finding that all 14 tested models (including GPT-4o, Claude, Gemini, Llama, and Qwen variants) fail to reliably navigate diverse narratives. The failure modes differ by model, but no model achieves strong performance across all six capabilities simultaneously. Architecturally, NarrativeBench is an evaluation-side contribution, not a model architecture. The key design choice is grounding queries in real Reddit posts rather than synthetic prompts — this ensures the queries reflect how actual users frame contested topics, including confirmatory and leading framings. The benchmark pipeline involves human annotation of narrative clusters from multilingual news corpora, mapping of which narratives appear in which languages, and structured evaluation of model outputs against these narrative maps. The 23 topics span geopolitical conflicts, domestic policy disputes, and international incidents where language-specific framing diverges meaningfully. The integrity profile is solid for a benchmark paper. The evaluation covers 14 models — a broad sweep rather than a cherry-picked comparison. The use of Reddit-grounded queries prevents the authors from inadvertently optimizing query framing for particular model behaviors. The 23-language, 23-topic matrix is large enough to resist topic-specific cherry-picking. The key vulnerability is the human annotation of narrative clusters: the paper's entire measurement apparatus depends on annotators correctly identifying and separating competing narratives across languages, and inter-annotator reliability on contested political topics is inherently noisy. The NeurIPS Dataset Track acceptance signals peer review found the methodology credible. The milestone to track is concrete: current models fail at the foundational capabilities (narrative inclusion, exclusion, identification) badly enough that higher-level capabilities can't even be meaningfully assessed for most models. The next milestone isn't a single number — it's a model that passes all three foundational capability tests with >80% reliability across at least 10 languages. That would be the minimum viable product for a narrative-aware information system. We're not close. The obvious experiment not run: fine-tuning or retrieval-augmented generation with explicit multilingual narrative databases to see whether the failure is in training data or architecture. The honest read is (a) — this is an evaluation paper accepted to a dataset track, and the authors' stated goal is to provide the benchmark, not the solution. Building the retrieval pipeline across 23 languages is a separate engineering effort they likely intend to enable for the community rather than execute themselves.