Imagine you're a hiring manager reviewing candidates, but sometimes you read their résumé, sometimes you watch their video interview, and sometimes you listen to the audio only. You'd expect your judgments to shift — but you'd hope that at least one format consistently gives you the best signal. This paper tests that hope for multimodal LLMs doing social-science video coding, and the hope does not survive. The committed claim: input representation — whether you feed an MLLM the raw video, extracted text, audio, or some combination — is a validity-relevant design choice that researchers currently treat as arbitrary but should treat as requiring explicit benchmarking per construct. No single representation reliably dominates. Across six open-weight MLLMs (the paper tests Gemma, LLaVA-family, and similar models) and three human-annotated news video datasets covering seven constructs, native-video macro-F1 scores cluster tightly at .70–.77 across models, but text-based performance swings wildly from .52 to .80. Three models do better with text; three do better with video. Combining modalities almost never helps. The more unsettling finding is the disconnect between classification accuracy and downstream validity. You might assume that if a model classifies individual videos more accurately, the aggregate statistics — prevalence estimates, regression coefficients — will be closer to the human-coded ground truth. They tested this explicitly across 42 comparisons of regression coefficients: the highest-F1 representation only best reproduced human-coded coefficients in 17 of 42 cases, barely beating chance. Even when each model uses its best-available representation, 45% of sample-level prevalence estimates diverge from the human benchmark by more than ten percentage points. That's not noise at the margins; that's a different story about your data. Architecturally, this is a benchmarking study, not a new model. The authors sit in the multimodal LLM evaluation family, using open-weight models as black-box classifiers and treating the input-representation decision as the experimental variable. The compute property they're leaning on is the recently acquired ability of open-weight MLLMs to natively ingest video (not just frames), which makes the text-vs-video comparison newly possible at scale without API costs. The integrity setup is respectable but bounded. All three datasets carry pre-existing human annotations — the gold standard for this kind of work. The benchmarks are external to the authors (human coders from prior studies), which limits circularity. But there's no pre-registration, no released code mentioned in the abstract, and the construct set is limited to news-related variables. The models tested are all open-weight, which is good for reproducibility, but the paper does not compare against GPT-4o or Claude — the models most social scientists would actually use. That omission matters. The milestone question is practical: at what reliability threshold do MLLM-coded video labels become safe to use without per-construct validation? The paper implicitly argues that threshold hasn't been reached. A concrete next target would be a shared benchmark suite — say, 10 constructs across 5 domains with pre-registered human annotations — where any new MLLM can be evaluated on representation validity in under a day. That infrastructure doesn't exist yet. The obvious next experiment the authors didn't run: testing closed-weight frontier models (GPT-4o, Gemini 1.5 Pro, Claude) under the same protocol. The honest read is (a) — cost and access constraints. These are social scientists, not an ML lab with API budgets. But it means the paper's findings may not generalize to the models practitioners actually deploy, which limits the practical bite of the warning.
