Imagine you're hiring a sous chef. You test knife skills, seasoning intuition, plating speed — each in isolation — and they ace every station. Then you ask them to cook a full five-course dinner for a table with dietary restrictions that evolve mid-service, and the kitchen collapses. That's the composition gap NPCBench documents. Models that nail individual medical tasks crater when those tasks must be jointly correct across a patient's longitudinal care trajectory. The committed claim: existing medical AI benchmarks are testing vocabulary, not clinical competence. NPCBench replaces isolated Q&A with an apprenticeship-structured evaluation for nasopharyngeal carcinoma (NPC), a head-and-neck cancer with complex staging, multimodal imaging, and treatment sequencing. The benchmark contains 4,777 component and consultation items plus a 100-patient CarePath cohort yielding 417 scored diagnostic-and-care tasks. The episode-level scoring demands that diagnostic AND treatment answers be jointly correct within the same care pathway — miss either, and the episode fails. Across 30 models (21 multimodal), the headline result is damning: strict CarePath success tops out at 23%. This isn't a matter of models being generally weak. Many score well on individual subtasks — guideline knowledge, single-image interpretation, isolated case planning. The failure mode is integration. When models must ground a treatment decision in visual evidence from MRI, track how that evidence changes over time, and produce a care plan that remains internally consistent across the episode, performance collapses. The paper calls this the composition gap, and it's the single most important finding. The architecture of the benchmark itself is the methodological contribution. NPCBench is organized into five tiers that mirror specialist training: foundational knowledge, case planning, longitudinal review, CarePath integration, and open-ended consultation. Paired controls test whether models are actually reading the images or pattern-matching from text. Physician calibration provides human baselines — oncologists complete the same tasks so you can see where machines trail and by how much. The evaluation is not gradient-based optimization; it's structured expert scoring with episode-level consistency requirements. Integrity is solid for a benchmark paper. The 100-patient CarePath cohort comes from real clinical records at Sun Yat-sen University Cancer Center, a major NPC referral center. Physician calibration provides an independent human baseline rather than letting models grade themselves. The five-tier structure means cherry-picking is harder — you can't hide a weak CarePath score behind strong guideline-knowledge numbers because the episode-level metric is reported separately. The main limitation: NPC is a single cancer type at a single institution, which the authors acknowledge. The milestone question is sharp: when does a model cross 50% strict CarePath success on a benchmark of this type? At 23% ceiling today, we're roughly a 2× improvement away from even a coin-flip level of longitudinal care-pathway consistency. The field needs to watch whether scaling (bigger models, more medical data) closes this gap or whether it's an architectural ceiling requiring new approaches to temporal reasoning and evidence grounding. The obvious experiment not run: testing whether retrieval-augmented generation (RAG) with explicit patient-timeline indexing narrows the composition gap. The authors structured the benchmark beautifully but evaluated only end-to-end model inference. My read is this is being saved — the benchmark is the contribution for this paper, and RAG/agent architectures applied to NPCBench will be the follow-up. The 100-patient cohort is small enough that scaling to multi-institution, multi-cancer validation is the other glaring next step, likely constrained by data access agreements rather than compute.
