Imagine you hire a brilliant consultant who aces the intake interview — sharp differential diagnosis, impressive recall — and then you discover they forget to order the follow-up labs, skip the staging workup, and never schedule the surveillance scan. That is the core finding of NPC-RubricBench: LLMs can diagnose, but they cannot sustain clinical reasoning across the phases of care that actually determine patient outcomes. The benchmark itself is the real contribution. Built from longitudinal records of 405 nasopharyngeal carcinoma patients across two Chinese cancer centers, it defines 941 patient-phase units across six sequential care phases — from initial work-up through treatment planning, delivery, and post-treatment surveillance. Each unit is scored against 17,688 patient-specific atomic rubric criteria spanning five evaluation dimensions, and critically, every assessment is restricted to information available at decision time. This is not a trivia quiz. It is a simulation of the rolling decision burden oncologists actually face. The headline result: strong diagnostic performance does not predict downstream competence. All 30 models — spanning general-purpose (GPT-4o, Claude, Gemini) and medically adapted variants — showed degradation as care phases advanced. But the failure modes are more interesting than the aggregate scores. In anatomical staging, 48.2% of errors were explicit decision errors — the model committed to the wrong answer. In initial work-up and post-treatment follow-up, the dominant failure was omission: 33.4% and 30.4% respectively. The model didn't get it wrong; it simply didn't do it. Medically fine-tuned models were worse on omissions than general-purpose ones, suggesting that medical adaptation may be narrowing attention rather than deepening it. The prompt-granularity finding is a genuine trap for deployment teams. When you give the model a broad, open-ended clinical question, Decision Quality drops across all 30 models. When you give it a detailed checklist-style prompt, Decision Quality rises — but Evidence Faithfulness and Safety Adherence decline. You can make the model more thorough or more faithful, but the current generation cannot be both simultaneously. This is a specification trade-off, not a tuning problem, and it has direct implications for how clinical AI interfaces are designed. Methodologically, the rubric-based approach is what distinguishes this from the dozens of medical LLM benchmarks published monthly. Each rubric criterion is patient-specific and phase-locked: you cannot get credit for mentioning the right drug if the patient hasn't been staged yet, and you cannot skip the staging just because you got the diagnosis right. The five evaluation dimensions — Decision Quality, Evidence Faithfulness, Safety Adherence, Reasoning Completeness, and Clinical Communication — give a structured decomposition of where failure actually lives, rather than a single accuracy number. The broader implication is a direct challenge to the current benchmarking paradigm in medical AI. Static, single-encounter benchmarks (MedQA, PubMedQA, USMLE-style tests) measure something real, but they measure it at a single point in the care continuum. NPC-RubricBench demonstrates that clinical reliability is phase-dependent: a model that passes a diagnostic benchmark might fail catastrophically in the treatment-planning or surveillance phases that follow. The paper argues, persuasively, that longitudinal phase-adaptive evaluation should become the standard for clinical AI deployment. What this does not show is whether the failure modes are fixable within current architectures or represent a deeper limitation. The omission problem — models failing to raise issues they should raise — is particularly concerning because it is invisible to the patient and often to the supervising clinician. The specification trade-off between thoroughness and faithfulness suggests that retrieval-augmented or multi-agent architectures may be needed, but that experiment remains unrun.
