Imagine you hire a new intern who aces the written test — scores in the top third of all applicants. Impressive. But then you shadow them on rounds and notice three things: they can't tell you when they're guessing, they sometimes get the right answer by ruling out options for the wrong clinical reason, and they don't actually look at the X-rays you hand them. That's the intern this paper describes — except the intern is a language model, and the exam is Spain's MIR, the national gateway to medical residency. The committed claim: open-weight LLMs in the 8-27B parameter range now score well enough on the MIR (1,552 questions across eight editions, scored under official penalty rules) to rank among the first 600-3,000 candidates, with 41-43 medical specialties still open to them. But this headline accuracy hides three structural failures that matter far more for clinical deployment. First, abstention never pays — even answers the models are least confident about land above the 25% random-chance breakeven, meaning the models cannot reliably flag their own uncertainty. Second, medical fine-tuning (the thing health systems would presumably do before deployment) reduced overconfidence but also reduced accuracy, without improving discrimination between right and wrong answers. The calibration knob and the accuracy knob are coupled in the wrong direction. The reasoning audit is the sharpest contribution. The authors manually evaluated correct answers and found that 13-38% of them arrived at the right choice by rejecting a distractor for a clinically unsound reason. The model picked the right door, but its explanation for why the other doors were wrong wouldn't survive a clinical review. Crucially, confidence scores did not flag these reasoning failures — a model was just as confident when its reasoning was garbage as when it was sound. This is the difference between an exam-taker and a clinician: exams reward the final answer, clinical practice requires the reasoning chain to be trustworthy. The image finding is stark in its simplicity. The authors removed or swapped contextual images provided with questions and measured the impact on accuracy: none. The models are not using visual information to answer questions, even when the question is designed around an image. This isn't a subtle degradation — it's complete indifference. For any health system planning multimodal clinical AI, this is a red flag: the model may appear to process images while ignoring them entirely. Methodologically, this paper does several things right. It uses the exam's own scoring rules (including penalty for wrong answers), tests across eight exam editions rather than cherry-picking one favorable year, and examines four models spanning different parameter counts and fine-tuning regimes. The reasoning audit and image ablation are the kinds of tests that exam-accuracy papers almost never run, and they're the tests that matter most. The Spanish MIR is a real, high-stakes, annually administered national exam — not a synthetic benchmark. The broader lesson here isn't about Spain or the MIR specifically. It's about the gap between "passes the test" and "ready for deployment" — a gap the field has been sleepwalking past by treating exam scores as deployment evidence. This paper names three specific dimensions of that gap (calibration, reasoning soundness, image utilization) and measures all three. The finding that medical fine-tuning makes calibration worse, not better, is particularly important: it suggests the obvious next step health systems would take actually degrades the safety property they need most. For any institution evaluating open LLMs for clinical use on local hardware, this paper is required reading. The models are smart enough to pass. They are not smart enough to know when they're wrong, to reason correctly even when they answer correctly, or to use the information you give them. That's a deployment problem, not an accuracy problem, and no amount of additional exam-score benchmarking will surface it.
