Feeding EHR Codes Back Through Language: Mistral-7B as a Longitudinal Patient Encoder

tEHR converts structured clinical codes into natural-language streams, then fine-tunes a 7B-parameter LLM as a patient timeline encoder. Beats specialized EHR models across five prediction tasks on MIMIC-IV and CPRD.

Imagine you inherit a vast warehouse of filing cabinets, each drawer labeled with a cryptic internal code — "ICD-10: E11.9" or "CPT: 99213." You know these codes MEAN something in English ("Type 2 diabetes" and "outpatient visit"), but the warehouse's inventory system treats them as opaque serial numbers. Every relationship between drawers has to be inferred from which ones tend to sit near each other, ignoring the fact that the labels on the front already tell you what's inside. tEHR's core move is to rip off those labels, read them aloud, and let a language model that already knows English do the heavy lifting. The committed claim: a causal language model (Mistral-7B-v0.3) pretrained on web text, when adapted via continued pretraining on natural-language-rendered EHR timelines and then discriminatively fine-tuned with LoRA, outperforms purpose-built structured EHR encoders on longitudinal clinical prediction tasks — not by learning better co-occurrence statistics, but by importing semantic priors about what clinical concepts actually mean. The evaluation covers four tasks on MIMIC-IV (a public ICU dataset) and pancreatic cancer prediction on CPRD (a UK primary care dataset) at horizons from 1 to 12 months. The paper reports that the Mistral-7B backbone achieves the strongest overall performance. Ablations are the real strength here: performance drops when you strip out natural-language descriptions (replacing them with raw codes), when you scramble the text-code mapping, when you remove numerical lab values, and when you skip either the continued pretraining or the LoRA fine-tuning stage. Interestingly, explicit elapsed-time descriptions between events add little — suggesting the model extracts temporal signal from event ordering itself. Architecturally, tEHR belongs to the "LLM-as-encoder" family that has been gaining traction: take a decoder-only causal LM, freeze or lightly adapt it, then bolt a task head on top. The two-stage adaptation pipeline (domain-adaptive continued pretraining → task-specific LoRA) is a known recipe from NLP transfer learning, applied here to the unusual modality of serialized clinical timelines. The 7B parameter scale is notable — large enough to carry rich semantic priors, small enough to fine-tune on clinical-scale data without nation-state compute. The method leans heavily on the assumption that natural-language descriptions of clinical codes are informationally richer than the codes themselves, which is a strong and testable bet. Integrity is mixed. MIMIC-IV is a community benchmark and publicly available, which is good. CPRD is a well-known UK dataset but access-controlled, limiting replicability. The paper compares against existing EHR models on the same tasks, though the abstract does not name specific baselines or give numeric deltas — a yellow flag. The ablation design is methodologically sound: removing descriptions, misaligning them, omitting numerics, and ablating each training stage are the right controls. No mention of pre-registration, and code availability is not stated in the abstract. The milestone question for this line of work is whether LLM-based EHR encoders can generalize across hospital systems without retraining. MIMIC-IV and CPRD are two different coding systems (US ICD/CPT vs UK Read/SNOMED), and tEHR's use of natural-language descriptions as a bridging representation is precisely the kind of approach that could enable cross-system transfer. The concrete next number: can a single tEHR model, pretrained on one hospital's data, achieve competitive performance zero-shot on a second hospital's EHR without task-specific fine-tuning? The obvious experiment not run is cross-institution zero-shot transfer. If natural-language descriptions truly provide a universal semantic bridge, a model trained on MIMIC-IV should perform non-trivially on CPRD without CPRD-specific tuning (or vice versa). The authors trained separately on each dataset. The honest read: they likely ran out of scope (this is already a multi-dataset, multi-task paper accepted at PMLR), and the cross-transfer experiment is almost certainly the next paper. If it doesn't appear within 18 months, that silence would be informative.