41 LLMs Give Different Mental Health Advice Depending on Who's Asking — and the Advice They Push First Is Often Stuff They Themselves Rate Poorly

Massive audit of 35,055 LLM responses reveals models serve clinicians richer guidance and lay users hedged simplifications — and lead with interventions they rate below neutral.

Imagine you walk into a restaurant and ask the chef what's good. Now imagine your doctor walks in and asks the same question. Same menu, same kitchen — but the chef describes the specials with technical detail to the doctor and hands you a laminated greatest-hits card with a warning not to order anything too spicy. That is what 41 language models are doing with mental health advice, and this paper caught them at it. The core claim: LLMs systematically vary the interventions they recommend based on who is asking — clinician, caregiver, or patient — even when the underlying clinical information is identical. Across 19 standardized psychiatric case vignettes and five questions per vignette, the team collected 35,055 free-text responses and extracted every named intervention using a validated LLM judge. They then had 29 of those same models rate the full intervention lists on a seven-point Likert scale for the same cases. The instrument is a serious piece of work: 100 pages of documentation, publicly hosted at mindbench.ai. The ladder finding is the one that should make you sit up. Models largely recommend what they rate well — coherence between generation and evaluation is decent. But when prompted about a specific intervention type, models more often named options they themselves rated below zero as initial recommendations, usually hedged with conditions. Translation: the model knows it doesn't think much of the intervention, but leads with it anyway when the question is framed a certain way. This is the equivalent of a doctor prescribing something they'd privately rate as marginal, because the patient asked about it directly. The footing effect — clinician vs. caregiver vs. patient — is the architectural finding. Many models adopt a consultation-style register for clinicians: more options, more nuance, more clinical terminology. Lay users get simplified, hedged advice. This is not necessarily wrong — clinicians can contextualize a wider option set — but it means the model is making triage decisions about information access based on perceived role, not clinical need. Nobody designed this explicitly; it fell out of RLHF training on web-scale data where clinician-directed text looks different from patient-directed text. Integrity is mixed but honest. The validated LLM judge for intervention extraction is a methodological contribution, but the whole pipeline — LLM generates, LLM extracts, LLM rates — is recursive. The authors know this and flag it. There's no ground-truth clinical panel saying 'this recommendation was appropriate for this vignette,' which means we know what models say but not whether what they say is good. The 19 vignettes are standardized but not pre-registered; the question battery was designed for this study. Code and corpus promised on publication. The practical upshot is this: if you're building or deploying any LLM-based mental health tool — triage bots, therapy augmentation, caregiver support — you now have a concrete instrument for auditing footing bias and recommendation–rating incoherence. The 35,055-response corpus is the largest structured dataset of LLM mental health recommendations published to date. The next milestone is connecting these recommendation patterns to actual clinical outcomes — right now we know what models say, not whether it helps or harms. The experiment the authors didn't run: testing whether users actually follow the hedged advice differently from the clinical-register advice, and whether footing-aware prompting can close the information gap without introducing harm. The honest read is (a) — that's a human-subjects study requiring IRB approval, clinical partnerships, and longitudinal tracking that a 17-author CS/psychiatry team could plausibly do next but couldn't scope into this already-massive paper.