Read the paper
Auditing black-box clinical prediction models with a large language model through queries alone
on GitHub
RUNS
+96%
RUNS
+95%
KEY METRIC
+80%
KEY METRIC
+99%
Imagine you're a food inspector who can't enter the kitchen. You can only order dishes — any dish, as many as you want — and taste what comes back. From the flavors, textures, and temperatures, you try to reverse-engineer the recipe, find contamination, and maybe even discover a new flavor combination the chef didn't intend. That's what this paper asks GPT-6 Astra to do with clinical prediction models: no weights, no code, no architecture — just query access and the returned predictions. The committed claim: an LLM, given only a prediction API, can build a faithful surrogate of a black-box clinical model (R² = 0.976), detect nearly all deliberately planted flaws (96%, 69/72 runs), and attempt to surface novel clinical relationships — but that last promise is where things get shaky. The authors trained four model families (logistic regression, EBM, gradient-boosted trees, neural net) on International Stroke Trial data, planted 24 specific flaws (under-predicted subgroups, reversed risk factors, implausible steps), mixed them with 8 clean models, and let GPT-6 audit blind with pre-committed judging rules. This is a controlled experiment on LLM interpretability, not a clinical deployment. The surrogate fidelity result is genuinely strong. A median of 128 terms reproducing log-odds at R² 0.976 beats a human analyst's 42-term additive surrogate at R² 0.959. The LLM is building richer approximations than a careful statistician, though at the cost of far more complexity. The flaw detection at 96% is impressive but needs context: a purpose-built script found all 24 planted flaws with 100% accuracy. The LLM is approaching scripted performance but hasn't matched it, and planted flaws are adversarial in a known, structured way — real-world model pathologies may not be. The integrity story is unusually careful for this kind of work. Judging rules were committed before any GPT-6 run and applied programmatically — not post hoc by the authors squinting at outputs. The 95% CI of 80-99% for the detection rate is honestly wide, reflecting only 24 distinct flaws across 72 runs (3 per flaw). The authors also report the uncomfortable numbers: 60% of 1,089 checkable spontaneous reports failed pre-specified checks, and while most of these turned out to be real-but-small effects on post-hoc inspection, 15% were outright hallucinations — things GPT-6 reported that simply weren't in the model. The knowledge-discovery angle is the weakest leg. Of 601 unexpected relationships the LLM extracted from clean models, only 11% held in patients randomised later (the temporal validation set), compared to 6% for random pairs. After checking against training data, the hit rate rises to 33% — but a regression scan's curated list hit 28-30%. The LLM is barely beating a simple scan and is far from reliable discovery. The authors are admirably transparent about this, but the implication is clear: LLM-generated 'insights' from model internals require heavy external validation and cannot be trusted at face value. Architecturally, this is a prompt-engineering and tool-use study, not a modeling advance. GPT-6 Astra uses a prediction-query tool iteratively, building surrogate models by probing the target across input perturbations. The method is model-agnostic by design — the same approach works across logistic regression, EBMs, GBMs, and neural nets — which is a genuine advantage but also means performance is bounded by the query interface's information content and the LLM's capacity to organize thousands of query-response pairs into coherent hypotheses. The paper lands a real result — query-only LLM auditing of black-box clinical models is viable and surprisingly faithful — while honestly documenting the limits. The flaw-finding is strong, the surrogate fidelity is strong, but the spontaneous reporting is noisy and the discovery claims are thin. For anyone building or regulating clinical AI, the takeaway is: LLMs can be useful auditors of deployed models, but their outputs need the same skepticism you'd apply to any automated analysis. Trust the structured flaw-detection; treat the free-text 'findings' as hypotheses requiring confirmation.
The analogy
Like a food inspector who can't enter the kitchen — you can order any dish and taste what comes back, then reverse-engineer the recipe, spot contamination, and maybe discover a new flavor. The inspector's palate is sharp for known problems but generates a lot of false leads when speculating about novel combinations.
Food inspector
Orders dishes, tastes outputs, reverse-engineers the recipe and spots contamination without entering the kitchen
LLM auditor
Sends prediction queries, reads outputs, builds surrogates and detects flaws without accessing model weights or code
Honest assessment
This paper delivers a genuinely well-controlled test of LLM-as-auditor for clinical ML. The fidelity and flaw-detection results are strong and honestly reported. The discovery claims are weak but the authors say so themselves. The missing piece is a head-to-head with standard interpretability tools and a test on real deployed models with unknown flaws. For clinical AI regulators and ML safety researchers, this is immediately useful as a proof-of-concept; for practitioners, it's a signal that query-only auditing is viable but not yet a replacement for existing tools.
What Is CLAIMS?
73
/ 100
BedrockNews applies a structured analytical framework called CLAIMS to academic papers and technical research releases.
How CLAIMS worksClaim
3/5
Ladder
4/5
Architecture
3/5
Integrity
5/5
Milestone
3/5
Successor
4/5
What is this paper actually committing to?
Real but bounded. Query-only LLM auditing is viable and tested rigorously, but the three promises split: fidelity and flaw-detection deliver, discovery doesn't.
The field fight
Can LLMs serve as reliable interpretability tools for black-box ML models, or are they sophisticated confabulators that give false confidence?
Side A
LLMs can explain, audit, and extract knowledge from models — a scalable alternative to manual interpretability
Held by: OpenAI interpretability team, various LLM-as-judge papers
Side B
LLM explanations are fluent but unreliable; they hallucinate structure and generate false confidence in high-stakes domains
Held by: SHAP/LIME community, clinical AI safety researchers, Rudin (interpretable-by-design)
Milestone tracker
Unlocks: Scalable third-party auditing of deployed clinical AI without requiring model access
Current
96%
detection rate
planted flaws, 4 model families
Next
90%+
detection rate
unknown flaws in deployed clinical models
~1-2 years if clinical access and regulatory willingness align
Verified Facts (12)
Mean R² of 0.976 for LLM surrogate models reproducing audited models' log-odds
R² of 0.959 for a 42-term additive surrogate fitted by an analyst
Planted flaw detected in 69 of 72 runs (96%; 95% CI 80% to 99%)
Purpose-written script found all 24 planted flaws
60% of 1,089 checkable reports failed the pre-specified check
15% of checkable reports were not found in the model
11% of 601 unexpected relationships held in patients randomised later (random pairs 6%)
Regression scan's strongest candidates: 28% temporal validation rate
After training-data check: 33% vs 30% for regression scan
Median surrogate model had 128 terms
Four model types: logistic regression, explainable boosting machine, gradient-boosted trees, neural network
24 flawed copies plus 8 clean models (32 total)
Practical Implications (4)
Regulators could mandate query-only LLM audits of deployed clinical models as a lightweight compliance check, without requiring model access from vendors
Clinical ML teams should not trust LLM-generated spontaneous findings without independent validation — 60% failure rate on checks and 15% hallucination rate
The 128-term surrogate approach could serve as a model documentation tool, producing richer approximations than manual analysis
Query-only auditing creates a new threat model: adversaries with API access can reverse-engineer proprietary clinical models at R² 0.976 fidelity
More latest news
Read all