A diagnostic model posts a headline accuracy figure on a curated set of published clinical cases. The press release writes itself: the machine beats the physicians.
I have read a lot of these. The engineering is often genuinely impressive. And the number is still answering a question nobody at the bedside asked.
A benchmark is an exam. A clinic is the real world.
An exam has bounded inputs, a fixed answer key and a candidate who knows they are being graded. Every case is complete. Every case is, by construction, a case — somebody already decided it was worth writing up.
A clinic has none of that. Patients arrive mid-story, with the relevant history missing, three comorbidities that were never in the training distribution, and a chief complaint that is not the thing that will kill them. The test set does not cough. It does not minimise its symptoms because it is afraid of the answer. It does not walk out before the workup finishes.
Most diagnostic AI dies in the gap between those two worlds. It dies in four specific places.
1. The evidence bar is lower than we would accept for a cheap drug
If you wanted to introduce a twenty-dollar generic into a treatment protocol, you would be asked for prospective, randomised evidence with pre-registered endpoints. Nobody would accept a retrospective accuracy figure on a convenience sample.
Diagnostic models routinely enter clinical workflows on exactly that evidence. Almost none have prospective validation in the setting where they will be deployed, against the clinicians they are meant to assist, on the population that will actually be exposed to them.
This is not a technical objection. It is the same objection epidemiology has been making about every other intervention for fifty years. There is no principled reason software should be exempt.
2. Retrospective accuracy is measured on a population that no longer exists
A model trained on one hospital’s historical records has learned that hospital’s referral patterns, its coding habits, its case mix, and the diagnostic thresholds its clinicians happened to use during the collection window.
Move it to a hospital with a different catchment population and the prevalence shifts. When prevalence shifts, sensitivity and specificity may hold, but positive predictive value does not — and positive predictive value is what the clinician reading the output actually experiences. A model that is “95% accurate” in a high-prevalence referral centre can generate mostly false positives in a screening setting, without a single line of code changing.
This is Bayes, not machine learning, and it is the most reliably ignored fact in the field.
3. Adoption is decided by workflow, not by AUC
Suppose the model is genuinely good. It still has to survive a twelve-minute consultation.
If the output requires four extra clicks, lives in a separate tab, or arrives after the decision point, clinicians will route around it. Not out of hostility — out of arithmetic. A tool that costs ninety seconds per patient costs a full clinic session per week, and it has to earn that back in a currency the clinician can feel.
The most common failure mode I have seen is not a bad model. It is a good model delivered at the wrong moment in the workflow, to a person who was not the one making that decision.
4. Silent failure — nobody is watching the drift
This is the one that worries me most, because it has no alarm attached.
Populations change. Referral pathways change. A new assay comes online with a different dynamic range. A coding guideline is updated. A model that was well calibrated at deployment degrades quietly, and the only signal is a slow drift in downstream outcomes that nobody is attributing to the software.
Drugs have pharmacovigilance. Devices have post-market surveillance. Diagnostic models, in most deployments, have nothing — no scheduled recalibration, no monitored performance metric, no named owner who is accountable for noticing.
The question that was never asked
“Can AI diagnose better than a doctor?” is a headline. It is not a question a health system can act on.
The question a health system can act on is narrower and much harder:
Does this tool, in this workflow, with these patients, improve an outcome we care about — and who is watching it when it stops?
Every clause matters. This tool, not the paper’s version. This workflow, with its real interruptions. These patients, at their real prevalence. An outcome, not a proxy for an outcome. And a named human being with the budget and the mandate to pull the plug.
What I would ask a vendor
If you are evaluating a diagnostic model, five questions separate the serious from the rest:
- What is the expected positive predictive value at our prevalence, not at yours?
- Where does the output appear in the clinical workflow, and what is the added time per encounter?
- What prospective validation exists, in a setting resembling ours, with pre-specified endpoints?
- What is the monitoring plan — which metric, at what cadence, reviewed by whom?
- What are the documented failure modes, and what does the tool do when it is uncertain?
A vendor who answers all five has thought about deployment. A vendor who answers the first with a benchmark number has thought about the demo.
Until diagnostic AI is validated the way we validate everything else that touches a patient — prospectively, in situ, with someone monitoring it afterwards — it will keep winning benchmarks and losing clinics.
I work with health organizations on exactly this evaluation: what to build, what to buy, and how to know whether it is still working six months later. Start with a call.