AI Got 13% of the Hardest Diagnoses Right. The Doctors Got 5.6%. | Edition 318

Ninety patients had already run out of medicine. Every one had been referred to the Undiagnosed Diseases Network — the NIH program that takes the cases nobody else can crack — and every one had spent a median of 7.6 years getting there. In the typical case, the symptoms had started at seven months old.

Researchers at Vanderbilt took those 90 cases, all of them eventually solved, and handed each patient intake summary to two AI models. ChatGPT-4o named the exact final diagnosis in 13.3% of them. Meta Llama 3.1 8B got 10.0%. The historical rate for those same cases going through clinical review was 5.6%.

The headline writes itself: AI more than doubled the doctors. That headline is also the least interesting thing in the paper. The winner missed 87% of the time — and the contest it won was never quite a fair fight.

What the study actually measured

The paper is Shyr and colleagues, published in JAMA Network Open in August 2025. The cases came from Vanderbilt University Medical Center arm of the Undiagnosed Diseases Network, diagnosed between November 2016 and April 2024. These are not textbook puzzles. They are patients who had already exhausted specialists, imaging and genetic testing without an answer.

Scoring ran on the Bond scale, a five-point system. A 5 meant the exact final diagnosis appeared somewhere in the model list of hypotheses. Two blinded physicians scored every output independently and a third resolved disagreements. Agreement between them was strong — a kappa of 0.93 for ChatGPT-4o and 0.88 for Llama — so the result is not an artifact of one generous grader.

A softer secondary measure counted differentials that named either the exact diagnosis or the right condition with the wrong genetic subtype. On that looser bar, ChatGPT-4o hit 23.3% and Llama 16.7%.

The economics are not close. ChatGPT-4o generated each differential for roughly three cents in about five seconds. The self-hosted Llama model cost nothing at all and took two minutes.

The caveat that is not in the paper

Research letters are short, and this one is no exception. The place where its central comparison gets stress-tested is the comment thread underneath it.

A physician reading the paper asked the obvious question: was the 5.6% clinical-review figure produced the same way as the AI figures? Same inputs, same time pressure, same blinded scoring?

The authors answered in November 2025, and the answer was no. The 5.6% reflects historical UDN performance. Clinicians did work from the same standardized intake summaries — but those diagnoses were made in the course of ordinary clinical care, which means they carried none of the timing constraints and none of the blinded adjudication that the AI outputs were held to.

That does not make the study wrong, and it is not a gotcha. The authors disclosed it plainly the moment they were asked, which is exactly how this is supposed to work. What it does mean is that the clean line — AI 13.3%, doctors 5.6% — is comparing two things that were measured differently. And that clean line is the one that travels.

The authors flag two further limits themselves. The models worked from intake summaries rather than complete workups. And every one of the 90 cases had eventually been solved. Nobody has tested these models on the cases that are still open, which is the population that actually needs help.

Why 13% is still a real number

An imperfect comparison does not make the result worthless, and it would be a mistake to read the caveat as a debunking.

Consider what three cents and five seconds buys. Nobody is proposing that a language model replace a clinical geneticist. The realistic question is narrower and much easier to answer: is it worth generating a free first-pass list of hypotheses before a human expert sits down with the file? On a case that has already consumed seven years, a differential that is helpful roughly one time in four — for the price of nothing — is difficult to argue against.

That is the version of this technology that survives scrutiny. Not a diagnosis. A starting point that costs almost nothing to produce and that a specialist can accept or discard in a minute.

The tools actually working in clinics are not chatbots

While the chatbot studies get the attention, the AI that is genuinely changing rare-disease diagnosis today looks nothing like ChatGPT. It does one narrow thing to one kind of data, and it has published sensitivity and specificity numbers attached to it.

In April 2026 the FDA cleared an ECG algorithm from Anumana — a company co-founded by Mayo Clinic and nference — to flag cardiac amyloidosis from a standard 12-lead ECG. It is the first clearance of its kind for that condition. In a validation study of more than 15,000 adults it caught 78.9% of cases while correctly clearing 91.2% of non-cases. Cardiac amyloidosis is exactly the kind of disease this helps with: serious, treatable, and routinely mistaken for ordinary heart failure. Mayo own newsroom profiled a 77-year-old patient in July 2026 whose case was caught this way after he arrived thinking he had pneumonia.

On the genetics side, Face2Gene has been in clinical use for years. Its underlying model, DeepGestalt, was trained on more than 17,000 photographs spanning 216 genetic syndromes and published in Nature Medicine in 2019. On 502 images it had never seen, its single best guess was right about 65% of the time, and the correct syndrome appeared in its top ten roughly 91% of the time.

Note the shape of that second number. Face2Gene is not built to answer a question. It is built to hand a clinician a short list worth checking. That is the same job the language models did in the Vanderbilt study, done better because the tool was built for one narrow task.

The ceiling is data — and it is not evenly distributed

Every one of these tools runs into the same wall. Rare diseases are rare. There are more than 7,000 of them, affecting an estimated 400 million people worldwide, but each individual condition may have only a few hundred documented cases, many of them in paper records that were never digitized. A pattern-matcher needs patterns, and for most of these conditions the patterns barely exist in machine-readable form.

Worse, the data that does exist is skewed. A 2017 study in Clinical Genetics tested Face2Gene on Down syndrome and found it recognized the condition in about 80% of Caucasian faces but only 36.8% of African ones. The researchers then retrained the system on African photographs and the gap narrowed considerably, and later evaluations of newer versions found ethnicity had much less effect. The specific number is a decade old and has improved.

The mechanism behind it has not. A model that learns from photographs inherits the demographics of whoever supplied the photographs. That is not a bug anyone can patch — it is a permanent property of the method, and it means every claim about one of these tools carries an unstated question: accurate for whom?

Where the money is going

On 20 July 2026, Anthropic opened the first themed call under its AI for Science program, aimed specifically at rare genetic disease. Selected researchers and early-stage biotechs get up to $50,000 over six months, split across a basic-research track and a drug-development track, with the Monarch Initiative named as an early partner.

Two details are worth noticing. The award is in Claude credits, not cash — compute rather than funding, which is a meaningfully different thing for a lab that needs to hire people. And the stated targets are curation, literature synthesis and shared terminology across fragmented datasets. In other words, the company is putting its money against the data problem rather than the model problem. That is the correct read of where the bottleneck sits.

What this changes for you

If you or someone close to you is stuck in a diagnostic odyssey, the honest summary is this. A chatbot is a hypothesis generator, not an answer. Used well, it produces a short list to bring to a doctor, and the evidence says that list is worth something more often than you might expect. Used badly, it produces a confident-sounding condition name that anchors everyone in the room — including the clinician — to the wrong idea.

The difference between those two outcomes is almost entirely about how the output is framed when it reaches a human. Bring a list of questions, not a conclusion.

And the broader lesson generalizes well past medicine. When you next see a number showing AI beating professionals at something, the useful question is not whether the number is real. It usually is. The useful question is whether both sides of the comparison were measured the same way — because in this case, the researchers themselves said they were not, and you had to read the comments to find out.