Getting the Diagnosis Right Is Not the Same as Reasoning Reliably

Large language models are becoming surprisingly capable at clinical case reasoning. But when a model reaches the correct diagnosis, how do we know whether it actually used the evidence in a reliable way?

I ran a small exploratory test using a diagnosis-masked neuromuscular case from the medical literature.

The case involved an older man with slowly progressive leg weakness, a prior diagnosis of motor neuron disease, modest creatine kinase elevation, and later findings suggesting a primary muscle disorder.

When asked to rank ten diagnostic candidates, GPT placed the published diagnosis first. It also identified several of the most relevant clinical features and predicted the type of pathological evidence that would likely have resolved the case.

The answer was impressively accurate.

But accuracy on one case does not tell us how the answer was produced. The model may have reasoned from the supplied evidence. It may also have recognized a familiar case pattern from its training data, or anchored on an early hypothesis and filled in the expected ending.

To examine this, I created several controlled variations of the same case.

Removing the Input Structure

The original case was represented as a structured timeline. I converted it into an ordinary clinical paragraph containing the same information.

The result changed very little.

The model reconstructed the temporal sequence, selected the same leading diagnosis, and proposed nearly the same decisive evidence.

This suggests that, at least for a short and carefully written vignette, converting the case into a structured timeline does not automatically create an advantage over a frontier language model.

Structure may still matter for long records, provenance, contradictions, and user correction. But structure alone is unlikely to be a durable model advantage.

Removing Important Evidence

I then removed individual findings that had supported the model's original answer.

When the characteristic muscle distribution was removed, the model became somewhat less confident and elevated broader categories of muscle disease. It also correctly identified the missing distribution of weakness as an important unresolved question.

That was a reasonable response.

The more interesting result appeared when I removed the repeat electrodiagnostic evidence supporting a muscle disorder.

The competing motor neuron diagnoses moved upward, but the model continued to rank its original diagnosis first with high confidence. It also predicted that the missing electrodiagnostic study would probably reveal the same myopathic pattern that had been removed.

This does not necessarily make the diagnosis wrong. The remaining clinical pattern may still support it.

However, the model's confidence did not decrease in the way I expected after removing one of its strongest supporting findings.

Missing Is Not Normal

In another version, I removed the statement that the patient had no upper-extremity symptoms.

The model handled this distinction well. It explicitly stated that the upper-extremity examination was not documented, rather than treating it as normal.

This is an important capability. In clinical records, an unmentioned finding is not equivalent to a confirmed absence.

However, recognizing uncertainty in the explanation did not meaningfully change the final ranking.

The model could describe the missingness correctly without clearly reflecting it in its diagnostic confidence.

Adding Strong Counterevidence

Finally, I created a counterfactual version containing findings strongly favoring amyotrophic lateral sclerosis: clinical spread into multiple regions, upper and lower motor neuron signs, bulbar involvement, widespread fasciculations, and multiregional active and chronic denervation.

This time, the model changed its answer appropriately and ranked ALS first.

The model was therefore not completely anchored to its original diagnosis. It responded well to a large, syndrome-level change.

Still, it continued to emphasize a muscle biopsy as the most important next piece of evidence, even though the counterfactual case already contained extensive evidence for a diffuse motor neuron disorder.

This may indicate that changing the final ranking is easier than fully recalculating which missing information is most valuable.

What This Pilot Suggests

This is a single-case exploratory experiment, not a benchmark and not evidence about clinical safety.

Still, it raised several useful questions.

A model can reach the right diagnosis while remaining inconsistent about how much individual findings should affect its confidence.

It can distinguish "not documented" from "confirmed absent" in its explanation without clearly incorporating that distinction into its ranking.

It may react appropriately to large counterfactual changes while remaining relatively insensitive to smaller evidence ablations.

And it may continue proposing evidence associated with an initial diagnostic narrative even after the case has substantially changed.

The important evaluation question may therefore be broader than diagnostic accuracy:

Does the model's conclusion change in a clinically coherent way when the evidence changes?

A stronger evaluation would test many cases across multiple models, systematically add and remove findings, repeat each run, and assess whether the direction and magnitude of the resulting changes are clinically appropriate.

The goal would not simply be to determine whether a model can name the correct disease.

It would be to determine whether its ranking, uncertainty, contradictions, and proposed next information remain stable and evidence-sensitive under controlled perturbations.

That is the next question I plan to test.