Case Study — The Differential
Building medical diagnosis models and the evaluation harness to stress-test them — perturbing cases, ablating evidence, injecting counterevidence, and measuring whether the model's conclusions shift in clinically coherent ways. Documented in public as a writing series.
At a glance
Large language models are getting good at naming the right disease. But accuracy on a case tells you nothing about how the answer was produced — the model may have reasoned from the supplied evidence, recognized a familiar case from training data, or anchored on an early hypothesis and filled in the expected ending. For clinical use, the difference matters more than the score. The question driving this project: when the evidence changes, does the model's conclusion change in a clinically coherent way?
Everything — experiment design, case construction, the perturbation protocol, analysis, and the public write-ups. This is my main ongoing project, built in public: every experiment becomes a post in The Differential series.
Ran a single-case pilot on a diagnosis-masked neuromuscular case from the literature. A frontier model ranked the published diagnosis first and predicted the pathological evidence that would have resolved the case.
Converted the structured timeline into an ordinary clinical paragraph — the model's answer barely changed, suggesting input structure alone is not a durable advantage on short vignettes.
Ablated individual supporting findings. Removing the characteristic muscle distribution lowered confidence appropriately; removing the strongest electrodiagnostic evidence did not — the model kept its diagnosis at high confidence and predicted the missing study would show the very pattern that had been removed.
Tested missing-vs-normal: the model correctly described an undocumented exam as undocumented rather than normal, but the distinction never showed up in its ranking.
Injected strong ALS counterevidence. The model switched its top diagnosis appropriately — but kept proposing the muscle biopsy from its original narrative as the most valuable next test.
Rebuilt the protocol around single-finding perturbations of one baseline case — characteristic muscle distribution, objective upper-motor-neuron signs, sensory abnormalities, a bulbar symptom, and a normalized lab value — then ran every condition three times in fresh conversations. Eighteen outputs total.
The leading diagnosis proved highly reproducible. Adding upper-motor-neuron findings reversed the top two diagnoses in all three repetitions, and the model preserved the conflicting myopathic evidence instead of erasing it. The characteristic muscle pattern reproducibly strengthened the myopathy; normalizing the creatine kinase reproducibly changed little.
Everything below the leading diagnosis was not reproducible. In the unchanged baseline, the rank and compatibility score assigned to ALS varied considerably between runs — so small evidence effects couldn't be separated from run-to-run variance, and a one-point score change can't be read as a change in diagnostic belief.
Found that the sensory variation exposed my benchmark rather than the model: it was proposing two coexisting processes, and a single 'compatibility' score conflated 'explains an important component' with 'explains the whole case alone'.
Traced the one output that was almost too stable: muscle biopsy proposed as the most informative missing evidence even where ALS had become the leading diagnosis — leaving open whether next-evidence choice is an information-gain estimate or the completion of a familiar diagnostic story.
Two experiments published as Parts 1 and 2 of the series. Repetition rewrote the conclusion: the leading diagnosis and the response to strong syndrome-level evidence were reproducible, while secondary ordering, score magnitudes, and calibration-related outputs were not — so reasoning quality is not one variable, and a single run can create a false impression of precision. The concrete outcome was product discovery: the original hypothesis (structure the vignette, label present/absent/normal/unknown, ask the model to rerank) is not a durable advantage, because the frontier model already does most of it well.
Two lessons so far. First, the benchmark is as much of the research problem as the model is — the sensory perturbation looked like a ranking failure until I realized my single compatibility score was asking two different questions at once, and a one-point ordinal change turned out to be within run-to-run noise. Second, and more expensive: my own product hypothesis didn't survive contact with the evidence. Structured timelines over a clean vignette don't beat a frontier model. What's left is the genuinely messy ground — years of fragmented records, contradictions between clinicians and tests, source provenance, one unifying disease versus several coexisting ones, analogous-case retrieval without diagnosis keywords, and the expected value of the next test. That's where the project goes next. This section updates as it progresses.