Series
Building and stress-testing clinical AI in public.
A build-in-public series on Diagnostic Odyssey — work aimed at patients whose diagnosis has taken years. The name is a double entendre: the differential diagnosis, the ranked list of candidate diseases, and the derivative, how much the output should move when the input changes. Each post is a real experiment designed to kill an assumption before I build on it. So far that has meant perturbing cases to test evidence sensitivity, measuring whether a frontier model can turn clinical narrative into a structure worth reasoning over, and now training on pancreatic CT where a wrong answer can't be argued into a disagreement about definitions. The experiments have retired more of my hypotheses than they have confirmed, which is the point.
Now
The audit came back: the pooled collapse was five folds scoring on disjoint scales, not a weak model. Re-scoring the saved checkpoints across 22 aggregation rules found no case-level signal in the spatial head either. Running the cheapest remaining test — can this pipeline memorise ten cases? If it cannot, the defect is a bug rather than a data-scale problem, and no amount of new architecture would have helped. The 30-case locked test set stays untouched.
A frontier model ranked the correct diagnosis first on a masked neuromuscular case. Then I started removing evidence — and its confidence didn't move the way it should. Part 1 of The Differential.
Jul 29, 2026 · 4 min read
I repeated every perturbation condition three times. The leading diagnosis was highly reproducible; the ordering and scoring below it was not — which killed my original product hypothesis. Part 2 of The Differential.
Jul 29, 2026 · 9 min read
50 manual extraction runs across two clinical NLP benchmarks. Source grounding was 100% and the unsupported-claim rate was zero — but the evidence units, entity boundaries, and temporal relations fell apart. Part 3 of The Differential.
Aug 1, 2026 · 12 min read
The text benchmark left me arguing about what counts as a finding. So I moved to pancreatic CT, where a wrong answer is just wrong. The first run scored 0.52 AUROC — and the way it failed was worth more than the score. Part 4 of The Differential.
Aug 4, 2026 · 9 min read
My cross-validation reported 0.61. The pooled out-of-fold number was 0.52. Both were correct, and the gap had nothing to do with the model — it was five folds scoring on five incompatible scales. Part 5 of The Differential.
Aug 6, 2026 · 9 min read