From Case Reports to CT Scans

The model did not hallucinate. The benchmark still said it was wrong.

That was where the previous experiment ended. GPT produced grounded clinical findings with valid source spans and no unsupported claims, but the content score collapsed because the model and the annotations often described the same narrative at different levels of granularity.

I could have spent the next few weeks repairing the matcher and completing the remaining locked cases. I still intend to return to that work. But before scaling it, I needed to answer a more basic question: was I testing a model, or mostly testing my definition of a finding?

I wanted the next experiment to fail more clearly.

That did not immediately mean "build a pancreatic cancer model." The decision came from working backward from the kind of healthcare product I actually wanted to build.

Starting with the product, not the model

Diagnostic Odyssey began as a system for people with long, unresolved medical histories. It would reconstruct timelines, normalize findings, retrieve similar cases, expose contradictions, and keep every conclusion tied to evidence.

The extraction benchmark showed why that problem matters. It also showed how many unproven layers sat between a clinical record and a useful recommendation. Before testing diagnostic retrieval, I had to validate extraction. Before validating extraction, I had to define equivalence between model outputs and annotations. Every layer created another place where a plausible score could conceal the actual failure.

So I wrote down stricter constraints for the next project.

It needed public data and labels. It needed a prediction that could be tested on held-out patients. The output had to be visible, not merely persuasive. Most importantly, it had to point toward a tool that a real physician could use—not another demonstration that an AI model could recognize a curated benchmark.

That narrowed the product idea to a specific workflow: use routine contrast-enhanced abdominal CT to identify pancreatic ductal adenocarcinoma, or subtle signals that justify specialist review, and route the case toward further workup.

The aim was not to let a model diagnose pancreatic cancer on its own. It was to test whether a model could act as a second-pass signal detector on imaging that already exists.

That was how I arrived at pancreatic cancer AI. The clinical target came first; the architecture came later.

Finding data I could actually build with

The project became feasible when I found PANORAMA, a pancreatic cancer CT dataset released for research.

Its development cohort contains 2,238 portal-venous contrast CT studies: 676 PDAC cases and 1,562 non-PDAC cases. The collection combines 482 manual masks with 1,756 automatically generated or weak masks and includes cases originating from MSD and NIH datasets. It was large enough to represent a real modeling problem and structured enough to support both patient-level classification and spatial evaluation.

That combination mattered.

A case label could tell me whether the model separated PDAC from non-PDAC scans. A lesion mask could tell me whether the model was looking in the correct place. The same experiment could therefore be evaluated at two levels:

Patient level: Does the model assign higher malignancy scores to positive cases?

Lesion level: Does its spatial prediction overlap the annotated lesion?

This was already cleaner than the text benchmark. AUROC could still mislead. Masks could still be noisy. Dataset shift would still matter. But a false positive could no longer be explained away as a disagreement about phrasing.

Turning the full idea into a runnable pilot

The long-term pipeline I had in mind contained three stages:

1. localize or segment the pancreas;
2. detect suspicious lesions or abnormal regions;
3. aggregate that spatial evidence into a patient-level malignancy score.

The final system would eventually need more than a single AUROC. I wanted sensitivity and specificity at a screening-oriented threshold, performance by tumor size or stage, false positives per scan, calibration, external validation, and ultimately a comparison between radiologists alone and radiologists using the model.

But the first run had a smaller purpose: determine whether the current training and scoring pipeline contained enough signal to justify another iteration.

I built a manifest of 198 cases. Before training, I separated 30 cases as a locked test set and did not use them for model selection. That left 168 development cases.

On those 168 cases, I ran five-fold cross-validation. Each patient received an out-of-fold prediction from a model that had not trained on that patient. The run configuration was fixed:

Setting	Value
Manifest	198 cases
Development set	168 cases
Locked test set	30 cases
Cross-validation	5 folds
Maximum epochs	25
Early-stopping patience	6
Batch size	1
Random seed	20260803
Output run	runs_retention_clean

The batch size of one was not an aesthetic choice. Volumetric CT training is memory-intensive, and the experiment had to fit within the GPU resources I could access. During the broader project, Colab availability itself became part of the workflow: sessions disconnected, GPU access hit usage limits, and a completed run had to be preserved carefully rather than treated as infinitely reproducible compute.

After the five folds completed, I combined the held-out development predictions into one out-of-fold table and evaluated the model once across all 168 cases. I also evaluated the spatial output against the available gold lesion masks.

That produced the first real result of the pancreatic cancer project.

The first result

Metric	Result
Raw OOF AUROC	0.5244
Raw OOF AUPRC	0.5238
Brier score	0.3126
Specificity at ≥90% sensitivity	0.1667
Gold lesion Dice	0.1086
Gold lesion detection sensitivity	0.7439

At the patient level, the answer was disappointing. The pooled out-of-fold AUROC was 0.5244—barely above chance. AUPRC was similarly weak at 0.5238. At an operating point chosen to retain at least 90% sensitivity, specificity fell to 16.7%.

That is not a deployable screening system. It would send far too many negative cases for additional review.

The spatial metrics told a more complicated story. Gold-lesion Dice was only 0.1086, so the model was not outlining lesions accurately. Yet lesion-detection sensitivity was 0.7439. It placed some predicted signal on roughly three quarters of the annotated lesions, even though the overlap was generally too small, too diffuse, or otherwise too inaccurate to produce a strong Dice score.

The model had not learned to segment the lesion well. But it may have learned a weak localization signal.

That gap—74.4% detection sensitivity versus 10.9% Dice—was more informative than a single headline score. It separated "the model sees nothing" from "the model often notices the area but represents it badly." Those failures lead to different next experiments.

Then the folds disagreed with the pooled result

When I calculated performance within each validation fold, the model appeared better:

Fold-wise metric	Mean ± standard deviation
AUROC	0.6109 ± 0.0447
AUPRC	0.6148 ± 0.0559

Reporting only these averages would make the pilot sound considerably more successful. It would also miss the central problem.

The predicted probabilities were compressed into an extremely narrow and unusually high range. In one fold, negative cases ranged from approximately 0.936 to 0.957. Positive cases occupied almost the same range. Other folds showed the same pattern around 0.94 to 0.95.

Within an individual fold, tiny score differences could preserve enough ordering to produce an AUROC above 0.60. Once predictions from all five fold-specific models were pooled, their score scales were not coherent enough to preserve that ranking. The raw OOF AUROC fell to 0.5244, and the Brier score of 0.3126 exposed how poorly the numerical confidence matched the outcomes.

This was not just a calibration footnote. A physician-facing system eventually needs a threshold. If one fold's model treats 0.944 as relatively low and another treats it as relatively high, a global threshold has no stable meaning.

The honest conclusion was therefore not "the model achieved 0.61 AUROC."

It was:

The first pipeline learned a weak spatial and within-fold ranking signal, but failed to produce stable patient-level probabilities across folds.

A better failure than the previous one

The result was bad, but it was useful in a way the clinical extraction score had not been.

In the earlier benchmark, poor content F1 opened an argument. Was the finding missing, paraphrased, split into smaller claims, or annotated at a different granularity?

Here, specificity of 0.1667 was simply poor. Dice of 0.1086 was simply poor. Predictions compressed around 0.95 were plainly not calibrated. The uncertainty moved from "what does correctness mean?" to "which component is producing the failure?"

That second question is much easier to test.

The lesion-detection/Dice gap tells me to inspect the actual masks: are predictions oversized, fragmented, shifted into nearby anatomy, or concentrated on only part of the lesion?

The fold-wise/pooled AUROC gap tells me to audit score construction and cross-fold comparability. I need to check patient grouping, fold assignments, label alignment, checkpoint loading, and the conversion from voxel-level output to a patient-level score.

The narrow probability range tells me not to treat calibration as cosmetic. It may also indicate that the current aggregation rule is saturating and discarding the information that the spatial model learned.

What I will test next

The next run should not merely add epochs. Twenty-five epochs and five completed folds were enough to reveal a structural problem. Training longer would make the same pipeline more expensive without proving that the scoring logic is sound.

The next iteration has four priorities.

First, I will audit the evaluation pipeline end to end, from the manifest and patient-level split through checkpoint loading and final OOF assembly.

Second, I will create visual error groups for true positives, false positives, missed lesions, and detected-but-low-Dice lesions. The spatial metrics need an anatomical explanation.

Third, I will compare alternative ways of converting spatial predictions into a case score instead of trusting one aggregation rule. Maximum activation, top-k aggregation, lesion volume, and a dedicated classification signal make different assumptions and should be evaluated separately.

Fourth, I will fit and assess calibration only on held-out predictions. A threshold cannot be selected responsibly from scores that are incompatible across folds.

The 30-case locked test set stays locked until those choices are frozen. It is not there to help me decide which version looks best. It is there to test the final decision once.

The real transition

So the move from clinical text to pancreatic CT was not: the first project was hard, so I chose a different AI problem.

It was: the first project taught me that I needed a tighter loop between a model's output, its ground truth, and the next engineering decision.

Diagnostic Odyssey asked whether AI could make a fragmented medical journey legible. The pancreatic cancer project asks a narrower question inside the same mission: can AI extract a visible, measurable signal from routine clinical data early enough to justify a second look?

I have not answered that question yet.

What I have now is a complete first training run, a locked test set that remains untouched, and a failure divided into specific parts: localization, overlap, aggregation, calibration, and cross-fold stability.

That is a much stronger starting point than a model that merely looks convincing.