The Model Wasn't the Problem. The Scoreboard Was.

The previous post ended with a failure split into parts: localization, overlap, aggregation, calibration, and cross-fold stability. The obvious next move was to start fixing them.

I did something else first. Before debugging any component, I audited whether the measurement itself could be trusted.

That turned out to be the whole story.

Two numbers, both correct

The pancreatic CT run produced these:

mean of the five per-fold AUROCs	0.6109 ± 0.0447
pooled out-of-fold AUROC	0.5244

Reporting the first number would have been easy and defensible. Five-fold cross-validation, mean and standard deviation, the format every paper uses. It reads as a weak but real result.

The second number is the one that matters, because it is the only one that describes a single model scoring a population of patients.

Neither is a mistake. The gap between them is the finding.

First, reproduce everything

I rebuilt the out-of-fold table from the rawest artifact the run saved and recomputed every recorded metric.

All nine reproduced with a delta of exactly zero: AUROC, AUPRC, Brier, threshold, sensitivity, specificity, gold-lesion count, Dice, detection sensitivity. Per-fold AUROC and AUPRC likewise, across all five folds.

The structural checks were clean too. 168 rows, 168 unique case identifiers, no duplicates, fold counts matching each fold's recorded validation size, binary labels, a perfectly balanced 84/84 split.

This matters more than it sounds. It rules out the entire category of explanation I would otherwise have spent a week on — mislabeled cases, leaked patients, a mis-assembled table, a wrong checkpoint loaded somewhere. None of that was happening. The numbers were an honest description of what the model did.

Which meant the problem was in what the model did.

The folds were not on the same scale

Here is the actual cause, and it is almost embarrassingly simple once you see it.

Each fold's model emitted probabilities in its own narrow band. Fold 0 put every case, positive and negative, between 0.936 and 0.957. Fold 4 put every case between 0.143 and 0.207. The other three sat at 0.51, 0.40 and 0.36.

Every one of the ten possible fold pairs had non-overlapping score ranges. Not partially overlapping. Disjoint.

So when the five folds are pooled into one table and ranked, the ranking is dominated by which fold scored the case. Fold 0's healthy patients outrank fold 4's cancer patients, by a wide margin, because fold 0's model happened to settle around 0.94 and fold 4's around 0.17.

Three measurements pin this down:

• 99.87% of the total variance in the predicted probability was between folds rather than within them. The largest spread inside any single fold was 0.0635, while the fold centres spanned 0.14 to 0.96.
• Fold identity alone, ignoring the model entirely, gave an AUROC of 0.5096. The offsets carried essentially no information about the label. They were pure nuisance.
• Removing the offset — ranking each case only against others in its own fold — brought the pooled AUROC to 0.6101, almost exactly the mean fold value of 0.6109.

That last number is a diagnostic, not a result. A future patient does not arrive with a fold label attached, so no deployable system can subtract these offsets. It tells me where the problem is. It does not tell me I have a model.

Why the scales drifted

The checkpoint for each fold was chosen by maximising validation AUROC plus a small Dice term, over epochs.

With 33 or 34 validation cases, about half of them positive, the standard error on validation AUROC is roughly 0.10. Taking the maximum over epochs of a quantity that noisy does not select the best epoch. It selects the luckiest one.

The chosen epochs were 1, 3, 1, 5, 7. Two folds froze at the first epoch.

Fold 0's training log shows what that looks like from the inside. Training loss fell steadily from 3.35 to 1.96 across seven epochs. Validation AUROC over the same epochs went 0.63, 0.49, 0.62, 0.56, 0.55, 0.45, 0.63 — no trend, just noise. And the threshold that achieves 90% sensitivity moved from 0.094 to 0.943 and back, within a single fold, epoch to epoch.

Nothing in the loss constrains where the probabilities sit. Only their ordering within a batch is trained. So the absolute output level was free to wander, each fold froze at a different point in that wander, and the five frozen levels had no reason to agree.

A model that scored cases while detecting nothing

The audit turned up a second defect, unrelated but worse.

The network has two heads. The lesion decoder produces a voxel map. The case score comes from a separate small network on max-pooled encoder features — a path that never touches the decoder.

Fold 3 reported a case-level AUROC of 0.588, comfortably mid-pack. It also produced zero suprathreshold lesion voxels. Not few. None, on any case, positive or negative. Its gold-lesion Dice and detection sensitivity were both exactly 0.

A model that assigns confident malignancy scores while pointing at nothing is not a detector. It is a number generator that happens to be graded on the right axis.

What this did to my earlier conclusion

In the previous post I wrote that the model had learned "a weak spatial and within-fold ranking signal." I no longer believe the first half of that, and I am much less sure about the second.

The 0.611 is the maximum of a noisy selection procedure. That biases it upward by an unknown amount. It is not a measurement of what the model learned; it is a measurement of how many chances the selection had to get lucky.

The cheap experiment before the expensive one

The natural response was to retrain with a better checkpoint policy. That is roughly seven hours of GPU time.

Instead I noticed that the five trained checkpoints still existed, and that the question I actually wanted to answer — is there case-level signal in the spatial head that the classification head was throwing away? — needs no training at all.

So: reload each checkpoint, re-run inference on its own validation cases, save the voxel maps the original run never saved, and compute every reasonable way of turning a voxel map into a case score. Maximum activation, top-k means at four values of k, four percentiles, thresholded volume and mass at three thresholds, largest connected component by volume and by mass, component count, intensity times extent. Twenty-two candidates.

168 forward passes. About twenty minutes.

A gate ran first: the re-scored classification probabilities had to reproduce the recorded ones, or the inference path differed from the original run and nothing downstream would mean anything. They matched.

The answer was no

The best of the twenty-two was 0.506, with a 95% confidence interval of [0.415, 0.593].

That is worse than the collapsed classification head, and its interval comfortably contains chance.

It is tempting to read a null result like this as "we haven't found the right aggregation yet." The numbers close that door. The twenty-two candidates are near-duplicates of one another — a maximum, a 99.99th percentile and a top-10 mean are largely the same statistic — so the effective number of independent tests is close to one. A single pure-noise score at this sample size has a median AUROC of 0.501. We got 0.506.

For comparison, if the twenty-two had genuinely been independent, selection alone would have pushed the best of them to around 0.585 on pure noise. We did not reach the ceiling that randomness would have produced.

And one more thing fell out. The fold-scale pathology was not specific to the classification head. Even for the best spatial aggregation, nine of the ten fold pairs still had disjoint ranges. It is not one head that is miscalibrated. The whole network's output level is a property of which fold trained it, not of the image in front of it.

A hypothesis I tested and threw away

Somewhere in this I convinced myself that the lesion masks might not be lesion masks — that they were really whole-pancreas segmentations, which would explain a model that fires on 82% of healthy cases and still overlaps most tumours.

It is a tidy story and it is wrong.

The gold masks have a median of 4,266 voxels and range from 88 to 93,739. At roughly millimetre resolution that spans about a 1 cm to a 5 cm lesion, which is what pancreatic tumours actually look like. And whether a lesion was detected is not explained by how big it was — the detected and missed groups are statistically indistinguishable in size, and the detected ones are if anything slightly smaller.

What does explain detection is the fold: 0.88, 0.94, 0.94, 0.00, 1.00. Again a per-fold global property, not anything about the case.

I am recording the dead hypothesis because checking it is the only reason I know it is dead.

Where this leaves the project

No case-level PDAC signal has been demonstrated by this pipeline, from either head, under any scoring rule I tried.

That sentence is deliberately narrow. It is a statement about a pipeline, not about the task, and not about the data. There is a version of this result that means "this is hard at 198 cases" and a version that means "something in the code has been broken the whole time," and I cannot yet tell them apart.

Which is the next test, and it is cheaper than everything above.

Take ten cases. Five with tumours, five without. Train until the model memorises them, and measure Dice on those same ten. No held-out set, no statistics, no confidence intervals — deliberate overfitting.

A model that cannot memorise ten examples has a bug. A model that can, and still fails on 168, has a data or capacity problem. Those two conclusions point in completely opposite directions, and twenty minutes separates them.

I am running it now.

Two things I am changing regardless

Save what you would need to debug. The original run could not be diagnosed from its own outputs. Voxel predictions were never written, so the aggregation comparison — the single most informative analysis available — was impossible until I re-ran inference to generate them. A run that cannot be interrogated afterwards has to be re-run to be understood, and that is a tax you pay every time.

Pre-register the scoring rule. The reason this audit was possible is that the metrics were saved. The reason it was necessary is that the scoring rule was decided after seeing the outputs. Whatever wins the next comparison gets frozen before the next training run, not after.

The 30-case locked test set has still never been opened. Not for calibration, not for threshold selection, not for choosing which examples to show you. It gets opened once, after the scoring policy is frozen — and if the overfit probe comes back the way I am half expecting, that is still a long way off.

---

The audit harness, the reports, and both follow-up notebooks are on GitHub at cv-audit. It runs on any k-fold run's out-of-fold table, and ships with a synthetic example that reproduces the failure described here, so you can see what a fold-scale collapse looks like without needing the medical data.