The Model Didn't Hallucinate. It Still Failed.

What 50 manual clinical extraction runs taught me about the real bottleneck in Diagnostic Odyssey

In the first two parts of The Differential, I tested whether the core of Diagnostic Odyssey should be a better diagnostic reasoning model.

The original vision was straightforward:

A patient with years of fragmented medical records uploads everything. The system reconstructs the diagnostic journey, identifies evidence that has been overlooked, retrieves analogous confirmed cases, and shows what information might distinguish the remaining possibilities.

Before building that system, I needed to understand what current frontier models could already do.

My earlier experiments suggested that GPT could reason surprisingly well over clean clinical vignettes. It responded consistently to large changes in evidence, such as adding upper motor neuron signs to a neuromuscular case. It was less stable when estimating the importance of smaller findings or ranking closely related alternatives, but clean-vignette diagnosis did not look like a strong product wedge.

So I moved one step backward in the pipeline.

Instead of asking whether GPT could diagnose a well-organized case, I asked:

Can GPT reliably transform an unstructured clinical narrative into the evidence structure needed to reason over a patient's longitudinal history?

The answer was more interesting than I expected.

GPT almost never invented its evidence.

It still failed to structure it.

The experiment

I evaluated GPT on two public clinical NLP benchmarks.

The first, CaseReportBench, contains published clinical case narratives with dense annotations of clinically relevant findings.

The second, E3C, contains clinical events, entities, factuality attributes, temporal information, and temporal relations.

For each case, I asked GPT to extract structured clinical information while preserving exact source evidence.

Depending on the dataset, the output included:

• clinical findings or events
• exact source quotations
• character offsets into the original narrative
• whether a finding was present, absent, or uncertain
• whether it was historical or current
• the experiencer of the event
• temporal relations between events

The model had to return valid JSON only.

I separated development and test cases in advance. The final evaluation consisted of:

• 20 locked CaseReportBench cases
• 20 locked E3C documents

Before the locked test, I also ran five development examples from each dataset.

That produced 50 manual runs in total.

Every run was performed in a new ChatGPT conversation with the same prompt version and the same model selection. Prompts, outputs, splits, scorer configuration, and file hashes were recorded in an experiment manifest.

This was intentionally tedious.

I wanted a baseline that could not be improved after the fact by quietly changing prompts, repairing outputs, or selecting favorable examples.

The operational result

All 40 locked-test outputs were present.

Thirty-eight were valid JSON.

Two were not.

One response inserted unescaped quotation marks inside a JSON string:

"concept_text": "Repetitive response consisting of the word "nothing""

The other used an illegal JSON escape sequence:

"source_quote": "a low fibrinogen \(152mg/dl)"

Neither failure came from a missing file or an incomplete copy. Both were preserved as raw model-output syntax failures.

The resulting JSON validity was:

38 / 40, or 95%

That is good enough for a chat interface.

It is not good enough for a production extraction pipeline that expects every output to be machine-readable.

But formatting was not the most important result.

The model grounded almost everything correctly

The first thing I expected to find was hallucination.

I assumed that, when asked to extract hundreds of findings from dense clinical narratives, the model would occasionally fabricate a quotation, paraphrase a sentence while presenting it as verbatim evidence, or attach a finding to text that did not support it.

That did not happen in the parseable locked-test outputs.

For CaseReportBench:

• 1,070 claims were generated
• exact source-quote grounding was 100%
• unsupported-claim rate was 0%
• submitted-offset accuracy was approximately 95.9%

For E3C:

• 1,459 entities were generated
• exact source-quote grounding was 100%
• unsupported-claim rate was 0%
• submitted-offset accuracy was approximately 99.7%

The offset errors were also small.

Across the complete experiment, 49 submitted offsets were incorrect. Every corresponding quote still appeared in the source text, and all 49 could be uniquely located. Twenty-five were off by one character. The remaining 24 were off by two to five characters. None differed by more than five.

This distinction matters.

The model was not making up clinical evidence and then hiding the fabrication behind a citation.

It was usually pointing to real text.

The primary failure was elsewhere.

The evidence was real, but the units were unstable

On the locked CaseReportBench cases, GPT generated 1,070 claims.

The corresponding narratives contained 384 gold claims in total.

The model therefore produced approximately:

2.79 predicted claims for every gold claim

This did not mean that 686 claims were hallucinated. The quote validator found that all of them were grounded in the narrative.

Instead, GPT frequently represented the same source evidence at a different level of granularity than the annotation scheme.

Consider a sentence such as:

Cerebrospinal fluid cell counts, protein, glucose, chloride, adenosine deaminase, and lactate dehydrogenase levels were within normal ranges.

A model can reasonably represent this as one finding:

• cerebrospinal fluid studies were normal

It can also represent it as several findings:

• CSF cell count was normal
• CSF protein was normal
• CSF glucose was normal
• CSF chloride was normal
• CSF adenosine deaminase was normal
• CSF lactate dehydrogenase was normal

Every claim is supported by the source.

But they are not the same evidence representation.

Across the locked CaseReportBench set, I observed:

• 126 fragmentation cases
• 121 merging cases
• 896 grounded predictions that did not match a gold evidence unit
• 104 gold claims that remained unmatched

The relaxed overlap span F1 was only 0.199.

Grounded-content F1 was 0.206.

In other words:

GPT was good at finding relevant clinical text, but poor at deciding what should count as one stable clinical fact.

That is a very different failure from hallucination.

Entity boundaries were inconsistent too

The same pattern appeared in E3C.

The strict entity-span F1 was 0.140.

When the scorer allowed partially overlapping spans, F1 increased to 0.407.

A model might extract:

low-grade fever for three days

while the gold annotation contains only:

fever

The model has identified the correct event, but it has selected a different boundary.

Some boundary differences are clinically harmless. A longer phrase may even preserve useful detail.

Other differences matter because they alter the identity of the event. A span may combine a symptom, its duration, its anatomical site, and a temporal expression into one object. Another run may split the same text into four objects.

Once event identity becomes unstable, every downstream operation becomes harder:

• deduplication
• longitudinal tracking
• contradiction detection
• state updates
• treatment-response linking
• temporal relation extraction
• retrieval of analogous trajectories

The problem is not merely that an annotation benchmark prefers one span boundary over another.

The deeper issue is that the model does not consistently decide what object it is tracking.

Time was the weakest part

A longitudinal diagnostic system cannot operate on a flat list of findings.

The difference between these two representations is critical:

Flat representation

• hand weakness
• leg weakness
• dysphagia
• immunotherapy

Longitudinal representation

• right-hand weakness appeared first
• bilateral leg weakness developed two years later
• immunotherapy was started under a suspected inflammatory diagnosis
• no objective improvement followed
• dysphagia appeared later in the course

The second representation contains a disease trajectory.

That trajectory is often more diagnostically informative than the presence of any single symptom.

GPT struggled to reconstruct it.

On the locked E3C test set:

• overlap entity F1 was 0.407
• overlap-matched assertion macro-F1 was 0.637
• overlap-matched temporality macro-F1 was 0.216
• mapped-endpoint temporal-relation F1 was 0.031
• relation endpoint coverage was approximately 52.6%

Only 240 evaluated relations had both endpoints mapped successfully.

Another 435 had one unmapped endpoint, and 195 had neither endpoint mapped.

When the correct endpoints were available, the model was not completely incapable of classifying the relationship. Relation-type accuracy on the small subset with mapped endpoints was approximately 64.7%.

But a temporal relation cannot be correct when the system has not established stable identities for the two events it is trying to connect.

The graph failed upstream.

This was not the result I expected

I began with a common assumption:

The central risk in applying GPT to medical records is that it will hallucinate unsupported clinical claims.

That risk still matters. This experiment was limited to two datasets, one model selection, one run per case, and carefully written extraction prompts. It was not a clinical validation study.

But within this experiment, hallucination was not the dominant failure.

The model's source grounding was excellent.

Its evidence structure was not.

A more accurate summary is:

GPT could find the facts, but it could not consistently maintain their identity, granularity, and temporal state.

This changes where I think the engineering work may belong.

A new system may not need to be better than GPT at locating clinical statements.

It may need to be better at turning those statements into a stable longitudinal patient model.

What the 50 manual runs actually bought me

Halfway through this process, I wondered whether I was simply doing benchmark work for its own sake.

That concern was legitimate.

A higher span F1 does not automatically produce a better medical product. Annotation conventions are not the same as clinical truth. A benchmark may treat a laboratory panel as one finding while a product may reasonably expose each analyte separately.

Still, the experiment eliminated several expensive assumptions.

1. I probably do not need to begin by training a medical extraction model

The frontier model already demonstrated strong verbatim grounding.

Building a custom model merely to find clinical statements would target a capability that may already be good enough for an initial product.

2. Clean-vignette diagnosis is probably not the moat

My earlier experiments showed that GPT already responds well to large, clinically meaningful changes in a clean case.

Attempting to build a system whose main value is "better diagnosis from a concise vignette" would place the product directly against rapidly improving general-purpose models.

3. The unresolved problem is longitudinal consolidation

The strongest remaining candidate is not fact retrieval or isolated diagnostic reasoning.

It is maintaining a coherent patient state across:

• years of notes
• multiple specialties
• repeated copied histories
• evolving symptoms
• conflicting observations
• diagnostic hypotheses
• tests ordered under those hypotheses
• treatments and subsequent outcomes

That is much closer to the original Diagnostic Odyssey vision.

4. I now have a reproducible baseline

The manual work also produced infrastructure that can be reused:

• dataset adapters
• a canonical schema
• source-span validation
• deterministic development and test splits
• a frozen scorer
• prompt and output hashes
• experiment ledgers
• locked-test reporting
• granular error analysis

The purpose of this infrastructure is not to make the repository look rigorous.

It is to prevent me from convincing myself that a more complicated system is better simply because its output looks cleaner.

But an evidence graph is not the product

It would be easy to take the benchmark results and immediately build a large evidence graph system.

That would be a mistake.

Patients do not want entity boundaries.

Clinicians do not want a graph because it has a higher temporal-relation F1.

The product value has to appear in an output that helps someone understand a difficult diagnostic journey.

A useful Diagnostic Odyssey result might look more like this:

Important changes over time

• Right-hand weakness was documented before lower-extremity weakness.
• Dysphagia was absent in earlier records and appeared later.
• Functional decline continued despite immunotherapy.

Repeatedly supported findings

• Sensory function was preserved across multiple examinations.
• Creatine kinase remained mildly elevated.
• More than one electromyography report described myopathic changes.

Conflicting evidence

• One examination documented an upper motor neuron sign.
• Two later examinations did not reproduce it.
• A rehabilitation note described sensory symptoms not reflected in the neurology history.

Evidence that weakened prior hypotheses

• A treatment given under the presumed diagnosis did not produce the expected objective response.
• Repeated testing did not show a finding usually expected under that hypothesis.

Unresolved discriminating questions

• Was weakness distributed across the specific muscle groups that distinguish the leading alternatives?
• Was the previous biopsy taken from an informative muscle and at an informative stage?
• Did the prior genetic panel cover the genes relevant to the current phenotype?

Every statement should link back to the exact source record.

That output—not the graph underneath it—is the product.

The next experiment is product-facing

My next step is not another 40-case extraction benchmark.

It is a direct comparison between two ways of producing a source-linked diagnostic brief.

A. Raw GPT

Give the model the longitudinal record and ask it to produce the brief directly.

B. Minimal evidence layer plus the same GPT

First consolidate repeated findings, preserve state changes, connect treatments to outcomes, and flag possible contradictions. Then ask the same model to produce the brief.

The comparison should measure outcomes that matter to the intended user:

• Were important clinical changes recovered?
• Were obsolete findings incorrectly presented as current?
• Were conflicting records surfaced?
• Were treatments correctly linked to subsequent outcomes?
• Were important negative findings preserved?
• Could every statement be verified against its source?
• How long did verification take?
• Did the additional pipeline complexity produce a meaningfully better result?

If the minimal evidence layer does not beat Raw GPT on those tasks, it should not exist.

A graph that improves benchmark F1 without improving the final diagnostic brief is research infrastructure, not product value.

The current hypothesis

The Diagnostic Odyssey hypothesis is now narrower than when I started:

Frontier models are already strong at locating and citing individual clinical facts. They may still need an external structure to maintain a reliable patient state across years of fragmented, duplicated, and conflicting medical records.

The phrase "may still need" matters.

This experiment did not prove that an evidence engine is necessary.

It identified the failure mode that such an engine would need to fix.

The next test is whether fixing that failure changes anything a patient or clinician actually cares about.

Conclusion

The model did not hallucinate its evidence.

Every submitted quotation in the parseable locked-test outputs could be found in the source narrative. Unsupported-claim rate was zero.

It still failed.

It produced unstable evidence units, over-extracted findings, inconsistently selected entity boundaries, struggled to preserve temporality, and rarely reconstructed the annotated relations between events.

The result was not:

GPT cannot understand clinical text.

It was:

Finding clinical facts and maintaining a longitudinal patient model are different problems.

That distinction is now the center of Diagnostic Odyssey.

The system I originally imagined may not require a new diagnostic foundation model.

It may require a layer that can take years of scattered records and reliably reconstruct:

• what happened
• when it happened
• what changed
• what was repeatedly confirmed
• what conflicted
• what treatment was tried
• what happened afterward
• and where every claim came from

The 50 manual runs did not prove that this layer will work.

They did something more useful for this stage of the project.

They showed me exactly what it has to earn the right to exist.