An AI hit 91% predicting ICD categories. Its scoring counted sepsis for a UTI as correct.

On June 27, 2026, researchers posted a preprint to arXiv reporting that a frozen medical large language model can recover a patient's primary diagnosis category from hospital records with 87.69% strict accuracy and 91.45% "medical accuracy." The work is careful, the authors are candid about its limits, and the number that'll get quoted is the 91.45%.

That number is going to end up on a vendor slide. It's worth knowing what the scoring rule quietly forgives before it does.

What the study actually measured

The researchers built a cohort of 13,645 admissions from MIMIC-IV, a public database of de-identified critical-care records. They took the ten most frequent primary ICD-10 codes in that cohort and consolidated them into seven diagnosis categories. The reported mean age was 64.0 years, and the cohort was 52.9% male. The admissions were split into 9,556 training, 2,042 validation, and 2,047 test records.

The task was a seven-way choice. Given a patient's structured data and a leakage-pruned discharge note, pick the right category. The best configuration, a probe trained on layer 32 of the frozen model with both inputs combined, reached 87.69% strict accuracy and 91.45% medical accuracy.

That's a legitimate result for what it is, which is a representation-probing study. It isn't a coding accuracy number, and the paper never claims otherwise.

The scoring rule credits sepsis for a UTI

"Medical accuracy" is not a stricter metric than strict accuracy. It is a looser one. The paper defines it as a fixed soft-matching rule "that credits clinically related but non-identical predictions," and it gives the example itself: predicting "Sepsis" for a "urinary tract infection" case "is counted as correct because both map to the Infection Related category."

Read that again with a coder's eye. The study's Infection Related category contains exactly two codes, A41.9 and N39.0. Sepsis and a UTI. The headline metric treats a model that confuses them as having gotten the answer right.

The authors flag this plainly in their limitations, noting that medical accuracy "is a clinically relaxed soft-matching metric and should be interpreted alongside strict accuracy and per-class recall." The paper is fine. The trouble starts when that number gets quoted without the sentence sitting next to it.

Where those seven categories land in CMS-HCC V28

We looked up each of the ten codes behind the seven categories against the CMS-HCC V28 model for PY2026 and the FY2026 ICD-10-CM tabular. The categories aren't risk-adjustment-neutral. Several of them straddle the line between a code that carries an HCC and a code that carries none.

Two of the seven categories pair an HCC-bearing code with a code that carries no HCC. Infection Related pairs sepsis with a UTI. Cardiovascular Event pairs an NSTEMI with unspecified chest pain, a symptom code. A model can be scored correct on either pair while landing on the wrong side of a risk-adjustment boundary worth half a RAF point.

Two categories rest on codes V28 stopped paying

The table holds a second lesson that has nothing to do with AI. Acute kidney failure (N17.9) mapped to HCC 135 under V24. Alcohol abuse with intoxication (F10.129) mapped to HCC 55. Neither maps to an HCC under V28, the model that pays for PY2026. Together those two categories account for 1,645 of the 13,645 admissions in the cohort.

Major Depression is the sharpest case. The study's code is F32.9, major depressive disorder, single episode, unspecified, and it carries no HCC in either model. Specify the severity and the picture changes. F32.1 and F32.2 map to HCC 155 under V28, and F32.3 maps to HCC 152. The category label "Major Depression" is precisely the thing that doesn't code. The specificity underneath it is what does, which is the same lesson our V28 guide and the RAF calculator exist to make concrete.

What the authors say the study does not show

The paper's limitations section is the most useful page in it, and it's doing the work a vendor deck usually won't.

The authors write that the task "does not capture the full multi-label and long-tail structure of production ICD coding." They note that MIMIC-IV and MIMIC-III are retrospective critical-care datasets, so "performance may not generalize to outpatient settings, non-ICU populations, or institutions with different documentation practices." Most risk-adjustment coding happens in exactly those outpatient settings. And they close by stating that the study "does not include prospective validation or assessment of workflow integration with professional coders."

That's an author telling you, in the paper, that no coder has tested this.

The miss-rate column is our arithmetic, not the paper's. It's simply the complement of each reported figure expressed as a frequency, and it's here because "91.45% accurate" and "wrong on one admission in twelve, before you even get to code selection" describe the same result.

Where a human coder is still required

Nothing in this study picks a code. It picks a bucket, from a menu of seven, on inpatient critical-care records, and it's graded on a curve that forgives sepsis for a UTI.

Everything that makes a diagnosis reportable happens after the bucket. Whether the encounter supports the condition. Whether the documentation carries MEAT for it. Whether the severity, the episode, and the laterality are specified enough to reach a billable code. Whether that code maps to an HCC under the model that pays this year, and not the model that paid three years ago. Every one of those is a judgment about a record, made by someone who can be held to it, and this paper doesn't claim otherwise.

The pattern is familiar. A study earlier this month graded an AI comorbidity reader against the billing codes already sitting on the claim, and agreeing with a billing code doesn't prove the record supported it. Here the yardstick is a category, and agreeing with a category tells you even less, because it never reaches a code at all. Both are worth reading. Neither replaces the coder QA pass on an AI-touched note.

When a tool's accuracy number lands on your desk, the number is the least interesting part. Ask what it was graded against, and at what granularity. If the answer is a category, you've learned nothing yet about a code. Look the code up in the encoder and read the note.