An ICD coding model scored 0.709 on a 50-code subset and 0.527 across all 15,761 codes

On July 11, 2026, four researchers posted a preprint to arXiv describing a new approach to automated ICD-10-CM coding, in which a language model walks down the code hierarchy level by level instead of scoring every code independently. The paper is worth a coder's attention less for the method than for the two accuracy figures it reports side by side, because the difference between them is the difference between a demo and a workday.

What the study actually measured

The authors trained systems to read de-identified inpatient discharge summaries from MIMIC-IV, a credentialed-access research database, and output ICD-10-CM leaf codes. Their best supervised model, which they call SFT-1+, was scored two ways: against a curated 50-code subset, and against the full space of 15,761 codes. On the 50-code subset it reached 0.709 micro-F1. On the full space it reached 0.527.

Both numbers describe the same system. Only the label space changed.

This is a preprint. It has not been through peer review, and the results are the authors' own reported measurements, not an independent evaluation.

The 50-code subset and the full code space are different tests

Fifty codes is about 0.3% of a 15,761-code label space. Scoring on a slice that size is a normal research convenience, and it's a poor proxy for a real queue, because the codes you actually argue about are the ones that didn't make the cut.

The macro-F1 gain being more than three times the micro-F1 gain is the tell. Macro-F1 weights every code equally, so a big macro gain next to a small micro gain says the movement happened in the tail, not in the codes the model already handled.

Where the model loses ground is the rare-code tail

The authors describe rare labels as the bottleneck their design is meant to relieve, and they report that the graph-constrained approach mitigates it. Mitigates is doing real work in that sentence. A tail that's less bad than it was is still the part of the code set where a risk-adjustment or specialty coder spends most of the day.

If you code diabetes with complications, cardiomyopathy subtypes, or anything where the fourth, fifth, and sixth characters carry the money, you live in the part of the distribution these systems are weakest on. That is the same pattern we covered when an earlier preprint reported 91% accuracy at ICD category level while its scoring counted a category hit as correct regardless of the code underneath it.

Structurally valid is not the same as clinically correct

The method's headline safety property is that walking the hierarchy guarantees a structurally valid billable leaf code every time. That's genuinely useful. It also means the output will always pass a format edit.

For a coder, this changes what a clean output proves. A code that exists, is billable, and carries the right number of characters has cleared the edit check and nothing else. Whether it matches what the provider documented is a separate question that no structural constraint answers. Treat "the system returned a valid code" as the start of your review, not the end of it.

Long notes broke the cascade design on 28 to 32% of cases

Buried in the results is a detail with an immediate documentation consequence. The authors compared their single shared policy against a three-specialist cascade, and report that the cascade overflowed its context window on 28 to 32% of full-space test notes. Their shared policy avoids that overflow.

The lesson isn't about the cascade. It is that discharge summaries are long enough to exceed what these systems can hold, and every design has some limit. When a note runs past that limit, something gets dropped, and what gets dropped is whatever sits furthest from wherever the system starts reading. A comorbidity mentioned once, deep in a hospital course narrative, is the first casualty. That's an argument for the thing coders have been asking providers for all along: state the assessed conditions in the assessment, not only in the middle of the narrative.

What this changes about your QA sample

Most AI-assist QA samples are drawn proportionally, which means they're mostly common codes, which means they mostly measure the part of the system that already works. If the reported gains concentrate in rare codes and the reported weakness is the tail, a proportional sample is close to blind by construction.

The fix isn't complicated. Stratify. Pull a fixed number of rare-code encounters into every sample regardless of how rarely they occur, and score them separately from the common ones, so you have your own micro and macro read instead of a single blended number. You can spot-check any code the tool hands back against the official descriptor and hierarchy in the code book or the encoder before you accept it, and hold the underlying diagnosis to the same MEAT standard you would apply to a human-coded chart.

Federal watchdogs have already flagged the oversight gap around automated coding accuracy, which we covered when GAO reported on AI in medical coding. Until that gap closes, the benchmark number a vendor quotes is only as meaningful as the label space it was measured on. Ask for that number.