In a coding study, AI hit 91.5% on CPT but 23.9% on ICD-10, with laterality the top error

On August 1, 2026, the Journal of the American Academy of Orthopaedic Surgeons published a study that asked a narrow question: can a public large language model read a clinical note and assign the correct billing codes? The models handled CPT procedure codes well and ICD-10 diagnosis codes poorly, and the authors report the single most common ICD-10 error was one thing a coder checks by reflex. Laterality. Left, right, or bilateral.

What the study tested

The researchers pulled 90 deidentified charts, evenly split across three orthopaedic hand surgeons and three procedures: cubital tunnel release, carpal tunnel release, and trigger finger release. They recorded the correct ICD-10 diagnosis code and CPT procedure code for each, then asked three models (ChatGPT 3.5, ChatGPT 4.0, and Gemini) to code the notes. Each note was posed four ways: zero-shot, one-shot, multishot, and chain-of-thought prompting. The study reports the work was limited to hand-surgery clinic and operative notes.

CPT held up, ICD-10 did not

Across all three models, CPT came back 91.5% correct and ICD-10 only 23.9%, the study reports. Two other results are worth a coder's attention. Neither the note's author nor the prompt style moved the numbers, and ChatGPT 3.5 produced less accurate ICD-10 codes than ChatGPT 4.0 or Gemini. The authors' conclusion is blunt: public-facing models "require additional optimization to interpret clinical documentation for coding purposes and are not ready for independent use."

Laterality was the sticking point

The paper names the failure mode directly: the most common ICD-10 error was incorrect or omitted laterality. When the authors rewrote the prompt to emphasize laterality specifically, they report accuracy improved (40%). Read that against what the FY2026 ICD-10-CM Official Guidelines actually require. Section I.B.13 says unspecified-side codes "should rarely be used," reserved for records where the side is not documented and cannot be clarified. A model that returns the unspecified-side code when the note says "right" is not being imprecise. It is producing a code the Guidelines tell you to avoid.

The codes behind the finding

Two of the three procedures the study coded live in the G56 family, where laterality is the only thing separating a specific code from an unspecified one. This is the shape of the error a coder is checking for, and it's the same category-versus-code gap a separate study hit when a model predicted the diagnosis bucket but not the billable code.

Cubital tunnel (lesion of ulnar nerve) mirrors carpal tunnel: G56.20 unspecified, G56.21 right, G56.22 left, G56.23 bilateral. The distance between the top row and the rest is exactly the distance an AI draft skips.

What the study did not test

This is 90 hand-surgery charts and three general-purpose chatbots, not a coding department. It didn't test the fine-tuned vendor coding tools or ambient scribes a practice actually buys. It didn't test the current model versions, and it says nothing about outpatient risk-adjustment charts or any other specialty. Take the 91.5% CPT result the same way: it graded three well-defined hand procedures, not the multi-code operative notes where CPT selection gets genuinely hard. The generalizable lesson isn't that AI is bad at coding. It's that these tools drop the specific detail, and laterality runs through much of the code set, from the injury and musculoskeletal chapters to the eye and ear.

Where the coder still owns the code

The study's own fix (tell the model to watch laterality) is a coder's habit, not a feature. When you review an AI-suggested or scribe-generated code, the laterality character is the first thing to check against the note, because it is the first thing the model gets wrong. If the provider did not document the side, the Guidelines let you code it from another clinician's documentation, and query the provider when the record conflicts, rather than defaulting to the unspecified code. That judgment (read the record, apply I.B.13, decide when to query) is the part no model in this study could do on its own.