General-purpose AI coded complex surgery at 5% to 12.5% exact-match, a new study reports

On July 15, 2026, the journal Plastic and Reconstructive Surgery - Global Open published a comparative study grading four ways of assigning CPT codes to surgical operative reports: three off-the-shelf large language models, a team of external professional coders, and a fine-tuned vendor tool. Every method was scored against a reference standard built by a four-member expert consensus panel, and the bar was exact agreement on the full code set.

The number a vendor slide will pull from this is the fine-tuned tool's 86.7%. The numbers worth a coder's time are all the others, and the footnotes under all of them.

What the study measured, and how it graded

The researchers took 120 de-identified plastic and reconstructive operative reports and split them evenly into three complexity tiers: low (1 to 2 CPT codes), medium (3 codes), and high (4 or more codes), 40 reports each. Four methods coded every report: baseline general-purpose LLMs, an external professional medical auditing service, and a fine-tuned hybrid tool the authors call ProCode. A panel of two board-certified plastic surgeons and two senior billing specialists, each with more than 20 years of experience, set the expert consensus reference standard.

Accuracy was defined strictly. A code set counted as correct only if every CPT code matched the reference with no omissions and no additions. Code order and duplicates were ignored, modifiers were excluded from the analysis, and no partial credit was given. The authors chose that exact-match rule to mirror real billing, where an incomplete or padded code set gets a claim denied.

The full accuracy table

Here is every method against every complexity tier, as the study reports it. The percentages are exact-match on the full CPT code set.

Two patterns jump out. Every method except the fine-tuned tool falls off a cliff as the code count climbs, and the general-purpose models fall hardest. On the high-complexity reports, the three off-the-shelf models the study tested landed between 5.0% and 12.5%.

Why the general-purpose models collapse on complex cases

The story in that first column is the one that matters most for a working coder. A single-procedure operative note is a fill-in-the-blank task, and a general model handles it more than half the time. A multi-code reconstruction is not. It needs the add-on logic, the bundling rules, and the specialty-specific exclusions that live in the CPT book and in a coder's head, not in the narrative text.

The study's own framing is blunt about this: the models were run out of the box, single pass, no human correction and no iterative prompting, because that is how a rushed office actually uses them. Under those conditions, a general chatbot got the full code set right on 1 in 20 to 1 in 8 complex operative reports. That is the number to keep when someone suggests pointing a general AI at your surgical queue.

Read the auditor number with the method next to it

The line that will get quoted out of context is the external auditors at 42.5% overall, and 22.5% on the hard cases. Taken alone it reads as "professional coders are worse than the AI." Read with the method, it says something narrower.

The auditors coded single pass, against a reference standard the study's own panel built, on a held-out set drawn from the vendor's data, and were graded on all-or-nothing full-set matching with no ability to query the surgeon. Real surgical coders send the operative report back with questions, work from the actual chart, and are judged code by code, not on a binary set match. Strip a coder of the query and grade the whole set as one pass-or-fail unit, and the number drops. The study measured a constrained task, not a coder's job. It is the same trap as grading an AI against the billing codes already on the claim: the yardstick decides the score.

The tool that won was built by the authors

The fine-tuned tool posted the best figure at every tier. It was also trained on roughly 55,000 operative reports, and, most important for how you read the result, six of the study's eight authors are affiliated with ProCode, Inc., the company that sells it. The tool being evaluated, the training data, and the paper making the claim all trace back to the same source. That does not make the 86.7% false. It means the number carries the weight of a vendor benchmark, not an independent one, and it belongs in the "the vendor reports" column until someone outside the company reproduces it.

Even taken at face value, the winning method missed the exact code set on 8 of 40 high-complexity reports, and modifiers, the part of surgical coding most likely to trigger a denial, were excluded from the scoring entirely.

Where a human coder is still required

Strip the branding off this study and the durable lesson is one every risk-adjustment and provider-office coder already lives: AI accuracy is a function of complexity, and the complex chart is exactly where the money and the audit risk sit. A tool that nails a single-procedure note can miss the multi-code reconstruction, the add-on service, the bundling exclusion, or the modifier that moves a claim.

So the coder's job does not disappear when an AI tool arrives on the desk. It moves to the code set as a whole. Did the tool drop a reportable code, or add one the note does not support? Are the add-ons and modifiers right, since this study did not even grade them? Does the documentation carry the procedure the code claims? Those are judgments about a record, made by someone who can be held to them, and no exact-match percentage in a vendor paper answers a single one. When one lands on your desk, ask what it was graded against and at what granularity, then look the codes up and read the note.