ResearchPractical AIProjectsOpen SourceAbout
Research/Three Local Models Read the Same Receipts. Repair Fixed No Totals.
Local AIDocument ExtractionEvaluationOutput ValidationOriginal Research

Three Local Models Read the Same Receipts. Repair Fixed No Totals.

Gemma 4 12B and Qwen3.6-27B got all 42 printed totals right on their first attempt. Qwen3.5-9B repeated its one wrong total, which passed the arithmetic check. An instruction and a repair pass changed some line-item scores while exact-total counts held across all three models.

Bart Porcincula · Published · Updated

Finding

On 42 eligible CORD v2 receipt transcripts, baseline, instructed and repaired exact-total counts were 41/42 throughout for Qwen3.5-9B and 42/42 throughout for Gemma 4 12B and Qwen3.6-27B. Arithmetic contradictions moved from 4 to 4 to 3, 2 to 1 to 1, and 0 to 0 to 0 respectively. Six repairs ran across the three deployments, with zero total fixes or breakages.

Receipt-arithmetic replication on 42 eligible CORD v2 receipt transcripts

The original frozen prompts and scorer, applied to the same 42 human-transcribed receipts. Eligibility favors explicit, arithmetically consistent receipts. Qwen3.5-9B Q4_K_M, Gemma 4 12B IT Q4_K_M and Qwen3.6-27B UD-IQ3_XXS. Each ran locally through the original llama.cpp build on an RTX 5060 Ti 16 GB, one model and one request at a time, with temperature zero, seed 808, 16,384-token context and reasoning disabled in the requests.

MeasurementValueNote
Qwen3.5-9B exact totals / baseline contradictions41/42 / 4
Gemma 4 12B exact totals / baseline contradictions42/42 / 2
Qwen3.6-27B exact totals / baseline contradictions42/42 / 0
Repairs triggered / printed totals fixed6 / 0

The same wrong total passed again

Qwen3.5-9B returned 434,500 for a receipt whose printed total was 93,500. It multiplied full line amounts by their quantities again, then calculated a consistent subtotal, tax and total. The arithmetic check accepted that combination. The rerun reproduced all 172 generated outputs from the original receipt study exactly, including that error.

Gemma 4 12B and Qwen3.6-27B both read the printed total correctly. Gemma still produced two arithmetic contradictions elsewhere in the same set of receipts.

What changed across the three deployments

Each model saw the same 42 eligible CORD v2 receipts reconstructed from human-transcribed words. The original selection excluded 58 of 100 receipts based on explicit-field and arithmetic requirements. The results apply to extraction from those transcripts. The experiment covers neither OCR nor images.

Each deployment ran the plain prompt three times and an arithmetic instruction once. A passive check and a one-retry repair procedure both started from the first plain answer. The check compares the extracted line amounts to the subtotal and compares the subtotal plus tax and service to the total. The scorer separately compares the extracted total to the printed total.

Local deploymentExact totals, plain / instructed / repairedContradictions, plain / instructed / repairedItem F1, plain / instructed / repairedRepair calls
Qwen3.5-9B41 / 41 / 414 / 4 / 3.895 / .895 / .9074
Gemma 4 12B42 / 42 / 422 / 1 / 1.909 / .930 / .9172
Qwen3.6-27B42 / 42 / 420 / 0 / 0.905 / .901 / .9050

Item F1 rewards matching normalized item names and full line amounts while penalizing missed and extra items. The 9B repair improved that score by correcting a line amount. Gemma's best item score came from the instruction. Its observed one-receipt reduction in contradictions had a paired 95% bootstrap interval of −7.1 to 0 percentage points. One of its identical-prompt repeats also changed an output on one receipt. Both Qwen deployments reproduced their first baseline outputs on both repeats.

What the repair was able to fix

All six repair attempts started from answers that already had the correct printed total. The only wrong total passed the arithmetic test and never reached repair. The 27B produced zero contradictions on this set, so its repair behavior went untested.

This comparison covers three quantized deployments, including their native templates. It leaves the effect of model size alone unresolved. The selected sample also gives us just one wrong total, so a general estimate of how often the check catches a wrong total would require more such errors.

Sources and public evidence

The original protocol explains receipt eligibility, the arithmetic contract and the item score. Original receipt-arithmetic study

The companion replication tests whether a table-summary instruction transfers across the same three deployments. Numeric-summary replication

The run report records model quantizations, controls, per-model results and hardware measurements. Replication report

The public release notes identify included summaries and configurations, excluded raw inputs and responses, and reproduction limits.

Cite this research

Bart Porcincula. "Three Local Models Read the Same Receipts. Repair Fixed No Totals.." Published 2026-11-09; updated 2026-11-16. https://porsync.com/research/receipt-arithmetic-checks-three-local-models

Permanent article URL