ResearchPractical AIProjectsOpen SourceAbout
Research/The AI's Receipt Total Added Up. It Was Still Wrong.
Local AIDocument ExtractionEvaluationOutput ValidationOriginal Research

The AI's Receipt Total Added Up. It Was Still Wrong.

On 42 eligible receipt transcripts, a local model produced four arithmetic contradictions. The only wrong total passed the arithmetic check: the model had multiplied line amounts by quantities again, then added its mistaken numbers correctly. One repair fixed a line amount, but total accuracy stayed 41/42.

Bart Porcincula · Published · Updated

Finding

On 42 eligible CORD v2 test receipts supplied as human-transcribed annotated text, Qwen3.5-9B Q4_K_M produced four baseline arithmetic contradictions and one wrong total. The wrong total was internally consistent and escaped the checker. One re-ask reduced contradictions to three and improved macro item F1 from 89.47% to 90.67%, while exact total accuracy remained 41/42. Two identical-prompt repeats matched baseline on all receipts. An arithmetic instruction corrected one tax/service swap without changing primary endpoints. This selected subset and its single wrong total cannot establish a general detection rate.

Receipt arithmetic consistency and source accuracy on 42 eligible CORD v2 transcripts

42 of 100 CORD v2 test annotations met frozen requirements for explicit fields and two gold arithmetic identities; 58 excluded with retained reasons. Human-transcribed words reconstructed by row and geometry, without category labels or gold parse in the prompt. Not an image or OCR-engine benchmark. Same local Qwen3.5-9B Q4_K_M runtime as the table-summary study, temperature 0, seed 808, thinking off. 168 initial calls plus four re-asks; one separate synthetic development call. Frozen primary scorer, post-hoc charge metrics and agent audit disclosed.

MeasurementValueNote
Eligible / excluded test receipts42 / 58selection favors explicit, self-consistent receipts
Baseline / instruction / repaired contradictions4/42 / 4/42 / 3/42
Exact total, baseline and repaired41/42 in both
Wrong totals flagged by arithmetic0/1one wrong total; too few for a general detection-rate claim
Macro item F1, baseline / repaired89.47% / 90.67%exact normalized name plus full line amount, averaged per receipt
Identical-prompt repeat changes0/42 in each of two repeatsraw and parsed outputs; not independent receipts
Scored model calls172168 initial plus four re-asks; development excluded

A convincing total with the wrong starting amounts

One receipt had full line amounts of 75,000 and 10,000, alongside quantities of five and two. The model multiplied the line amounts by those quantities again. It produced a subtotal of 395,000, tax of 39,500 and total of 434,500.

Those numbers add up. The printed total was 93,500.

An arithmetic check approved the output, so a repair step triggered only by arithmetic failures never looked at it again. This was the only wrong total in my test. It is a useful failure example, with a denominator far too small to estimate how often the problem occurs generally.

What counted as a receipt in this test

I used public CORD v2 test annotations, credited to NAVER CLOVA and Park et al. under CC BY 4.0. I reconstructed text from human-transcribed word regions, ordered by row and geometry. The model saw neither category labels nor the gold structured answer.

This measures extraction from annotated text. It does not measure image reading or an OCR engine.

Of 100 test receipts, 42 qualified and 58 were excluded: 35 lacked an explicit subtotal, nine had discounts or unmodelled charges, eight failed the required gold arithmetic, and six had missing or ambiguous fields. I did not hand-correct gold amounts or infer subtotals to retain more examples.

The required identities, within one rupiah, were: item line amounts sum to subtotal; subtotal plus tax and service equals total. Price explicitly meant the full line amount, without multiplying by quantity again. This selects for simpler, explicitly self-consistent receipts; 20 of the 42 contain only one top-level item.

Repeats, an instruction and one repair opportunity

Qwen3.5-9B Q4_K_M ran through llama.cpp on one RTX 5060 Ti 16 GB, with thinking off, temperature zero and one processing slot. The code, inputs and runtime were frozen. There was no grammar forcing JSON, output cleanup or hidden retry.

I ran the baseline three times, added an arithmetic-check instruction in a fourth call, then applied two procedures to the original baseline: a passive checker and a single re-ask for any invalid or arithmetically contradictory output. There were 172 scored model calls, including four repairs, after one separate synthetic development example.

ApproachArithmetic contradictionsExact totalMacro item F1
Baseline4/4241/4289.47%
Each of two identical-prompt repeats4/4241/4289.47%
Added arithmetic instruction4/4241/4289.47%
Passive check of baseline4/4241/4289.47%
One re-ask on flagged baseline outputs3/4241/4290.67%

All outputs passed JSON and schema checks. Both baseline repeats matched the original raw and parsed outputs on every receipt. They are repeated measurements of the same receipts, not another 84 independent examples.

The instruction changed one structured answer: it corrected a tax/service swap of 12,733 and 8,330. Their sum was unchanged, so the primary arithmetic and total metrics could not see the improvement. Post-hoc field checks retain it separately.

The four flags were useful, but aimed at another problem

Two receipts omitted printed tax while copying the correct total. Another combined duplicate items but retained only one copy's amount, violating the declared full-line-amount contract. The fourth used a unit price of 12,000 where the amount for two items was 24,000.

One re-ask fixed that fourth case. It corrected a real line-item error and improved the item score. The other three contradictions remained. All four flagged receipts already had correct totals, so there were zero total fixes and zero total breakages.

The only wrong total belonged to the internally consistent example above. Arithmetic therefore caught 0 of 1 wrong totals here. That is a description of this sample, not a credible population detection rate.

What the item score does and does not mean

Item F1 measures the balance of missed and extra items, matching exact normalized names and full line amounts, then averaging per receipt. A shortened but recognizable name can fail this strict match. A zero-price modifier can count as an extra item relative to CORD's top-level menu taxonomy without being invented text.

The agent reviewed all ten baseline/instruction error or charge-assignment cases and all four repairs. The notes distinguish these scoring penalties from arithmetic failures. This was not independent human validation. Charge-field measures were added after the run; the frozen primary scorer was not changed.

What this changes in an extraction workflow

I would report source accuracy and internal arithmetic separately. A failed sum can reveal an omitted charge or incorrect line amount and give a repair prompt something concrete to address. A passed sum provides no direct assurance that the amounts came from the correct places in the receipt.

This run supports a limited conclusion: the check detected four declared-contract violations and helped repair one line amount, while missing one coherent but wrong extraction. The small selected subset, human-transcribed inputs and single model leave broader performance unmeasured.

The table-summary study shows another version of the same verification problem: numbers can come from the source while the sentence relating them is false.

Sources and public evidence

The full results include exclusion reasons, frozen hypotheses and scoring limits. Public summary and manifests describe the 42-receipt grid and repairs; receipt text and individual traces are not distributed. The run used no remote model API or subscription.

Does a receipt extraction become trustworthy when its amounts add up?

A passed sum establishes internal consistency under the declared line-amount rules. Source accuracy needs a separate check. Here one re-ask fixed a line amount, but the only wrong total already added up and escaped repair. The 42 selected transcripts and single wrong total are too limited to estimate general detection performance.

Cite this research

Bart Porcincula. "The AI's Receipt Total Added Up. It Was Still Wrong.." Published 2026-11-08; updated 2026-11-16. https://porsync.com/research/ai-receipt-total-adds-up-still-wrong

Permanent article URL