The Numbers Matched the Table. The AI Summary Still Got the Facts Wrong.
I tested summaries of 200 public CO2 tables with a small local model and a numeric checker. The checker rejected supported approximations while letting false comparisons and trends through. In an agent review of 50 passing summaries, 11 contained clear factual contradictions. A clean numeric-membership score measured much less than factual accuracy.
Bart Porcincula · Published · Updated
Finding
On 200 five-year CO2 snapshots across 113 countries, a frozen numeric-membership checker rejected 13/200 plain Qwen3.5-9B summaries and 6/200 with an added numbers-only instruction. The paired country-cluster bootstrap interval for the change was −7.32 to 0.00 percentage points. Agent review of all 21 flagged numeric occurrences found 15 false flags, four legitimate flags and two uncertain cases; review of a seeded 50 passing summaries found 11 clear factual contradictions. Guarded outputs had zero membership failures by construction, which does not establish factual correctness.
Numeric membership and factual claims in 200 local-model table summaries
200 five-year snapshots across 113 countries, 1990–2023, from public OWID coal, oil and gas CO2 data. Five separate development snapshots. Qwen3.5-9B Q4_K_M on llama.cpp b11753-d8d92973c, RTX 5060 Ti 16 GB, temperature 0, seed 808, thinking off, one slot. 406 scored calls including six repairs. Frozen numeric checker; post-run agent audit, not independent human validation. Membership counts are not factual-error rates.
| Measurement | Value | Note |
|---|---|---|
| Plain / instruction-only summaries rejected | 13/200 / 6/200 | numeric membership, not factual accuracy |
| Instruction minus plain rejection rate | −3.5 percentage points | paired country-cluster bootstrap 95% interval −7.32 to 0.00 |
| Audited false / legitimate / uncertain numeric flags | 15 / 4 / 2 | all 21 flagged occurrences; agent review |
| Passing summaries with clear contradictions in audit sample | 11/50 | seeded passing sample; not all outputs, not independent human review |
| Immediate / one-reask template fallbacks | 6/200 / 3/200 | both start from the identical instruction-only output |
| Scored model calls | 406 | 400 initial plus six re-asks; five development calls excluded |
A correct number can support an incorrect sentence
One passing summary called gas, at 182.887 million tonnes, the largest source. Oil was 207.743. Another said a quantity rose from 20.738 to 19.831.
The numbers were available in the input. The statements about them were false.
I tested whether a simple number checker could help a small local model summarize tables faithfully. It could enforce a narrow rule about which numbers appeared. Its own audit showed why that rule was insufficient for approving a summary.
What I ran
I sampled 200 five-year snapshots across 113 countries from the public Our World in Data CO2 dataset. Each table showed coal, oil and gas emissions in million tonnes, with three decimal places. The underlying Global Carbon Budget data are credited in the retained data manifest. The sampled years span 1990–2023, with up to two non-overlapping windows per country.
Qwen3.5-9B, compressed to four bits, ran locally through llama.cpp on an RTX 5060 Ti with 16 GB of memory. Thinking was off, temperature was zero, and requests ran one at a time. Five development examples used countries outside the test set. Inputs, prompts, code and runtime hashes were frozen before inference.
The checker extracted digit-based numbers and compared them with an allowed set: table cells, extrema, selected differences and percentage changes, rounded to zero, one or three decimals. It normalized million-unit expressions and excluded integers from 1900 to 2100 as years. It did not validate which source a number described, the direction of a trend, or words such as “largest”. Spelled-out numbers were outside its scope; excluding years also creates a possible collision with real amounts.
Four ways to produce a summary
| Approach | Checker rejections | Template fallbacks |
|---|---|---|
| Plain three-sentence summary | 13/200 | 0 |
| Add an instruction to use table numbers or direct calculations | 6/200 | 0 |
| Check that same instructed summary; replace any failure with a template | 0/200 | 6/200 |
| Check it; allow one re-ask, then use the template if needed | 0/200 | 3/200 |
The guarded approaches reuse exactly the instruction-only first attempt. The template reports first and last cell values. Their zero rejection counts follow from that construction; they are not evidence of zero factual errors.
Adding the instruction reduced observed summary rejections from 6.5% to 3.0%. The paired bootstrap, resampling by country, gives a 95% interval of −7.32 to 0.00 percentage points for that change. It includes zero. Plain summaries always ran before instructed summaries within shuffled snapshots, and this experiment has no separate identical-prompt repeat. I would not generalize a prompt advantage from this run.
The checker rejected supported statements
There were only 21 flagged numeric occurrences across the two unguarded approaches, so the agent reviewed all of them against their tables. Fifteen were false flags, four were legitimate flags and two were uncertain. This was agent review, not an independent human audit.
The false flags exposed gaps in the allowed set. A difference of 130.906 supports “more than 130 million”. A value of 2.727 can reasonably become “approximately 2.73”, but two-decimal rounding was not allowed. Supported upper bounds and some cross-series calculations also failed membership.
At least 15/21 flags, or 71.4%, were false alarms in this audit. A guard that replaces these summaries is discarding supported text as well as catching mistakes.
Passing was also a weak assurance
The agent reviewed a seeded sample of 50 passing summaries and found 11 clear contradictions involving direction, ranking, extrema, chronology, attribution or numerical relationships. Nine additional summaries used “steady” or “consistent” for non-monotonic series; those looser judgments are counted separately.
This sample does not establish an error rate for all four approaches. The annotation was performed after the run, and there was no independent human reviewer. It does establish concrete cases where the frozen check passed a false claim.
A later exploratory direction heuristic also struggled: only 10 of its 32 fuel-and-sentence flags were confirmed in agent review. It often attached a verb to the wrong fuel or interval. I retained that result separately and did not change the frozen scorer to improve its score.
What I would check before approving a dashboard summary
I would keep numeric membership as one limited diagnostic and separately check the claims a reader will act on: which series is largest, which interval increased, and whether the cited change belongs to that series. A verifier needs examples of supported rounding and bounds as well as incorrect numbers.
This is one model, three related indicators and a convenience public dataset. It does not show that automated summary verification is impossible. It shows that a zero score on this particular guard says little about the truth of the relationships in the prose.
The receipt arithmetic study tests a related limit: numbers can agree with each other while disagreeing with the source.
Sources and public evidence
The full results and limitations describe the protocol, scoring and agent audit decisions. The public package includes aggregate results and frozen configuration metadata; exact inputs and raw traces remain outside this release. Public summary and manifests document the run of 406 scored calls; the individual call traces are not distributed. The run used no remote model API or subscription.
Can a number checker approve an AI-written dashboard summary?
This checker was insufficient for approval: it rejected supported approximations and passed false comparisons and trends. Audit both accepted and rejected text against the source, and test the relationships readers will act on. Zero numeric-membership failures after a guard is not a measurement of factual accuracy.
Cite this research
Bart Porcincula. "The Numbers Matched the Table. The AI Summary Still Got the Facts Wrong.." Published 2026-11-08; updated 2026-11-16. https://porsync.com/research/ai-summary-numbers-match-table-claims-still-wrong
Permanent article URL