ResearchPractical AIProjectsOpen SourceAbout
Research/A Numbers-Only Instruction Halved Flags on One Local Model
Local AIData SummariesEvaluationOutput ValidationOriginal Research

A Numbers-Only Instruction Halved Flags on One Local Model

The original 13-to-6 drop in checker flags reproduced on Qwen3.5-9B. Gemma 4 12B went from zero to one, while Qwen3.6-27B went from six to five. A review of passing summaries found claims that misused valid source numbers.

Bart Porcincula · Published · Updated

Finding

On the same 200 five-year CO₂ tables from 113 countries, a numbers-only instruction changed numeric-membership flags from 13 to 6 for Qwen3.5-9B, 0 to 1 for Gemma 4 12B, and 6 to 5 for Qwen3.6-27B. All three paired 95% country-cluster intervals included zero. In a seeded sample of 50 passing summaries per model, agent review found 11, one and two clear contradictions respectively.

Numeric-summary replication of numeric membership and factual claims on 200 CO₂ tables

The original frozen prompts and numeric checker, using 200 five-year public CO₂ tables from 113 countries. Qwen3.5-9B Q4_K_M, Gemma 4 12B IT Q4_K_M and Qwen3.6-27B UD-IQ3_XXS. Each ran locally through the original llama.cpp build on an RTX 5060 Ti 16 GB, one model and one request at a time, with temperature zero, seed 808, 16,384-token context and reasoning disabled in the requests. The post-run audit used one agent reviewer and a seeded sample of passing outputs.

MeasurementValueNote
Plain / instructed summaries flagged on Qwen3.5-9B13/200 / 6/200
Plain / instructed summaries flagged on Gemma 4 12B0/200 / 1/200
Plain / instructed summaries flagged on Qwen3.6-27B6/200 / 5/200
Clear contradictions among 50 passing summaries per deployment11 / 1 / 2single agent reviewer, descriptive sample

The original drop reproduced on the original model

I asked three local models to summarize the same 200 public CO₂ tables. Each produced a plain summary and another with an instruction to use numbers from the table or direct calculations. The original Qwen3.5-9B reproduced all 406 generated outputs exactly. The other deployments each moved by one flagged summary, in opposite directions.

Local deploymentPlain flaggedInstructed flaggedChange in flag rate, paired 95% interval
Qwen3.5-9B13/2006/200−3.5 pp [−7.3, 0.0]
Gemma 4 12B0/2001/200+0.5 pp [0.0, +1.5]
Qwen3.6-27B6/2005/200−0.5 pp [−3.5, +2.5]

The intervals resample by country because some countries contribute two tables. Each interval includes zero. Gemma started with zero flags, leaving no room for a decrease, and its instructed summaries used 1,271 numeric tokens compared with 735 in its plain summaries. That gave the checker more numbers to evaluate.

What passed and what got rejected

The checker tests whether a numeric token matches a table value or an allowed calculation. In the seeded review of 50 passing summaries per deployment, one agent reviewer found 11 clear contradictions for the 9B, one for Gemma and two for the 27B. These counts describe the audited samples of passing text.

In Gemma's Denmark summary, two valid coal values were attributed to 2003 even though one belonged to 2005. In the 27B's Spain summary, gas rose from 39.970 to 68.893, about 72%. The model said it more than doubled. Both endpoints passed membership.

Gemma's only flag came from a supported bound. Its Benin summary said all displayed gas values were below 0.111, and the maximum was 0.110. The 27B audit found two wrong sums, ten supported numeric occurrences and three uncertain approximations among 15 flagged occurrences. Counts of flagged numbers and counts of flagged summaries use different denominators.

What a retry preserved

Both guard strategies began with the same instructed answer. Immediate replacement used a fixed endpoint template on six, one and five summaries for the 9B, Gemma and 27B respectively. Allowing one retry lowered those fallback counts to three, one and one. On the 27B, retry rewrote a wrong Latvia total using verified components. A wrong Benin total survived retry and was removed by fallback. Gemma's supported upper bound also went to fallback.

Both guards finished with zero membership flags on every deployment. The template achieves that score by reporting first and last table cells, at the cost of dropping the original interpretation. A summary that passes still needs its years, comparisons and trends checked against the table.

Sources and public evidence

The original study documents the public data, the rounding rules and the limits of the numeric checker. Original numeric-summary study

The companion replication follows what an arithmetic guard repairs in receipt extraction. Receipt-arithmetic replication

The run report includes the paired comparisons, audit decisions and per-deployment configuration. Replication report

The public release notes identify included summaries and configurations, excluded raw inputs and responses, and reproduction limits.

Cite this research

Bart Porcincula. "A Numbers-Only Instruction Halved Flags on One Local Model." Published 2026-11-09; updated 2026-11-16. https://porsync.com/research/numeric-summary-checks-three-local-models

Permanent article URL