The Quote Was Real. The Contract Answer Was Still Wrong.
A literal quote check caught zero of 49 answers on contract fields labeled absent across three local deployments. It also discarded seven answers that passed the recall scorer. The same payment-allocation passage fooled all three deployments on a governing-law question.
Bart Porcincula · Published · Updated
Finding
On the same 630 field–excerpt pairs, the quote check kept all 19 labeled-absent answers from Qwen3.5-9B, all 17 from Gemma 4 12B and all 13 from Qwen3.6-27B. Requiring a quote changed overlap recall by −6.7, +1.9 and −1.0 percentage points, respectively. Strict value match fell on all three deployments.
Contract-field extraction replication on CUAD contract excerpts
The original 630 field–excerpt pairs cover five contract fields, with 315 labeled present and 315 labeled absent. Gemma 4 12B Q4_K_M and Qwen3.6-27B UD-IQ3_XXS completed 3,780 requests through llama.cpp b11753. The historical Qwen3.5-9B Q4_K_M run used b11408. The comparison is between quantized deployments on the same cases.
| Measurement | Value | Note |
|---|---|---|
| Labeled-absent answers removed by quote check | 0 of 49 across three deployments | |
| Overlap-correct answers removed by quote check | 7 across three deployments | |
| Gemma overlap recall, baseline to quote requirement | 285 → 291 of 315 | |
| Qwen 27B overlap recall, baseline to quote requirement | 293 → 290 of 315 |
Three models cited the same payment clause
I asked three local deployments to extract the governing law from a contract excerpt. Each cited a passage that mentioned the Colombian Civil Code. The passage explained how to apply a payment when a debtor owes several debts. The excerpt gave no governing-law designation. All three quotes passed a check that looked for the quoted words in the excerpt.
The filter removed seven answers that passed recall scoring
I reran the contract-field extraction study on Gemma 4 12B and Qwen3.6-27B, using the same 630 field–excerpt pairs as the original Qwen3.5-9B run. The quote arm asked for verbatim evidence and a value, with an explicit marker for an unstated field. The offline filter kept non-null answers only when the evidence appeared in the excerpt after normalization.
| Deployment | Answers on labeled-absent fields, before → after | Overlap-correct answers, before → after |
|---|---|---|
| Qwen3.5-9B, original | 19 → 19 | 275 → 272 |
| Gemma 4 12B | 17 → 17 | 291 → 287 |
| Qwen3.6-27B | 13 → 13 | 290 → 290 |
Each column uses 315 cases per deployment. The 49 labeled-absent answers are outputs across runs on a shared sample. The check verified where the words came from. The payment example shows how a genuine quote can still fail to support the field being extracted.
The original recall drop was larger
Relative to the baseline, requiring a quote cut overlap recall by 6.7 percentage points on the original 9B run. The same change was +1.9 points on Gemma and −1.0 point on the 27B. Both new estimates had 95% contract-clustered intervals spanning zero. The original stricter value-match score fell on all three deployments, by 15.2, 12.7 and 18.7 points respectively. That score uses normalized substring matching and can penalize longer values even when they share the expected words.
The frozen baseline already told the model to use null when the field was unstated. Its paired schema arm used the identical wording and enforced a nullable-value schema. Both new deployments had identical headline counts in those two arms. The quote arm changed the prompt, schema and way of signaling abstention together.
The results cover these deployments and this excerpt sample. Absence labels can be noisy, and I spot-checked semantic support on individual examples. Contract-field extraction replication report
Original contract-field extraction study
The public release notes identify included summaries and configurations, excluded raw inputs and responses, and reproduction limits.
Cite this research
Bart Porcincula. "The Quote Was Real. The Contract Answer Was Still Wrong.." Published 2026-11-09; updated 2026-11-16. https://porsync.com/research/local-ai-contract-extraction-quote-check-replication
Permanent article URL