Gemma Invented Zero Search Parameters and Returned 307 Invalid Replies
Gemma 4 12B made 885 tool calls without inventing an argument key. It still failed the final-answer format in 307 of 600 attempts, compared with 72 on Qwen3.5-9B. A third local deployment stopped at the development gate after four searches used category values outside the catalogue.
Bart Porcincula · Published · Updated
Finding
On the same 150 synthetic shopping requests under four tool descriptions, Gemma 4 12B passed 241 of 600 tasks and Qwen3.5-9B passed 527. Gemma produced 76, 77, 77 and 77 invalid finals across the four conditions. Qwen produced 19, 7, 28 and 18. Both invented zero argument keys. GPT-OSS 20B passed 6 of 10 development tasks against a threshold of 7 and recorded no test outcomes.
Tool-filter replication on a synthetic product catalogue
The original 500-product SQLite catalogue and 150 generated requests, with four attempts per request under different tool descriptions. Gemma 4 12B IT Q4_K_M and GPT-OSS 20B MXFP4 ran locally through llama.cpp b11753 on an RTX 5060 Ti. The run reused the original prompts, 16K context, one slot, temperature zero, reasoning disabled and a four-call cap. The original Qwen3.5-9B Q4_K_M run supplies the reference results.
| Measurement | Value | Note |
|---|---|---|
| Gemma / Qwen invented argument keys | 0 in 885 / 0 in 1,696 calls | |
| Gemma / Qwen invalid finals | 307 / 72 of 600 each | |
| Gemma / Qwen strict task successes | 241 / 527 of 600 each | |
| Gemma / GPT-OSS supported development successes | 10 / 6 of 10 each | The test gate required at least 7 of 10. |
The search worked. The next step still needed a readable reply.
A product search can return rows successfully while the assistant's final reply is unusable by the software that called it. The original tool-filter study found 72 such format failures on Qwen3.5-9B. I kept the same requests, catalogue and four versions of the tool instructions and ran Gemma 4 12B through the same local harness.
Gemma made 885 tool calls with zero invented argument keys. It missed the final format in 307 of 600 attempts. Of those, 163 contained a JSON object plus extra text, 134 were prose and 10 contained tool-call text. The next step expected a single JSON record.
The one-sentence change had a different result on Gemma
| Tool description | Gemma invalid finals /150 | Qwen invalid finals /150 |
|---|---|---|
| Plain description | 76 | 19 |
| Adds “These are the only filters this tool supports” | 77 | 7 |
| Also adds a disclosure instruction | 77 | 28 |
| Adds schema annotations and server rejection | 77 | 18 |
The shorter instruction went from 19 to 7 format failures on Qwen and from 76 to 77 on Gemma. Both planned paired comparisons on Gemma, each against the plain description, had exact p = 1.00. Strict task success also checked whether the answer satisfied the request. Gemma passed 241 of 600 attempts and Qwen passed 527.
A valid argument name can carry a value the catalogue cannot match
GPT-OSS 20B cleared the required format on all 10 supported development requests. It got six tasks right, below the seven required to enter the test. Each of its four misses used an out-of-catalogue category value such as “Jackets,” “jacket” or “shirt.” The tool accepted the category argument and returned an empty result because the catalogue matches category strings exactly. The tool description listed the argument and left the allowed values implicit.
Scope and records
These results cover the tested quantized deployments, including their chat templates, one synthetic catalogue and narrow request templates. All conditions already asked the model to disclose what it could not verify. The test held the output contract fixed, so it leaves the effect of enforced JSON or a repair pass open.
The first run documents the four conditions and the scoring correction. Original tool-filter study
The local report records the development gate, format breakdown and paired comparisons. Tool-filter replication report
The public release notes identify included summaries and configurations, excluded raw inputs and responses, and reproduction limits.
Cite this research
Bart Porcincula. "Gemma Invented Zero Search Parameters and Returned 307 Invalid Replies." Published 2026-11-09; updated 2026-11-16. https://porsync.com/research/local-ai-tool-filters-gemma-replication
Permanent article URL