No Invented Search Filters, but 72 Answers Failed the Format Check
I tested whether telling a local AI that a search tool's filters were its only filters would stop it inventing new ones. Across 600 attempts on 150 synthetic shopping requests, it invented none. The more visible problem was finishing: 72 replies failed the required output format, and one further reply searched the wrong category. The shortest added instruction had the fewest format failures in this run, but that pattern did not recur on Gemma 4 12B (76 versus 77 invalid finals). The original filter-error question hit a zero-error baseline.
Bart Porcincula · Published
Finding
On 150 generated product-search requests run once under each of four tool-description conditions, Qwen3.5-9B Q4_K_M made 1,696 tool calls with zero invented parameters. Among 333 readable finals on 400 unsupported-filter attempts, the corrected structured-field scorer found zero false application claims or silent constraint drops; 67 unsupported attempts had invalid finals. Across all 600 attempts, 72 failed the output contract and one further attempt used incorrect category values, leaving 527 task successes. Invalid-final counts were 19/150 with plain descriptions, 7/150 with an only-filters sentence, 28/150 with an added disclosure instruction, and 18/150 with schema annotations plus server rejection. All arms shared an instruction to disclose unverifiable requirements; one catalogue, narrow templates, and a post-start scoring correction limit the conclusion.
Tool-filter descriptions and final-answer validity on a synthetic product catalogue (Nov 2026)
500 seeded products in SQLite; 150 generated requests, with 50 supported, 50 requiring unavailable attributes and 50 requiring filtering returned fields. Four arms, 150 attempts each, in one shuffled queue. Qwen3.5-9B Q4_K_M, llama.cpp b11753-d8d92973c, RTX 5060 Ti 16 GB, one processing slot, temperature 0, thinking off, 16,384-token context, four tool calls maximum. Ten separate interface-validation requests preceded the 600 test attempts. Inputs and model-loop code were frozen; a false-claim scorer correction after an early trace is disclosed and legacy scores retained. No final-answer grammar, repair prompt or retry was used.
| Measurement | Value | Note |
|---|---|---|
| Test attempts / distinct requests | 600 / 150 | the same requests under four conditions; one pass per condition |
| Invented parameters | 0 in 1,696 tool calls | attempted unknown arguments would count even if rejected |
| Unsupported attempts: readable / invalid finals | 333 / 67 | 400 attempts total; unreadable finals are failures, not clean answers |
| Structured false claims / silent drops in readable unsupported finals | 0 / 0 | corrected scorer; prose claims are outside this metric |
| Invalid final, plain / only-filters sentence / disclosure / rejection | 19 / 7 / 28 / 18 of 150 | no repair or retry in any arm |
| Supported task success, same arm order | 47 / 48 / 50 / 49 of 50 | overlapping intervals do not establish equivalence |
| Unavailable-attribute task success, same arm order | 40 / 47 / 30 / 37 of 50 | must disclose unavailable evidence and return no product IDs |
| Correct returned items after filtering observable fields, same arm order | 44 / 47 / 42 / 46 of 50 | matching output does not establish the model's internal process |
| Overall task successes | 527 of 600 | 72 invalid finals and one further category-value error |
| Invalid-final types: tool-call text / extra prose / malformed JSON | 54 / 15 / 3 | all counted as failures of the requested output contract |
The search filters were valid. The reply still had to work.
Suppose a shopping assistant receives a request for blue jackets under $50. Its search tool can filter by category, price, brand and stock status, but has no colour filter. The assistant could invent a colour parameter, ignore the colour requirement, or read the search results and select blue items itself.
I wanted to know whether one sentence in the tool description would help: “These are the only filters this tool supports.”
I tested 150 generated shopping requests under four conditions, giving 600 attempts with the same small local model. Across 1,696 tool calls, it invented no parameters. Among readable final answers, the structured claims also showed no unsupported filter falsely reported as applied or silently dropped.
There was another problem to count: 72 attempts failed the final-answer format, and one further attempt used the wrong category values. A system that needs to read the reply automatically would still have work to do.
What the assistant could search and know
The catalogue contained 500 synthetic products. Search accepted category, minimum and maximum price, brand, and stock status. A second tool could retrieve a product by its ID. Results included colour, size and rating, so the assistant could check those fields after searching.
The requests fell into three equal groups:
- Supported: every requirement had a search parameter.
- Unavailable evidence: a required property, such as waterproof or machine washable, appeared nowhere in the catalogue. The correct response was to explain the limitation and return no product IDs.
- Visible in the results: colour, size or a minimum rating had no search parameter, but could be checked in the returned product records.
That distinction matters. A missing search filter does not always mean the assistant cannot answer. Sometimes the information is available through another step. Sometimes the information does not exist in its tools at all.
Each request also named a category, brand, price bound and stock requirement. These were narrow templates, and the stock condition was redundant with the chosen price bound. Every supported or observable-field request had a matching product.
The four descriptions
All conditions used the same final-answer instructions: return a single JSON object, a machine-readable record containing product IDs, applied constraints and requirements that could not be applied. Every condition already told the assistant to disclose unverifiable requirements.
| Arm | Change to the tool description or handling |
|---|---|
| T0 | Plain parameter documentation |
| T1 | T0 plus the sentence saying these are the only filters |
| T2 | T1 plus an instruction to report unsupported filters as not applied |
| T3 | T2 plus schema annotations forbidding unknown arguments and server rejection of them |
For T0–T2, the server would ignore unknown arguments while retaining them in the log. T3 would reject them. Attempted invention counted in every condition, including rejected attempts. No condition used a decoder rule to force final answers into valid JSON.
I ran Qwen3.5-9B, compressed to four bits, through llama.cpp on one 16 GB graphics card. Requests ran one at a time, with randomness turned down, thinking off and a maximum of four tool calls. Nine of ten separate supported validation requests passed before the frozen test queue began. There was one test pass per condition, with no retry or repair prompt.
Results: no invented filters, uneven finishing
| Arm | Supported success /50 | Unavailable evidence handled /50 | Matching items after filtering /50 | All tasks passed /150 | Invalid final /150 |
|---|---|---|---|---|---|
| T0: plain | 47 | 40 | 44 | 131 | 19 |
| T1: only-filters sentence | 48 | 47 | 47 | 142 | 7 |
| T2: add disclosure instruction | 50 | 30 | 42 | 122 | 28 |
| T3: add rejection | 49 | 37 | 46 | 132 | 18 |
The headline filter-error counts were zero across all conditions. On the 400 attempts requiring an unsupported search filter, 333 finals were readable and 67 were invalid. The zero false-claim and silent-drop counts apply to the readable structured records. They do not turn the invalid replies into successes, and they do not grade every claim in the prose.
T1 had the fewest invalid finals here: 7 of 150, compared with 19 of 150 for plain descriptions. An exploratory paired comparison made after the run gives p=0.00754, without adjustment for the six format comparisons. A later Gemma 4 12B replication did not repeat this pattern: invalid finals went from 76 to 77 of 150. This result should not be treated as a general improvement. It does not show that the sentence reduced invented filters, because none occurred in the baseline.
The extra disclosure instruction did not improve finishing in this run. T2 had 28 invalid finals, including many attempts on unavailable attributes. Rejecting unknown arguments in T3 could not demonstrate a benefit against invention either: the model never sent any unknown arguments to reject. The rejection path was checked separately by tests.
What the failures looked like
Of the 72 invalid finals, 54 were tool calls written as text instead of a final answer, commonly after the four-call budget. Another 15 contained JSON with extra prose, and three contained malformed JSON. They failed the requested contract; I did not extract the JSON or ask the model to try again.
One more response had valid JSON but failed the task. Asked for Dune bags, it searched with category values “Dune bags” and then “Dune”, instead of separating brand “Dune” from category “bags”. It received no results and reported that none were available.
That example used allowed parameter names. The values were wrong. Counting invented parameters alone would have missed the failure.
A scoring correction I had to make
When 34 test traces were complete, a correct refusal exposed a defect in my scorer. The assistant returned no products because a required property was unavailable, but truthfully listed the supported filters its search had applied. The scorer marked those claims as false because there were no returned items, overlooking the tool-call evidence.
I added a separate reporting correction and three regression tests. The model prompts, tools, requests, settings and queue stayed frozen. The original scores remain in the raw traces: their false-claim counts were 16, 19, 8 and 16 across T0–T3. The corrected counts are zero. This correction happened after seeing an early outcome; I am not presenting the corrected scorer as frozen before the run.
Every logged tool result was replayed against the catalogue, all 600 request-and-condition pairs were checked, and the raw final responses were checked against the stored parses. The source, inputs, runtime details, scoring amendment and a seeded 21-case trace audit are retained with the results.
What this changes in how I test a tool-using assistant
I would keep four checks visible: whether the call uses allowed parameters, whether the chosen values retrieve the right records, whether the selected records satisfy the request, and whether the next step can read the answer. These results show why a clean score on the first check is insufficient.
The common instructions already asked the model to acknowledge missing evidence. One catalogue and a few request templates also leave plenty untested. I cannot use this run to say that invented filters are solved, that extra instructions generally hurt, or that strict argument handling makes final answers reliable.
For a real product-search assistant, this gives me a concrete test plan: include requirements its tools can apply, requirements it can verify only after reading results, and requirements it cannot verify at all. Then count incorrect results and unusable replies alongside invented arguments.
The earlier test of absent contract fields measures a different failure: supplying information missing from a document. Its rates should not be transferred to this tool-search task.
What should I check before connecting an AI assistant to product search?
Check that it uses allowed search parameters, supplies the right values, returns products meeting the request, and produces a reply the next step can read. This study had no invented parameters but still had output-format and search-value failures. Include requirements visible only in returned records and requirements absent from the available information. The results come from one synthetic catalogue and do not establish reliability on your inventory or customer requests.
Cite this research
Bart Porcincula. "No Invented Search Filters, but 72 Answers Failed the Format Check." Published 2026-11-08; updated 2026-11-08. https://porsync.com/research/local-ai-tool-filters-valid-parameters-invalid-answers
Permanent article URL