The Check That Fails Without Telling You: A Required Phrase Missed 9% of the Time
Part 4 of 6. A common way to wire an AI into code is to ask it to start its reason with a fixed phrase, then test for that phrase. It looks reliable and it is not: the model sometimes just doesn't write it, and your code concludes nothing happened. A separate yes/no field did not have the problem.
Bart Porcincula · Published
Finding
When Qwen3.5-9B itself said an outage was reported, the code check that the reason starts with the required phrase silently failed on 5 of 55 cases (9.1%, 95% interval 1.8 to 18.2) at temperature 0, while a separate boolean field failed on 0 of 52; a tolerant matcher recovered none of the 5.
Required-phrase check versus a typed field (Oct 2026)
500 support tickets, with the model asked whether an outage was reported. In the phrase arms the reason must begin with a fixed prefix and code tests for it; in the field arm a separate boolean carries the answer. A silent miss is a case where the model's own answer is yes but the code check fails. Arms: phrase at temperature 0, phrase buried in a long prompt, boolean field, phrase at temperature 0.7 (three runs).
| Measurement | Value | Note |
|---|---|---|
| Phrase check at temperature 0: silent misses | 5 of 55 (9.1%) [1.8, 18.2] | the model said yes but the reason lacked the required prefix |
| Phrase buried in a long prompt: silent misses | 2 of 52 (3.8%) [0.0, 9.6] | not worse than the short prompt; difference within noise |
| Boolean field: silent misses | 0 of 52 | typed field, same model, same tickets |
| Phrase at temperature 0.7, three runs: silent misses | 3/65, 3/52, 5/54 (4.6%, 5.8%, 9.3%) | mean 6.6% |
| Phrase at temperature 0.7: false fires | 3, 1, 1 | the prefix sometimes appeared when the model had said no |
| Misses recovered by a tolerant matcher | 0 of 5 | the reasons had no prefix to normalise |
| Agreement of the model's own incident call with the dataset label | 64-65% in every arm | secondary: the model's idea of an outage is narrower than the label |
The short version
A lot of AI glue code works like this: tell the model to begin its reason with the word OUTAGE when an outage is reported, then write if reason.startswith("OUTAGE"). It reads cleanly and it passes a quick test. I measured how often it quietly fails. All six use the same setup: the open Qwen3.5-9B model compressed to 4 bits, run through llama.cpp on one 16 GB RTX 5060 Ti, at temperature 0 (the setting that makes a model as repeatable as it gets), on 500 customer-support tickets. Each experiment's questions, thresholds and analysis were written down before it was run.
- At temperature 0 the check missed 9.1% of the time (5 of 55 cases where the model itself said an outage was reported). The failures were not typos or capital letters. The model wrote a reason without the required prefix.
- Nothing flags it. The code takes the "no prefix" branch, the ticket is treated as no outage, and no error is raised.
- A separate yes/no field missed 0 of 52. Same model, same tickets, but the answer lives in its own typed field instead of in the wording of a sentence.
- Cleverer matching did not help. I tried a tolerant matcher that ignores case and punctuation. It recovered none of the five, because there was nothing to normalise.
Takeaway: do not make correctness depend on the model writing a magic phrase. Have it fill in a field with a fixed set of allowed values, and let the free-text reason be only an explanation for a human.
What I compared
| Arm | How the answer reaches your code |
|---|---|
| Phrase, temperature 0 | the reason must start with a fixed prefix; code checks for it |
| Phrase buried | the same, but the instruction sits in the middle of a long prompt |
| Boolean field | a separate "outage reported" yes/no, enforced by the output format |
| Phrase, temperature 0.7 | the same as the first, with randomness on, three runs |
A "silent miss" is the case that matters: the model says yes in its own words, but your code's check does not fire. A "false fire" is the reverse.
Results
| Arm | Model said yes | Silent miss | False fires |
|---|---|---|---|
| Phrase, temp 0 | 55 | 5 (9.1%) | 0 |
| Phrase, buried | 52 | 2 (3.8%) | 0 |
| Boolean field | 52 | 0 | 0 |
| Phrase, temp 0.7 (3 runs) | 65 / 52 / 54 | 3 / 3 / 5 | 3 / 1 / 1 |
I expected the buried instruction and the higher temperature to make the phrase worse. Neither did, within the noise of 55 cases. The honest reading is narrower: the phrase approach is non-zero (roughly 4 to 9% here), the typed field was zero. The order among the phrase variants is not established.
The phrase approach also failed in the other direction when randomness was on: the prefix occasionally appeared when the model had said no, so the check can fire when it should not.
Why it matters beyond this example
The cost is not the 9%. It is that the failure is silent and looks like a normal result. A missed outage is filed as "no outage", a missed escalation as "no escalation". You would find it only by auditing against a second source. Whenever your code branches on how a model phrased something, you have a check that can fail without raising anything.
What I would not claim
- 55 positive cases per arm is small. Rates of 4 to 9% differ by two to five cases.
- The model's reason is written before the separate field in this setup, so the field benefits from the reason having been written first.
- One model and one prompt wording. A larger model may follow the phrase rule more reliably; I did not test that.
Part 4 of 6. Previous: made-up values on missing fields. Next: answer first, explanation after.
Cite this research
Bart Porcincula. "The Check That Fails Without Telling You: A Required Phrase Missed 9% of the Time." Published 2026-10-29; updated 2026-10-29. https://porsync.com/research/the-phrase-check-that-silently-stops-matching
Permanent article URL