Two Local Models Followed Every Written Rule and Missed Many Implied Ones
Qwen3.6-27B and Gemma 4 12B passed all 252 explicit-rule checks across roughly 4K, 16K and 32K tokens. With the rules implied, Qwen passed 52–55% and Gemma passed 24–29%. Giving each model its own extracted rule list never raised its score.
Bart Porcincula · Published · Updated
Finding
At each of three context lengths, both models passed 42 of 42 explicit-rule probes. Qwen passed 23, 22 and 22 with implied rules and 19, 20 and 21 with a same-model extraction pass. Gemma passed 12, 10 and 11 with implied rules and 12, 8 and 10 with extraction.
Instruction-following replication of stated and implied rules
Six synthetic rule types crossed with seven seeds and three target context lengths. Two local model deployments, four scored conditions and one extraction call per case produced 1,260 calls. The run used Qwen3.6-27B UD-IQ3_XXS and Gemma 4 12B Q4_K_M on llama.cpp b11753 with one slot, 40,960-token capacity, q8_0 KV cache, temperature zero and reasoning disabled.
| Measurement | Value | Note |
|---|---|---|
| Explicit-rule passes across both models | 252 / 252 | |
| Qwen implied, then implied plus extraction | 23 / 22 / 22, then 19 / 20 / 21 | Each count is out of 42 at 4K, 16K and 32K, respectively. |
| Gemma implied, then implied plus extraction | 12 / 10 / 11, then 12 / 8 / 10 | Each count is out of 42 at 4K, 16K and 32K, respectively. |
The gap was already there at 4K
I put a rule near the start of a synthetic work session, filled the middle with unrelated turns and tested the rule at the end. In one version the rule was written out. In another, the model had to infer it from a clue. The probe and its checker stayed the same. For example, the test checked whether an identifier came back in snake_case when the convention was either stated or implied.
| Model and condition | ~4K | ~16K | ~32K |
|---|---|---|---|
| Qwen3.6-27B, stated | 42 | 42 | 42 |
| Qwen3.6-27B, implied | 23 | 22 | 22 |
| Qwen3.6-27B, implied plus extracted rules | 19 | 20 | 21 |
| Gemma 4 12B, stated | 42 | 42 | 42 |
| Gemma 4 12B, implied | 12 | 10 | 11 |
| Gemma 4 12B, implied plus extracted rules | 12 | 8 | 10 |
Each cell is out of 42 probes. The original Qwen3.5-9B run also found a large gap at 4K. The two new deployments showed little change in their implied-rule scores from 4K to 32K.
Asking the model to list the rules added a call
The extraction pass asked the same model to list applicable rules from the implied history. I put its list directly before the probe and kept the original history available. Across all six model-and-length comparisons, the extra call produced five lower scores and one tie. The largest observed drop was four probes out of 42.
Scope and records
The replication used six synthetic rules and one quantized deployment per model. The checks score the output, and some formats can appear even with the neutral history. The results cover a same-model extraction pass. The original study records its separate cross-model rewrite test. Original stated-and-implied-rules study
The public record contains per-rule counts, paired intervals and model settings. Raw responses are not distributed in this release. Instruction-following replication evidence
The public release notes identify included summaries and configurations, excluded raw inputs and responses, and reproduction limits.
Cite this research
Bart Porcincula. "Two Local Models Followed Every Written Rule and Missed Many Implied Ones." Published 2026-11-09; updated 2026-11-16. https://porsync.com/research/local-ai-stated-vs-implied-rules-replication
Permanent article URL