ResearchPractical AIProjectsOpen SourceAbout
Research/Giving a Local AI the Runbook Raised Incident-Triage Accuracy to 57%. A Bigger Model Reached 80%
Local AIRetrievalIncident TriagePre-RegisteredOriginal Research

Giving a Local AI the Runbook Raised Incident-Triage Accuracy to 57%. A Bigger Model Reached 80%

I gave two local models the same incident reports, retrieved runbook pages and system state. The 9B got all five required fields right on 56.9% of incidents. The bigger Coder got 80.3% right. The 9B finished more correct incidents per hour across different engines, but the accuracy gap was far outside the five-point margin I set for swapping one for the other.

Bart Porcincula · Published · Updated

Finding

On 3,000 synthetic incidents, Qwen3.5-9B with retrieved documents and system state scored 56.9% versus 80.3% for Coder IQ1_M on Niko1221's Strata engine with the same prompt. The paired gap was -23.4 percentage points (90% CI -25.0 to -21.9), outside the pre-set ±5-point equivalence margin. The 9B delivered 1,301 correct tasks per hour versus 1,233 for the Coder.

Synthetic incident triage with retrieved runbooks and system state (Nov 2026)

Three local setups on the same 3,000 incidents across three difficulty tiers. An incident passes only when category, root cause, affected systems, next action and escalation destination are all correct. The cloud comparison used a shared 120-item subset.

MeasurementValueNote
9B with incident text only0/3,000the escalation destination requires the runbook
9B with retrieval and state56.9%1,301 correct per hour
Coder with the same inputs80.3%1,233 correct per hour
9B minus Coder accuracy-23.4 points90% CI -25.0 to -21.9
Shared 120-item subset59.2% / 82.5% / 99.2%9B / Coder / Claude Haiku

The short version

An incident report may tell you something broke. The runbook and system state tell you who to page and what to do next. I built 3,000 synthetic incidents to see whether a 9B model with those inputs could come within five accuracy points of a much bigger Coder running on the same 16 GB card.

SetupAll five fields rightCorrect incidents per hour
Qwen3.5-9B, incident text only0.0%0
Qwen3.5-9B, with runbook and state56.9%1,301
Coder IQ1_M on Niko1221's Strata engine, with the same inputs80.3%1,233

The zero has a simple explanation. The incident alone omits the escalation destination, and a pass requires all five fields. With the runbook, the 9B got the escalation destination right every time. The next action (75.1%) and the list of affected systems (76.9%) were its two weakest fields. The larger model got both right more often.

What counts as a workable swap?

I set the margin at five points before scoring. The 9B trailed by 23.4 points, with a 90% interval of 21.9 to 25.0 points behind. Both ends are well outside the margin. Its slight lead in correct tasks per hour comes from answering faster, while the Coder gets more of a fixed queue right. Those are different decisions for a team choosing between the two setups.

On the same 120-item subset, Claude Haiku got 119 right. The Coder scored 82.5% and the 9B 59.2%. I ran Haiku in batches of ten via subagents, so that comparison is about scored answers on the subset. I have comparable timing only for the local arms.

Where the result applies

The incidents were synthetic and carried exact codes, and keyword retrieval found the correct document on every item. A real incident report may start with vague symptoms and a harder search. The 9B ran on llama.cpp and the Coder on Strata, so throughput combines model and engine effects. A later same-model comparison found a roughly sixfold engine gap on this incident pool. I used this same incident pool for a same-model comparison of the two engines.

Can giving an AI the company runbook make a smaller model good enough?

Providing the same reference material did not make the smaller model an acceptable replacement under this study's accuracy rule. The smaller model completed more correct incidents per hour, while the larger model got more incidents fully right. This is evidence about the tested incident task, not customer-service accuracy generally; choose an acceptance rule that reflects the consequences of a wrong result.

Cite this research

Bart Porcincula. "Giving a Local AI the Runbook Raised Incident-Triage Accuracy to 57%. A Bigger Model Reached 80%." Published 2026-11-07; updated 2026-11-08. https://porsync.com/research/local-model-incident-triage-retrieval-16gb

Permanent article URL