Run the Same AI Prompt Twice: How Much of a "Change" Is Just Noise?
Part 1 of 6. Before you can say an edit to a prompt changed the answers, you need to know how much they change when you edit nothing. I measured that on 500 support tickets. Edits that look cosmetic, like reordering the options, changed more answers than a deliberate improvement did, and the improvement made the model slightly less accurate.
Bart Porcincula · Published
Finding
On 500 support tickets with Qwen3.5-9B, re-running an identical prompt changed 0 of 500 queue answers when one request ran at a time and 21 of 500 (4.2%) when four ran together; reversing the option order changed 111, adding a one-line definition per option changed 135 and lowered queue accuracy by about 4 points, and a whitespace-only edit changed 38.
Prompt-edit noise floor on 500 support tickets (Oct 2026)
Five prompt arms (same prompt re-run, whitespace-only, reversed option order, the ticket noun reworded, and a one-line definition per queue option) run on 500 customer-support tickets asking three questions each (queue, 10 options; priority, 3 options; incident yes/no), at one request at a time and four at a time. Counts are answers that differ from the baseline run. Attributable flips subtract the same-concurrency re-run floor, with paired bootstrap 95% intervals.
| Measurement | Value | Note |
|---|---|---|
| Same prompt re-run, one request at a time (queue answers changed) | 0 of 500 | two separate re-runs; the model is fully repeatable in this mode |
| Same prompt re-run, four requests at once (queue answers changed) | 21 of 500 (4.2%) | other identical pairs: 19, 20, 23, 28 (3.8-5.6%) |
| Whitespace-only edit (queue answers changed) | 38 of 500 | attributable +17 [5, 29] above the 21 floor |
| Rewording of the ticket noun (queue answers changed) | 42 of 500 | attributable +21 [9, 33] |
| Reversed option order (queue answers changed) | 111 of 500 | attributable +90 [72, 108], about 5 times the floor |
| A one-line definition per queue option (queue answers changed) | 135 of 500 | attributable +114 [94, 134] |
| Queue accuracy: baseline vs definitions | 33.0% vs 29.4% | the edit written to help lowered accuracy by about 4 points |
| Invalid answers in any run | 0 | answers were constrained to the allowed labels |
The short version
When someone says "I changed the prompt and 20 answers changed", the useful question is how many would have changed if they had changed nothing. I measured that. All six use the same setup: the open Qwen3.5-9B model compressed to 4 bits, run through llama.cpp on one 16 GB RTX 5060 Ti, at temperature 0 (the setting that makes a model as repeatable as it gets), on 500 customer-support tickets. Each experiment's questions, thresholds and analysis were written down before it was run.
- Run one request at a time and nothing changes. Same prompt twice: 0 of 500 answers differ.
- Run four at once and about 4% change on their own. Same words, same model, 21 of 500 queue answers flipped, in a repeat at the same four-slot setting. Any claim of "the edit changed 20 answers" at that setting is zero evidence.
- Edits that look cosmetic still beat that floor. Whitespace alone: 38. Rewording one noun phrase: 42. Reversing the order of the options: 111, about five times the floor.
- The edit written to help made things slightly worse. I added a one-line definition for each of the ten queues. It changed 135 answers, 27% of them, and net moved queue accuracy from 33.0% to 29.4%.
Takeaway: report how many answers changed alongside a same-prompt baseline, and judge a prompt change by accuracy with a margin of error, not by how different the outputs look.
Why measure "nothing changed" first
A language model run on your own hardware is only as repeatable as the way you serve it. With one request at a time and the randomness turned off, it gives the same answer every time, so any difference after an edit is caused by the edit. Serve several requests at once, which is how you get throughput, and the order in which tickets land in a batch nudges the arithmetic. That is a real, small, unavoidable amount of change. Call it the floor. Everything an edit does has to be read against it.
What I changed
All edits were to the instruction for the "queue" question (which of ten teams should handle this ticket):
| Edit | What it is |
|---|---|
| Whitespace | extra blank lines and spacing, no words changed |
| Reworded noun | "customer-support ticket" reworded as "support request" |
| Reversed options | the same ten options, listed in the opposite order |
| Definitions | one short line explaining each of the ten queues |
Results
| Edit | Queue answers changed (of 500) | Above the floor of 21 |
|---|---|---|
| Same prompt again | 21 | none (this is the floor) |
| Whitespace | 38 | +17 (5 to 29) |
| Reworded noun | 42 | +21 (9 to 33) |
| Reversed options | 111 | +90 (72 to 108) |
| Definitions | 135 | +114 (94 to 134) |
The size of the change did not track the size of the edit. Whitespace moved about as many answers as rewording one phrase. Reordering the options, which changes no information, moved about as many as adding definitions, which changes a lot. The model reads the whole prompt as a picture, and position matters.
The two other questions (priority, and whether it is an incident) were quieter, and the definitions edit left them alone, which confirms nothing leaked between questions.
Flips versus accuracy
Accuracy stayed within about 2 points for the cosmetic edits, even when more than a fifth of answers changed, because answers flip in both directions. The definitions edit did not help: queue accuracy fell from 33.0% to 29.4%. The labels in this ticket set are noisy (33% accuracy on a ten-way choice), so more explanation does not make the model agree with them more. Flip counts and accuracy answer different questions, so report both.
What I would not claim
- One model, one task, one 500-ticket sample. Because the labels are noisy, there is little room for any prompt to look better.
- The same-prompt floor is specific to this server setup. Other serving programs have their own floors.
- I am not saying cosmetic edits are bad. I am saying you cannot call them neutral without checking.
Part 1 of 6 in the series on small-model failure modes. Next: what one appended sentence does to the same model.
Cite this research
Bart Porcincula. "Run the Same AI Prompt Twice: How Much of a "Change" Is Just Noise?." Published 2026-10-29; updated 2026-10-29. https://porsync.com/research/same-prompt-twice-how-much-changes-small-model
Permanent article URL