ResearchPractical AIProjectsOpen SourceAbout
Research/Write Down Your Bet Before You Test the AI
Pre-RegisteredEvaluationMethodologyAI Strategy

Write Down Your Bet Before You Test the AI

Most AI results are explained after the fact: run it, look at the numbers, then decide what they mean. I do it the other way round. Before a single test runs, I write down the question, the score that counts as a pass and how much wobble I'll treat as noise, then lock the page so it can't be quietly edited. This is why that habit matters for anyone buying or building with AI, with a live example where I had to change my method halfway and wrote the change down instead of hiding it.

Bart Porcincula · Published

Finding

Fixing the question, the pass mark and the noise level in writing before any result exists makes an AI test hard to bend after the fact; in a small-model study the noise floor was set at 0.017 on a 0-1 score before scoring began, and a mid-study change of method was logged as a dated amendment before the affected results were read.

Pre-registration record for a small-model study (Oct 2026)

What was fixed in writing before scoring began in one study: the hypotheses, the noise floor, the pass rules, and the amendments made along the way, each dated and written before the results it affects were read.

MeasurementValueNote
Noise floor set before scoring0.017twice the run-to-run spread of the same setup on a 0-1 score; differences smaller than this are treated as no difference
Hypotheses fixed in advance5each with its pass rule written before any result
Mid-study changeslogged as dated amendmentsincluding a faster way of scoring, written down before the affected results were read
Held-out test setsealed, never openedthe plan said to stop before opening it if the safety test failed, and it did fail

The short version

When someone shows you an AI result, the most useful question is not "how good is it?" It is "when did you decide what good meant?" If the answer is "after I saw the numbers", the result tells you less than it appears to. There are a dozen ways to measure any AI system, and picking the flattering one afterwards is easy, usually without anyone meaning to.

My fix is boring and it works: write the bet down first. Before any test runs, I record the question, what counts as a pass, and how big a difference has to be before I'll call it real. Then I freeze that page so it can't be quietly edited. Anything I change later gets a dated note saying what changed and why.

Why this matters to you

If you are deciding whether to adopt an AI tool, a vendor's demo is a bet that was placed after the results were in. The cheapest protection is to ask for the pass mark in advance: what score would make you say this isn't working? If nobody has an answer before the pilot starts, the pilot can't fail, and a test that can't fail isn't a test.

What "noise" means, and why I fix it first

Run the same AI setup twice and you'll get slightly different scores. That wobble is not a finding, it is just noise. So before testing anything, I measure the wobble and set a bar of twice that. In the study below it came to 0.017 on a score between 0 and 1. A difference smaller than that is reported as "no difference", even when it looks exciting.

A live example, including the awkward part

I have just finished a study on whether a small AI that lives on my own computer can learn to do a job by copying a much larger, paid one. The questions, the pass marks and the noise bar were all written down before the first score. Most of my bets lost, and they are recorded as losses: of the five questions, one came out as predicted, three did not, and one was a safety test that my own pass rule failed.

Partway through, one stage turned out to be painfully slow. Checking the work item by item was taking about 30 seconds each, because of how the program was juggling memory on a busy graphics card. I found a faster route that gives the same answers (it picked the same answer on the handful of items I compared, with small differences in the fine detail of its confidence).

I could have swapped it in and said nothing. Instead I wrote a dated note before any of the affected results were read: what changed, why, which items were scored the old way and which the new way, and that I had not yet seen the outcome. That note is the point of the whole exercise. The method changed; the reader can see exactly where, and judge whether it matters.

What to take from this

  • Ask for the pass mark before the pilot. If there isn't one, you're being shown a demo, not a test.
  • Ask what counts as noise. A "5% improvement" inside the wobble is not an improvement.
  • Expect changes, and ask to see them. Plans meet reality. The question is whether the change was logged before or after the result.
  • Look for a sealed final exam. Good tests keep a set of questions nobody tunes against, and open it once.

None of this makes a result true. It makes it checkable, which is the most you can fairly ask of any evidence.

Cite this research

Bart Porcincula. "Write Down Your Bet Before You Test the AI." Published 2026-10-22; updated 2026-10-22. https://porsync.com/research/write-down-your-bet-before-you-test-ai

Permanent article URL