ResearchPractical AIProjectsOpen SourceAbout
Research/Does an AI Grader That Can See the Answer Just Agree With It? Not Quite. The Explanation Did.
EvaluationAI GradingAnchoringLocal AIPre-RegisteredOriginal Research

Does an AI Grader That Can See the Answer Just Agree With It? Not Quite. The Explanation Did.

Part 6 of 6. A second AI checking the first one's work is a common safeguard. The worry is that a grader shown the proposed answer simply agrees with it. I tested two small graders on 734 answers, half of them wrong. Showing the answer did not make them agree more. Showing the first AI's reason did, and both graders were weak either way.

Bart Porcincula · Published

Finding

Across 734 proposed answers (367 wrong), showing Gemma-4-E4B the proposed label did not change its catch rate of wrong answers (35.4% shown against 36.5% blind, gap +1.1 points [-4.4, +6.3]), but adding the classifier's one-line reason cut catch by 12.0 points and a stated 95% confidence by 6.0; the Qwen3.5-9B grader's blind arm simply reproduced its own labels (100% agreement), so it could not test anchoring.

Anchoring in small AI graders, two models, five conditions (Oct 2026)

734 items built from the part 1 baseline run (queue and priority answers): 367 wrong proposals and 367 right. Two graders, Qwen3.5-9B and Gemma-4-E4B, each Q4_K_M, at temperature 0 with grammar-constrained outputs. Conditions: grader answers blind (picks its own label), is shown the proposal and says correct/incorrect, is also shown the classifier's one-line reason, is also told the classifier was 95% confident, or judges a random label instead. Catch rate is the share of wrong proposals flagged; false alarm is the share of right proposals flagged. Paired bootstrap 95% intervals.

MeasurementValueNote
Gemma, blind: catch / false alarm36.5% / 34.6%it disagrees about as often when the answer is right as when it is wrong
Gemma, shown the answer: catch / false alarm35.4% / 22.1%catch gap vs blind +1.1 pt [-4.4, +6.3], no anchoring effect on catching errors
Gemma, shown the answer and the reason: catch / false alarm23.4% / 15.0%extra catch drop -12.0 pt [8.4, 15.5] vs shown alone
Gemma, shown the answer and a 95% confidence: catch / false alarm29.4% / 16.9%extra drop -6.0 pt [3.3, 9.0]
Qwen, blind: catch / false alarm0.0% / 0.0%degenerate: it agrees with its own label on 100% of items, because the proposals came from the same model at temperature 0
Qwen, shown the answer: catch / false alarm18.5% / 13.6%
Qwen, shown the answer and the reason: catch / false alarm9.8% / 7.1%extra drop -8.7 pt [5.7, 11.7]
Qwen, shown the answer and a 95% confidence: catch / false alarm14.4% / 10.1%extra drop -4.1 pt [1.6, 6.8]
Random label shown, share flagged as incorrect (Qwen / Gemma)79.3% / 80.0%n = 580 wrong, 154 right; confounded, because random labels are not plausible near-misses

The short version

Checking one AI's work with another is a standard safeguard, and the standard worry is anchoring: if the checker sees the proposed answer, it will nod along. I tested that directly with two small graders. All six use the same setup: the open Qwen3.5-9B model compressed to 4 bits, run through llama.cpp on one 16 GB RTX 5060 Ti, at temperature 0 (the setting that makes a model as repeatable as it gets), on 734 answers taken from a classifier's output on those tickets, half of them wrong. Each experiment's questions, thresholds and analysis were written down before it was run.

  • Showing the answer did not make the stronger-signal grader agree more. Gemma caught 36.5% of wrong answers judging blind and 35.4% when shown the answer. The difference is inside the noise.
  • Showing the first AI's one-line reason did. With the reason, Gemma's catch rate fell by 12 points (to 23.4%) and Qwen's by 9. A stated 95% confidence cut it by 6 and 4 points.
  • Both graders were poor at the job anyway. Gemma flagged about a third of the wrong answers, and it flagged right answers almost as often blind (34.6%). Qwen caught under a fifth.
  • One of the two comparisons did not work. The answers being graded came from Qwen, so Qwen judging blind just reproduced its own label every time. It cannot tell us about anchoring.

Takeaway: what suppresses a small grader is the persuasive-sounding justification, not the answer. If you use a model to check a model, do not pass it the first model's reasoning or confidence, and measure how many wrong answers it really catches before you rely on it.

The setup

I took the classifier's answers on queue and priority questions, found 367 that were wrong and 367 that were right, and asked each grader in five ways:

ConditionWhat the grader sees
Blindthe ticket and the question; it picks its own label, and I compare
Shownthe ticket and the proposed label; it says correct or incorrect
Shown + reasonthe same, plus the classifier's one-sentence reason
Shown + confidencethe same, plus "the classifier reported 95% confidence"
Random labelthe same yes/no question about a label picked at random

The grader's output is restricted to the allowed words, so there is no format noise. I measured the catch rate (wrong answers flagged) and the false alarm rate (right answers flagged).

Results

GraderConditionCatch (wrong)False alarm (right)
GemmaBlind36.5%34.6%
GemmaShown35.4%22.1%
GemmaShown + reason23.4%15.0%
GemmaShown + confidence29.4%16.9%
QwenBlind0.0%0.0%
QwenShown18.5%13.6%
QwenShown + reason9.8%7.1%
QwenShown + confidence14.4%10.1%

Notice that the false alarm rate also falls as more is shown. The grader becomes more agreeable all round, not more discerning. Catching fewer wrong answers is only the visible half of it.

The Qwen blind row is not a result

The proposals came from Qwen at temperature 0. Asked the same question without seeing the proposal, Qwen gave its own label on every one of the 734 items. So "blind" for Qwen is self-reproduction, its catch is zero by construction, and the gap between blind and shown for Qwen says nothing about anchoring. Only the Gemma comparison tests it. The extra drops from the reason and confidence are still valid within each grader, but Qwen's reason came from Qwen, so part of that effect is self-agreement.

Do they just say yes?

No. Asked about a random label, both graders called it incorrect about 80% of the time. So they are not rubber stamps. But the classifier's wrong answers are plausible near-misses and the random ones are not, so this does not measure anchoring. It tells you that these graders reject obvious nonsense and accept plausible nonsense.

What I would not claim

  • This does not show anchoring is absent. It shows that for these two small graders, the answer alone did not matter and the justification did. A larger grader may behave differently.
  • The ticket set's own labels are noisy (the classifier is right 32 to 40% of the time), so "wrong" and "right" proposals are looser than in a clean dataset.
  • One sample of 734 items.

Part 6 of 6. Previous: answer first, explanation after. Start of the series: run the same prompt twice.

Cite this research

Bart Porcincula. "Does an AI Grader That Can See the Answer Just Agree With It? Not Quite. The Explanation Did.." Published 2026-10-29; updated 2026-10-29. https://porsync.com/research/does-an-ai-grader-that-sees-the-answer-agree-with-it

Permanent article URL