ResearchPractical AIProjectsOpen SourceAbout
Research/Teaching a Small AI to Read Contracts: It Got More Careful, Not More Thorough
Small Language ModelsFine-TuningContract ReviewEvaluationPre-registrationOriginal Research

Teaching a Small AI to Read Contracts: It Got More Careful, Not More Thorough

I tried to train a small AI model, small enough to run on one ordinary computer, to find 41 kinds of clause in real contracts and quote them word for word. Compared fairly, with both versions reading the contract the same way, training made it far more careful (fewer false claims, almost no invented text) and slightly less thorough, and left the overall score unchanged. The bar I set in advance was not cleared, so the main comparison against a larger model never ran.

Bart Porcincula · Published · Updated

Finding

On 41-category contract-clause extraction, fine-tuning (LoRA) a roughly 4-billion-parameter model that reads contracts in 1,000-token chunks raised precision (the share of flagged clauses that were really there) from 0.78 to 0.93 and verbatim-quote fidelity from 87% to 97%, lowered recall from 0.87 to 0.82, and left the overall score (macro-F1) unchanged: 0.697 against 0.698 for the untuned model reading the same chunks (validation, 95% interval -0.043 to +0.061). No tuned version cleared the pre-registered bar, which was set against the untuned model reading the whole contract at once (0.727).

Contract-clause extraction: small-model training experiment (Oct 2026)

Finding which of 41 clause types a contract contains, and quoting the supporting text word for word, on public CUAD contracts: 347 for training, 61 for tuning decisions (validation), 102 held back for a final test. A larger model (one run) and an untrained small model were scored on the test set; four trained variants of the small model were scored on validation. One consumer graphics card. The overall score (macro-F1) averages across all 41 clause types and balances catching clauses against false alarms; 1.0 is perfect. Scored by a fixed script against a protocol written in advance.

MeasurementValueNote
Larger model, test set (overall score)0.783one run, 102 contracts
Untrained small model, test set0.7156.8 points behind; margin of error -10.0 to -3.0
Untrained small model, whole contract at once, validation set0.727the baseline the pre-registered bar was set against
Untrained small model, same 1,000-token chunks as the trained versions, validation set0.69878.3% of flagged clauses were real; it found 86.9% of those present; 86.5% of its quotes appeared word for word
Best trained version (S3), validation set0.69792.5% of flagged clauses were real; found 81.6%; 97.1% of quotes word for word; vs untrained on the same chunks -0.1 points, range -4.3 to +6.1
All four trained versions (S1 / S2 / S3 / S4)0.585 / 0.671 / 0.697 / 0.668the last round, with more clause-heavy examples and more training, got worse
Bar set in advance (trained version beats the whole-contract untrained model on validation)Not clearedso no trained model earned a gated run on the test set
S3 on the test set, exploratory (run later as a routing signal in a follow-up study)0.718 vs 0.705against the untrained model on the same chunks, 101 contracts; +1.3 points, range -2.3 to +4.9

The short version

I wanted to know whether a small AI model could be trained to do contract review nearly as well as a larger one. Contract review is a good test: the answers are checkable, the documents are long, and a wrong or invented quote is costly.

  • What training fixed: the trained model made far fewer false claims (92% of the clauses it flagged were real, up from 78%), and it almost never made up text (97% of its quotes appeared word for word in the contract, up from 87%).
  • What it did not fix: the trained model missed more clauses than the untrained one (it found 82% of them, against 87%). It became more careful, not more thorough. The overall score did not move.
  • The outcome: none of the trained versions cleared the bar I set in advance, so the main question, a trained small model against a larger one, was never tested. That is "unanswered", not "no".

Corrected 2026-10-12. The first version of this piece compared the trained model, which reads contracts in short chunks, against an untrained model that read each whole contract at once, and attributed the whole gap to training. A like-for-like untrained run now exists. The section "A correction" below explains what changed.

Takeaway: before training a small model for document work, decide which mistake costs more. If a wrong or invented quote is the expensive error, training helped. If a missed clause is the expensive error, it did not.

What I was testing

The question was narrow, and I wrote it down before running anything. Can a small open model (Gemma 4 E4B, roughly 4 billion parameters), trained on the public CUAD contract collection with a lightweight method called LoRA, land within five points of a larger model at two jobs? The first job is deciding which of 41 clause types a contract contains (for example "non-compete" or "source-code escrow"). The second is quoting the supporting text word for word.

The test plan, the data split (347 contracts for training, 61 for tuning decisions, 102 held back) and the rules for what could be changed after seeing results were fixed in a pre-registration file first. The 102 test contracts were reserved for a single final comparison.

The starting point

On the 102 test contracts, the larger model scored 0.783 overall. The untrained small model, given each whole contract in one call, scored 0.715. That 6.8 point gap is real: comparing the two contract by contract puts the difference at -6.8 points, with a 95% range of -10.0 to -3.0. The untrained small model had a recognisable weakness. It found most clauses, but only about three quarters of its quotes (75%) actually appeared in the contract word for word.

Four rounds of training

Training examples have to be short, so the trained versions read each contract in short passages (about 750 words each) and the results are merged: a clause counts as present if any passage says so. I fixed that merge rule before any test run. The four trained versions differ in how many passages with no clause in them were kept, and how long they trained. All four were scored on the validation set.

The untrained baseline in my plan read the whole contract at once, so it differs from the trained versions in two ways, not one. The table therefore shows both untrained versions: the whole-contract one my bar was set against, and an untrained run reading the same passages as the trained versions, which is the fair comparison.

VersionOverall scoreFlagged clauses that were realClauses foundQuotes word for word
Untrained, whole contract0.72782.4%86.7%83.2%
Untrained, same passages0.69878.3%86.9%86.5%
S1 (110 training steps)0.58592.5%62.1%96.4%
S2 (300 steps, fewer no-clause examples)0.67194.4%76.1%97.3%
S3 (600 steps, fewer still)0.69792.5%81.6%97.1%
S4 (900 steps, fewest no-clause examples)0.66890.3%78.4%97.3%

What training bought, and what it didn't

Against the untrained model reading the same passages, training bought two things, immediately and consistently. Precision rose from 0.78 to about 0.93, meaning the trained models made far fewer false claims that a clause was present. And quote fidelity rose from 87% to 97%, meaning they almost never invented text.

It did not buy recall, the share of real clauses the model actually finds. Every trained version missed more clauses than the untrained one. The closest, S3 (recall 0.816), still trailed it (0.869). The trained models lean toward answering "absent". The rare clause types, such as source-code escrow and unlimited-license clauses, were the weakest.

The two effects cancel on the overall score. S3 scored 0.697 against 0.698 for the untrained model on the same passages, a difference of -0.1 points with a margin of error from -4.3 to +6.1. Against the whole-contract model the gap is 2.8 points (range -7.3 to +3.9), and almost all of that comes from reading in passages, not from training. Either way, with 61 contracts the data cannot say whether S3 is slightly worse, equal or slightly better. It can say that S3 is not meaningfully better, which is what my bar required.

The last round made it worse

The first three rounds improved steadily, which suggested the answer was more of the same. The fourth round pushed further, with an even more clause-heavy training mix and about 2.2 passes over the data. It got worse (0.668), losing both recall and precision against S3. That is the signal to stop. I ran out of cheap levers at this scale rather than out of patience.

I also got something wrong along the way. I had assumed the third round saw only part of the training data and that training longer was the obvious next step. It had already covered it about 1.3 times. That claim is corrected in the project write-up.

Where the bar landed

The bar I set in advance was that a trained version must beat the untrained (whole-contract) one on validation before it earns a single run on the test set. None did, and the main planned comparison, trained small model against the larger model, did not happen. That verdict stands as written: I am not moving the bar after the fact.

A correction

The first version of this piece made two statements that turned out to be wrong.

  • It said the untrained model also read the contract in passages. It did not. It read each whole contract in one call, so the original comparison changed the input and the training at the same time. A later study (the cascade test) needed an untrained model reading the same passages, and that run is the fair baseline now shown above. The practical conclusion is sharper, not weaker: training trades recall for precision and quote fidelity at an unchanged overall score.
  • It said S3 was never run on the test set. The same follow-up study ran it there, as an input to a routing signal rather than as a contender. On 101 test contracts it scored 0.718 against 0.705 for the untrained model on the same passages (+1.3 points, range -2.3 to +4.9). That number is exploratory: it chose nothing, and it does not change the verdict above.

Limits

  • The bar was set against a baseline that read whole contracts, while every trained version read passages (see "A correction").
  • Four trained versions were chosen using the same 61 validation contracts, so their validation scores are, if anything, optimistic.
  • The larger model is a single run of a mid-tier model, not a ceiling.
  • One small base model, one random seed, one set of instructions.
  • A small repair rule (when the model's reply was cut off one bracket short of valid, I added the missing bracket) was derived from validation failures and then frozen for every version. It moved the first version from 0.537 to 0.585.
  • 102 test contracts cannot reliably separate a 2-3 point deficit from a tie, so a smaller gap would have been reported as inconclusive rather than a win.

What I would take from it

  1. Training a small model trades completeness for care. If your cost is a wrong or made-up quote, the trained model is the better extractor even at a similar overall score. If your cost is a missed clause, it is not.
  2. Set the bar first, then respect it, and check the baseline sees what the contender sees. The protocol stopped me from promoting a version that had not earned it. It did not stop me from comparing a chunked model against a whole-contract one.
  3. Watch for the point where the curve turns. Three rounds of improvement made a fourth feel obvious, and it was not.
  4. Report the negative result with its uncertainty. "Not cleared, and here is how uncertain we are" is a finding.

Nothing from this experiment is deployed. The data is the public CUAD set. The protocol, scores and scripts are in the project repository.

Cite this research

Bart Porcincula. "Teaching a Small AI to Read Contracts: It Got More Careful, Not More Thorough." Published 2026-10-03; updated 2026-10-07. https://porsync.com/research/tuning-a-small-model-precision-not-recall-cuad

Permanent article URL