Extracting the same fields across a document set

Extraction is the task these systems are genuinely good at, and the one most often done in a way that cannot be checked.

The usual approach is to ask for a summary of each document and read the summaries. That produces something, and it produces something different every time, and there is no way to tell whether document 34 was handled the same way as document 4.

Structured extraction fixes that, and it comes with something more valuable than the output: a way to measure whether it worked.

Define the fields before you touch a document

Write the field list first, on paper, before any tooling. For each field, three things:

• The name, exactly as you will use it
• The permitted values — a date, a number, one of a fixed set of options, or free text as a last resort
• What it means when the field is absent

That third one is where extraction quietly breaks. If a contract has no termination-for-convenience clause, the correct output is not present, not an empty cell and certainly not a plausible guess. Empty and absent are different findings, and a system that conflates them will hand you a spreadsheet where you cannot tell which is which.

Prefer fixed options to free text wherever the substance allows. “Monthly / quarterly / annually / other” is checkable. “Payment frequency” as free text gives you eleven spellings of quarterly.

Build the answer key

Take twenty documents. Fill the fields in by hand. That is the answer key.

This is the small-scale version of test, evaluation, verification and validation. The NIST AI Risk Management Framework Core calls for repeatable evaluation processes, metrics and methods to be documented rather than inferred from a convincing demonstration.

Ninety minutes, and it is the entire difference between a tool you can defend and one you cannot. Pick the twenty deliberately: a few clean standard ones, a few with the awkward drafting you know exists in the set, at least two where a field is genuinely absent, and one scan if scans are in scope.

Do this before you run anything. If you build the key afterwards you will read the machine’s answer while filling it in, and the key becomes worthless.

Run, then read the errors separately

Run the extraction across the twenty. Compare. Then split the errors into two piles, because they mean different things and have different fixes.

Wrong value. The field was found and the answer is incorrect. Usually an ambiguous field definition — two clauses could both plausibly be “the termination provision” and you did not say which governs. Fixable in the instruction.

Wrong presence. A field reported as present that is absent, or absent that is present. This is the serious one. A wrong value is visible when someone reads the row; a fabricated presence looks exactly like a real finding, and nobody re-reads the contract to check.

Track them separately. A tool at 90% overall might be at 97% on values and 60% on presence, and those are two completely different situations. One is usable with a review step. The other is not usable.

What to do with the number

You now have a figure — say, forty-seven of fifty fields correct across twenty documents, with the errors concentrated in one field.

Three things follow, and the third is the one that matters.

Publish the field-level breakdown, not the headline. “94%” invites a challenge you cannot answer. “Forty-seven of fifty, with two of the three errors on the governing-law field where the drafting is inconsistent” invites a conversation about the governing-law field, which is a conversation you can have.

Put the review gate where the errors are. If presence errors cluster in one field, that field gets checked every time and the others get sampled. A uniform review step across all fields wastes the measurement you just took.

Say what the number does not cover. Twenty documents from one deal type is not evidence about a different deal type. A number from clean PDFs says nothing about scans. State the boundary alongside the figure, every time, or somebody will eventually apply it where it does not hold.

The honest limit

This gives you a review queue with a known error profile. It does not give you a reviewed document set.

The difference is that you now know where to look and roughly how often you will find something — which is a great deal better than reading fifty agreements at uniform attention, and considerably worse than having read them. Both halves of that sentence are worth saying out loud when you present it.