What does the evaluation permit us to release?
Separate observed results, inference and release authority, so an attractive score cannot quietly approve a different product.
AI tooling7 min read
A blank field can be the correct answer
Twelve synthetic invoices reveal the trade between automatic coverage, unsupported accepted values and the review work an extraction policy creates.
A completed form can conceal unfinished reading
Ten of twelve invoice totals have appeared in the form. That looks like progress until someone asks where the numbers came from. In the synthetic example below, one of those ten has no support in its source. A count of completed fields would give it exactly the same credit as a correct extraction, even though it is the value we most need the workflow to stop.
Allowing a blank is the beginning of a better design, not the whole design. The next decision is which non-blank values the workflow accepts without review. A cautious policy can leave more work for colleagues; a permissive one can let an ordinary-looking error enter another system. Both consequences belong in the business case.
I would keep the candidate outputs fixed and change the acceptance rule before comparing any headline success rates. That lets us see what the workflow has actually bought: fewer reviews, fewer unsupported automatic values or some combination of the two. The twelve documents here are synthetic, and the arithmetic says nothing about the performance of an AI model or OCR supplier. It does show what a permissive acceptance rule can hide.
The supplied fixture contains annotated totals and invented candidate outputs. Seven candidates are correct with usable source locations. Two more have the correct value but insufficient automatic evidence. One contains an unsupported value. Two contain no candidate total. By holding that set constant, we can examine the decision around the tool rather than mistake a more permissive acceptance rule for a more capable model.
Make a blank carry information
A total needs a record of the candidate value, currency context, extraction status and source location where available. The status should explain why a value remains unresolved. Absent means the document does not supply it; ambiguous means the source cannot yet support one answer; a processing failure means examination did not complete. Those distinctions tell the reviewer whether to inspect the source, seek another document or rerun the extraction.
The record below illustrates the ambiguous case: a candidate value is present, but it is not automatically accepted. Keeping the value can help a reviewer. Its presence must not quietly override the status. Similarly, null is not zero, and an omitted property is not the same thing as a property explicitly set to null. JSON Schema's documentation explains those structural distinctions.[1][2]
A schema can insist on a status field and reject an unknown status name. It has no way to read the cited page and decide whether the number is a total or a subtotal. Check what you can outside the model: whether the text exists at that location, whether its currency is permitted and whether the record is consistent enough to use. An unresolved check should remain visible rather than being tidied into an answer that merely looks plausible.
Keep the candidate and the eventual human correction separately. If a reviewer turns an unsupported 12000 into the annotated 1800, that is a recovery by the workflow, not evidence that extraction was correct. Combining repaired values with original outputs would make the next evaluation look better while concealing the review effort responsible for it. The business needs both measures, particularly if review capacity is already limited.
json
{
"document": "invoice-09",
"field": "total_minor",
"candidate": 5000,
"status": "ambiguous",
"source_location": null,
"reason": "Two candidate totals require source review",
"automatic_acceptance": false
}The candidate can exist without being accepted
The ambiguous invoice-09 record illustrates field semantics rather than the three acceptance-policy totals.
Scroll sideways to see all columns.
| Field | Illustrative value | Consequence |
|---|---|---|
| candidate | 5000 | Useful to the reviewer; not automatically accepted |
| status | ambiguous | Two totals still require source review |
source_location | null | No supporting location established |
automatic_acceptance | false | Downstream action must not use the value as settled |
Three policies, one set of outputs
Policy A accepts every non-blank candidate, automatically handling ten of the twelve documents, or 83.3 per cent. Nine accepted values match their annotations, while one is unsupported. That gives an unsupported share of 1 divided by 10, or 10 per cent, among automatic acceptances. The remaining two documents go to review. Reporting coverage without that accepted-set calculation would omit the policy's most consequential error.
Policy B accepts only a found status with a usable supporting location, using flags assigned in the fixture. Seven of twelve pass, or 58.3 per cent; five need review. All seven are supported and correct in these toy rows. The two correct values held back are important too: the policy cannot accept them automatically without the evidence its rule requires, even though we can see the answer sheet.
Policy C accepts nothing automatically and sends all twelve for a human pass. Its automatic coverage is zero. That does not make the completed process infallible: people can misread documents, and an absent or illegible total may require a replacement from the supplier rather than another look at the same image. Human involvement is a different route through uncertainty, not a mathematical guarantee of truth.
The policies demonstrate a coverage-risk trade-off: accepting fewer predictions can change the error rate within the accepted set. Selective prediction studies that relationship.[3] This example does not implement the SelectiveNet method or claim its research results transfer to invoices. It borrows the distinction between deciding an answer and deciding whether to act on that answer. A single accuracy score would hide the operating choice we are trying to make.
Price the work that comes back
Assume, only for illustration, that each routed document receives a three-minute initial review. Policy A allocates six minutes, Policy B fifteen and Policy C thirty-six. Those are 2 × 3, 5 × 3 and 12 × 3 respectively. They exclude follow-up, correction after downstream acceptance and the business consequence of the unsupported value. Calling them total cost would imply evidence the example does not contain.
Invoices 08 and 09 are the awkward ones. We can see that their candidate totals match the annotations, yet Policy B still sends them to a person. The acceptance rule does not have our answer sheet. At the moment it must decide, the support is insufficient, even though a later review will vindicate the number. Treating those referrals as wasted effort would ask the policy to know the outcome of work it has not yet had done.
To review the ambiguous record, a colleague needs the original source alongside the candidate location. The available outcomes should include confirmation, correction, a request for clearer material and an unresolved result with a reason. Highlighting the suggested value and asking for approval does little to address the missing evidence. The screen needs to make independent examination practical if the human review is expected to improve the decision.
If five reviews will not fit into the team’s day, loosening the rule is only one possible response. Better source material, fewer eligible document types or a faster side-by-side source view may change the work more usefully. The team can examine which of those changes would help the five referrals. Otherwise connecting the automation may simply move work into a review queue that has no capacity to finish it.
The acceptance rule changes the work, even when extraction stays the same
Twelve synthetic candidates: nine correct non-blanks, one unsupported non-blank and two blanks. Review time assumes a three-minute initial pass per routed document. These are illustrative calculations, not model performance results.
Scroll sideways to see all columns.
| Policy | Automatic coverage | Unsupported accepted | Initial review allocation |
|---|---|---|---|
| A · accept non-blank | 10/12 = 83.3% | 1/10 accepted = 10% | 2 × 3 min = 6 min |
| B · require status + support | 7/12 = 58.3% | 0/7 in this fixture | 5 × 3 min = 15 min |
| C · review every document | 0/12 = 0% | No automatic acceptances | 12 × 3 min = 36 min |
Let the trial challenge the acceptance rule
In an actual trial, establish the annotations before reviewers see the tool’s suggestions. If two reviewers disagree about which of two printed amounts is the total, that is a source-truth dispute to resolve or retain as ambiguous. Scoring either candidate against an unquestioned answer would turn that dispute into an apparent extraction error. Include conflicting totals, absent fields and poor scans in material the business is permitted to use, then keep separate held-out documents for assessing changes. Otherwise repeated repairs can improve familiar examples without answering whether the policy handles another ambiguous invoice.
Report automatic coverage, unsupported accepted values, review outcomes and time separately. Inspect a sample of automatically accepted records too. Otherwise the workflow learns about errors only when someone complains downstream, and a policy can look excellent simply because it sends its observable uncertainty to humans while leaving quiet errors unexamined. Do not present an uncalibrated confidence value as an evidence threshold without testing what it predicts.
The downloadable fixture lets a reader recalculate the three policies and change the assumed review time. It contains annotated totals and candidate outputs rather than document images or an extraction-service run. Use it to choose the acceptance comparisons an actual trial should preserve: which unsupported values escaped, which correct values were referred and how much source review those referrals required.
A blank is useful when it prevents an unsupported total becoming an accepted one. It is less useful when referrals pile up faster than anyone can read them. Before these values feed another system, I would want to see both sides of that choice: how the acceptance rule handles unsupported numbers and how the team finishes the reviews it creates. The twelve-row calculation gives a starting point for that conversation, not a reason to trust a filled-in box.
Working on a workflow like this? Hamsa helps finance and operations teams connect their tools and handle the exceptions, starting with one workflow.
See the Payment Operations PilotReferences & further reading
Primary sources behind this article. Living documentation may change.
Practical engineering. A human perspective.
More about Hamsa Software and the person behind it. Editorial notes.A few more things
worth a read.
Back to the reading room 
The prompt did not change. The system did.
Version the resolved configuration around an AI response so comparisons, investigations and rollback refer to the system that actually ran.
The notice was withdrawn. Why is the assistant still using it?
Freshness follows source dependencies through indexes, summaries, caches and in-flight answers—not just the original URL.