FRFree audit
← All articles

AI document extraction with human review: a practical guide

How a small business can let AI read invoices, forms and certificates while people check what matters: thresholds, cross-checks, review screens, audit trail.

Published on

A supplier invoice arrives as a PDF. An AI tool reads it in seconds and proposes the supplier, invoice number, date, amount and bank details. Handy. But if the bank details have changed since the last invoice, is that a reading error, a genuine change of bank, or a fraud attempt? The tool can’t tell. Someone has to look.

That’s what “human in the loop” document processing is about: getting the speed of AI on simple documents without letting through the things that matter. Re-checking everything defeats the purpose. Letting everything through is reckless. The work is in organising reviews so that only the exceptions reach a person.

What AI reads well, and where it breaks

On typed, consistent documents, current tools do a good job of:

  • recognising the type of document (invoice, delivery note, certificate, ID);
  • picking out header details: issuer, number, date, total;
  • summarising a long document;
  • spotting that an expected document is missing or isn’t the one you expected.

They are much less reliable with:

  • handwriting and poor scans;
  • tables, especially across several pages, and invoice line items;
  • ambiguous documents: a pro forma or a final invoice, a certificate that is current or expired.

Generative models have a weakness of their own: they can produce a plausible value that appears nowhere in the document. That’s why every extracted value should point back to where it came from on the page. If you can’t show where it’s written, don’t trust it.

Be wary of headline accuracy figures, too. A “99%” rarely says whether it counts fields or whole documents, or which documents it was measured on. The only figure that matters is the one you measure on your own documents.

The machine prepares, a person decides the exceptions

The principle is simple: the tool prepares everything, and a person only looks at what deserves it. A document goes to review when:

  • the tool isn’t confident about a field;
  • a business rule fails;
  • a required field is missing;
  • the document touches something sensitive, whatever the tool’s confidence.

Everything else goes through, with a record, and a sample is spot-checked from time to time.

Confidence thresholds and always-check fields

Most extraction tools return a confidence score per field. Microsoft says as much in the documentation for its document reading service: when accuracy is critical, the score can be used to decide whether to accept a value automatically or flag it for human review. The same page notes that not every field returns a score.

Two cautions:

  • A confidence score isn’t a guarantee. It’s a statistical estimate. A value read with high confidence can still be wrong.
  • Some fields are always checked, whatever the score. A change of bank details, an amount above a limit you set, a due date or expiry date, a legal identifier such as a company or VAT number.

Thresholds are set per field and per document type, not once and for all. Start cautiously, watch what reviewers correct, then adjust.

Cross-check against data you already trust

Before asking a person, ask your own data. Many errors show up with a simple comparison:

  • does the invoice match the purchase order and the delivery note?
  • are the bank details the ones on file for this supplier?
  • do the lines add up to the total?
  • are the dates consistent with each other?
  • is the name on the document the name on the file?

These checks work without AI, and catch a reading error just as well as an error in the document itself.

The review screen and the audit trail

Reviews have to be quick, or they’ll be rushed. A good screen shows the original document next to the extracted values, marks the flagged fields and offers three actions: accept, correct, reject.

For the audit trail, keep:

  • the original document, always linked to the record;
  • the values as extracted;
  • the corrections made;
  • who approved it, and when;
  • which tool or model version was used.

Watch out for rubber-stamping. When a tool is right nine times out of ten, people end up approving without looking. The EU AI Act even names the risk: “automation bias”, the tendency to over-rely on a system’s output. Two simple counters: track the correction rate (if it drops to zero, the review may have become a formality) and re-check a few automatically accepted documents now and then.

The EU rules in plain English

Three points, without the legal detail.

GDPR. Article 22 gives people the right not to be subject to a decision based solely on automated processing when it produces legal effects or similarly significantly affects them. Where such a decision is allowed, the person must at least be able to obtain human intervention. European data protection authorities have made clear that a token human step doesn’t count.

The AI Act. The human oversight requirement in Article 14 applies to “high-risk” systems. Reading invoices isn’t one of them in itself. But the list of high-risk uses includes, for example, filtering job applications and assessing a person’s creditworthiness, so a document pipeline that feeds those decisions moves into that category. The Act was amended in July 2026, and the high-risk obligations for those listed uses now apply from 2 December 2027. Separately, organisations that use AI systems must take measures to build their staff’s AI literacy.

Data protection guidance. France’s data protection authority, the CNIL, recommends starting from a concrete need, defining which uses are allowed and forbidden (for example, not putting personal data into the tool, or not letting it make decisions) and checking what the provider does with the data you send.

When not to use AI

  • Volume is very low. Typing ten documents a month costs less than setting up and monitoring a pipeline.
  • The document feeds an important decision about a person. Hiring, credit, housing: AI can help sort, but the decision stays human and must be explainable.
  • A single error is unacceptable and you’ll check everything anyway. Then AI adds a step without removing one.
  • The data is sensitive and nothing governs sending it. No clear legal basis, or a provider that may reuse what you send.
  • The document is already structured. A software export or a standard-format file can be read without AI, more safely.

Two examples

For a file that needs several supporting documents, AI can recognise each document, file it and spot what is missing, while a person checks doubtful cases and decides the file is complete. That’s our use case on supporting documents filed and chased. For PDF data sheets, where a unit error would end up on a public website, see PDF data sheets turned into a product database.

Among our client projects, the prospect summaries built from sales calls follow a rule we recommend everywhere: if a piece of information isn’t in the source, it is marked “not mentioned”, never guessed.

If documents are piling up and you’re wondering what AI could do with them, let’s talk. Sometimes the answer is “not much”, and that’s worth knowing too.

Sources

Is a process costing you time?

Describe it in a few lines. We will get back to you to set up the audit.

Request a free audit