Practical intelligence for accountable AI programmes.
Intelligent Document Processing
A practical, technology-neutral guide to controlling documents from secure intake and extraction through human review, acknowledged delivery and retention.
ModelFold Editorial Team9 min read
An intelligent document processing pipeline succeeds only when it controls the entire document journey, not merely the extraction result. A model might read every requested field correctly while the service accepts the same file twice, sends one result to the wrong queue and records completion before the destination accepts it. The extraction worked, but the service failed. A production design therefore covers intake, preparation, classification, extraction, validation, routing, review, delivery and the information lifecycle, with enough evidence for operators to locate a document, understand each decision and recover failed work.
Key operating principles
A sequence becomes operable when every logical stage has a written contract covering what it accepts, what it durably produces, what permits progression, who owns it and where failure goes. Use eight logical stages: capture, preprocessing, classification, extraction, validation, routing, human review and retention. A deployment may combine several stages in one service, but it should not hide their distinct outputs or controls. Unsupported input, bounded retry, validation failure, review and successful delivery should remain visible outcomes.
Create a stable identity at intake and associate the original, permitted derivatives, versions, decisions, history and accepted result with it. This traceable record lets operators distinguish a delayed case from a lost one and a legitimate retry from duplicate processing. Use the matrix during design workshops rather than after procurement: an empty owner, output or failure cell exposes an operating gap before it becomes a production incident.
| Stage and owner | Accepted input | Durable output | Progression control and failure route |
|---|---|---|---|
| Capture — intake owner | Document from an authorised channel, source metadata and processing purpose | Preserved original, stable identity, receipt, duplicate status and initial state | Acceptance controls; continue, reject, quarantine or request recapture |
| Preprocessing — document operations owner | Preserved original and declared document constraints | Normalised pages, text, layout, quality facts, transformations and page lineage | Quality and reproducibility checks; continue, bounded retry, recapture or specialist review |
| Classification — taxonomy owner | Prepared pages, text, layout and approved taxonomy | Class, packet boundaries, taxonomy version, confidence where available and selected schema | Known-class policy; extract, reclassify, review assembly or route as unknown |
| Extraction — schema owner | Classified pages and a versioned extraction schema | Raw and normalised candidates, types, omissions, source locations and processor version | Schema and provenance checks; validate, retry, raise a schema exception or seek specialist review |
| Validation — control owner | Extracted candidates, rules, reference data, source metadata and confidence policy | Field and document results, reason codes, severity and proposed route | Declared rules; straight-through eligibility, retry, recapture, quarantine or review |
| Routing — workflow owner | Validation result, current state, destination, policy and priority | State transition, route reason, priority, attempt count and acknowledgement expectation | Allowed transitions and retry limits; deliver, queue, quarantine or record terminal exception |
| Human review — queue owner | Original evidence, candidates, source locations, failed controls, history and permitted actions | Confirmed or corrected record, reason, reviewer identity, time and reintegration state | Review authority; approve, correct, recapture, escalate, reject or leave unresolved |
| Retention — information owner | Source and derivative artifacts, final output, review history, metadata and approved policy | Retention class, protected state, hold or transfer status, disposition and deletion evidence | Owner-approved lifecycle; maintain, hold, transfer, dispose, delete or escalate missing policy |
Accept documents through a controlled boundary, preserve the received original, and create a reproducible preparation package for later stages. Use layered acceptance controls appropriate to the threat model: authorised channels, format allowlists, type and signature checks, generated storage names, size and decompression limits, segregated storage, and scanning where appropriate. No single check establishes that a file is safe, so record rejection, quarantine and recapture as explicit states rather than letting questionable content reach parsers or models silently.
Preparation may produce native text, OCR text, corrected orientation, page order, layout information, quality findings and a record of each transformation. Quality signals can produce false positives, so they are evidence for routing, not infallible verdicts. Rotation or enhancement can improve readability, but it cannot restore content that was cut off or never captured. Route that case to recapture or a named exception. Keeping the original unchanged also lets reviewers determine whether a defect came from the source or from processing.
Keep classification, extraction and validation logically distinct because each answers a different question and produces different evidence. Classification identifies a document or page type, establishes packet boundaries and selects the relevant extraction schema. Its output should include the class, taxonomy version and confidence where available. Unknown, ambiguous and mixed classes need a named route instead of being forced into the nearest familiar schema, which can make a technically successful extraction structurally wrong.
Extraction returns candidate fields, tables or entities and should retain raw and normalised values, declared types, omissions, processor version and source location where available. Page coordinates, reading order and table relationships help a reviewer trace a value back to the source. Validation then tests those candidates against a declared contract. It is a separate logical stage for presence, type, format, range, duplicate, reference and relationship checks. Passing those tests means the result conforms to declared controls; it does not establish that the document is authentic or that its substantive claims are true.
Confidence is a routing signal, not a substitute for validation or evidence of substantive truth. Raising a confidence threshold generally increases precision while reducing recall because more predictions, including some correct ones, are excluded. Set straight-through and review policies from representative documents for the relevant class, field, use and consequence; no vendor example supplies a universal cut-off. Measure false acceptance and false rejection together, then let accountable owners decide which cases may proceed and which need review.
A document pipeline is only as dependable as its least explicit hand-off.
Route every success, failure and uncertainty through an explicit state transition with a reason and an accountable destination. Straight-through delivery, bounded retry, recapture, quarantine, specialist handling, human review and terminal exception should not collapse into one error queue. Carry the current state, reason, priority, attempt count, destination and expected acknowledgement so operators can find loops, orphaned cases and duplicate delivery attempts. Retry ceilings must be explicit; otherwise a recoverable problem can become an invisible processing cycle.
A useful review packet provides the original evidence, candidate value, source location, failed controls, relevant confidence signals, processing history and only the actions the reviewer is authorised to take. Queue ownership, age, capacity, target response time and unresolved-case escalation are controls, not administrative details. A human branch cannot protect the service if nobody can service it. Limit access to the context needed for the task and provide clear options to confirm, correct, request recapture, escalate or reject.
Record reviewer identity, time, reason, before-and-after values, disposition and reintegration outcome. A correction should not automatically become approved training data because it may still need quality, consent and governance checks. Finally, do not mark processing complete until the destination acknowledges the accepted output or the pipeline records a named delivery failure. That boundary prevents a successful extraction or queue exit from being mistaken for successful business delivery.
The pipeline remains controlled by applying owner-approved lifecycle rules, monitoring the complete service and governing every material change. Treat the source, derivatives, extracted data, review record and operational logs as distinct artifact classes when assigning metadata, access, holds and disposition. A controlled record may include the source, permitted intermediate artifacts, final structured output, confidence data, validation outcomes and processing history, but retention should be justified rather than indefinite by default.
There is no universal retention period: records, privacy, security, legal and business owners must determine the applicable rules for each document class and jurisdiction. Retention is an owner-approved lifecycle covering capture, maintenance and use, holds, transfer, disposition and evidence of authorised deletion where applicable. U.S. federal records guidance offers useful planning categories, but it is not a rule for South African private-sector organisations. Local requirements and active obligations must govern the actual policy.
Finish the design workshop by checking whether every stage has an owner, accepted input, durable output, progression control, named failure route, measurable service objective and recoverable state. Bring security specialists into intake and access design, and records and privacy owners into lifecycle decisions. Obtain qualified advice for jurisdiction-specific or regulated requirements. Legal, clinical, credit, insurance, tax and other high-stakes judgements must remain with appropriately qualified human authority; validation and review routing do not remove the underlying risk. Fill the matrix before selecting services or setting automation targets, because unresolved contract cells reveal what technology alone cannot fix.
A practical logical model has eight stages: capture, preprocessing, classification, extraction, validation, routing, human review and retention. A deployment may combine stages within one service, but each stage should still have an explicit input, output, progression control, owner and failure route.
Classification identifies the document or page type, establishes packet boundaries and selects the appropriate schema or processor. Extraction uses that context to return candidate fields, tables, entities and typed values, ideally with enough page or geometry provenance to trace each result to its source.
Human review should be an explicit route for defined quality, rule, confidence or consequence conditions, not a general dumping ground for errors. Reviewers need appropriate source evidence, clear authority and permitted actions, while the queue needs an owner, service objective, capacity and escalation path.
There is no universal confidence threshold. Evaluate representative documents for the relevant document type, field and downstream use, then weigh false acceptance against false rejection and the consequences of each. Confidence should inform routing alongside validation rules and human authority.
Distinguish the received original, derivatives, extracted data, review history and operational logs rather than assigning one rule to everything. Appropriate organisational owners should set metadata, access, hold, transfer, retention and disposition requirements for each document class and jurisdiction, including deletion evidence where required.
This article was researched using the following sources:
We report on how AI actually lands inside a business. Our work starts from named sources, separates what we found from what we think, and uses AI assistance for research and drafting under documented editorial controls. We are not a substitute for individual expert review.
A practical South African guide to mapping real workflow behaviour, testing approvals, defining handoffs and resolving exceptions before automation.
A practical field-research method for testing workflow complaints, locating recurring constraints and deciding whether AI deserves a controlled trial.
A practical method for building reproducible AI evaluation scenarios that cover routine work, difficult boundaries, known failures and prohibited behaviour.
