The backfill arrived after the live update

Start with the supplier that can move backwards

The backfill reads supplier V-404 at revision 7. Before it writes, the live integration delivers revision 8. If the destination keeps whichever request arrives last, the backfill can successfully put the old supplier back. Both processes may report success. That is why I would decide what a historical run may replace before concentrating on how quickly it can reach its last page.

This fictional migration moves 10,000 supplier records into a new search system while the live integration remains active. The run needs a defined set of supplier identities, a transformation revision and a rule for observations that have become older than the destination. Population and value history are different: a supplier can belong to the migration while the revision read earlier is no longer the version search should keep. The job description has to say how that overlap is handled.

First explicitly ask whether the destination is meant to reflect historical facts or the current interpretation of old records. Rebuilding search usually wants current searchable state. Recreating an old report may need values and rules from a past date. Both are called backfills in ordinary conversation, yet their evidence requirements differ sharply. If the source no longer retains historical revisions, a current-record replay cannot honestly recreate the earlier report; that limitation belongs in the job description before anyone starts comparing totals.

Let the newer supplier revision stay current

At the destination, retain enough source identity and revision information to compare an incoming observation with what is already stored. If revision 8 is present, the delayed revision 7 backfill should not replace it merely because its HTTP request arrived later. The comparison must use the source’s documented ordering, not a guess based on a display timestamp.

For V-404, the proposed rule is to accept revision 7 into an empty destination, accept revision 8 over 7 and reject 7 after 8. An equal revision with identical transformed content is a repeated observation. An equal revision with different content requires investigation: perhaps the transformation revision changed, or the source’s revision contract does not cover all relevant fields.

The search document also depends on the transformation that built it. Call two transformation revisions M-3 and M-4: they may build different documents from the same supplier record. A guard using only the supplier revision could reject a legitimate rebuild as an observation already accepted. Give the transformation change its own generation or replacement policy, so the next engineer can tell whether the destination is protecting fresher source data or blocking a new representation.

Deletions need an explicit path too. A record absent from a partial page is not proof of deletion. A complete snapshot may support a membership comparison, while a change stream may provide deletion markers. Preserve a tombstone or equivalent revision evidence where needed so a delayed historical write cannot recreate a record that the live integration already removed.

Which ten thousand records does this run mean?

For the supplier example, a manifest might name export X-18, its generation time, 10,000 record identities and transform revision M-3. A replay of that manifest can process the same inputs if the export remains available. If the source offers only current records, the manifest should say so: it identifies the population and observations, not a reconstructible historical snapshot.

One option is a source snapshot or an export with a stable identity. Another is a documented query boundary, such as records created before a cutoff, together with an appropriate change-capture route. The correct choice depends on the source. A timestamp filter alone does not guarantee historical values remain unchanged while pages are read.

PostgreSQL’s transaction-isolation documentation illustrates the distinction within a database: successive statements at Read Committed can see different committed data, while Repeatable Read uses a stronger snapshot boundary. That does not imply an external paginated API supplies the same guarantee. A long export also has operational costs, so choose and test the source’s actual supported mechanism rather than borrowing the word snapshot from a database manual.

Cursor pagination helps traverse a collection according to the API’s contract. Stripe, for example, documents object-based cursors and a has_more indicator for its v1 list interface. That tells a caller how to request subsequent pages. It does not, by itself, establish that every object’s value is frozen for the duration of an application’s backfill. Traversal and consistency are separate questions.

The checkpoint has a crash boundary

Suppose the backfill writes suppliers 301 through 400 to the destination, then crashes before storing its checkpoint. On restart it may repeat those records. That is acceptable only if repeated application is safe under the chosen destination contract. Storing the checkpoint first would create the opposite failure: a crash could skip records that were never written.

A suitable transaction can couple the write and checkpoint when both live in one database. Across an API boundary, the local checkpoint cannot promise that the remote write committed with it. Use stable identities, repeat-safe replacement or the receiver’s documented deduplication, and retain uncertain outcomes for reconciliation. The restart plan needs to fit that arrangement, rather than assume that saving a local position made the external work atomic.

A checkpoint needs a defined work unit and a distinction between attempted and acknowledged records. Page numbers may shift and provider cursors may expire, so retain the manifest and source boundary necessary to restart deliberately. For the supplier job, a position labelled 400 is useful only with the ordering and export identity that explain which records preceded it and whether their writes were acknowledged.

A useful interruption exercise stops the job before a destination write, after the write but before checkpointing and after checkpointing. Observe the resulting destination population after resumption. The purpose is to establish whether records can be missed, repeated or made stale, not merely whether the worker process restarts without an exception.

Compare the destination before calling it complete

For the 10,000-record manifest, track accepted, already-current, superseded, rejected and unresolved outcomes. The categories should account for the expected population without counting retries as new suppliers. A run can reach the end while leaving unresolved failures; report that distinction instead of awarding a complete status because has_more became false.

Then compare the intended destination membership and relevant revisions with the source boundary. A simple row count can hide one missing supplier and one duplicate. A checksum can help detect a difference, but investigation still needs record-level evidence. Keep enough information to identify which records and transformation revisions explain a mismatch.

Reserve capacity for the live route while the historical job runs. Finishing the backfill sooner is a poor trade if current updates spend the day waiting and the search system becomes stale again. Set an acceptable recovery window, then watch both the historical progress and live delay against it. The task has two obligations during the overlap, even though only one of them has a conspicuous progress counter moving towards 10,000.

There is no universal requirement to run online. For a small, low-traffic system, a short agreed maintenance window and a stable export may be simpler and safer than a complex overlap protocol. The trade-off is business interruption versus coordination work. Make that decision openly rather than assuming that an online migration is required without comparing its coordination cost.

For X-18, account for the 10,000 supplier identities under M-3 and the chosen conflict rule. A newer live revision can make a superseded outcome correct; an uncertain write remains unfinished work even after pagination ends. Leave enough detail to explain why V-404 stayed at revision 8 and what a resumed run may still apply. Completion then describes the intended destination, rather than the worker having run out of pages to request.