The next page may no longer start where you left it

A small result can look remarkably complete

Thirty familiar records and a sensible total can make the first page look like a complete report. If hundreds more remain, the output may be useful-looking rather than useful for the decision someone intends to make. The integration needs to know how the endpoint continues and what counts as the end. A successful response proves it retrieved a page; neither its familiar customers nor its plausible totals tell us that it traversed the requested population.

GitHub's current REST guidance documents continuation through Link headers and rel=next. It also notes that pagination parameters differ between endpoints.[1] Following the documented continuation is the first requirement. It is not the last: reaching the end of a changing list does not automatically establish that the collected records represent one coherent population. Traversal and consistency are different promises, and the word page supplies neither without a contract.

Five invented identifiers are enough to make the failure inspectable: [5, 4, 3, 2, 1], fetched two at a time. The small Python program runs controlled deletion, insertion, snapshot and checkpoint cases without calling an API. Every missing or repeated identifier is then visible. These are examples of possible traversal behaviour, not claims about GitHub’s ordering, cursors or snapshot guarantees. The endpoint’s own contract would have to tell us which of those behaviours a real export can rely on.

The second page moved underneath the offset

At the beginning, the list is [5, 4, 3, 2, 1]. The first page returns [5, 4]. Before the next request, identifier five is deleted, leaving [4, 3, 2, 1]. A request for two records at offset two now returns [2, 1]. The integration finishes with [5, 4, 2, 1] and never fetches three. No page request failed. The omission is caused by applying a position in an earlier arrangement to a later arrangement.

Deduplicating the collected identifiers does not repair this case. Three was never fetched, so there is no duplicate operation that can recover it. In a separate insertion case, adding six before the next offset request yields [4, 3] for page two, repeating four from page one. Deduplication can remove that repeated four, but it still does not define whether the newly added six belongs in the export. Correcting repeated observations and defining the desired population remain separate tasks.

Retaining snapshot S1 fixes the original list in the model. Its pages stay [5, 4], [3, 2] and [1] while the separate live list changes. The retained copy provides that guarantee. An opaque cursor, by itself, tells us little: it might refer to a snapshot, a position or another traversal strategy. Before describing the pages as a frozen population, find out which meaning the API supplies. The shape of the continuation token cannot answer a question about the data it traverses.

For comparison, the toy code asks for immutable identifiers below the last seen value, four. After deleting five, it still obtains [3, 2, 1], avoiding this offset-shift omission. That depends on stable identifier ordering. Records entering a filter or changing their sort key pose different questions, as do changes to the contents already fetched. Before choosing the alternative, establish whether the endpoint can preserve the population the export needs; replacing an offset with a last-seen identifier does not decide that population on its own.

Describe the population the export is meant to represent

An export might mean records present at the start, records currently matching when each page is read, or all changes since an earlier checkpoint. Those are different products. A snapshot can suit a coherent report while becoming expensive to retain for long-running traversal. A live list can suit an activity screen while being unsuitable as an authoritative historical export. The consumer should know which meaning applies before it compares totals or makes decisions based on absence from the file.

A cut-off sounds like a solution until an older record changes its status or amount after the cut-off. Created before this time constrains creation, not necessarily the later contents of every selected record. A stable sort also needs a tie-breaker when records share its primary value. Those are endpoint questions a pagination helper cannot infer. If the available consistency is weaker than the decision needs, keep that limitation visible and choose an appropriate export or reconciliation route, rather than make the helper promise a population it cannot hold still.

There is a practical distinction between no next link and no more matching business data. The former is a traversal observation under the endpoint's rules. A partial response, permissions change or query change may alter what that traversal can see. The integration should retain the query scope and any response indicators of incompleteness rather than label every end condition complete. A report over one account should not silently become a report over whichever account a refreshed connection happens to select midway through the run.

A low-impact activity screen may need only eventual freshness and a Refresh action. A durable snapshot adds work that such a view may not justify. An operational handover file has a different requirement because someone could act on an omission. Its population and recovery contract need to support that decision. Naming the intended use helps choose the appropriate consistency guarantee without imposing the strongest export mechanism on every paginated endpoint.

A continuation checkpoint is a commitment about processed work

The resource's proposed checkpoint records account A1, filter status=open, descending identifier order, snapshot S1, next offset two and committed identifiers five and four. The surrounding fields are important. A continuation value belongs to a query and population context. Reusing it after changing a filter or account can skip an unknown part of a different result set while looking like a successful resume. Store enough identity to reject that mismatch rather than hope the cursor will explain itself.

A checkpoint can lose records even against the stable snapshot. In the negative case, next offset two is saved before [5, 4] commits to the destination. Interrupt there and restarting on S1 retrieves only [3, 2, 1]. The first two identifiers are missing from the output even though they were fetched. The program checks that membership calculation; it does not build a transactional destination. What it exposes is the actual persistence question: which durable work is the saved progress entitled to claim?

Reversing the sequence can repeat a page if output commits but the checkpoint does not. A destination able to recognise already-applied records can make that safer, provided its update semantics are explicit. An append-only file cannot simply pretend repetition does not matter, and a mutable record may need a version as well as an identifier. Decide how page processing and progress recording share a boundary, or how replay is made harmless, before describing resume as automatic recovery.

When a continuation expires, starting again at page one and appending is not automatically recovery. The original snapshot may be unavailable, and the destination may not tolerate replay. Check the endpoint’s options and keep the export visibly interrupted or incomplete while deciding what to do. A stopped run can leave a recoverable job. A partial file announced as complete invites someone to act on its omissions before the team has even noticed that a recovery decision remains.

Keep a record of why the traversal ended

A compact run record can retain the query identity, snapshot or consistency assumptions, pages followed, last committed checkpoint and termination reason. Record errors and any known exclusions as well. It need not contain entire sensitive payloads to answer whether a run followed its intended population. A count of fetched rows is useful context, but the five-record deletion case shows why it cannot independently certify that the right identifiers were collected.

Identifier three disappears in the deletion example without a single failed page request. That is the result to keep beside a successful traversal counter. Continuation rules explain the next request; consistency rules explain the population; durable processing explains when progress may advance. Recording those together lets someone account for the export they actually have. Reaching the last page is still important, but it cannot settle a question the list changed underneath us.