hamsa software
← All articles
Engineering7 min read

The bookings restored, but one document was missing

A disposable booking-and-attachment rehearsal exposes two failures that a successful database query cannot detect.

Crossing geometric bars surround a circle, suggesting the linked database and attachment pieces that must be checked during recovery.

A readable database can still lack an attachment

Three restored bookings can be present and readable while the person opening B2 still cannot get its document. If D2 never entered the backup set, counting the database rows will not tell us. The rehearsal has to follow the reference from booking to attachment and try the service action that needs it. That is a more demanding question than whether the database opens, but it is also closer to what someone means when they ask whether the bookings have been recovered.

The disposable rehearsal uses three invented bookings and text attachments, with no production access. It first restores an incomplete set, then a present but incorrect file, then the expected files. Those executed checks follow a small service action: open the document belonging to B2. They do not measure recovery duration. Starting there makes a missing dependency visible before a larger rehearsal has to coordinate real access, volume and application startup. A successful database connection is useful, but the colleague opening the booking still needs its document.

Its recovery unit is a set: database snapshot, attachment manifest, attachment bytes and a configuration record stating that outbound work is disabled. Each booking refers to one document. That relationship gives the rehearsal a meaningful check beyond counting tables. A real service would have additional dependencies, but starting with one action somebody needs to perform is more informative than starting with the list of file extensions a backup tool happens to recognise.

Restore the set and challenge the relationships

The three bookings, B1–B3, point to D1.txt–D3.txt. Python’s connection backup interface copies the SQLite database, while a separate manifest records the expected attachment digests.[1][2] Copy only D1 and D3 and the booking rows still look complete. The reference to D2 survives; the bytes needed to follow it do not. That is why the database and attachment store must be treated as a recovery set, even when separate tools copy them. Neither copy operation can infer the missing part of the application’s promise.

In the first restored directory, all three booking rows are readable and D2.txt is absent. Add D2 but put wrong bytes at D3: the existence check now succeeds, while the digest check fails for D3. Restore the expected D3 and both checks clear. These are the observed local results. They ask different questions of the same booking journey. The person opening B2 needs the referenced file to exist; the person opening B3 also needs it to be the retained document rather than another file with a convincing name.

The demonstration repairs its set from unchanged source copies. After an actual loss, the expected bytes must instead come from an intact retained backup or another authorised source. Protect the manifest as well as the files: if one process can alter both, matching their digests will not reveal the substituted expectation. The comparison establishes byte identity with the recorded value. Its usefulness therefore depends on retaining a trustworthy record of which document version B3 was supposed to recover.

Attachments stay unchanged while this snapshot is made. A live application can be less cooperative: a document might be replaced after its database reference is read but before its bytes are copied. The database snapshot can then be consistent by itself while the combined set is not. Versioned document identities, a coherent backup boundary or an application-specific consistency procedure must preserve that relationship under the actual write model. Rehearse a replacement as well as an absent file, so the team knows whether the restored reference identifies the version it intended to retain.

A recovery claim has more than one clock

After taking the snapshot, the fixture inserts booking B4 into the source database. The restored copy contains B1–B3 and deliberately does not contain B4. No amount of checking the restored copy can recover that post-snapshot write from this backup. This gives a concrete difference between successful restoration and an acceptable data-loss window. The restored history may be exactly the history the backup retained, while still being older than the business can tolerate. A recovery review can accept this omission only by comparing that retained point with its stated data-loss tolerance; successful attachment checks cannot alter the answer.

A recovery-point target concerns how much recent committed work may be absent after recovery. A recovery-time target concerns how long the business can wait for a defined service to become usable again. The endpoint matters: database query available, booking documents accessible and full customer service restored can occur at different moments. Quoting one duration without saying which capability returned leaves the person planning operations with an incomplete promise.

In the real rehearsal, start and end the clock at the agreed service stages: obtaining access, provisioning dependencies, restoring the set and completing the booking journey may each account for part of the wait. Keep delays in the record. If the database is ready while the document account remains unavailable, the service has reached one stage rather than finished recovery. The business can then compare the capability actually returned with the time it can tolerate being without it.

Backup frequency is only one input to the data-loss question. Check whether backups actually completed, whether the latest coherent set can be retrieved and whether the intended history includes externally acknowledged actions. A restored booking system may be missing local records for work a supplier already accepted after the snapshot. Reconciliation with external evidence is then a separate recovery obligation. Starting the database does not rewind another organisation's system to the same point in time.

Three restores of the same three booking records

Executed disposable SQLite and text-file checks compare readable bookings with their attachment dependencies.

Scroll sideways to see all columns.

Rehearsal stageDatabase rowsAttachment checksOutbound configuration
Incomplete setB1, B2, B3 readableD2.txt missingDisabled
D2 supplied; D3 wrongB1, B2, B3 readableD3.txt digest mismatchDisabled
Expected file set repairedB1, B2, B3 readableAll expected bytes matchDisabled
Post-snapshot B4Absent from restored snapshotNot recoverable from this setNo external replay implemented
Fixture repair uses unchanged source copies. A real loss requires intact retained backup versions; matching digests alone do not authenticate a compromised manifest.

Prevent the rehearsal from creating new live work

Restored workers can create new live work from old records. A reminder sent during rehearsal still reaches a real customer, however clearly the database is labelled test. This example carries outboundEnabled false and checks it, with no sender or external client present. A real rehearsal needs controls that block the actual email, payment, webhook and supplier paths before workers start. Checking one flag is sufficient only for the path it really governs; another adapter must not inherit permission merely because the main worker is paused.

A recovery screen can also borrow health from production. If its booking view reads the surviving live attachment store, the incomplete restored set may appear to work. Verify the configured paths or endpoints used by the actual journey, rather than only the backup directory you intended to use. Otherwise the rehearsal is examining a mixture of recovered and live dependencies. The unavailable store in a real incident may be exactly the part that quietly rescued the test.

Use the restore procedure for the actual database too. PostgreSQL’s SQL-dump guidance describes role and ownership prerequisites, objects outside a single database dump, and plain psql continuing after errors unless configured otherwise.[3] A procedure that only says import the backup leaves those decisions to the covering engineer. Name the intended command and failure handling, then verify the resulting service stage. The SQLite booking exercise supplies the relationship to check; it cannot supply PostgreSQL’s restore semantics or the roles a different deployment will need.

Access to backup storage, encryption material and instructions also needs to survive the failure being rehearsed. Requiring an unavailable application to retrieve the recovery credentials can delay restoration even when the archive is intact. The procedure needs an independently available, permitted access path, with people authorised to use it. Exercising that path establishes an operational prerequisite; it does not require placing secret material inside the ordinary runbook.

Keep the restore rehearsal inside its own recovery set

Configuration and dependency boundaries discussed in the booking rehearsal.

Follow the labelled arrows. On a narrow screen, scroll sideways if needed.

Keep the restore rehearsal inside its own recovery setRestored database: B1–B3. Rows must resolve attachments inside the recovered set, without silently borrowing surviving production files. Restored attachments. Use the manifest and expected bytes. A live attachment store would conceal an incomplete recovered set. Restored configuration: outboundEnabled false. Rehearsal workers must not create live email, payment or supplier effects. The fixture checks this value and has no network sender. From Restored database: B1–B3 to Restored attachments: Resolve only restored dependencies. From Restored configuration: outboundEnabled false to Restored database: B1–B3: Constrain rehearsal behaviour.Resolve onlyrestoreddependenciesConstrain rehearsalbehaviourRestored database:B1–B3Rows must resolve attachmentsinside the recovered set,without silently borrowingsurviving production files.Restored attachmentsUse the manifest and expectedbytes. A live attachmentstore would conceal anincomplete recovered set.Restored configuration:outboundEnabled falseRehearsal workers must notcreate live email, payment orsupplier effects. The fixturechecks this value and has nonetwork sender.
The fixture verifies its configuration value and has no network sender; it does not establish isolation of a production deployment.

Define what usable means before the next rehearsal

For this booking action, usable means the expected bookings and their document bytes are available while outbound work remains disabled. Login, authorisation, search and background processing would add other service stages to a real recovery. Choose those journeys with the people depending on them. Include an intentionally absent dependency so the procedure has to recognise an incomplete return, rather than treating a successful startup as the end. B2 is a useful first case because someone can see exactly what remains unavailable even while the database is healthy.

Keep the backup identifiers, procedure, environment, errors and repairs so the next rehearsal can challenge the same relationships. Missing D2 escaped the database checks; wrong D3 passed the existence check. Those are different failures with different remedies. Even after repairing both, post-snapshot B4 stays absent because it lies beyond the retained history. A credible recovery account can explain those three outcomes without letting the success of one check stand in for the service the business still needs back.

Working on a workflow like this? Hamsa helps finance and operations teams connect their tools and handle the exceptions, starting with one workflow.

See the Payment Operations Pilot

References & further reading

Primary sources behind this article. Living documentation may change.

  1. Python: SQLite connection backup interface
  2. SQLite: online backup API
  3. PostgreSQL: SQL-dump restoration and prerequisites
Hamsa Jama

Practical engineering. A human perspective.

More about Hamsa Software and the person behind it. Editorial notes.

A few more things
worth a read.

Back to the reading room
Engineering7 min read

What does £240 belong to?

A readable table preserves relationships between values, row labels, column headings and actions across different ways of reading.

Engineering7 min read

When two calculations agree on the wrong answer

A shadow comparison exposes differences. Independent rule examples explain whether those differences are bugs, corrections or changed assumptions.