How to stop an automation that still has work to do
Retire intake, accepted jobs and external authority in a deliberate order, so a late event cannot revive an obsolete process.
Automation7 min read
The fallback is where you find out what you automated
A worked restart manifest: distinguish completed manual work, an uncertain send, a changed record and an untouched enquiry before turning automation back on.
The backlog has changed while you were away
The automation has been down, but the booking work has carried on. People have made appointments, corrected addresses and replied to customers. When the worker returns, those enquiries may still be sitting in its queue. Treating the queue as a list of unfinished jobs would reward everyone’s efforts with another round of bookings and confirmations. The restart needs to understand what happened while it was away.
That is why I would rehearse the return from manual working as carefully as the pause. A fallback can keep the service useful during an outage and still leave a difficult restart behind it. The worker needs to recognise completed actions, changed instructions and uncertain sends when it claims each enquiry. A list prepared at the beginning of recovery can already be out of date by the time the first action runs.
Take an invented maintenance-booking service whose intake stays open while automatic booking and confirmation are paused. Staff work through the same operational records as the software, so an enquiry’s identity, version and action history can record their changes. The four enquiries below let us examine that return journey without suggesting that a real company suffered this outage. Each has remained in the backlog for a different reason.
The four enquiries in this example need different restart decisions. One is complete; another has an uncertain send outcome. A third has been corrected since a worker read it, and the fourth is untouched. All four can remain in a stored backlog. Queue membership tells the worker where to look, but the current record must determine whether any action is still due.
Read the four histories before choosing an action
E101 is version 4. A colleague booked the appointment manually and recorded that its confirmation was sent. The relevant obligation is complete, so the restarting worker should skip that action. Preserve the history rather than deleting the enquiry from view: a support colleague may still need to explain the booking, and another delivery of the original input should not recreate it.
E102 is version 2. A draft booking exists, and a confirmation attempt has an unknown outcome. That is different from an unsent confirmation. The next step is to establish whether the original send completed, using the provider’s supported lookup or retry contract. A second attempt with a new identity may send another message. Until the uncertainty is resolved, keep this enquiry outside the ordinary automatic lane.
E103 is now version 3 because someone corrected the address manually. A worker still holds version 2. Even if neither version is marked complete, the old work proposal is stale. Reject that proposal and reload the current record. The correction might change the service area or appointment availability; the worker cannot safely reuse an earlier decision merely because the enquiry identity stayed the same.
E104 remains version 1 and has no recorded action. It is eligible for processing, provided the service is currently in automatic mode and the worker can claim this version. Eligibility is not ownership. Another worker may reach the same conclusion, or a colleague may amend the enquiry before the claim is committed. The final check belongs where ownership changes.
The backlog is four different histories
An original synthetic restart manifest distinguishes completed manual work, an uncertain send, a stale worker view and untouched work.
Scroll sideways to see all columns.
| Enquiry | Current history | Worker view | Restart decision |
|---|---|---|---|
E101 | Version 4 · manually completed; confirmation known | Version 4 | Skip completed action |
E102 | Version 2 · draft exists; send outcome unknown | Version 2 | Reconcile original send; do not create another |
E103 | Version 3 · address corrected manually | Version 2 | Reject stale claim; reload current record |
E104 | Version 1 · untouched | Version 1, current mode | Eligible for conditional claim |
Make the claim conditional, not conversational
A worker that reads a record, decides it is available and later writes its name into the owner field has left an interval for another change. Instead, include the expected version and permitted state in the update condition. PostgreSQL’s UPDATE documentation describes conditions determining the rows changed and the returned affected-row result. [1] A zero-row result is useful information: the proposal did not acquire the record it expected.
The rejected claims are worth looking at alongside the successful one. E104 moves from unprocessed version 1 to claimed version 2; a second worker expecting version 1 changes nothing. E103 also changes nothing, because the worker’s expected version 2 predates the manual address correction at version 3. Removing the version condition would make both stale proposals look easier to process, at the cost of losing the very check that protects the colleague’s correction.
The control record also carries an operating mode and generation number. Advancing the generation when automatic processing resumes invalidates local claims made under the earlier generation. A worker must present the generation it was authorised to use. This makes a stale worker visible instead of assuming that every process received the pause notification. The condition must be enforced by the system accepting the claim, not merely remembered in worker code.
The two numbers answer different questions. The generation says which period of automatic working authorised this worker; the enquiry version says which customer details it used. Neither can stand in for the other, so both need protection in the transition that acquires the record. In production, that includes choosing transactions and isolation that coordinate changes to the control record. The small local example cannot show that every process in a distributed service observes the pause at the same instant.
A claim has two independent preconditions
The restart fixture checks enquiry version and operating generation at the transition that acquires work.
Follow the labelled arrows. On a narrow screen, scroll sideways if needed.
A local claim cannot recall an email
Suppose a worker sent a confirmation before the pause took effect, then lost the response. A new generation can prevent its later local update from being accepted under stale authority, but it cannot reverse the send already accepted by the email provider. E102 therefore still needs reconciliation after the local operating mode has been repaired; the generation check establishes a different fact.
AWS’s account of idempotent APIs explains how a caller-provided request identity can distinguish a retry from a new intention, and why recording that identity with its effects requires careful service-side handling. [2] Use the actual provider’s contract; a local label named idempotency_key does not make an arbitrary remote operation repeatable. Preserve the original action identity and parameters while resolving an uncertain outcome.
A claim timeout needs similar care. It may let another worker investigate or take over unfinished local work, but it does not prove that the previous worker never reached the remote service. Lease expiry describes elapsed ownership time, not the absence of effects. Where the provider cannot establish the outcome, the safe result may be a manual decision and a delayed customer reply rather than an automatic duplicate.
Manual completion deserves the same scrutiny. A colleague may have finished the booking and still be unable to tell whether its confirmation was sent. Marking the enquiry complete should not erase that distinction. Record which obligation finished and which remains uncertain, so the next worker does not either repeat the whole job or skip the unanswered part. An honest gap in the record is easier to recover than a comfortable status that means different things to different people.
Rehearse the return journey, not just the outage
The downloadable example puts the four histories into an in-memory database using Python’s sqlite3 module. [3] You can inspect the claim statement beside the decision for each enquiry. E104’s successful claim and the rejected repeat belong together: the first gives one worker the current record, while the second stops another from acquiring the same old version. E103 tests a different obligation. Its rejection protects the colleague’s corrected address even though the enquiry still needs work. None of these local changes sends a confirmation.
Changing one of the four histories is a good way to see where the decision comes from. Confirm E102’s send and ask what, if anything, remains due. Increase E104’s version before claiming it, or put the control record back into manual mode, and follow the rejected proposal. The database guard stops an outdated transition; it cannot decide what a freshly read customer history means. That decision still belongs to the booking policy.
For the real workflow, rehearse a pause after creating a booking draft but before recording a send outcome. Ask an authorised colleague to handle selected enquiries manually, including one correction, then resume processing. Account for each record and each external effect. Check the operator view too: can someone tell completed work from unresolved work without reading raw logs or asking the original author?
Some services should stop cleanly rather than ask staff to improvise a manual substitute. Where manual working is necessary, though, it becomes part of the system’s history. I would want the person restarting this service to account for E101’s completed booking, E102’s uncertain send and E103’s corrected address before celebrating E104’s successful claim. That is what a useful fallback leaves behind: enough shared information for the software to rejoin work that people have kept moving.
Working on a workflow like this? Hamsa helps finance and operations teams connect their tools and handle the exceptions, starting with one workflow.
See the Payment Operations PilotReferences & further reading
Primary sources behind this article. Living documentation may change.
Practical engineering. A human perspective.
More about Hamsa Software and the person behind it. Editorial notes.A few more things
worth a read.
Back to the reading room 
A budget that three workers can share
Enforce task limits through atomic reservations and repeat-safe settlement, including the uncomfortable case where a timed-out operation may still be spending.
What exactly did that approval authorise?
Bind an assistant’s approval to the action, target and material revision, then enforce that binding when the write happens.