How to stop an automation that still has work to do
Retire intake, accepted jobs and external authority in a deliberate order, so a late event cannot revive an obsolete process.
A useful runbook says how to recognise the case, what evidence permits the action, when to stop and what proves completion. A CRM timeout gives those instructions something concrete to do.
“Rerun the job” is a surprisingly large decision to leave in a runbook. In a CRM-create timeout, the destination may already hold the customer while the local record has no result. The person taking over needs a way to recognise that situation before following the instruction and creating another customer. The useful procedure starts with what to check, not with a command whose safety its absent author was expected to explain.
In this invented case, the local operation says outcome_unknown and has no external identifier. The destination may have created the record before the reply was lost. Depending on the provider’s contract, another attempt could retrieve the intended result or make a duplicate. A short instruction can still handle the case, but it needs to distinguish those possibilities before asking someone with production access to act.
Google’s SRE guidance identifies clear escalation paths and well-defined operational procedures as resources supporting people on call. [1] The small runbook below applies that principle to a specific synthetic case. It is not a universal CRM recovery command: it assumes a destination with a trustworthy lookup by the stable operation reference, and a local reconciliation action designed for this purpose.
The entry condition is precise: operation crm-create:C17, correct tenant and environment, state outcome_unknown, external_id absent and local revision 4. The operator needs read access to the operation and the authorised destination lookup, plus permission for the local reconciliation action. None of those permissions should be obtained by borrowing an administrator’s token pasted into the runbook. Link to the approved access path and the duty role instead.
Some nearby cases need another procedure. This is a create, not a customer update; an unanswered outcome, not a known validation rejection. A local link to a different remote customer is a reason to stop, not overwrite it until the screen looks right. Give the operator the appropriate escalation route and the identifiers to take with them. That leaves a useful boundary they can recognise without having to infer what the author probably meant.
Keep a short read-only summary before any mutation: local operation, customer, tenant, submitted identity fields, current revision and known provider reference. The person should be able to check that they are investigating the right object. A production screen that differs unexpectedly from the documented fields is another stop condition, not a reason to infer which unfamiliar button probably does the same thing.
The procedure also needs an escalation owner and the team’s maintained contact route. While waiting, the operator should keep the operation held, prevent another uncontrolled attempt and preserve its unresolved status for the person handling the customer-facing work. Naming a general role without a functioning duty arrangement leaves the same recovery decision unowned when the integration’s author is unavailable.
The first decision is a lookup by the stable operation reference, not a broad search for a similar company name. In this fixture, the synthetic destination result binds operation crm-create:C17 to customer C17 and remote identifier X77. Verify the relevant tenant and identity fields under the actual provider contract. An address or display name alone may identify several records, so a plausible-looking match is not sufficient evidence to link one.
Exactly one trustworthy matching result permits the chosen recovery branch: record X77 locally as the result of this operation. Do not issue another remote create. The local action changes outcome_unknown to reconciled and advances revision 4 to 5 only if the expected state, missing link and revision still hold. PostgreSQL’s conditional UPDATE model supports expressing such a precondition with the mutation and inspecting whether a row changed. [2]
If the conditional update changes no row, read the operation again. Someone else may already have reconciled it, or a background process may have advanced the revision while the lookup was being checked. The zero is telling the operator that this transition no longer fits, rather than asking them to force it through. Compare the current state with the procedure before deciding whether any work remains.
If the lookup returns no result, several results or a mismatched identity, this branch stops. Absence in one search is not automatically proof that no create was accepted or remains in flight. AWS’s idempotency discussion explains why intent and uncertain outcomes matter to retry design. [3] The real integration needs its documented provider-specific path; a generic runbook should not turn a failed lookup into permission to mint a new identity and try again.
Original CRM-create recovery procedure, assuming the destination supports a trustworthy lookup by the stable operation reference. The chosen branch links an existing remote record instead of creating another one.
Follow the labelled arrows. On a narrow screen, scroll sideways if needed.
X77 and all identifiers are synthetic; no CRM lookup or remote mutation was performed. A real runbook must use the destination’s actual supported contract.The downloadable local branch uses the authored destination result to link X77 at revision 5. Repeating the old revision-4 transition changes nothing. Read those outcomes together: one records the justified relationship, while the other preserves it against an obsolete recovery request. The in-memory database does not query or write a CRM. Its useful check is narrower and inspectable: when the expected state, missing link or revision no longer holds, this recovery branch must stop rather than overwrite the current relationship.
The runbook’s final verification should read back the same business identity, recorded remote link and final state. Where appropriate, inspect the destination through the supported read path too. The expected result is one linked customer record for the original intention, not merely an exit code of zero. Capture the evidence needed for the operational record without dumping private customer contents into a general incident channel.
Keep the original uncertain attempt visible after the link is repaired. Its missing result may explain a recurring gap in the integration, or why the local completion time is later than the destination’s creation time. Reconciliation establishes the current justified relationship; it does not rewrite the earlier uncertainty out of existence. The next investigator can then see both why X77 is linked now and why somebody had to recover the link at all.
Define what to do if read-back after reconciliation shows another link or an unexplained state: stop further attempts, retain the identifiers and escalate through the supported route. Repeating the mutation could erase evidence of the race or incorrect assumption. The procedure needs this branch even after an apparently successful local action, because the completion claim depends on reading the expected relationship back from the system.
The CRM fixture’s successful local branch preserves the original uncertain attempt.
Scroll sideways to see all columns.
| Verification | Expected result | Limit |
|---|---|---|
| Operation/customer binding | crm-create:C17 belongs to customer C17 | Not a name-only remote match |
| Conditional local action | Revision 4 outcome_unknown becomes revision 5 reconciled | No new remote create performed |
| Read-back | External identifier X77 remains linked | A zero exit code alone is insufficient |
| Unexpected link or state | Stop and retain identifiers | Do not repeat mutation until display looks right |
Ask someone who did not write the integration to use the runbook against safe synthetic states: the existing X77 case, no trustworthy match, multiple matches, already reconciled and a changed revision. Watch where the document asks them to infer missing context. Its purpose is to improve the instructions before a real incident supplies the stress.
Keep commands and application links specific to the maintained implementation. Show required parameters, whether a step is read-only and the expected shape of its result. Separate specimen commands from live commands so a copied example cannot silently target the wrong environment. A long catalogue of every possible administrative tool is less useful than a short supported path with clear prerequisites and stop conditions.
Revisit the runbook when the integration’s states, access route or destination contract changes, and give that work an owner. A recent safe exercise or substantive review is more informative than a fresh formatting date. An older procedure may still be right; a document edited yesterday may describe a button that no longer exists. Maintaining this one means checking that its decision path still finds and justifies the original customer link.
For this branch, the person taking over can finish with crm-create:C17 linked to verified X77, reconciled at revision 5, without issuing a remote create. If the lookup or local conditions differ, they have a stop route with the relevant identifiers ready. That is the useful independence a runbook can provide. The integration’s author does not need to be present to explain which uncertainty the next command would otherwise turn into another customer.
Working on a workflow like this? Hamsa helps finance and operations teams connect their tools and handle the exceptions, starting with one workflow.
See the Payment Operations PilotPrimary sources behind this article. Living documentation may change.

Practical engineering. A human perspective.
More about Hamsa Software and the person behind it. Editorial notes.