What the failed rollback revealed

The first explanation is often the most available one

The importer starts rejecting rows after a release, so rolling the release back is a reasonable thing to try. In this invented incident, it changes nothing. That failed rollback is worth keeping near the centre of the review: the old code is still using the new parser setting, and the troublesome row contains a quoted comma. Removing the unsuccessful action would make the story neater while losing the clue that changes its explanation.

A recovery attempt can teach us something even when it does not recover the service. Here it asks whether changing the code version changes the result, and the answer directs attention towards a condition left in place. Another person reading the incident should be able to follow that turn. Otherwise the final configuration explanation arrives as an answer without the investigation that made it more convincing than the initial deployment theory.

Google’s SRE postmortem guidance emphasises contributing causes, learning and effective preventive actions rather than blaming individuals. [1] For a small team, that need not mean a sprawling document. It means a record honest enough to preserve uncertainty and precise enough that someone can tell what the follow-up work is supposed to change.

Give the incident a mechanism you can inspect

The original teaching incident uses an importer with two modes: a proper CSV reader and a simple comma split. A configuration change selects simple_split. Both the current and previous code versions in the scenario use whichever mode is configured. The first problematic row arrives later: a company name contains a comma inside quoted text. The input and the selected mode interact; the release timestamp alone does not explain that interaction.

The literal row is "North, Ltd",42. Under the agreed CSV format, the name is one field and 42 is another. Python’s CSV reader understands quoting according to its dialect settings. [2] The fixture returns two fields with csv.reader. Splitting on every comma returns three pieces, so the importer’s two-column check rejects it. There is no need to invoke an intermittent network problem to explain this particular failure.

A plain row, North Ltd,42, succeeds in both modes. A genuinely malformed unquoted row, North,Ltd,42, is rejected in both. Those controls matter. Without the plain row, we might think the importer is universally broken. Without the malformed row, we might “repair” the incident by accepting every three-field record and silently assigning the wrong name or amount. The mechanism distinguishes valid variation from invalid input.

The downloadable parser example keeps all three specimens available under both modes. Preserve the rejected malformed row when repairing the quoted one: allowing every extra piece would make the original symptom disappear by weakening the contract. For the incident review, keep the selected mode beside each result. The authored timeline below uses those local behaviours to follow an investigation; a real review would need its own configuration history and affected records to establish whether the same explanation fits.

Keep four kinds of statement separate

At 09:00, the example configuration changes. At 09:05, a quoted-comma row reaches the importer. Those are observations in the authored record. “The new code cannot handle commas” is an early hypothesis. At 09:10, the code is rolled back while simple_split remains selected, and the row still fails. That result weakens the code-only explanation; it does not prove every aspect of the previous release was harmless.

At 09:14, intake is paused to limit further affected imports. That is a mitigation. It changes exposure to the failure while the cause remains under investigation. At 09:20, comparing the exact row under both modes reproduces the distinction. That is evidence of a mechanism. At 09:25, restoring the agreed CSV mode handles the valid quoted row. None of these steps should be renamed “root cause” merely because it appears near the end of the timeline.

The explanation now has a mechanism. simple_split does not implement the agreed quoted-field format, the incoming row exercises that difference, and the rollback leaves the selected mode untouched. The tests had missed the mode/specimen combination. Keeping the same row while varying the mode and code version makes those links inspectable: both code versions reject it under simple_split, so code version alone cannot explain the behaviour. “Bad data” would miss that the row is valid; “human error” would miss the supported configuration that allowed the failure. Neither label tells a maintainer what to change. The exact comma, quoting and effective mode do, without requiring the review to turn every unsuccessful step into part of an inevitable story.

There can still be unanswered questions. Why was the alternative mode available for this source? Which check allowed the configuration change? Did any earlier records parse into two plausible but incorrect fields instead of failing? A review should retain those questions with owners and next steps. Reproduction of one rejection does not establish the full affected population or prove that all previous accepted output was correct.

Make the follow-up smaller than a slogan

“Improve testing” cannot tell the team when it is finished. A better action is to add the quoted-comma specimen to the supported-format contract, run it under every permitted importer mode and prevent unsupported source/mode combinations from being selected. Another action can record the effective parser mode and configuration revision with each import. Both follow directly from the mechanism rather than from a desire to add a reassuring number of tasks.

The rollback action needs a more precise improvement than “make rollback better”. Would restoring a reviewed code-and-configuration pair be appropriate, or should the tool show the configuration it will leave behind? Either can fit a real service. What matters here is that the operator can identify the effective combination before assuming the old code restores the old behaviour. That would have made the unchanged simple_split setting visible during this attempt.

Give each action an accountable role, a completion condition and a realistic review point. For the contract case, completion is a checked specimen and a rejected unsupported configuration, not merely a ticket moved to done. For evidence logging, completion includes finding the effective mode for an example import without recovering it from somebody’s memory. The document becomes useful when its claims about improvement can be inspected later.

The follow-up scope needs to address the conditions established by the reproduction. A rewrite may be justified if the design keeps permitting unsupported modes, but a bounded configuration guard and contract specimen may prevent the demonstrated failure more directly. Conversely, a parser patch is incomplete if the same unsupported combination remains selectable. Explain which condition each proposed action removes and how completion will be verified.

Recovery has a population as well as a cause

Before replaying records, identify the affected cohort from the actual time window, source, mode and outcome evidence. A failed-before-write record may be safely eligible for another attempt under the real import contract. A record with an unknown or partial effect needs a different disposition. “The parser is fixed” is not enough to classify every previous attempt as if nothing happened.

Keep recovery counts by justified outcome: accepted correctly, rejected as genuinely malformed, corrected under a reviewed rule or still unresolved. Do not measure success only by the absence of errors after intake resumes. A backlog can disappear through deletion or silent misparsing too. The same exact specimens that explained the mechanism can help inspect whether the recovery path now preserves the intended fields.

Read the review as the person who may encounter a quoted company name next month. They need to find the active mode, distinguish valid quoting from a genuinely malformed row and know which imports remain uncertain. A document mainly explaining who made the unfortunate change will not help them do those things. A specimen and a recoverable configuration history can, particularly when the apparent quick fix has already been tried.

The unsuccessful rollback belongs in the review because it rules out an attractive shortcut for the next operator. Keep it with the literal row and the setting that survived it. They can then inspect the effective code/configuration pair before choosing another action, while the record investigation decides which imports need replay or correction. Restoring CSV mode explains why the valid row now passes; it does not make the uncertain earlier outputs disappear.