When two calculations agree on the wrong answer

Agreement is useful evidence, with a limit

Run the replacement calculation alongside the old one and you can examine differences without immediately changing what the customer pays. That is useful when the real input population is hard to anticipate. There is still a question the comparison cannot settle by itself: which result should the tariff produce? If both implementations inherited the same mistaken rule, a reassuringly empty difference report will leave the mistake exactly where it was.

The old route may have kept its tariff table longer than the application team realised. A replacement using that same table could reproduce every price and still charge under the wrong revision. A deliberate correction creates the opposite problem: the disagreement report fills with prices the business actually wants to change. The comparison needs the tariff behind each answer before either pattern can support a release decision.

For a fictional delivery-price calculation, suppose a package costs £8 up to and including 5 kg, then £1.50 for each additional started kilogram. This is an invented commercial rule, chosen so the boundary is easy to inspect. The old implementation charges by rounding the whole weight upwards and subtracting five. The proposed implementation applies the rule directly after the 5 kg allowance.

Capture the input, tariff revision and both results so each difference can be replayed. Keep an independently worked expected answer alongside the important cases. The old implementation supplies something valuable to compare with, but the team is replacing it for a reason. Measuring a week of agreement should not quietly turn that implementation into the authority for every price the new version is allowed to charge.

Freeze the same question for both implementations

Start with a stable input representation. Is weight supplied in kilograms or grams? Is it a measured value or an already-rounded invoice value? Does the rule apply to each parcel or the combined shipment? If the old system receives an earlier snapshot than the new one, their difference may say more about the integration than either calculation.

For this example, use integer grams and a tariff revision identified as T-4. A 5,000 g parcel costs 800 pence. A 5,001 g parcel costs 950 pence. A 6,000 g parcel still costs 950 pence, while 6,001 g costs 1,100 pence. These boundary examples express the intended started-kilogram rule without relying on binary floating-point comparisons around a decimal weight.

Give both implementations the same captured weight, tariff revision and destination class. Keep the comparison date separate from the tariff’s effective date. When someone replays the case next month, they need the rule that applied to the accepted quote, not the table currently easiest to retrieve. Otherwise a difference caused by changed source material can look like a calculation changing its mind, and the investigation starts in the wrong place.

Shadow execution needs an explicit boundary around side effects. If the new path reserves stock, sends a message or writes a charge as well as calculating a price, invoking it beside the old path may repeat those actions. Isolate the pure calculation or route its effects into a deliberately controlled receiver. The deployment label alone cannot prevent the function's ordinary code from acting externally.

Look for the shape of disagreement

Imagine six captured weights: 4,000, 5,000, 5,001, 6,000, 6,001 and 8,200 g. A candidate that mistakenly charges only complete kilograms above the allowance would agree at 4,000, 5,000 and 6,000 g. It would undercharge at 5,001, 6,001 and 8,200 g. Three matches out of six are much less informative than the pattern: the candidate mishandles partial additional kilograms.

The expected T-4 prices are respectively £8, £8, £9.50, £9.50, £11 and £14. The faulty floor-based candidate produces £8, £8, £8, £9.50, £9.50 and £12.50. Each disputed case is £1.50 low. That repeated delta points towards a boundary rule, not a random numerical tolerance issue.

Explicitly record which captured input records never reached the comparison. A new parser might reject unusual weights before the candidate calculation runs, leaving the matched population looking reassuringly ordinary. Count exclusions and failures separately from comparable outputs. Otherwise a candidate can appear accurate because the difficult cases vanished at an earlier stage. The comparison boundary should include an explicit outcome for missing input, rejected input and calculation failure, with enough evidence to investigate each one.

Look for the features shared by the disagreements: a threshold, destination, effective tariff, missing input or rounding stage. Retain the cases under those groups. Our three £1.50 undercharges total £4.50, but another set could contain overcharges and undercharges that cancel. The net would then look settled while individual customers still received the wrong prices. An explained pattern is more informative than a small overall delta.

A tolerance should follow a documented numerical contract. A one-penny allowance could conceal a different rounding stage, while a percentage allowance can admit a large absolute error on a high price. T-4 specifies exact pence, so its expected results can be compared exactly. If the team introduces a tolerance, it needs evidence that the resulting differences are permitted rather than an easier way to accept unexplained cases.

The existing answer is a witness, not the judge

Use the written tariff and independently worked boundary cases to adjudicate the disagreement. If the old system also uses the faulty floor rule, both paths will agree and the shadow comparison will miss the defect. The independent cases are what expose the shared mistake. This is why comparison and correctness testing complement each other rather than compete for one impressive pass rate.

There can also be legitimate historical differences. Perhaps the old route used tariff T-3 for shipments booked before a change, while the candidate applies T-4 to every replay. The calculation may be correct for T-4 and still wrong for that shipment. The review must distinguish a mathematical defect from a rule-selection defect and from an intentional business-policy change.

Give each material difference a reason someone responsible can act on: candidate bug, deliberate correction of old behaviour, changed input or an unresolved tariff rule. The last category needs a person authorised to decide that commercial policy. Developers can expose the ambiguity without quietly settling it just to unblock the release. Keep the decision with the case so a later reviewer can see why disagreement was accepted or why the candidate had to change.

The six weights were selected to expose boundaries, so their failures reveal a mechanism without estimating how often it occurs in real parcels. A production shadow can add frequency and impact evidence, provided the observation period, input population and exclusions are recorded separately. Keep those claims distinct: an intentionally difficult case set establishes different information from a representative distribution of actual shipments.

Shadowing should lead to a decision

Before enabling the comparison, decide what would justify moving the official calculation. One reasonable gate is that every observed material disagreement has an accepted explanation, independent boundary cases pass and the proposed configuration handles the required input population. That is more demanding, and more useful, than requiring an arbitrary percentage agreement with the old system.

Plan how to stop the comparison too. A permanent second calculation consumes resources and can leave confusing diagnostic records long after the migration. Retain the cases and explanations that became regression tests. Remove temporary collection that no longer serves a decision, subject to the system’s retention needs.

The release should preserve the route back where it remains meaningful. If a new tariff representation changes persisted quotes, reverting only the code may not recreate the previous behaviour. Record the calculation and tariff revisions attached to an accepted quote so later questions can be answered from the actual decision, not from whatever implementation happens to be deployed today.

There are cases where a simple direct replacement is enough. A low-impact calculation with a complete specification and strong independent tests may not need a production shadow period. Shadowing is especially useful when actual input diversity is difficult to anticipate, but that advantage does not make live traffic an oracle. It provides another source of evidence.

For T-4, resolve the partial-kilogram differences against the frozen inputs and independently worked prices. Then keep those boundary cases with the tariff revision as regression material. The old system has helped find the pattern without being appointed to judge the rule. A later question about an accepted quote can return to the calculation and tariff actually used, rather than whatever answer the current deployment happens to produce for it.