Why an optional field can break an old consumer

It was only one extra field

The producer adds one optional field and changes none of the old values. One reader carries on, another rejects the response, and a third writes the new reference under the amount heading in a CSV. All three can parse JSON. The apparently small change has reached three different agreements about what to do with it, which is why I would inspect the downstream result before calling the addition compatible.

Optional means the field may be absent under the contract. It says nothing by itself about whether an old reader tolerates the field when present. That needs an extension agreement covering the actual consumers, including closed generated clients and scripts nobody thinks of as an API integration anymore. A small producer diff is useful information about the change; it is not a complete inventory of the assumptions that will meet it.

The original fixture here gives one old and one new payload to three small authored consumers. It does not simulate a real provider incident. The deliberately different outcomes make the hidden assumptions visible: a named-field reader, a strict property reader and an exporter which treats object values as a positional record. All three can parse a dictionary; that is not the same as all three agreeing about an interface.

Watch the extra value travel

Payload v1 contains customer_id C17, amount_minor 1200 and currency GBP. Payload v2 adds reference PO-7 between the identifier and amount in the authored dictionary. The permissive consumer explicitly selects customer_id, amount_minor and currency by name. It returns the same tuple for both payloads. Under this narrow task, the extra reference does not change the values it uses.

The strict consumer requires exactly the three known keys. It accepts v1 and rejects v2. JSON Schema distinguishes properties from constraints on additional properties; its documentation shows how additionalProperties can restrict or reject extras. [1] The fixture uses its own explicit key check, not a claimed execution of a schema validator. Either way, a strict boundary can be intentional. A response contract that promised a closed shape cannot simply redefine the reader as unreasonable when it enforces that promise.

The positional exporter is the most revealing reader here because it produces a file. It writes the old headers customer_id, amount_minor and currency, then exports every object value. With reference added, the row becomes C17,PO-7,1200,GBP: three headings, four cells, and PO-7 beneath amount_minor. The strict reader has at least made its disagreement visible. This one has returned an apparently usable result with the wrong meaning attached to a cell.

These outcomes leave three different release decisions. The named-field reader needs its selected values preserved; the strict reader needs the promised extension policy settled; the exporter needs a fixed column mapping. Treating every failure as an objection to optional fields would miss the silent CSV corruption, while treating every rejection as a defective consumer would dismiss an intentionally closed contract. Keep the actual output from each old reader with the candidate payload so the release discussion can address the agreement that changed.

Repair the assumptions at the right boundary

The exporter should project an explicit ordered set of named fields into its output contract. Adding a new JSON property should not automatically add a CSV column, and a missing required named property should produce a useful failure rather than shift the rest of the row. RFC 8259 defines a JSON object as an unordered collection; position is not the business contract to borrow from it. [2] The Python fixture preserves a particular insertion order only to make the defect reproducible.

The strict reader needs an agreement rather than a universal rule about permissive parsing. If the response is meant to grow, deliberately accept the permitted unknown fields while checking the ones the consumer uses. If the boundary promises a closed shape, coordinate a version or migration instead. A producer cannot promise that shape and then be surprised when a reader takes it seriously. Both policies have legitimate uses; the release needs the one these parties actually agreed.

A reader may permissively accept an external response while using a strict internal command schema. That can be a sensible separation: the outside interface evolves, while the internal operation has a controlled set of accepted arguments. Preserve that boundary with an explicit mapping. Forwarding the whole response directly into another action makes the producer’s extension policy the recipient’s effective write policy, which may be a rather different agreement.

Preserving unknown fields for round-tripping can be intentional, but it also carries data into logs, exports and later requests. Define which values are retained and for what purpose, and project only what each downstream task needs. Otherwise a reader that correctly tolerates the reference property could forward it into another system whose contract never included that information. Parsing and onward disclosure are different decisions.

Shape compatibility cannot protect a changed meaning

Now imagine no new field at all. The producer changes amount_minor from the final amount due to the amount before discount, while retaining the same name and integer type. Every toy reader can accept the payload. The person using the resulting figure receives a different financial meaning. A structural check has no way to infer that a familiar integer has changed its definition.

Document the semantic contract alongside the shape: units, currency relationship, inclusion rules, null and absence meanings, and the moment in the workflow the value describes. Give a small example with an expected result. If the final amount is 1200 minor units after a discount, state that directly. A consumer comparison can then ask whether the new response preserves the business result, rather than only whether deserialisation avoided an exception.

Optionality has a similar semantic edge. Missing reference might mean not supplied, not applicable or unavailable from this source. An empty string may or may not mean the same thing. Decide before introducing the field, especially if a later consumer will use it to choose a route or display a status. Otherwise the first convenient default can turn an unknown into a false fact, even when the added property is syntactically well formed.

Enum additions deserve the same attention. A new status can be structurally a string and still break a consumer which assumes all values are one of three known states. The right response might be an explicit unrecognised state, a controlled stop or a mapped fallback. Silently treating it as the most convenient existing state can preserve uptime while changing the business outcome. Compatibility needs a policy for that ambiguity as well as a parser setting.

Test the change where the result is used

A useful contract exercise runs old consumers against the candidate payload and inspects their output. For the named reader, check the selected values. For the strict reader, check the intended rejection or extension policy. For the exporter, inspect headers, column count and exact cells. A successful response from the producer cannot prove these downstream results, and a parser-only test would miss the mislabelled CSV.

Choose specimen values that make a wrong result obvious. PO-7 is useful because it does not resemble an amount. An all-numeric fixture could let the same positional shift create a plausible-looking number, leaving a column-count check as the only visible complaint. Include missing and unexpected fields too. The point is to expose the boundary’s assumption in the consumer’s output, rather than accumulate successful producer requests that never inspect where the values end up.

For a controlled set of internal consumers, the producer can coordinate a migration and record which versions have been checked. For a public or poorly inventoried interface, assume less knowledge of downstream assumptions and maintain the promised extension rules. Versioning helps communicate a changed contract; it does not repair an unexamined export or document a semantic change by magic. The downstream result still needs inspection.

This reference addition has three findings: the named reader keeps its values, the strict reader rejects the candidate, and the exporter exposes a positional mapping defect. They need different repairs and an agreed extension policy. That is more useful than deciding whether optional additions are safe as a class. The producer’s change really is small; the evidence of compatibility has to come from the places where people rely on its result.