The prompt did not change. The system did.

The configuration behind an unchanged prompt

The maintenance summary has lost an approval condition, although the prompt has not changed. Perhaps retrieval no longer supplies the decisive passage. Perhaps the model retained it and a parser discarded it. A model alias may also reach another deployment. I would follow the condition through those components before asking someone to rewrite an instruction that still says the same thing.

Prompt versions preserve important instruction history. An application's behaviour also depends on evidence, tools, model configuration and surrounding code, so the prompt version alone cannot identify the system that produced a disputed result. Keeping those component identities together makes comparisons more precise: a reviewer can distinguish an instruction change from a different source packet or a parser that discarded part of the response.

Take a fictional maintenance-summary assistant whose prompt asks it to preserve approval conditions. The next release changes how source passages are retrieved and how the response represents those conditions. A summary now loses one. The team needs to compare the retrieved passage, raw answer and displayed result to find where it disappeared. The figure gives the two configurations their exact component labels so that those comparisons can be recorded.

Call the earlier release R-12 and the candidate R-13. Both use prompt P-7; R-13 replaces retriever Q-3 with Q-4 and schema S-2 with S-3. A small release manifest keeps those identities together with the parser build used for the run. The next colleague can then compare the source packet and response shape without reconstructing the deployment from people’s recollections.

A manifest might name the model deployment, prompt digest, retrieval configuration, source corpus generation, tool schema, response schema, parser build and application revision. Use stable component identifiers that another colleague can resolve, or recorded content digests where appropriate. A label such as latest is convenient for routing and poor evidence unless the record also captures what it resolved to.

Include the policy that determines which tools are available. Two runs can share a tool schema yet expose different operations because the actor or task has different permission. For an evaluation, retain a safe description of that capability set and the relevant policy revision. The comparison should not mistake an appropriately refused tool call for a model regression, or credit a model with success when the test accidentally granted it broader access than production permits.

Record what ran as well as what the deployment intended. A worker can still have Q-3 cached after the release declares Q-4. Copying the release note into the run record would hide that difference precisely when it matters. Capture the effective configuration where the component is used and identify the environment and route. Then the investigation can follow the worker’s actual inputs instead of repeatedly checking a correct deployment file.

Source provenance needs particular care. A corpus generation can identify an indexed collection, but a disputed answer also benefits from the actual retrieved source identities and revisions. If the source can change after the request, a later fetch of the same URL may not reproduce the evidence. Preserve the relevant record or a controlled reference according to the privacy and retention policy.

The SLSA provenance specification asks how a software artefact was produced and which inputs and dependencies were involved. That is a useful question for this assistant too, without claiming SLSA compliance. The manifest helps locate the run that lost the approval condition; the retained source and outputs let a reviewer investigate why it happened.

Change one thing when you need a causal answer

For the lost qualification, compare R-12 with a candidate that changes only Q-3 to Q-4. Keep the source corpus, prompt, schema and parser fixed. Inspect whether the decisive passage still reaches the model. If it does not, the retrieval change offers a concrete explanation to investigate before tuning the wording of P-7.

Then examine S-2 to S-3 separately where practical. A new schema may omit a field for conditions, or a parser may drop unrecognised properties. The model might have preserved the qualification in its raw response while the application removed it before display. A final-output-only evaluation would detect the failure but leave its origin ambiguous.

Once the substitutions are understood, evaluate the components together. Q-4 may return a shorter passage that needs clearer instructions, while S-3 represents its conditions differently. Changing one thing at a time helps locate an effect; it does not promise that the complete candidate will behave like those isolated comparisons. Give the maintenance-summary task its intended final configuration and inspect the resulting qualifications before making the release decision.

Keep the input cases stable and record any deliberate changes to expected outcomes. If the new release is meant to enforce a different policy, label that change rather than pretending the old and new answers are judged by one unchanged rule. Otherwise the evaluation can turn an intentional correction into an apparent regression, or excuse a regression as an undocumented policy update.

Rollback is a compatibility question

If R-13 causes trouble, restoring P-7 accomplishes nothing because it never changed. Restoring the old model alone may also be irrelevant. The rollback target should identify a compatible configuration, including the retrieval and parsing contracts that produced the previous behaviour.

Suppose S-3 stores conditions as a structured list while S-2 expects one text field. Work already generated under S-3 may not be readable by the old parser. Rolling back the application can therefore create a new failure even if fresh requests would behave as before. Decide how to handle in-flight and persisted outputs across the schema boundary.

The same issue appears with tools. A restored prompt may request an argument the current tool no longer accepts. A rolled-back retriever may depend on an index generation that has been deleted. A previous model may have been retired. A configuration manifest makes these dependencies visible, but the operating plan must still preserve or rebuild the resources required by the promised fallback.

For a small application, a versioned configuration file, representative cases and a deployment record may be enough. Identify the resources the old route needs and decide what happens to outputs made under the new schema. The colleague taking over should be able to find and exercise the rollback target. Previous needs to resolve to a working prompt, index and parser, rather than become another instruction to ask whoever remembers the last release.

Preserve useful evidence without building a private-data warehouse

A run record can include component identities, source references, validation outcomes and timings without indiscriminately logging every document. Where raw inputs or outputs are necessary for a controlled evaluation, restrict access and define retention. Hashes can establish equality of retained artefacts, but a hash cannot reconstruct an artefact that was never preserved.

Secrets should not become configuration provenance. Record the identity or version of the credential mechanism where useful, not the credential value. Likewise, a source reference should not expose a private customer name in a public diagnostic URL. Reproducibility needs deliberate evidence, not maximal copying.

A practical change record for R-13 would state that retrieval and output structure changed, identify the paired cases used, explain material disagreements and name the compatible rollback route. Attach the manifests to the evaluation results so a reviewer can distinguish the candidate tested from the candidate deployed. Those are sometimes different versions by the time the meeting happens.

There is a limit to exact reproduction. A hosted service may change, some model generation is variable and a source may no longer be available. State which part of the run can be replayed and which evidence merely documents it. A precise limit is more useful than either promising perfect reproduction or declaring investigation impossible because the system contains a language model.

For R-13, investigate Q-4 and S-3 even though P-7 stayed the same. Follow the retrieved passage into the raw response and then the parsed result to find where the qualification disappeared. Keep that trail with the compatible rollback target and the limits of what can be replayed. The disputed answer then gives the next engineer a set of concrete differences to examine, instead of an apparently unchanged prompt that somebody has to persuade to behave as before.