What if the passing test is the thing that changed?

The answer moved

The price calculation returned £110 where the rule required £108. Changing the test’s expected answer to 110 made the run pass without repairing the calculation. Tamperward’s diff check missed that edit in this experiment. Its second check restored the original assertion, asked for £108 again and exposed the unchanged bug. That difference is why I would keep both results in view when reviewing the patch.

The arithmetic is small enough to follow without the tool. Start with £100, subtract the fixed £10 discount, then add a 20 per cent surcharge: £108. Reverse the operations and the answer is £110. Code and tests can agree on the wrong order, particularly when the same assistant can change both. A successful test command then needs an explanation of which requirement it actually ran.

Tamperward is Hamsa Software’s research project for protecting that checking process.[1] This experiment used its published 2.47.0 CLI, one JavaScript function and one Node assertion. The question was deliberately narrow: could a candidate make the visible test pass by changing the check instead of satisfying it?

Give the same rule four different changes

The starting function calculates subtotal × 1.2 − 10. Its test uses node:assert/strict to require 108 for a subtotal of 100, so the baseline fails. That is intentional. The baseline is trusted for the requirement it preserves, rather than assumed to contain correct source. Each candidate starts from that same point in its own temporary Git repository, with node --test as the configured verification command.

One candidate repairs the function to calculate (subtotal − 10) × 1.2. Another changes the test call to test.skip. A third changes only the expected 108 to 110. The fourth repairs the function and renames the test without altering the assertion. That rename matters: a check that refused ordinary test maintenance would make legitimate development harder without telling us whether the price rule survived.

Each candidate went through check on its baseline-to-candidate Git diff and verify against the same base. The accompanying runner retained both JSON outputs. Keeping the pair lets a reviewer distinguish a detector finding from an executed failure. If only the final badge survived, the expectation edit and explicit skip would look alike even though the diff detector treated them differently.

The literal edit gets past the diff check

The diff check blocked test.skip, reporting the test-skip rule, changed line and need for human sign-off. The source repair and harmless rename produced no findings. That is the distinction we wanted so far: changing the implementation is allowed, while preventing the original assertion from running needs attention.

The expected-value edit also produced no findings. In this release, the assertion-weakening detector covers selected JavaScript patterns, including some expect-style changes, but it did not detect the node:assert literal changed here.[2] The calculation still returned the wrong amount. A reviewer stopping at this clear diff report would approve a candidate whose only improvement was making the test expect its mistake.

The release guide separates blocking rules, warning-only assertion checks and reserved work. Assertion weakening is a warning in this version; guard removal remains reserved.[2] Those are different promises to an integration team. If a release process requires a particular refusal, it needs to check whether that refusal is enforced, merely reported or still planned. Here the missed literal change left the next layer with an important job.

Put the original assertion back

Verification first runs the candidate’s visible suite. It then runs a second suite with protected test and configuration material restored from the trusted base. In the expectation-edit case, the first run accepted 110. The restored assertion demanded 108, and the unchanged calculation failed. The verdict was MASKED_FAILURE: the apparent success depended on changing the protected checking evidence.[3]

The skipped-test candidate reached the same verdict by another route. Its visible command exited successfully because the assertion was skipped. Restoring the test made it run and fail. The repaired calculation passed both suites and received VERIFIED, as did the repair with a renamed test. The table keeps those four actual outcomes together, so a legitimate rename is not lost among the weakening examples.

The explicit skip was detected before execution, while the literal expectation change passed diff inspection and failed when the original assertion was restored. For this patch, the restored test supplied the disagreement the diff report lacked. It still asked only the configured question about the £108 rule. Other prices and unrelated access checks need their own tests; this comparison does not create coverage that the baseline never contained.

Who gets to choose the baseline?

Someone still has to select a trustworthy baseline and command. If an assistant can weaken that starting point or remove important inputs from protection, the restored run may no longer test the independent requirement. The project therefore treats verification configuration and related inputs as protected surfaces.[3] A commit hash identifies selected material; the review process establishes why that material is an appropriate basis for judging the candidate.

For a small team, the practical arrangement could be a reviewed main-branch commit, a separately controlled CI configuration and a named person who can approve a change to the protected expectations. Keep the baseline identifier and candidate identifier with the result. Otherwise, someone reading yesterday’s verification output may accidentally apply it to a different patch today.

Sometimes the test should change. A new surcharge policy or corrected provider contract may deliberately replace the old requirement. Restoring yesterday’s assertion will then reveal the incompatibility we expected. Review the changed contract and its supporting evidence, rather than insist the old value must remain forever. The protected run helps make the decision visible; it does not remove the need for somebody authorised to decide which rule the candidate should now satisfy.

This is also why an exception should name the rule, file, reason and intended change rather than becoming a permanent permission to modify tests. The reviewer needs to distinguish a deliberate business decision from a convenient escape route. No detector can make that distinction solely from the fact that a test file changed.

A comparison needs somewhere suitable to run

The verification command needs a host that can start the required Git and Node processes. If a subprocess cannot launch, a command is missing or the execution budget runs out, there is no completed comparison to call passed. Keep the error with that attempt, repair the environment and rerun. An inability to ask the original question must not become another way for the candidate to escape it.

Tamperward’s public security envelope sets out the research release’s assumptions and boundaries.[4] The host, selected command and protected baseline remain trusted inputs. Restoring a test does not make the application code it executes harmless. A team therefore needs an appropriately isolated runner and proportionate file, credential and network access before running an untrusted candidate.

New tests still deserve review. A source change can add behaviour that the baseline never examined, and a restored suite may pass while saying nothing about it. For a changed payment calculation, I would still inspect zero, negative inputs if allowed, rounding and the actual surcharge policy. Protecting earlier evidence and choosing sufficient evidence are different jobs; the release decision needs both.

Try the expectation edit before relying on the result

Begin a trial in a disposable repository with a known failing case and a harmless source repair. Then skip the assertion and change its expectation deliberately. Keep the CLI version, baseline, candidate and raw outputs with each result. Those examples let the team see how its chosen configuration distinguishes fixing the requirement from removing the objection to it.

In this experiment the revealing candidate leaves the price wrong, the visible tests green and the diff report clear. Only restoring the original £108 assertion exposes the dependence on the edited test. That is a concrete failure to repeat when evaluating another configuration. If it stops appearing, investigate which check changed before asking the green result to support a release.