What does the evaluation permit us to release?
Separate observed results, inference and release authority, so an attractive score cannot quietly approve a different product.
AI tooling7 min read
Who chooses the next step?
A service-desk case set separates uncertain wording, missing evidence, exploratory research and absent authority—four reasons a workflow can stop, with different architectural answers.
An unfamiliar sentence is not an unfamiliar process
“The printer stops after the second page” is an uncertain description of a fault. It may still belong to a familiar process: establish the model, retrieve the approved guide, interpret the problem and draft a reply. The wording needs judgement; the next business step may not. I would make that distinction before deciding the service desk needs an agent to choose its route.
That distinction is easy to lose when every workflow containing an AI call acquires the agent label. Anthropic's December 2024 article separates predefined workflows from agents whose models dynamically direct their processes and tool use.[1] The terminology is useful when it helps locate a decision. It is less useful when it turns a practical design choice into a contest over which arrangement sounds more advanced.
The question is where flexibility will actually help. A fixed sequence can contain a useful language interpreter, and an exploratory investigation can have firm limits on what it may read or change. A hybrid can remain the right architecture for this particular job. Draw the choices the service desk must make, then identify which next step really depends on something discovered during the work rather than on a rule the business already knows.
Use a fictional office-equipment service desk to make that reasoning concrete. The examples below are design cases, not observations from customers or model tests. The proposed tool can read approved material, ask for missing details and draft a reply. It cannot book a visit or change an account. That scope lets us examine route selection without confusing an interesting research step with permission to make a customer commitment.
Draw the fixed route honestly
Begin with validate the equipment identity, retrieve the approved guide, interpret the request and draft supported guidance. Give each stage an outcome for missing or contradictory information. A serial number absent from the request can take a known clarification branch. Two conflicting serial numbers require resolution. Neither case becomes an argument for broad exploration simply because the happy path cannot continue immediately.
A model could interpret the ordinary description as an approved issue category and return the supporting passage with any uncertainty. The surrounding workflow can still require clarification when the equipment identity is missing. Keeping those decisions explicit means that a fluent answer about the wrong model cannot bypass the identity check merely because the interpretation step produced a plausible category and useful-looking guidance.
Fixed branches have a cost when every unfamiliar phrase acquires another condition. After a while the route can become a collection of examples that is harder to understand than the requests it handles. Ask whether the business state changed or the customer simply described it differently. Better interpretation can handle new wording inside the same process. A new kind of commitment or investigation may need a genuinely different route, rather than one more instruction added to the existing prompt.
For this design, a known fault code with a matching approved guide stays on the fixed path. A missing model goes to clarification. A contradictory identity also goes to clarification, with the conflict stated. These decisions can be tested without asking a model to invent the next action. The model's role is to interpret or explain within the authorised route, not make the existence of the route itself probabilistic.
Recognise a discovery that changes the route
Now consider a request whose model is known but whose fault does not appear in the approved guide. It may be appropriate to search a permitted manufacturer knowledge base, inspect several candidate explanations and report what was established. Which source to examine next can depend on the first finding. That is a more credible use for model-directed exploration than guessing the missing serial number.
The exploration still needs an exit. It can produce a supported draft, a question or a handover describing the gap. If no permitted source covers the fault, the system should not widen its own source policy until an answer appears. A route chosen dynamically is useful only within the authority the business intended to delegate. The ability to search is not authority to certify a repair or promise a visit.
An unavailable approved lookup is another kind of stop. The guide might cover the fault perfectly; the service just cannot return it at the moment. A bounded retry or a useful handover addresses that dependency. Sending an agent off to search elsewhere can change the allowed source set while leaving the broken lookup untouched. Record why the route stopped, so the colleague receiving it knows whether to restore a service or investigate a gap in the guidance.
Finally, a request to change an account lies outside this draft-only tool's scope. More reasoning cannot supply missing permission. The appropriate route is a clear explanation and the authorised service channel. Treating every refusal as an agent failure would reward the system for crossing the very boundary that makes the tool acceptable to operate.
Compare architectures on the cases that separate them
The six cases are therefore a known fault, missing model, conflicting identity, unknown but in-scope fault, unavailable lookup and out-of-scope account action. The figure records which kind of uncertainty each introduces. These cases help prevent a broad question about agents from collapsing into a demonstration where the chosen architecture is shown only the work it already handles well.
A fixed workflow can describe all six, but the unknown-fault branch may become a constrained research sub-process or an immediate human handover. A bounded agent can select research steps dynamically, but still needs deterministic checks for identity and authority. A hybrid can keep intake and commitments fixed while delegating the uncertain research route. The relevant comparison is what each design makes easier to explain, change and recover.
Do not count only successful answers. Record where a person intervenes and what evidence they receive. A research handover containing source references and a precise unresolved question can reduce work. A longer transcript of unsuccessful tool calls may increase it. Conversely, a fixed handover after every unfamiliar phrase can leave the tool unable to perform interpretation that would have been perfectly reasonable to delegate.
Somebody has to maintain the branches, source list, tool contracts and cases used to check the result. Model-directed flexibility can reduce code changes for unusual investigations, then ask more of the person untangling a failed one. A fixed route can make that handover simpler while leaving engineers with more branches to maintain. Compare both workloads with the people who will do them; the architecture’s apparent simplicity may depend on whose calendar is being left out.
Six requests that separate interpretation from route selection
A fictional office-equipment assistant can read approved sources, request clarification and draft guidance. It cannot book visits or change accounts. These are proposed design decisions, not model test results.
- Known model and covered fault
- Fixed path
Retrieve the matching approved guidance; language interpretation need not choose a new route.
- Model missing
- Clarify
The route is known, but required identity data is absent. Do not guess a model.
- Two conflicting identities
- Resolve conflict
Explain the contradiction before retrieving guidance for either device.
- Known model, uncovered in-scope fault
- Bounded research
The next approved source may depend on earlier findings; return support or a precise gap.
- Approved lookup unavailable
- Dependency recovery
Follow the agreed delay or handover rule rather than confusing service failure with a research question.
- Request changes an account
- Authorised channel
A draft-only tool has no authority to perform this action, however well it understands the request.
Let the boundary move with evidence
For an actual trial, keep the business scope and source set comparable and examine the six cases through each proposed design. Note which choices were prescribed, which were model-directed and which required a person. A model-backed prototype can provide useful outputs, but a hand-worked route comparison is already enough to expose a design that depends on missing permission or unexplained defaults. Do not label a planned path as a measured success.
Look at the failures that cause the most additional work. If fixed branches continually defer ordinary wording, improve interpretation before adding general autonomy. If bounded research repeatedly follows the same approved sequence, make that sequence explicit. If the tool needs genuinely different investigation paths, retain the flexibility and give the resulting evidence a clear structure. Architecture can change without the team having been wrong to start more simply.
Write down what would justify expanding the delegated route. New source access, new external actions and a different customer group are distinct changes, not an undifferentiated next phase. A draft-only tool can be valuable for a long time. It does not need to apologise for stopping before a consequential action, particularly when the person authorising that action benefits from the draft's explanation.
For this service desk, I would keep equipment identity and customer commitments explicit, allow interpretation of ordinary descriptions and try bounded research where the next approved source genuinely depends on earlier findings. A workflow, agent or hybrid could implement that division. Put each through the six requests and inspect what reaches the colleague who takes over: which sources were checked, why the route stopped and which unresolved question they can now address.
Working on a workflow like this? Hamsa helps finance and operations teams connect their tools and handle the exceptions, starting with one workflow.
See the Payment Operations PilotReferences & further reading
Primary sources behind this article. Living documentation may change.
- Anthropic: Building effective agents Published 19 December 2024
Practical engineering. A human perspective.
More about Hamsa Software and the person behind it. Editorial notes.A few more things
worth a read.
Back to the reading room 
The prompt did not change. The system did.
Version the resolved configuration around an AI response so comparisons, investigations and rollback refer to the system that actually ran.
The notice was withdrawn. Why is the assistant still using it?
Freshness follows source dependencies through indexes, summaries, caches and in-flight answers—not just the original URL.