There may be no policy to find

The policy appears between two documents

Can a colleague claim £180 for a home monitor they have already bought? An internal assistant finds a travel reimbursement policy and an equipment note about monitors, combines them into a yes and suggests manager approval. Neither document says that a home monitor already purchased can be reimbursed, although each supplies words that make the answer sound plausible. The search has found related material; the answer has quietly supplied the missing policy.

That is an awkward mistake for an internal tool because the person asking has a reasonable expectation that it describes the organisation’s rules. They may repeat the answer to somebody else, and a policy owner may never see the original question. A more hesitant tone would not repair that. The product needs to preserve the missing rule and give the colleague a useful next step instead of expressing an invented permission more politely.

The example here uses original synthetic passages, available in the linked JSON fixture. D1 says rail travel for approved client visits may be reimbursed when agreed in advance. D2 says employees should use the equipment request form for a chair or monitor. These statements are enough to answer some questions. They are not enough to establish retrospective reimbursement for an already-purchased home monitor.

The distinction is the point of this article. Useful search is not only a matter of finding related material. The system needs to decide what the material supports, what the question still leaves unspecified and when the organisation itself has not supplied an answer. Those outcomes deserve different interfaces, because the person asking needs to know how to continue.

Related words are an invitation to inspect

Retrieval can perform well at its immediate task and still leave the answer unsupported. A passage about reimbursement is relevant to an expense question; a passage about equipment is relevant to a monitor. The connection makes them worth reading. It does not permit every sentence that could be assembled from their combined terminology.

The original RAGAs paper separates evaluation dimensions including relevant retrieved context, faithful use of that context and answer quality. [1] That separation is useful even in a modest internal assistant. A single good-looking response can hide which stage worked and which failed. Inspect the retrieved passages before assuming the answer problem requires a better search index or a different model.

Read the proposed yes clause by clause. Does a source cover home use, an already completed purchase and the amount involved? Does it say a manager can approve afterwards? A passage may support one part while leaving the others open. Following each claim back to its words is more revealing than asking whether the overall answer sounds sensible. In this case, the plausible connection is precisely the part the documents do not supply.

Sometimes one small question is enough

Now add D3: managers may approve equipment requests up to GBP 150 through the equipment request form. The user asks, can my manager approve the monitor request? The source contains a rule, but the question omits the amount and whether this is a request through that process. The assistant need not defer everything to a policy owner. It can ask the particular facts needed to apply the available rule.

A clarification is useful when the answer determines how a source rule applies. Here, the amount determines whether the cited limit covers the request, and the process determines whether this is the type of request the passage describes. Explaining that connection briefly helps the person supply relevant information. Asking for broad context without identifying such a connection can create another exchange without bringing the decision closer.

Do not use a clarifying question to cover a missing policy. Asking where the person works or which monitor they bought cannot establish a reimbursement rule absent from the collection. Before asking, identify how the answer could resolve the uncertainty. If no likely response would supply the missing organisational authority, the next step belongs with the policy owner, not another turn of conversation.

A good assistant can therefore be quite concise without pretending certainty. It might say that the available rule concerns equipment requests through the form and ask the amount. It should not quietly convert that rule into permission for a retrospective claim. The wording preserves the boundary between applying a known rule and making a new one.

A conflict is not a request for the larger number

Suppose the collection contains both D3 with a GBP 150 limit and D4 with a GBP 200 limit. There is no metadata establishing which controls, whether one supersedes the other or whether they cover different groups. Selecting the newest-looking paragraph or the more generous limit would resolve the text conflict by inventing an authority decision.

The response should name the disagreement, cite the two passages and explain what is needed to resolve it. That may be confirmation from the owner or better authority metadata. If one source is clearly marked superseded, the conflict changes: the system can follow the current rule and retain the older passage only as history. Document age alone is not the same as that explicit relationship.

A missing rule feels different from a missing amount, and the interface should explain the difference. One person can supply the price; they cannot supply the organisation’s authority to reimburse it. A conflict between £150 and £200 needs another kind of resolution again. Treating all three as “something went wrong” makes a search service that reached an honest limit look broken, while giving the person no clue about who can move the question forward.

Make the refusal useful enough to trust

For Q2, the already-purchased home monitor, a useful response identifies the precise gap: the supplied passages do not establish retrospective home-equipment reimbursement. It can still point to the equipment form as related information, clearly labelled. Then it routes the unresolved policy question to a named role or maintained contact route, without inventing a promise that the claim will be accepted.

Keep enough context with that handoff that the person does not need to start again. Include the question, sources considered and the missing rule, while respecting the access arrangements around the documents. The policy owner can then answer the actual uncertainty. A generic instruction to ask somebody else merely returns the work with an additional page to navigate.

Repeated home-equipment questions are useful information for the document owner. There may be a missing paragraph, or a relevant policy outside the collection. Publishing the approved answer at its proper source can help everyone who asks next. One manager’s ad hoc reply should not silently become that source, though. Someone still needs the authority to make the rule and a maintained place where later readers can find the version that governs them.

Test both the useful answer and the useful pause

The fixture contains four authored expected behaviours. Q1 asks how to request a monitor and can be answered directly from D2. Q2 exposes the reimbursement gap. Q3 needs the missing amount and request context. Q4 exposes the conflicting limits. These are not model scores. They are a small, inspectable account of how the product should behave before a model is evaluated against it.

In a trial, save actual retrieved passages and responses for these cases and a broader independently labelled set. Review unsupported answers separately from unnecessary deferrals. A system that refuses everything will avoid some errors while failing its purpose. An automated judge can assist evaluation, but its verdict should not become unquestioned ground truth for the exact policy distinctions the team has not yet resolved.

Anthropic’s effective-agents discussion notes that a single model call with retrieval may suffice for many tasks. [2] Begin with a flow simple enough to inspect. More searches or a longer agent run may help find missing evidence, but they cannot create organisational authority when the evidence is absent. Keep a stopping rule so the tool does not keep looking merely to avoid admitting that it has reached the collection’s limit.

The four questions now lead to four understandable continuations: use D2’s equipment form instruction, take the reimbursement gap to its owner, supply the facts needed for D3, or resolve the conflicting limits. The assistant can be helpful in each case without saying yes in all of them. For the already-purchased monitor, a longer search might find another document; it cannot make the travel and equipment passages confer a permission they never contained.