The right paragraph from the wrong handbook

A citation can be accurate and still answer the wrong question

The citation can be impeccable and the answer still belong to someone else’s case. A faithfully quoted handbook paragraph may cover another market, product class or policy period. That is especially easy to miss when its vocabulary is exactly what the question uses, giving it a strong place in text search. Before asking which passage sounds relevant, the application needs to know which documents are entitled to answer this particular question.

Consider an invented internal support process for arranging post-dispatch follow-ups. Its documents distinguish market, new or refurbished products, and effective dates. These are fictional operational rules, not a retailer's real policy or a statement of customer rights. The question concerns a UK refurbished order dispatched on 12 December 2024. Before ranking passages by textual relevance, the application already has several facts that determine which handbooks could legitimately supply the answer.

The worked corpus contains three short documents. P1 covers current UK new-product follow-ups after thirty calendar days. P2 covers current UK refurbished follow-ups after fourteen calendar days, with a manual-coordination exception for pilot orders. P3 is an archived other-market refurbished policy with a thirty-day interval. A search for refurbished follow-up could find P3 very attractive. Its vocabulary is relevant, while its applicability is wrong in two independently identifiable ways.

Ask eligibility before asking relevance

The Python example supplies each document’s market, product and effective interval, together with permitted document identifiers. Access, market, product and date are checked separately. All three documents may be read in this invented account, but P1 covers the wrong product and P3 the wrong market and period. Only P2 applies. Keeping those reasons separate is useful when the answer is challenged: permission to read a handbook is not permission to use its rule for every order.

The date rule is explicit: effective-from is inclusive and effective-until is exclusive, with no end meaning still open. The selected date is the order's dispatch date under this proposed workflow. Another process might apply the policy in force when the follow-up is arranged or when an agreement was made. Those are different questions. Using the machine's current date by habit would erase the business choice about which event determines applicability. Record that choice with the policy metadata and the query context.

Give P1 an illustrative ranking score of nine, P2 five and P3 ten. These numbers are hand-assigned to expose the ordering, not measured similarity, BM25 or embedding scores. Searching everything selects P3. Filtering eligibility before ranking selects P2. Taking the top two and filtering afterwards produces nothing, because P3 and P1 already displaced the only valid source. All three outcomes were checked in the example. The empty result is particularly instructive: the right document exists, but the earlier truncation has made it unavailable to the later eligibility check.

Retrieval engines differ in their filtering strategies and performance trade-offs, so the application needs to inspect where eligibility is enforced. The requirement is to prevent ineligible passages from displacing the evidence that applies. For a tiny handbook collection, controlled source selection before text search may be clearer than a vector pipeline. Either approach must preserve the candidate boundary, including when a truncated result set would otherwise exclude P2 before filtering.

Preserve the qualifiers when the paragraph travels

P2’s ordinary rule adds fourteen calendar days to dispatch on 12 December 2024, giving 26 December. The example uses Python date and duration arithmetic for that calculation.[1] The answer needs to carry the conditions that made the calculation relevant: UK, refurbished, the supplied dispatch date and ordinary non-pilot handling. A date copied without those qualifications can travel much further than the rule that justified it.

The exception belongs with the ordinary rule too. A pilot order requires manual coordination in this invented handbook, so the system should not automatically apply the calculated ordinary date to it. If pilot status is unknown, the question remains unresolved. It can return the conditional interpretation or obtain the missing controlled order attribute. Assuming non-pilot because the user did not mention the exception would make a complete-looking answer depend on an unstated default. The most relevant paragraph still needs its conditions.

Anthropic’s 19 September 2024 Contextual Retrieval article discusses context lost when documents are split into chunks, and adding chunk-specific context to help retrieval.[2] In this handbook, the practical loss could be P2’s title, scope or adjacent pilot exception. A sentence saying fourteen days can be retrieved accurately while losing the information that makes it apply to this order. Keep that qualifying context beside the passage. Otherwise the later answer has to reconstruct a condition the publishing process already knew and could have preserved.

One proposed chunk record therefore retains document ID, title, market, product and effective interval alongside the relevant passage and exception. Metadata used for eligibility should come from an authorised publishing or curation process, not be rewritten by the answer model because a different scope produces more matches. A model-generated explanation can help people read a chunk; it should not quietly become the authority for which handbook applies. Keep the original wording available so the added context itself can be checked.

Distinguish missing knowledge from missing applicability

If the product classification is absent from the controlled order, the assistant cannot select between P1 and P2 on the strength of the question's wording alone. If all eligible documents are unavailable, it has a different problem from a genuinely empty eligible corpus. If no document covers the requested date, that is different again. An application that reports all three as no results leaves the operator without a useful next step and can encourage a prompt change that merely loosens the rules.

The useful response names the missing piece: confirm the product class, restore authorised handbook access or refer the uncovered period to its policy owner. None is repaired by falling back to highly ranked P3. Nor does unavailable access tell us that the hidden document has nothing useful to say. Explain the limitation the operator can act on, while keeping any details about inaccessible documents within the normal disclosure rules. That is more helpful than broadening the search until something supplies an answer.

Hard eligibility metadata needs maintenance. Tagging P2 as new would systematically exclude the right policy even if its text were otherwise easy to find. The publishing review should compare scope fields with the document, test boundary dates and exercise a question from each policy family. Including P3 as an intentionally attractive but ineligible source tests the actual failure described here. Easy same-market queries would not reveal whether that exclusion works.

For an informal assistant over one small, uniformly applicable handbook, these dimensions may be unnecessary. Adding elaborate market and date fields when no such distinctions exist can create work without improving an answer. They become useful when the organisation already maintains different rules for different cases. The search system should preserve that existing distinction. It should not ask a similarity ranking to rediscover a boundary the business already knows how to state.

Review the evidence path, not just the final sentence

A practical review record for this question names the controlled order attributes, eligible document P2, the exclusion reasons for P1 and P3, the selected passage, the pilot exception and the fourteen-day calculation. This is more informative than recording only the answer and citation. If the result is challenged, a colleague can inspect whether the wrong policy was selected, a qualifier was lost or arithmetic was applied under the wrong condition. Those are different repairs.

The final answer can still be short: under the invented ordinary rule, the follow-up date is 26 December, with pilot orders needing separate coordination. Its confidence comes from the path behind it. The controlled order matches P2’s market, product and period; the calculation uses that rule; P1 and P3 have explicit exclusion reasons. If someone disputes the result, the team can locate a selection, qualification or arithmetic error instead of arguing about whether the quotation was accurate. It may have been accurate all along.