Twenty awkward emails can change the decision

The score improved; the service might not have

In the twenty-email example below, the policy with the better accuracy score misses two requests that need urgent attention. The less accurate policy catches them, then sends more work to the priority queue than the team can handle. I think that is a more useful starting point for an evaluation than choosing a target percentage. Both policies have something going for them, and neither number tells the service owner which one they can actually run.

Start with what happens after the classifier answers. Here, access-critical messages need the priority queue, while ordinary enquiries stay in the standard queue. A short, politely phrased access problem can need attention sooner than a long general question. What matters is the work the message describes and the response the service owes, rather than how convincingly the customer manages to sound urgent.

We will compare two invented policies on 20 teaching cases: eight access-critical and twelve ordinary. Their outcome counts are deliberately constructed, not produced by a model or drawn from a client inbox. This lets us follow the arithmetic and change the assumptions openly. A real evaluation would need independently labelled records, an appropriate sample and actual predictions saved against a versioned configuration. A useful real case record would keep the message, relevant account context, expected route and reason together. For example, “I can sign in but cannot open the service I paid for” may belong to access support even without the words locked out. Inspecting the two missed cases could reveal a missing concept rather than a generally weak classifier. The toy counts cannot tell us that cause; only their underlying records could.

Count the mistakes in both directions

Policy A correctly sends six critical messages to priority and misses two. It sends one ordinary message to priority unnecessarily and leaves eleven correctly in the ordinary queue. Seventeen of twenty decisions are correct, giving 85% accuracy. The two missed access requests remain consequential even though they account for a small share of the score; the confusion matrix identifies them separately from the unnecessary escalation.

Policy B catches all eight critical messages but sends four ordinary messages to priority too. It leaves eight ordinary messages correctly routed, giving sixteen correct decisions out of twenty: 80% accuracy. The score falls by five percentage points. Whether the service becomes worse depends on the consequence of the two critical misses avoided and the extra priority work created.

A confusion matrix keeps those counts visible. The scikit-learn documentation defines its entries as actual versus predicted classes and explains the related classification metrics. [1] Use explicit axis labels and counts, not just a heatmap. Otherwise a reader can reasonably misread a normalised percentage as a number of messages, or misunderstand which direction contains the missed cases.

For this example, critical recall is the proportion of actual critical requests that reach priority: 6/8 for A and 8/8 for B. Precision asks a different question about the requests sent to priority: 6/7 for A and 8/12 for B. Neither replaces the other. The team needs to know both whether important work is missed and how much work arrives in the priority lane.

Give the errors consequences, carefully

Suppose, purely for planning, a missed critical request costs 20 effort-or-harm units and an unnecessary priority route costs 1. Policy A costs 2 × 20 + 1 = 41 units. Policy B costs 0 × 20 + 4 = 4 units. The lower-accuracy policy looks preferable under that rule. These units express a judgement about relative consequences; they are not pounds lost, customer outcomes measured or a universal weighting for access support.

Giving a missed case a weight of 1 instead of 20 reverses the preference: A now costs 3 units and B costs 4, although neither has changed a single prediction. That is not a defect in the arithmetic. It reveals a judgement we were making about the mistakes. Put that judgement beside the score so the people responsible for the service can disagree with the assumption instead of being asked to accept a recommendation whose reasoning is hidden.

Scikit-learn’s threshold guidance distinguishes estimating a class score from deciding which action follows it. [2] That distinction is useful even if your classifier is a prompted language model rather than a conventional statistical estimator. The model supplies an output; the application chooses the route and consequence. A default decision threshold or a natural-language instruction should not quietly become the business’s risk policy. Some consequences should be treated as constraints rather than traded away through a low average cost. The business may require a particular class of access problem to reach a person within a stated period, regardless of how many cheap ordinary routes are improved. Write that requirement alongside the weighting. A cost worksheet can expose choices, but it cannot authorise a trade the service has promised not to make. Keeping a hard requirement visible prevents a convenient numerical optimum from answering the wrong question.

The priority queue has to fit the policy

Then the queue intervenes. A needs seven priority places and B needs twelve. If only ten can be handled within the required window, B has not become an operating solution simply by catching every critical case in the example. Those extra ordinary enquiries will be read by the same people who must help the critical customers. More generous routing can therefore delay the very work it was intended to protect.

There are several possible responses, with different obligations. Increase capacity, improve the decision boundary, collect a missing input before routing or give ambiguous cases a separate review path. Do not simply hide two messages from the queue to make the numbers fit. The route needs an owner and an agreed response window, and the team should know which work will wait when arrivals exceed the assumed volume. The answer sheet lets us identify B’s four unnecessary escalations after the fact. The live priority queue does not arrive with those errors labelled. Dropping two items to reach ten could therefore discard the very access requests B was chosen to protect. A separate review route needs a way to make that decision, not merely a smaller capacity target.

Avoid translating the capacity limit directly into a fixed sample quota. The twenty cases here are a teaching batch, not a day’s arrival pattern. A real priority queue depends on when work arrives and how long different cases take. Use actual operational evidence before promising a response improvement. The useful connection is that classification decisions create work somebody must be available to complete.

Keep the small collection honest

A small evaluation set is valuable because people can inspect it. Include clear access problems, ordinary enquiries and genuinely awkward language, with the expected operating rule recorded. Ask another person to label a sample independently and resolve meaningful differences. If the team cannot agree whether a case is critical, that disagreement belongs in the task definition rather than being silently charged to the model.

Keep a stable comparison set and separate cases for checking whether prompt changes generalise. Repeatedly tuning against the same twenty messages can produce a better rehearsal rather than a better service. Record the source material, prompt, model configuration and action policy used for each run. A changed policy may explain a changed confusion matrix even when the underlying language model stays the same.

Be cautious with confidence values. A model saying 90% confident does not establish that nine out of ten similar claims are correct. Calibration concerns the relationship between predicted probabilities and observed frequencies; its documentation explains that separate evaluation problem. [3] Until you have evidence for that relationship, use the number as an unverified output, not a guaranteed way to select which customer can safely wait.

Choose the next trial

The downloadable worksheet computes the counts, accuracy, recall, priority volume and two cost scenarios discussed here. You can change the cost weights and see the decision move. That reproducibility is useful precisely because the assumptions remain yours to inspect rather than arriving concealed behind a recommendation.

A real pilot is a chance to read the missed messages, not just count them. Keep customer sending and irreversible actions outside the trial while the routing decision is assessed. Ask what made each critical case disappear into the ordinary queue, and what the unnecessary escalations have in common. Missing account context, an overlooked phrase or overlapping queue definitions may call for a narrower change than replacing the whole model. Save enough of the original record to tell those explanations apart.

Neither policy has earned deployment if its mistakes or workload fall outside what the service can support. That still leaves the trial with a useful result: perhaps the classifier needs account context, perhaps the priority definition needs resolving, or perhaps ten places will never cover the demand. The next trial can examine that particular question. Choosing the better accuracy figure would leave those different problems conveniently unresolved until customers found them.