AI misclassifies customer requests: how should workflows handle confidence scores?

Calibration asks a practical question: among predictions scored near 0.8, is the model correct roughly 80% of the time on representative data? If accuracy is only 60%, it is overconfident; if accuracy is 90%, it is underconfident. Calibration does not make every prediction more accurate, but it makes thresholds more meaningful for workflow decisions.

Build a separate, human-labeled validation set that resembles production: short and long messages, typos, multiple languages, multi-intent emails, OCR output, new customers, and out-of-scope requests. Plot a reliability diagram or group scores into buckets, then compare mean confidence with observed accuracy. For probabilistic classifiers, Brier score or log loss can add useful evidence. Re-evaluate whenever the model, prompt, taxonomy, or data distribution changes; old thresholds should not be carried forward automatically.

At minimum, retain top1_intent, top1_score, top2_intent, top2_score, margin = top1 - top2, model or prompt version, and extracted entities. A high top score with a small margin means two intents are competing. For example, refund_request = 0.82 and payment_dispute = 0.80 is more ambiguous than refund_request = 0.82 with the next intent at 0.25.

Then check None or out_of_scope, language, input completeness, and required entities. A change_booking result cannot safely run without a booking ID, new date, and requestor verification. Finally, apply deterministic business rules: is the customer authorized, is the request within policy, does the transaction exist, and can the action be rolled back? These controls matter more than trying to increase confidence through prompting alone.

1. Auto-process. Use this route only when an intent-specific threshold and minimum margin are satisfied, the input is in scope, required entities are complete, and the action is low risk or reversible. Adding a tag or placing a ticket in a queue usually carries less risk than issuing a refund.

2. Ask for clarification. Use this when two intents are close, a message contains several goals, or the customer can supply missing information. Ask a concrete question: “Do you want to disable renewal for the next term, or cancel service immediately?” Avoid an unhelpful “Could you clarify?”

3. Review queue. Use this for material actions, conflicting data, or rare cases that still need resolution. The reviewer should see the original request, leading intents, entities, failed rules, and proposed action—not only a score.

4. Fail closed or route out of scope. Use this when the request cannot be identified, shows prompt-injection signals, lacks authorization, or fails downstream validation. A safe stop should not discard the ticket: record it, notify the appropriate party, and provide a defined recovery path.

Misclassifying general_question may only send a ticket to the wrong queue and be easy to correct. A false positive for cancel_service, refund_payment, change_bank_account, or reset_access can cause financial loss, disruption, or a security incident. These categories should not share one threshold or the same automation path.

Create an intent-level matrix covering the impact of acting incorrectly, the impact of missing a request, reversibility, verification requirements, and review SLA. Optimize thresholds using weighted error costs rather than only maximizing model-wide F1. High-risk intents may still require business confirmation at very high confidence. Low-risk intents may tolerate a more flexible threshold when reconciliation and fast correction exist. Threshold values must come from the system's own validation set and shadow traffic—not from a vendor example.

First, normalize the input and preserve an immutable original with a request_id. The classifier should only return a structured result: intent, score, top candidates, entities, model version, and a concise auditable rationale. It must not call the business action directly.

Next, a policy engine reads versioned, intent-specific configuration: auto_threshold, review_threshold, minimum_margin, required entities, risk tier, and allowed actions. It also checks authentication, customer state, duplicate requests, and business constraints. Persist the routing decision and evidence before invoking downstream systems.

Use an idempotency key in the action path so retries do not create duplicate effects. Keep the same conversation or request ID in the clarification path so new answers are joined to the original request. Use a lease or atomic claim in the review path so two employees cannot handle the same ticket. If the model or policy service fails, follow the declared safe route rather than bypassing the gate.

Track precision, recall, and F1 per intent, the confusion matrix for neighboring intents, calibration error by score bucket, and out-of-scope detection. At workflow level, add auto-processing rate, clarification rate, review rate, reviewer override rate, false automatic-action rate, resolution time, and rollback incidents.

Pay special attention to precision within the auto-process route: of all requests on which the system acted automatically, how many were truly correct? This is often more important than overall accuracy when the objective is risk containment. At the same time, monitor recall for urgent intents so an overly conservative threshold does not hide important requests. Segment results by language, channel, customer cohort, and input type to expose drift concealed by averages.

Test a new policy by replaying historical data, then use shadow mode, where the system calculates a decision without executing the action. Increase the automated share only after acceptance criteria hold over a representative period. Reviewer overrides and verified customer corrections should feed a governed evaluation dataset after cleaning and label confirmation.

Common questions about confidence scores in AI workflows

Should 0.8 be the default confidence threshold?

No. Measure score distribution, precision, recall, calibration, and error cost per intent on the system's own data.

Can high confidence replace business validation?

No. Confidence concerns the classifier's label choice. Authorization, contract state, required data, and transaction rules still need deterministic validation.

When should the workflow ask the customer rather than request review?

Ask when one specific question can resolve the ambiguity, such as choosing between disabling renewal and immediate cancellation. Risky, conflicting, or verification-dependent cases belong in review.

Does the top-1 versus top-2 margin replace a confidence threshold?

No. Margin is an additional signal. Both scores may be low or both high; out-of-scope detection, required entities, and risk policy still apply.

Does every low-confidence request require human review?

Not necessarily. The workflow can ask for missing data, route out-of-scope requests appropriately, and reserve authorized review for cases requiring judgment or sensitive action.