AI Uncertainty: Why Confidence Scores Are Not Enough

TL;DR

AI uncertainty is not one measurement. Token entropy describes prediction variability, semantic entropy examines variation in meaning, and calibration tests whether confidence estimates correspond to observed correctness. None replaces current evidence or grants permission to act. For an enterprise assistant, define the claim being evaluated, preserve missing evidence as missing, and provide a deliberate abstention path. Measure errors together with the proportion of requests answered. The goal is not an assistant that always sounds certain. It is one whose uncertainty changes what the application does next.

Introduction

Return to the order-processing incident from Part 1. An AI assistant recommends restarting an instance because older tickets associate the symptoms with a suspected memory leak. This time, the recommendation includes a confidence score: 96%.

The percentage is illustrative, not a measured result. But it exposes an important design question: 96% confidence in what?

The assistant might be estimating that its explanation sounds consistent with the retrieved tickets. The application might be reporting agreement among several generated answers. Alternatively, it might have a separately evaluated estimate of recommendation correctness. Each interpretation supports a different conclusion.

Meanwhile, no one has checked whether the remaining instances can carry the workload. A precise-looking score does not fill that evidence gap.

Part 1, AI-Assisted Decisions: When Repetition Becomes False Confidence, examined the human side of this problem. This article examines the model signals and the application decisions built around them. The scope is an advisory language-model application with fixed weights during inference, access to approved records, and separately collected operational evidence. Production authorization remains outside the assistant.

The research establishes the distinctions discussed below. The incident examples, decision record, and evaluation workflow are proposed engineering applications, not reported deployment results.

Give Each Uncertainty Signal a Defined Job

Before adding a confidence indicator to an interface, decide what it measures and what decision it is allowed to influence.

The following distinctions draw on the semantic-uncertainty research of Farquhar and colleagues and the calibration work of Guo and colleagues. Evidence sufficiency is an application requirement in this proposed design, not another entropy measure.

Do not average these into a general-purpose trust score and let the average override mandatory checks. In our incident, agreement among answers cannot compensate for missing capacity evidence. The appropriate control is to obtain that evidence or withhold the recommendation that depends on it.

An uncertainty signal should have a defined meaning, an evaluation method, and an operational consequence.

Token Entropy Describes Prediction, Not Reality

Applying Shannon’s definition from the foundation article to a next-token distribution gives:$$H_t = -sum_i p_i log_2 p_i$$

Here, $p_i$ is the probability assigned to token $i$ at generation step $t$, conditioned on the prompt and preceding tokens. Base-two logarithms give entropy in bits.

The calculation describes that distribution at that point. It is not a global reliability score for the model. A high-probability next token is also not equivalent to a high probability that every claim in the eventual response is correct.

The distinction is easy to see in the incident example. Several phrasings can express the same restart procedure. Conversely, a model can strongly prefer a familiar explanation that the incident evidence never established. Counting wording alternatives does not resolve the causal question.

Record Which Distribution You Measured

Hugging Face’s Utilities for Generation distinguishes unprocessed logits from processed generation scores. Its documentation also describes temperature scaling and token filtering, which can change the distribution used for sampling.

Record whether an entropy value was computed before or after those transformations. Comparing a raw-distribution value with a filtered-distribution value mixes different measurements.

If an interface supplies only the chosen token’s probability or a truncated list of alternatives, those values do not generally determine full-vocabulary entropy. Renormalizing the visible alternatives produces a distribution over that subset, not a reconstruction of the missing probabilities. Label the measurement accordingly.

Lower Temperature Is Not a Factual Check

For fixed logits under standard temperature scaling, a lower positive temperature concentrates probability on higher-scoring tokens. In Hugging Face’s documented generation workflow, the temperature setting affects sampling when do_sample=True.

That can alter output variability. It does not collect current inventory, inspect dependency health, or validate a copied diagnosis. Repeatedly obtaining the same answer under a concentrated sampling distribution is therefore not an independent test of the answer.

Choose decoding settings for the task, then evaluate them. Do not treat reducing randomness as a substitute for checking facts.

Semantic Entropy Examines Meaning, Not Just Wording

Farquhar and colleagues’ 2024 Nature paper addresses the difference between uncertainty about wording and uncertainty about meaning. Its semantic-entropy methods group answers by meaning and target confabulations, a particular class of arbitrary, incorrect generations. The paper explicitly distinguishes these from consistent errors and systematic reasoning failures.

For our operational example, consider three candidate recommendations:

A useful grouping method should preserve the difference in the third row. For this application, it should also preserve differences in target, environment, prerequisites, and action scope. Two answers that mention the same service and command are not necessarily equivalent.

The authors’ Oxford research explanation discusses checking semantic equivalence and warns that context affects grouping. For an engineering deployment, test the grouping component against domain-reviewed examples. An evaluator that overlooks a negation or merges incompatible preconditions can hide the disagreement you need to detect.

Agreement Still Needs External Evidence

Suppose ten illustrative samples produce six recommendations to drain first and four to restart immediately. Using those observed group frequencies, the empirical entropy is:$$-0.6log_2(0.6)-0.4log_2(0.4)approx0.971text{ bits}$$

This is a constructed calculation over manually defined groups, not a benchmark or a complete implementation of the published method. It demonstrates disagreement in the observed samples. It does not mean the answer has a 97.1% chance of being wrong, and subtracting the value from one would not create a correctness probability.

If all ten samples fall into one group, the empirical value becomes zero. They could agree on the right procedure or on the same unsupported procedure. Ten samples also cannot rule out meanings that were simply not observed.

Varying the random seed produces additional model outputs, not additional measurements of the service. The samples may be independent draws under a fixed generation setup while still sharing the same missing telemetry and stale source material.

Use disagreement to identify where investigation may be valuable. Do not use agreement to declare that investigation unnecessary.

Calibration Needs a Specific Event

Guo and colleagues define confidence calibration in terms of whether predicted probabilities represent correctness likelihood. Their 2017 study concerns classification models, not a universal confidence field for conversational AI.

For a defined task, calibration has an observable interpretation: among comparable predictions assigned probabilities near 0.9, approximately 90% should be correct, allowing for sampling uncertainty. It is a relationship across predictions, not a certificate attached to one answer.

The event being scored must come first. For our assistant, producing valid JSON, correctly identifying a failure cause, and recommending a recovery action that meets its prerequisites are three different events. A score calibrated for one must not be relabeled as confidence in another.

Self-Assessment Can Be Useful Without Being Authoritative

The evidence does not justify dismissing all model self-assessment. Kadavath and colleagues’ 2022 research found encouraging self-evaluation performance under the formats and tasks they tested, while also reporting difficulties with knowledge-probability calibration on new tasks.

Jiang and colleagues separately studied calibration on question-answering tasks using T5, BART, and GPT-2. Their 2021 findings showed that answer probabilities were not reliably calibrated as provided and examined methods to improve them. Those results concern the evaluated models and settings, not every later deployment.

The practical conclusion is to treat a self-reported score as a candidate signal. Keep its origin visible and test it against independently reviewed outcomes before interpreting it as a correctness probability. Adding a percentage field to a prompt does not perform that validation.

Do Not Hide a Critical Claim Inside an Overall Score

In the incident answer, the claim that earlier restarts preceded recovery may be supported by the records. The claim that a memory leak caused the failures may remain unresolved. The claim that restarting now is appropriate requires current capacity and dependency evidence.

Evaluate those claims separately. An overall response score can obscure exactly which unsupported statement the operator would rely on.

Calibration and error detection are also different objectives. As a mathematical example, an estimator that assigns 0.9 to every case in a population with 90% correctness can be calibrated overall while offering no way to identify the incorrect cases. A routing policy needs evidence that its signal helps select the cases worth answering, not just an acceptable overall average.

Finally, keep calibration temperature scaling separate from generation-temperature tuning. Guo and colleagues fit a scaling parameter against a held-out validation set. Choosing a lower sampling temperature because the text looks more consistent is not the same procedure.

Measure Abstention Alongside Accuracy

Selective prediction allows a system to withhold some predictions rather than answer every request. Geifman and El-Yaniv’s SelectiveNet studies a model with an integrated reject option and evaluates the tradeoff between risk and coverage. It is not a ready-made policy for a language-model operations assistant.

For this proposed application, define the unit of evaluation as one requested operational recommendation. A request that receives observations but no recommendation remains an abstention for that decision.

Using incorrect recommendations as the error measure:

Coverage = requests receiving a recommendation / all evaluated requests

Selective risk = incorrect recommendations / recommendations delivered

The following values are constructed, not measured. Both policies are applied to the same hypothetical set of 1,000 requests.

The second policy has a lower error rate among the requests it answers, but it sends more work elsewhere. Whether that tradeoff is acceptable depends on error consequences, review capacity, delay, and the value of the unanswered requests.

The numbers do not imply that stricter confidence thresholds always reduce errors. A consistently wrong group of answers might receive high scores. Test the actual routing rule on held-out cases rather than assuming its risk will improve monotonically.

Report unnecessary abstentions on answerable cases and review high-consequence errors separately. Also measure the final outcome of escalated requests; passing work to a human does not demonstrate that the end-to-end decision succeeded.

A system that answers nothing has zero coverage. Its selective risk is undefined because there are no delivered recommendations to evaluate.

Make Uncertainty Change the Next Step

An abstention should identify the blocked decision, the missing or conflicting evidence, and the next responsible role. It should not simply display a warning beneath a recommendation that remains ready to execute.

For the incident assistant, a useful response would distinguish verified observations from the withheld conclusion: the previous tickets document recovery after restart, but current remaining capacity has not been established, so the assistant cannot support restarting this instance now. Route the capacity check to the incident owner under the approved response process.

The following proposed flow keeps model signals and system evidence separate until claim-level review. Neither path can grant execution permission.

Preserve the Reason the Recommendation Stopped

This illustrative application-owned record captures the distinction between model-reported confidence, sampled disagreement, and validated evidence. It is not an existing product schema, an executable policy, or a validated scoring implementation.

assessment:
  request_id: incident-0042
  requested_decision: restart_instance_now
  target: orders-instance-3

  model_signals:
    self_reported_confidence: 0.96
    semantic_cluster_counts: [6, 4]
    empirical_semantic_entropy_bits: 0.971
    calibrated_correctness_probability: null

  evidence_checks:
    target_inventory: verified
    remaining_capacity: missing
    dependency_health: stale

  routing:
    disposition: abstain_from_recommendation
    reason_codes:
      - remaining_capacity_not_verified
      - dependency_health_not_current
    owner_role: incident_commander

  execution_authorized: false

Replace the identifiers and evidence checks with the actual workflow. Bind each evidence result to its source, environment, observation time, and validation rule in the application’s evidence store. Trusted application components must populate or verify those results; the model cannot mark its own prerequisites satisfied.

The null calibrated probability is deliberate. No calibrated estimate is available, which is different from estimating a zero probability of correctness. Similarly, stale dependency evidence is not proof that the dependency is unhealthy.

Successful handling means the application withholds this operational recommendation, explains why, and routes the required check. These fields do nothing without application logic enforcing that behavior. A missing or timed-out assessment component should not silently turn into a passing result.

Evaluate the Routing Policy, Not Just the Model

Build the evaluation around the decisions the service will support. Include current and stale records, missing telemetry, contradictory sources, wrong-environment evidence, and cases where the familiar recovery action is inappropriate. Preserve only information available at the decision point.

Have domain reviewers define acceptable recommendations and escalation paths before reviewing the candidate system’s outputs. Permit multiple defensible answers where the evidence supports them. A model-based evaluator can assist, but validate its decisions against human-reviewed examples rather than letting it provide an unquestioned answer key.

Separate cases used to develop prompts or fit calibration from the final test set. Keep records derived from the same incident together so copied tickets do not appear as independent examples on both sides of the evaluation. Reserve later periods or different incident families to test changed conditions.

Compare the existing workflow with an evidence-gated baseline, then compare that baseline with evidence gating plus the proposed uncertainty signal. This makes it possible to see whether the uncertainty mechanism adds value beyond simply checking the prerequisites.

Review results by incident family and consequence, including sample counts and uncertainty around observed rates. Track response latency, repeated-generation cost, and review workload alongside error and coverage measures. The Oxford authors note that semantic-entropy assessment adds generation and comparison work; measure that overhead in the application rather than assuming it is negligible.

Assign a domain owner to the correctness rubric and evidence requirements, and an evaluation owner to scoring, thresholds, and regression cases. Record the model identifier, prompt revision, retrieval snapshot, decoding settings, and evaluator revision. Reassess the routing policy when those inputs or the supported task change.

Conclusion

AI uncertainty becomes useful when it informs a specific decision. Token entropy describes a prediction distribution. Semantic analysis can expose disagreement about meaning. Calibration tests the relationship between a score and observed correctness. Evidence checks determine whether the information required for this decision is present and applicable.

None of those functions should be collapsed into an unexplained percentage. Start with one recurring recommendation, define its critical claims, and specify what would make the application answer, investigate, or abstain. Evaluate errors and coverage together, including what happens after escalation.

The important question is not whether the assistant can produce a confidence score. It is whether the application responds appropriately when that confidence is unjustified.

The final article, AI Feedback Loops: Building Systems That Stay Correctable, carries these requirements into ongoing operations: reviewed knowledge, external authorization, versioned releases, and corrections that do not turn yesterday’s unsupported answer into tomorrow’s authority.

External References

• Hugging Face: Utilities for Generation
• Nature: Detecting hallucinations in large language models using semantic entropy
• Oxford Applied and Theoretical Machine Learning Group: Detecting hallucinations in large language models using semantic entropy
• Proceedings of Machine Learning Research: On Calibration of Modern Neural Networks
• Anthropic: Language models (mostly) know what they know
• ACL Anthology: How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering
• Proceedings of Machine Learning Research: SelectiveNet: A Deep Neural Network with an Integrated Reject Option

The post AI Uncertainty: Why Confidence Scores Are Not Enough appeared first on Digital Thought Disruption.