AI Output Verification: Confidence Is Not Evidence

TL;DR

AI output verification should assess specific claims against appropriate evidence, not assign a blanket trust score to a polished response. Separate supportability, capacity, isolation, and resilience, then record what supports each conclusion and where that support stops. This article develops an evidence contract for an infrastructure assistant evaluating two Kubernetes clusters, including a worked example where normal capacity looks sufficient but host-failure capacity does not. The result is a scoped decision record with limitations, ownership, and review triggers. It is not permission to deploy.

Introduction

An infrastructure assistant recommends deploying two Kubernetes clusters on an existing virtualization estate. Its explanation is clear, its sizing sounds reasonable, and its final sentence says the design is ready.

Before accepting that conclusion, consider what the organization is being asked to believe. The component combination is supported. Capacity is sufficient. Access isolation meets the requirement. The service can tolerate a host failure. These are different claims, but the assistant has packaged them into one recommendation.

The foundation article, Schrödinger’s Cat and AI: Plausible Is Not Proven, separated a generated answer from a verified claim. This first follow-up turns that distinction into an operating pattern: an evidence contract connecting each material claim to its scope, sources, assessment, and limitations.

The running example is hypothetical. The organization wants two Kubernetes clusters for an internal inference service, with defined access-isolation and single-host-failure requirements. No particular vendor configuration is assumed to be supported. The evidence contract is a proposed application pattern, not an industry standard or a guarantee of correctness.

The objective is to produce a recommendation that another engineer can inspect, challenge, and update without starting the investigation again.

Define the Claim Before You Judge the Answer

Start by replacing “Can we deploy two clusters?” with a scoped decision. Identify the platform, environment, target resources, component versions, workload profile, and acceptance criteria.

“Isolation” needs a definition. Separate administration, restricted workload access, and independent failure domains are different requirements. Likewise, “survive a host failure” should specify which workloads remain protected, what interruption is acceptable, and whether the service must retain its normal performance objective.

For this scenario, decompose the recommendation as follows.

Decide which claims are mandatory before evaluating the answer. The workload and platform owners should control that requirement set. The assistant can identify additional questions, but it should not quietly remove a difficult requirement to complete its recommendation.

Three supported claims do not compensate for one failed mandatory requirement. Treat the overall decision as a set of conditions that must be satisfied, not an average of favorable observations.

AI Confidence Is a Signal, Not a Support Statement

NIST’s Generative Artificial Intelligence Profile identifies confabulation as a risk in which generated content can be confidently presented while erroneous. A polished explanation is therefore not sufficient evidence about the environment it describes.

However, the opposite claim, that model confidence is always meaningless, goes too far. In Language models (mostly) know what they know, Anthropic researchers found useful self-evaluation and calibration behavior under studied conditions. They also reported difficulty calibrating one form of knowledge prediction on new tasks.

The practical distinction is between a confidence estimate evaluated for a particular task and an unexplained percentage attached to an answer. Neither should substitute for information the assessment requires but does not have.

For the cluster decision, a confidence score cannot supply missing inventory, establish an undocumented compatibility condition, or demonstrate recovery behavior. Use a validated confidence signal to help prioritize review, not to waive mandatory evidence.

Asking the model to reconsider can help expose a weak argument. But another answer agreeing with the first, without new evidence, does not independently establish the state of the infrastructure. Agreement is an observation about the answers, not a measurement of the target environment.

Build an Evidence Contract Around Each Material Claim

An evidence contract should make the relationship between a conclusion and its support explicit. At minimum, retain the claim, scope, acceptance criterion, evidence identifiers, assessment method, reviewer or checking component, assessment time, and known limitations.

Notice the separation in the proposed workflow below. The model helps formulate the claims and explain the result. Evidence collection and assessment remain identifiable activities, rather than being implied by confident prose.

Use assessment states consistently. Supported means applicable evidence satisfies the defined criterion within its recorded conditions. Contradicted means applicable evidence opposes the claim. Unresolved means evidence is insufficient. Conflicting means relevant evidence disagrees and the discrepancy needs reconciliation.

These are workflow labels, not absolute truth categories. A retrieval failure is not evidence that a configuration is unsupported. A missing test is not a failed test. An access-denied response is not proof that a resource does not exist.

Use deterministic checks for structured comparisons and calculations, and domain review where interpretation is required. The assistant can help draft an assessment, but a field it generates saying supported should not constitute its own proof.

Make Retrieval Prove Its Relevance

Retrieval-augmented generation, or RAG, does not eliminate the distinction between finding a source and supporting a claim. Gao and colleagues’ Enabling Large Language Models to Generate Text with Citations evaluates fluency, answer correctness, and citation quality separately.

NIST’s generative AI profile also recommends reviewing sources and citations during pre-deployment evaluation and ongoing monitoring. For this workflow, translate that principle into three checks: the source exists, the relevant passage supports the claim, and its conditions match the target environment.

A design guide may describe a supported deployment pattern without establishing that the installed versions qualify. A successful laboratory report may cover the correct software but a different workload. Both sources can be relevant without settling the decision.

Look for Exceptions, Not Just Confirmation

Require the assessment to examine limitations, exclusions, and version-specific notes, not merely retrieve a sentence containing “supported.” Preserve the condition attached to a favorable statement when summarizing it.

When sources disagree, compare their authority, scope, versions, and revision history. A newer general article should not automatically override a release-specific support statement. Conversely, an older statement should not survive an explicit applicable correction just because it was retrieved first. Escalate unresolved conflicts to the relevant owner.

Record the source revision and supporting passage. Two pages repeating the same underlying statement are not two independent confirmations. A checksum can help establish that a stored artifact has not changed, but it cannot establish that the artifact was correct.

Keep Retrieved Content Out of the Control Path

OWASP’s LLM01:2025 Prompt Injection describes indirect injection through external content and cautions that RAG does not fully mitigate it. A retrieved document can therefore be both useful evidence and a source of instructions that should not control the workflow.

In this design, documents may inform an assessment, but they cannot change the required checks, expand collector permissions, or set the final evidence status. Enforce those boundaries in the application. Labeling a passage “untrusted” is useful context, not a complete security control.

When Normal Capacity Passes but Failure Capacity Does Not

Consider a simplified memory calculation for the proposed deployment. These figures are invented to illustrate the decision, not measured results or a sizing recommendation.

Assume four identical hosts, each with 256 GiB available to the protected workloads after platform overhead. Existing protected demand is 640 GiB. The proposed clusters add 192 GiB. All protected demand must remain accommodated after losing one host, with no reliance on memory overcommit or workload shedding.

The arithmetic is straightforward:

Healthy-host capacity:       4 x 256 GiB = 1,024 GiB
Existing protected demand:                    640 GiB
Proposed additional demand:                   192 GiB
Combined protected demand:    640 + 192 =      832 GiB

Healthy-host headroom:      1,024 - 832 =      192 GiB
Capacity after one failure:   3 x 256 =        768 GiB
Failure-state shortfall:       832 - 768 =      64 GiB

An assistant looking only at normal operation could conclude that there is enough memory. Under the stated host-failure requirement, the same inputs contradict the claim that all protected memory demand fits after losing one host.

This is a necessary-condition check, not a full availability analysis. It assumes ideal placement and does not assess CPU, accelerator capacity, storage, networking, or recovery time. Failing it is enough to reject this memory plan under the stated assumptions. Passing it would not prove that the inference service meets its availability or performance objectives.

The useful response is specific: revise the capacity or workload plan before recommending production deployment. Do not convert the failed condition into “probably fine,” and do not discard the separate supportability assessment merely because the capacity plan needs work.

An Evidence Record for the Two-Cluster Decision

The following YAML shows how to preserve that distinction. It is an illustrative record, not an executable policy or vendor schema. All identifiers and findings are fictional; the scope and requirement records would contain the exact environment, workload, and acceptance criteria.

schema: evidence-record/v1
record_id: evidence-042-v2
request_id: cluster-expansion-042
scope_record: scope-042-v1
requirements_record: requirements-042-v1
assessed_at: "<UTC assessment timestamp>"

claims:
  component_support:
    required: true
    status: supported
    evidence: [support-review-091]
    limitation: captured_configuration_only

  single_host_failure_memory:
    required: true
    status: contradicted
    evidence: [inventory-881, demand-plan-042, calculation-017]
    finding: protected_demand_exceeds_remaining_capacity_by_64_GiB

  access_isolation:
    required: true
    status: unresolved
    missing: [boundary_test_against_requirements]

  workload_acceptance:
    required: true
    status: unresolved
    missing: [representative_performance_and_recovery_tests]

assessment_owner: platform-review-team
recommendation: revise_before_deployment_review
reassess_on:
  - component_or_support_guidance_change
  - inventory_or_workload_requirement_change
  - placement_or_dependency_change

Replace the identifiers, claims, owners, and review triggers with the actual workflow requirements. Resolve evidence identifiers through an access-controlled store containing the underlying records and their provenance. The checking service should reject missing required claims, unknown status values, and evidence references it cannot validate.

Successful processing produces a partial assessment with a clear blocker. In this example, the support review remains useful, the memory condition is contradicted, and isolation and workload testing remain unresolved. A generic “verification failed” would lose that distinction.

The reader-facing answer could be:

The captured component configuration has supporting documentation. The current memory plan falls 64 GiB short of the stated single-host-failure requirement. Access isolation and workload behavior still require testing. Revise the capacity plan, then complete the outstanding validation before submitting the deployment for approval.

A more defensible recommendation does not have to be longer than a confident one. It has to expose the condition that changes the decision.

Keep Evidence Fresh Without Rechecking Everything

Assign validity rules by evidence type. A version-specific support assessment needs review when the configuration or applicable guidance changes. Capacity evidence needs a much shorter operational window and a separate allocation mechanism. A failure test needs reassessment when a relevant dependency, placement rule, or workload condition changes.

Preserve historical results. An assessment that was supported yesterday should remain part of the record, even when its evidence is no longer eligible to support today’s decision. Record that loss of current validity and reassess the dependent claim rather than silently rewriting the past result.

Use a combination of expiry and change-triggered review. Events provide a prompt to reassess, while periodic refresh helps cover missed notifications. Recheck material environment conditions at the later deployment boundary, because a valid observation is still not a reservation.

Give evidence collectors scoped read access and explicit query limits. Keep sensitive raw artifacts separate from the requester-facing summary. A timeout or permission failure should produce an unresolved finding with a cause, not a fabricated result or a reason to grant the assistant unrestricted access.

Measure Useful Uncertainty, Not Just Answer Completion

Evaluate whether the assistant identifies which claims it can support and explains what would resolve the others. Track unsupported positive recommendations, unnecessary holds, and time to a useful assessed answer separately. Preserve the case counts and denominators so a change in workload mix does not hide a regression.

Assign ownership to the evidence, not merely to the chatbot. The platform team owns support and inventory interpretation. The workload owner defines demand and acceptance criteria. The security owner defines the isolation boundary. The application owner ensures that requirements and evidence states cannot be bypassed by the generated response.

Start with one consequential decision and its material claims. A low-risk explanatory answer does not need the same assessment depth as a production deployment recommendation. Add checks when they can change the decision, expose a meaningful limitation, or support accountable review, not simply to make the record look more complete.

Conclusion

AI output verification becomes useful when it stops asking whether an entire answer is trustworthy and starts asking what supports each material claim. Scope, source applicability, current state, acceptance criteria, and unresolved findings belong in the decision record.

For the two-cluster proposal, the evidence supports a partial answer and exposes a specific capacity problem. That is progress. The assistant has helped identify the next engineering task without disguising missing evidence or a failed requirement as readiness.

Begin with the recommendation your team is most likely to act on. Break it into claims, assign the evidence and owner for each, and make mandatory gaps visible before the proposal reaches a change workflow.

Confidence can help guide attention. Evidence determines what the recommendation can legitimately claim.

The next installment, AI Agent Authorization: Approval Is Not a Blank Check, addresses the separate question of who may act once the evidence is sufficient.

External References

• NIST: Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile
• OWASP Gen AI Security Project: LLM01:2025 Prompt Injection
• Association for Computational Linguistics: Enabling Large Language Models to Generate Text with Citations
• Anthropic: Language models (mostly) know what they know

The post AI Output Verification: Confidence Is Not Evidence appeared first on Digital Thought Disruption.