AI Context Sensitivity: Same Evidence, Different Decisions

TL;DR

AI context sensitivity becomes an operational problem when irrelevant presentation changes alter a decision that should remain stable. But conflicting answers can also reflect different tasks, missing evidence, changed execution settings, or a grader that misreads a valid response. Define the decision before comparing answers. Preserve the evidence and policy, inspect what actually reached the model, and distinguish a known requirement violation from an evidence gap. Consistency matters, but consistent correctness is the objective.

Introduction

Consider a hypothetical architecture review. An AI assistant recommends holding a change. A second review recommends approval. Both responses are articulate, both reference the same platform, and both appear to explain their conclusions.

Those are different operational outcomes, not merely different writing styles. Yet the disagreement alone does not tell us what failed.

Did both reviews receive the same evidence? Were they applying the same release criteria? Did an earlier conversation influence one review? Did the evaluation compare actual decisions, or simply look for a particular phrase?

The foundational article, The Double-Slit Experiment and AI: Why Context Changes the Answer, established a bounded analogy: interpreting an observation requires understanding the conditions that produced it. This installment takes that lesson into enterprise engineering without treating conventional AI behavior as evidence of a quantum mechanism.

The question is not whether context should influence an answer. It should. The question is which changes should alter a decision, which should not, and how to locate the failure when the distinction breaks down.

The Decision Is the Unit of Comparison

Consider a deliberately narrow review requirement: two application nodes must occupy independent power failure domains.

In this hypothetical case, the authoritative records identify both node-1 and node-2 as belonging to power domain A. Both nodes report healthy, but the policy evaluates independence, not current health. Under that policy, the correct result is to hold approval for this requirement.

Now compare two requests:

Apply the supplied power-domain policy to these records.

Determine whether these records satisfy the supplied power-domain policy.

These requests are intended to ask the same question. Different sentence structure in the response is acceptable. A switch from hold to approve, with the same policy and facts, requires investigation.

By contrast, asking whether the application is currently responding is a different task. A system can be healthy now while failing the stated resilience requirement. Different answers to those questions would not establish a context-sensitivity defect.

Keep Violations and Evidence Gaps Separate

Suppose the domain for node-2 is unknown rather than A. That is not evidence of independence, but it is also not proof that both nodes share a domain.

For this exercise, hold means a known policy violation, while insufficient_evidence means the requirement cannot yet be assessed. Both prevent approval, but they lead to different work: remediate the placement or obtain the missing evidence.

The assistant remains advisory throughout this series. Its answer assesses one requirement, not complete production readiness, and it has no deployment authority. The domain records are assumed accurate for the exercise; differently named domains would not, by themselves, prove physical independence in a real environment.

Separate Four Reasons an Answer Can Change

Start with four common explanations before attributing a changed answer to model unreliability.

A grader is the rule, program, or reviewer that scores the response. It belongs in the investigation, not outside it.

Sclar and colleagues’ prompt-format study demonstrated substantial performance differences across meaning-preserving formatting changes in the models and few-shot settings they tested. Their work supports evaluating multiple plausible formats rather than treating a single format as definitive.

Hua and colleagues’ 2025 paper, Flaw or Artifact? Rethinking Prompt Sensitivity in Evaluating LLMs, adds an important qualification. In their experiments, heuristic evaluation methods, including rigid answer matching, accounted for much of the observed sensitivity. Using model-based judges substantially reduced performance variation across prompts.

These studies use different models, tasks, and methods. Neither provides a universal sensitivity rate for a current enterprise application. Together, they support a practical position: test for behavioral variation, but validate the measurement before declaring the model unreliable.

Separate Meaning From the Output Contract

A freeform answer of “do not approve” may mean the same thing as “hold.” A grader should not call it a wrong decision solely because the wording differs.

However, an application may explicitly require the JSON value hold. Returning freeform prose then violates the interface contract, even if the underlying recommendation is correct. Report the protocol failure separately from the decision error. Combining them into one score conceals what needs fixing.

Also check execution conditions. A changed model revision, tool configuration, or decoding setup makes the comparison a different experiment. The table is a diagnostic starting point, not an exhaustive list of possible causes.

The Same Data Is Not Necessarily the Same Context

A team may say two requests used the same knowledge base. That does not establish that the model received the same records.

Anthropic’s Effective context engineering for AI agents describes context management as broader than writing a prompt. It includes instructions, tools, external information, and message history. The relevant inspection point is the information actually supplied during the interaction, not everything the application could potentially retrieve.

For a retrieval-augmented generation application, I would inspect the following boundaries. Notice that the grader is downstream of the recorded answer; it does not change that answer unless feedback is explicitly introduced into the workflow.

Find Where the Decisive Fact Disappeared

Return to the power-domain case. Suppose an assembly step summarizes the records as “two healthy application nodes” and removes the domain values. The final reviewer no longer receives the fact that determines the policy outcome.

That is evidence loss, not a clean test of equivalent prompt wording. Anthropic’s discussion of context compaction explicitly warns that aggressive summarization can discard information whose importance becomes apparent later.

If the full domain records are present but the assistant still approves, investigate how it applied the requirement. If the assistant correctly holds approval but the grader marks the result wrong, investigate scoring. These failures can produce the same dashboard symptom while requiring different corrections.

Liu and colleagues’ Lost in the Middle found that the position of relevant information affected performance in the long-context tasks and models they studied. That supports testing evidence placement. It does not prove that every current model behaves identically or that placing important information first is a universal solution.

For diagnosis, preserve the exact assembled context alongside record identifiers and versions. A source collection name and model identifier are not enough to establish what the application asked the model to assess.

Evidence Order and Workflow Order Are Different Experiments

Reordering independent records can be a presentation test. Changing a multistep interaction is a workflow test.

Suppose one workflow asks the assistant to make a preliminary approval recommendation before receiving the complete evidence. Another presents the complete evidence before requesting any judgment. Even if both eventually contain the same source records, the final decisions occur under different conditions.

In the first workflow, the earlier recommendation remains in the conversation if the application carries it forward. The final review now includes a model-generated judgment as well as the underlying evidence. That judgment is not an additional infrastructure fact.

My proposed comparison would preserve the complete policy and evidence while testing a fresh review against a review that includes the earlier recommendation. Any difference would justify further investigation; one conflicting pair would not establish the cause or frequency of the behavior.

There is another boundary: some ordering carries meaning. Reversing independent timestamped records while preserving their timestamps may be useful. Reversing the steps of a recovery procedure changes the procedure.

Declare what each transformation preserves: the facts, their provenance, their chronology, and the requested decision. When those change, classify the run as a different scenario or workflow, not an equivalent prompt variant.

Request a Decision Record, Not a Performance of Certainty

A confident explanation can make an inconsistent decision harder to challenge. I would require a compact decision record that identifies the outcome, governing policy, supporting evidence, unmet requirement, and conditions for reassessment.

The following JSON illustrates an application output contract for the shared-domain case. The identifiers are synthetic, and the structure is not a vendor API format or an executable control.

{
  "decision": "hold",
  "requirement": "independent_power_domains",
  "policy_version": "power-domain-policy-v1",
  "evidence_snapshot": "review-001-v1",
  "evidence_ids": ["node-1", "node-2"],
  "basis": "Both records identify power domain A.",
  "reassessment_trigger": "Updated, verified domain placement"
}

Replace the policy and evidence identifiers with values that resolve to records your application actually retains. The outcome should mean only that the stated requirement passed, failed, or could not be assessed.

Then validate three separate properties: whether the response satisfies the schema, whether the cited evidence supports its factual statements, and whether the policy implies the decision. A valid JSON object can contain an incorrect conclusion. Real evidence identifiers can accompany an explanation that misrepresents those records.

Turpin and colleagues’ 2023 study of generated explanations demonstrated that models could rationalize answers influenced by biasing prompt features without acknowledging those influences. That finding does not make explanations useless. It means a plausible explanation is not sufficient evidence of a faithful account of internal computation.

Use the decision record as something an engineer can inspect independently, not as proof that the model reasoned correctly because it described a convincing process.

Define Stability Before Measuring It

An application should not be equally sensitive to every change.

For the power-domain review, equivalent wording should preserve the decision. Verified movement into independent domains should change the result for this requirement. Removing a necessary fact should produce an evidence-gap outcome, not an invented value.

Those expectations form a small behavioral contract:

The missing companion to stability is responsiveness. An assistant that always returns hold may be consistent, but it fails valid approval cases. An assistant that always approves may be equally consistent and substantially more dangerous in this workflow.

Include cases that should pass, cases that should fail, and cases where the evidence does not support a determination. Report incorrect approvals separately from unnecessary holds and unresolved evidence gaps. The remedies and operational consequences differ.

The architecture owner should define the requirement and expected outcomes. The application team should own context assembly and execution. The evaluation owner should validate the cases and graders. Changing the definition of success after observing the answers defeats the purpose of the test.

Diagnose Before Patching the Prompt

Before changing instructions, freeze one disputed case. Preserve the policy, authoritative records, assembled context, conversation history, model configuration, output, and grading result. Keep sensitive evidence in an appropriately controlled store rather than copying it indiscriminately into logs.

Compare the two runs at those boundaries. Did the necessary record enter retrieval? Did its decisive fields survive assembly? Did the assistant apply the right requirement? Did the grader recognize the actual decision?

Next, repeat the unchanged condition to establish whether outcomes already vary before any presentation change. Then vary one declared factor at a time. Otherwise, an existing run-to-run difference may be incorrectly attributed to the prompt revision.

Treat this as diagnosis, not certification. A repeated correct answer does not establish coverage of other architectures, and zero observed failures does not prove zero risk. Record the scenario, conditions, observed outcomes, and limitations together.

For a known policy violation, I would treat an incorrect approval as a release-blocking defect in this advisory workflow. The appropriate correction depends on where the failure occurred, not on which prompt happens to produce a reassuring answer next.

Conclusion

The useful question is not whether an AI can produce two different answers. It is whether the difference is justified by a changed task, changed evidence, or changed policy.

When none of those changed, inspect the assembled context, execution conditions, and grading process. That avoids repairing the wrong component and prevents articulate explanations from concealing incorrect decisions.

Reliable AI decision support must preserve the verdict when irrelevant presentation changes, revise it when material facts change, and identify when the available evidence is insufficient. Consistency without those distinctions is not enough.

The next article turns this model into a small Python test harness, using controlled cases, equivalent requests, evidence-order variations, and separate checks for decision correctness and response validity.

External References

• arXiv: Quantifying Language Models’ Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting
• ACL Anthology: Flaw or Artifact? Rethinking Prompt Sensitivity in Evaluating LLMs
• ACL Anthology: Lost in the Middle: How Language Models Use Long Contexts
• arXiv: Language Models Don’t Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting
• Anthropic: Effective context engineering for AI agents

The post AI Context Sensitivity: Same Evidence, Different Decisions appeared first on Digital Thought Disruption.