AI-Assisted Decisions: When Repetition Becomes False Confidence

TL;DR

Repetition can make an explanation familiar without making it correct. In AI-assisted operations, a tentative diagnosis can spread through tickets, runbooks, and generated summaries until it looks like established knowledge. The answer is not to abandon experience or distrust every AI recommendation. Separate observations from interpretations, preserve the origin of supporting evidence, and rehearse the conditions that should change a decision. A useful operating habit makes the right action easier while keeping exceptions visible. Confidence should follow evidence, not the number of times an explanation appears.

Introduction

Consider an illustrative incident. An order-processing service slows down. An engineer restarts an instance, response times recover, and the ticket records a suspected memory leak. Several later tickets reuse that explanation, and the restart becomes a familiar recovery procedure.

An AI assistant searches the incident history and recommends the same action. Its explanation is coherent, cites several internal records, and matches the team’s experience. The recommendation feels well supported.

But the original investigation never established the cause. The team observed recovery after a restart. It did not demonstrate why the service degraded or which change during recovery resolved the symptom. Several documents now support the same narrative, but their diagnostic claims trace back to one unverified assumption.

The foundation article, Entropy in AI and Human Thinking: Why Certainty Is Not Accuracy, distinguished physical entropy from uncertainty and correctness. This first follow-on addresses the human operating problem: how do we develop reliable habits without training ourselves to trust familiar explanations automatically?

The focus is human judgment in AI-assisted operations. Assume the assistant provides advice, engineers can inspect the underlying evidence, and production actions remain subject to established controls. The incident and review workflow below are proposed engineering examples, not reported results from the cited studies.

Familiarity Is Not an Independent Source

Fazio and colleagues’ 2015 study, Knowledge does not protect against illusory truth, found that repetition increased perceived truth even when participants possessed relevant knowledge. The authors describe situations in which people relied on processing fluency, how readily a statement could be processed, rather than available knowledge.

These were controlled experiments about truth judgments, not studies of enterprise incident management. They do not establish that experienced engineers inevitably ignore evidence. They do challenge the assumption that possessing relevant knowledge automatically protects every judgment.

In our incident scenario, a separate information-quality problem compounds that risk. A copied diagnosis is not an independent confirmation of the diagnosis. A generated summary of that diagnosis is not a new observation either.

The distinction becomes clearer when we separate document lineage from evidence:

The diagram does not imply that later incidents contain no useful information. They may provide additional observations about symptoms and recovery. The problem is treating a repeated causal claim as though each document independently established it.

For a consequential recommendation, inspect where the claim originated, what evidence was added later, and whether the evidence applies to the current environment. A recently edited runbook is not necessarily a recently validated runbook.

Count independent support for the claim, not appearances of the claim.

Separate Recovery From Root Cause

Two questions should remain distinct: what action is justified to restore service, and what evidence explains the failure?

Google’s Effective Troubleshooting chapter presents troubleshooting as an iterative process of forming hypotheses and testing them against observations. It also prioritizes mitigating a major outage rather than delaying recovery until the root cause is established.

That distinction is important here. An approved restart may be the correct recovery action without proving the memory-leak hypothesis. The operational decision can be justified while the causal explanation remains open.

Ask What Would Distinguish the Explanations

For this illustrative service, compare plausible explanations against evidence that could help separate them:

These are investigation directions, not universal diagnostic signatures. More than one mechanism can be involved, and a resource limit can be a downstream symptom rather than the initiating fault.

A useful test should produce different expectations for competing explanations. Collecting more screenshots of the same high-latency chart adds less value than obtaining the observation needed to distinguish them.

Preserve relevant evidence when doing so is safe and proportionate. Do not turn a demand for perfect diagnosis into a reason to delay urgent recovery. Where evidence cannot be preserved, record the gap instead of filling it with a confident explanation.

The incident can close with a precise statement: service restored using the approved procedure; root cause not yet confirmed. That is more useful than a definitive label the investigation did not earn.

Rehearse the Decision, Not Only the Procedure

Mental rehearsal can contribute to performance, but the evidence has boundaries. Bernardi and colleagues studied skilled pianists and found performance improvements after mental practice, with greater overall improvements after physical practice. Their work supports a specific motor-learning claim, not a complete training method for infrastructure engineers.

The engineering application proposed here is to rehearse both the procedure and its decision boundaries, then exercise them against a representative system or scenario. A remembered command sequence is only one part of operational readiness.

A weak walkthrough asks whether an engineer can explain the restart sequence. A stronger walkthrough asks which target can safely be removed from service, what must be checked first, what evidence should be retained, and what would make the restart inappropriate.

Practice the Exception as Deliberately as the Normal Path

Use one exercise where the recovery procedure fits: sufficient remaining capacity, understood dependencies, and satisfied approval requirements. Then repeat the scenario with one material condition changed, such as another instance already being unavailable.

The correct response should change when that condition changes. An engineer who recognizes that the familiar action is now unsafe has demonstrated something the normal-path exercise could not test.

Add a misleading-success variant as well. In the simulated timeline, latency recovers after the restart, but dependency telemetry also shows a coincident recovery. Ask what can reasonably be concluded. The expected answer should preserve uncertainty about causation rather than reward the most decisive-sounding diagnosis.

This is how rehearsal becomes more than repetition: the exercise requires prediction, observation, and revision. It prepares the engineer to recognize when a learned pattern applies and when it must be reconsidered.

Put an Independent Judgment Before the AI Recommendation

Buçinca, Malaya, and Gajos studied interventions designed to make people engage more deliberately with AI advice. In their 2021 experiment, cognitive forcing reduced overreliance relative to the simple explainable-AI approaches tested. The designs that reduced overreliance most also received the least favorable subjective ratings.

That tradeoff matters. Adding review friction may improve a particular decision process while making it less pleasant to use. The finding does not justify slowing every interaction or assuming that any mandatory extra step improves judgment.

For training and planned reviews, a practical adaptation is to capture a short initial assessment before revealing the AI recommendation. Record the leading explanation, a plausible alternative, the evidence supporting each, and the next observation that would help distinguish them.

Then expose the AI recommendation and compare claims, not confidence of presentation.

The separation is about the initial assessment, not statistical independence. Human and AI may still share the same incomplete evidence or inherited assumptions. Agreement should therefore prompt the question, “What supports our shared conclusion?” rather than end the review.

Disagreement deserves the same treatment. A reviewer should not reject an AI recommendation simply to demonstrate independence. The useful behavior is changing position when stronger evidence warrants it.

Reserve this fuller process for exercises, planned changes, and decisions where the consequences justify the effort. During an active incident, preserve the approved emergency-response path rather than requiring a lengthy assessment before time-critical mitigation.

Build a Decision Record That Preserves Uncertainty

A decision record should make it possible to reconstruct what was known at the time. Do not rewrite the initial assessment after the outcome becomes clear. Add the correction as a subsequent entry so the learning remains visible.

For the incident scenario, use a compact record such as this:

Replace the generic fields with the actual service, evidence identifiers, validation criteria, and owner. Define what recovery means before judging the result. In an order-processing service, that might include successful completion of test transactions and acceptable queue behavior, not merely a responsive health endpoint.

Keep recovery confidence separate from diagnostic confidence. Evidence may justify confidence that the service recovered while providing little support for a specific causal explanation.

The record also needs an operational owner. Assign diagnosis to the relevant engineer, recovery validation to the incident owner, and promotion of conclusions into approved guidance to the knowledge owner. One person may hold several roles, but none should disappear into a generic instruction to “review later.”

Test Whether the Habit Survives Changed Conditions

A pilot should test behavior, not simply ask whether participants liked the workflow or felt more confident afterward.

Build a small exercise set from de-identified incidents. Preserve only the information available at the original decision point, and keep the eventual outcome separate until the assessment is complete. Include familiar cases, changed-precondition cases, and cases where the evidence does not support a definitive diagnosis.

Have domain reviewers establish acceptable actions and escalation paths before examining the assistant’s answers. Some cases should permit several defensible next steps. Do not force an artificial single-answer key onto a genuinely uncertain situation.

Compare the existing workflow with the proposed review pattern using comparable cases and balanced assignment. Reusing the exact same case immediately can turn the exercise into a memory test, so use matched variants and account for order effects.

Measure whether engineers identify unsafe recommendations, recognize copied claims, choose useful checks, and revise their conclusions appropriately. Also track decision time, unnecessary escalation, and rejection of correct advice. Reducing acceptance of bad recommendations is not enough if the process also makes the team reject useful assistance indiscriminately.

Finally, test a later, changed scenario without the assistant. Treat this as an assessment of retained decision capability, not a requirement to operate production without approved tools. The aim is to see whether the engineer can explain and apply the decision boundary rather than merely repeat a previously displayed answer.

These are proposed evaluation criteria. They do not imply a measured improvement or guarantee that the workflow will benefit every team.

Make Correction Easier Than Defending the First Answer

Google’s Postmortem Culture: Learning from Failure emphasizes examining contributing conditions without blaming individuals and turning the review into concrete preventive actions. That is an appropriate operating principle for AI-assisted decisions as well.

When a recommendation was accepted incorrectly, investigate what made it appear reasonable. Was the original evidence unavailable? Had a tentative diagnosis lost its qualification? Did the reviewer have enough time and authority to challenge the action?

A human approval step is not a substitute for giving the reviewer the information and authority needed to make a meaningful decision. Correct the missing access, misleading record, or ambiguous responsibility, rather than ending the review with “be more careful.”

When stronger evidence changes the conclusion, update the affected guidance and add a regression exercise that tests the same mistake. Preserve the history, but distinguish superseded claims from current guidance.

Reward the engineer who changes a position for a good reason. The objective is not to defend the earliest explanation. It is to improve the next decision.

Conclusion

Repetition is valuable when it develops a capability that remains responsive to evidence. It becomes dangerous when a familiar explanation no longer has to justify itself.

For AI-assisted decisions, start with one recurring incident or change procedure. Trace the supporting claims to their original evidence, distinguish recovery from diagnosis, and rehearse the exception that should stop the familiar action. Capture what was known, what remained uncertain, and what would justify a different decision.

The goal is neither constant skepticism nor automatic trust. It is an operating habit that uses experience efficiently while remaining willing to revise it.

A well-trained team does not just recognize the usual answer. It recognizes when the usual answer no longer fits.

The next article, AI Uncertainty: Why Confidence Scores Are Not Enough, examines the model side of that problem: what uncertainty signals can tell us, what they cannot establish, and when an assistant should abstain.

External References

• Scholars@Duke: Knowledge does not protect against illusory truth.
• Frontiers in Human Neuroscience: Mental practice promotes motor anticipation: evidence from skilled music performance
• arXiv: To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making
• Google SRE: Effective Troubleshooting
• Google SRE: Postmortem Culture: Learning from Failure

The post AI-Assisted Decisions: When Repetition Becomes False Confidence appeared first on Digital Thought Disruption.