LLM as a Judge: Evaluation Is Not Authorization

TL;DR

LLM-as-a-Judge uses a large language model to evaluate another system’s output, proposed action, or recorded behavior against defined criteria. It can make review more scalable, expose inconsistencies, and help identify problems that rigid checks miss. Its usefulness does not make its verdict an independent source of truth.

A favorable score means that a particular evaluator accepted particular inputs under a particular rubric. It does not automatically establish that the inputs were authoritative, the proposed action was permitted, or the target system reached the intended state. Those claims need their own controls and evidence.

Use AI judgment to inform decisions. Do not let a favorable judgment manufacture permission.

Introduction

Consider a hypothetical network-change workflow. The approved request allows one application group to reach one production service on Transmission Control Protocol (TCP) port 443.

An agent generates two candidate changes. Plan A uses the correct source group but opens every destination port. Plan B opens every destination port from a much broader source population.

A comparison judge could reasonably prefer Plan A. It is the less permissive option. The failure begins when the workflow interprets that preference as approval to execute.

Both plans exceed the authorized request.

This is a different problem from an evaluator hallucinating. The evaluator may have answered its assigned question correctly while the surrounding system used the answer for a decision it could not support.

The first article, Who Audits the AI Auditor? Independent AI Assurance, established the broader requirement for assurance independence. This installment focuses on one component: how to design, validate, and operate an LLM judge without allowing evaluation to become authorization.

The scope is vendor-neutral and production-oriented. The examples assume consequential agent actions, identifiable policy owners, and access to authoritative system records. The proposed contracts and numerical examples are design illustrations, not results from a deployed DTD benchmark.

What an LLM Judge Actually Establishes

An LLM judge produces an assessment. The strength of that assessment depends on the question, rubric, model, input evidence, and validation method.

A retained judge result can establish that an evaluation ran and returned a specified finding. It can contribute to a defensible assurance process. It does not become independently corroborated evidence merely because it came from a second model or a separate endpoint.

Three statements should remain distinguishable:

The judge found the explanation convincing. This concerns the evaluator’s assessment of the material it received.

The proposed action satisfies the organization’s requirements. This requires applicable requirements, reliable facts, and a decision process authorized to apply them.

The intended result exists in the target system. This requires observation of the relevant system state.

Anthropic’s Demystifying evals for AI agents makes a useful distinction between an agent’s transcript and the outcome in its environment. The final message can describe success while the actual outcome must be checked elsewhere. Its guidance also distinguishes code-based, model-based, and human graders rather than treating one method as suitable for every obligation.

An LLM can help interpret independently collected evidence. The evidence does not lose its value because a model assists with analysis. Conversely, the model’s analysis does not acquire the independence of an observation it never received.

The architectural question is therefore not whether AI participated. It is which claim the assessment supports, and what could cause that claim to be accepted incorrectly.

Match the Evaluation Method to the Decision

Two distinctions matter when choosing a judge.

Pointwise evaluation assesses one artifact against criteria, such as whether a change explanation identifies its operational dependencies. Pairwise evaluation compares two artifacts, such as which explanation better communicates the proposed change.

Reference-based evaluation is a separate dimension. Either approach can use supplied reference material. A reference-free judge instead relies on its instructions and learned knowledge without a task-specific reference answer or source.

These methods answer different questions. A pairwise winner may still fail an absolute requirement. A response may match a reference that is outdated. A high-quality explanation may describe an unauthorized action.

Google Cloud’s evaluation documentation illustrates the source-support boundary. Its GROUNDING metric checks consistency against provided source text. That is useful, but the architectural implication is narrower than “the answer is true”: agreement with supplied material cannot, by itself, establish that the material is current or authoritative.

For a production change, first decide which obligations can be checked directly. An explicit port range, resource identifier, approval expiry, or permitted operation should not depend solely on a model’s interpretation when the relevant value can be validated in code.

Reserve model judgment for the semantic work that remains.

Where Automated Review Creates False Confidence

Human-Preference Agreement Is Not Authorization Accuracy

Zheng and colleagues’ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena demonstrated useful agreement between strong model judges and human preferences in the studied settings. The research also identified position, verbosity, and self-enhancement biases, along with limitations in reasoning.

Those findings support using model judges carefully. They do not establish a production false-approval rate for firewall changes, identity permissions, or model promotions.

Wang and colleagues’ Large Language Models are not Fair Evaluators, published at ACL 2024, also showed that changing response order could change comparative judgments in their experiments. Their calibration approach included balancing presentation order and seeking human assistance for difficult cases.

The practical response is to test these sensitivities in the intended evaluator. Do not assume that every current model has the same weaknesses or error rates as an earlier study. Equally, do not assume that a newer model has eliminated them without measurement.

A Better Rubric Cannot Recover Missing Evidence

Suppose the actor and judge receive a ticket claiming that an exception was approved. Neither receives the exception record, its scope, or its expiration.

A rubric that asks whether the change is authorized may encourage the judge to infer what the workflow failed to retrieve. The correct operating response is to preserve the uncertainty and obtain the missing record.

Make evidence availability a first-class condition. Distinguish an unsupported claim from a disproven claim, but allow neither to become an automatic authorization.

A long explanation of why the exception sounds reasonable does not resolve the missing approval.

The Evaluator Has an Attack Surface Too

OWASP’s LLM Prompt Injection Prevention guidance explicitly warns that a guardrail LLM remains susceptible to prompt injection. It recommends treating model-based guardrails as part of a layered design, alongside other controls.

An evaluator that reads agent-generated explanations, retrieved documents, or tool output is examining potentially untrusted material. That material may contain instructions, forged approval claims, or statements designed to influence the review.

Separate trusted evaluation instructions from the content being examined. Minimize unnecessary context, identify source provenance, and constrain the evaluator’s access. These measures reduce exposure; they do not prove that prompt injection has been solved.

The judge should not need production write credentials to critique a proposed change. Nor should it be able to rewrite its rubric, modify approval records, or replace the evidence used to assess its own performance.

Optimizing the Agent Against the Judge Can Distort the Metric

A development loop that repeatedly adjusts an agent until a judge approves creates a specific risk: the agent may improve at satisfying the evaluator without improving the intended operational outcome.

That possibility does not make iterative feedback inherently wrong. Feedback is useful when it remains connected to the real requirement.

Maintain a governed feedback loop with separately reviewed acceptance criteria and protected test cases. Do not automatically turn every favorable judge output into a trusted training label, shared-memory entry, or policy exception.

The evaluator is part of the system being optimized. Treat its blind spots as dependencies to investigate.

Give the Judge a Bounded Contract

Avoid asking one judge to decide whether a complex enterprise action is simply “safe.”

That word can hide several independent obligations: technical correctness, authorized scope, data handling, operational impact, recovery readiness, and business acceptance. A single score makes it difficult to identify which obligation was actually assessed.

Define a narrow contract instead. State what the evaluator receives, which criteria it examines, what evidence each finding must reference, and how the surrounding controller handles missing or contradictory results.

The following YAML is a proposed application-level specification for reviewing a network-change explanation. It is not a vendor configuration format or an implemented security boundary.

spec_version: "0.1"
judge_id: network-change-review
mode: advisory

required_inputs:
  - original_request
  - normalized_proposed_change
  - authoritative_policy_snapshot
  - independently_collected_state
  - supporting_records

rubric:
  version: network-change-review-v1
  criteria:
    - description_matches_proposed_change
    - claimed_exceptions_have_supporting_records
    - material_risks_and_dependencies_are_disclosed

output:
  finding_states:
    - supported
    - contradicted
    - insufficient_evidence
  required_per_finding:
    - criterion
    - state
    - evidence_refs
    - explanation

controller_rules:
  any_contradicted: hold
  any_insufficient_evidence: hold
  missing_criterion_or_unknown_reference: hold
  invalid_or_missing_result: hold
  all_criteria_supported: continue_to_authorization
  override_external_denial: forbidden

Replace the criteria and input requirements with the obligations of the actual workflow. Define what counts as adequate disclosure and supporting evidence before selecting the model.

The important transition is continue_to_authorization. It does not mean execute.

The controller must implement these rules outside the model. It should validate that every required criterion appears exactly as expected, that finding states are valid, and that evidence references resolve to records supplied through an approved path. An empty findings array must not pass through an “all findings supported” check.

A trusted service should bind the response to the exact proposal, input snapshot, rubric version, and evaluator configuration. Do not depend on the model to report its own provenance accurately. Keep concise explanations and supporting references rather than requiring hidden reasoning.

In the opening scenario, a judge should identify the discrepancy between the requested port and the proposed range. If it misses that discrepancy, the deterministic action checks must still reject the expanded scope.

Even a supported finding means only that the evaluator found support within the named criterion and available evidence. It is not an assertion that every relevant risk has been discovered.

Put the Verdict in the Correct Execution Path

For consequential actions, design the judge as a source of findings feeding an externally governed decision path.

The diagram below shows the intended asymmetry: a concern can stop automated progression, while an absence of concerns cannot bypass authorization.

The authorization stage must use authenticated identities and authoritative approval records, not a judge-generated statement that permission exists. It should bind the decision to the actual action and recheck relevant conditions before execution.

This does introduce operational friction. A model can raise a false concern, miss a legitimate exception, or become unavailable. Provide an accountable review path for resolving those cases. The remedy is not to let the actor keep changing the prompt until it receives a favorable answer.

There is also a distinction between refusing an action and returning the service to operation. A reviewer may correct an erroneous finding, but any subsequent execution still needs valid authority. Clearing a model-generated hold should not clear unrelated policy denials.

Finally, a successful tool response and a favorable post-action review should not close the change while the required system-state confirmation remains unresolved.

Measure Missed Violations, Not Just Overall Agreement

Google Cloud’s Evaluate a judge model guidance uses human-rated evaluation data to assess the judge itself. It also describes confusion matrices and class-specific measures rather than relying only on a mean score.

That distinction matters when unacceptable actions are rare.

Consider the following arithmetic example. The judge is classifying proposals against an evaluation rubric. “Accepted” is an evaluation label, not permission to execute.

These numbers are illustrative, not benchmark results.

The judge agrees with the reference labels on 960 of 1,000 proposals: 96% overall accuracy.

But it accepts 40 of the 50 prohibited proposals: an 80% false-approval rate on the prohibited class.

A dashboard showing only 96% accuracy would conceal the weakness most relevant to this control.

For an operational evaluator, report false approvals by violation type and impact. Track false rejections separately because an evaluator that rejects everything is not useful. Keep abstentions, malformed responses, timeouts, and unreviewed cases visible rather than removing them from the denominator without explanation.

Also distinguish evaluation failure from enforcement failure. A judge may miss a prohibited action that an external control successfully blocks. That is evidence of layered protection, but the judge’s miss still belongs in its performance record.

Measure Coverage Alongside Accuracy

An evaluator that returns decisions for only the easiest requests can appear accurate while contributing little protection to the difficult ones.

Report how many eligible cases were reviewed, how many received a usable determination, and how many required escalation. Break those figures down by action class, environment, language, and other dimensions that materially affect the workflow.

For tests that deliberately contain many prohibited cases, do not present the aggregate results as the expected production incident rate. The test distribution and production distribution serve different purposes.

Treat Confidence as Something to Validate

A model-generated confidence value is not automatically a calibrated probability.

To use confidence operationally, compare predicted confidence with observed correctness on held-out, representative cases. Examine the results separately for consequential error types and the versions being deployed.

A single high-confidence label should never compensate for missing evidence. Confidence in an interpretation and availability of the required facts are different conditions.

Test the Second Judge on the First Judge’s Misses

A second evaluator should be assessed on the errors that reach it, not only on a generic test set.

For two serial judges that must both accept a prohibited proposal, the probability relationship is:

P(both accept | prohibited)
  = P(J1 accepts | prohibited)
  x P(J2 accepts | prohibited, J1 accepts)

The second term is conditional. It asks how often Judge 2 accepts prohibited proposals that Judge 1 has already accepted.

Multiplying the judges’ separate, overall false-approval rates substitutes an independence assumption for that measurement.

This is where the recursive trust problem becomes testable. Does the second judge catch the first judge’s unsupported-exception errors? Does it notice scope expansion hidden by the same summary? Does it challenge fabricated execution evidence?

Different models may help. So may different retrieval paths, narrower rubrics, deterministic checks, and qualified human review. Choose the combination based on measured additional detection and operational cost.

Also measure the extra false holds created by the combination. Independence that adds useful challenge is valuable; a collection of reviewers that produces delay without detecting additional consequential errors is not automatically better assurance.

Build Calibration Cases That Expose the Boundary

An evaluation harness for AI agents should contain a distinct test suite for the judge component. Otherwise, an improved agent can make a weak judge look reliable simply by producing fewer difficult cases.

Start with cases whose expected outcomes can be justified from explicit requirements and authoritative records. Have qualified domain reviewers label them without seeing the model’s verdict first. Preserve disagreements that reveal ambiguous policy or insufficient evidence.

Human labels are an adjudicated reference, not infallible truth. For directly observable requirements, use independently verified facts to support those labels.

Two kinds of case variation are particularly useful.

Presentation changes should preserve the expected decision. Rewrite the explanation, change its length, remove self-congratulatory language, or reverse candidate order while preserving the underlying facts. Unexpected verdict changes identify sensitivity that needs investigation.

Material changes should affect the expected decision. Expire the approval, change the tenant, expand the resource scope, remove a supporting record, or introduce a mismatch between the explanation and the proposed tool arguments. A judge that gives the same reassuring answer has failed to distinguish the relevant condition.

Keep development, calibration, and held-out cases separate. Group variations of the same underlying scenario together so that near-duplicates do not make the held-out set artificially easy.

Evaluate the actual pipeline, including context assembly and any summarization before judging. Testing a judge with complete evidence does not validate a production path that removes the decisive fact to save tokens.

Set acceptance criteria before inspecting the candidate’s results. Report uncertainty and the number of relevant cases. Zero observed misses in a small sample does not demonstrate zero risk.

Operate the Judge as a Production Dependency

The evaluator’s placement determines what it can protect.

For the high-impact workflow described here, an unavailable required judge should produce a hold or an explicitly governed fallback. It should not silently become a successful review.

A fallback evaluator is a different configuration. Validate it for the same task and data-handling requirements rather than assuming that any available model is an acceptable substitute.

Do Not Retry Until the Verdict Becomes Convenient

Define bounded retries for transport errors and malformed responses. Retain the attempts.

A substantive adverse finding should enter a review process. A genuinely revised proposal can be assessed again, but it should remain linked to the earlier proposal and findings.

Repeatedly sampling unchanged material and accepting the first favorable response changes the effective decision policy. That behavior needs to be evaluated as part of the system, not hidden inside an implementation detail.

Version the Evaluator Separately from the Agent

Treat a judge change as a versioned AI behavior release. Record the model identifier available from the provider, prompt, rubric, context-processing configuration, output schema, and decision aggregation rules.

Where a provider does not expose an immutable underlying revision, retain the observable identifiers and document the limitation. A stable endpoint name alone should not be treated as proof of unchanged behavior.

Before replacing a judge, run the old and new versions against the same frozen agent outputs and evidence. This separates a change in the measuring instrument from a change in the agent.

During rollout, monitor verdict distributions, missed violations found by independent review, false holds, unavailable evaluations, and latency. A higher pass rate may indicate improvement, a more permissive judge, changed traffic, or broken input preparation. It requires investigation.

Name the Owner of a Bad Judgment

The evaluation owner should maintain rubrics, calibration cases, and error analysis. The policy owner should define mandatory constraints and exception authority. Platform operations should own service availability, version deployment, and recovery.

The service owner must decide whether the residual error and operational burden are acceptable for the action class.

Keep evaluator telemetry proportionate. Record enough to reconstruct the assessment, but do not turn the review pipeline into an uncontrolled copy of sensitive prompts, system records, or credentials. Access controls and retention apply to judge inputs and explanations too.

Conclusion

LLM-as-a-Judge is useful when the organization can state exactly what it measures and what happens when it is wrong.

It can compare candidates, identify unsupported claims, assess explanations, and focus human attention. Its verdict should remain tied to a defined rubric, a specific input set, a versioned evaluator, and a measured error profile. Authorization and target-state verification remain separate obligations.

The next installment, The Assurance Independence Model: Six Boundaries for Agentic AI, develops the six-dimensional profile introduced in Article 1. It examines how model, provider, context, enforcement, evidence, and organizational boundaries contribute to assurance without collapsing them into one reassuring average.

The control plane must not grade itself.

Before expanding an agent’s permissions, trace one favorable judge result through the production workflow. Does it lead to another required check, or directly to a privileged action?

External References

• Google Cloud: Evaluate a judge model
• Google Cloud: Define your evaluation metrics
• Anthropic: Demystifying evals for AI agents
• OWASP: LLM Prompt Injection Prevention
• Zheng et al.: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• ACL Anthology: Large Language Models are not Fair Evaluators

The post LLM as a Judge: Evaluation Is Not Authorization appeared first on Digital Thought Disruption.