Running the Recursive Trust Benchmark: Your First Reviewer Pilot

TL;DR

Start a Recursive Trust Benchmark pilot by proving the measurement path before comparing reviewers. Validate the source cases, keep the answer key outside the candidate environment, freeze the trial assignments, and retain responses without silently repairing them. Account for every planned trial, including invalid and missing results.

The original sixty-case starter supports decision-case development, not production execution testing. This companion adds an offline preparation and response-accounting utility without changing the original cases or their reference decisions. The local helper tests use synthetic responses; no model or infrastructure comparison is reported here.

A reviewer score is useful only when the inputs, missing results, and limits of that score remain visible.

Introduction

Imagine the first pilot produces an impressive result: every response included in the report agrees with its reference decision.

Then someone asks how many trials were planned. Eight never produced a usable response. The scoring process simply omitted them. Another response contained contradictory decision fields, and the parser retained the last one. Nobody checked whether the candidate could read the answer key from its mounted workspace.

The problem is no longer only the reviewer. The experiment cannot support its own conclusion.

The original ten-part series ended with The Recursive Trust Benchmark: Test AI Assurance. This practical companion takes the next step: operating its decision-case starter with explicit input isolation, response validation, and complete trial accounting.

The scope is deliberately bounded. The pilot assesses advisory decisions against authored reference labels. It does not grant permissions, execute infrastructure changes, or measure whether a production control stops a prohibited action. The general evaluation harness for AI agents remains the broader architecture companion.

Know What the Starter Actually Contains

The supplied version 0.1 archive contains sixty synthetic cases across ten domain groups. Each group has two permit, two deny, and two hold reference decisions. All are development material pending independent domain review.

Its six variants cover a permitted proposal, presentation-only variation, unauthorized parameters, revoked approval, unavailable approval evidence, and a changed resource revision. The policy is intentionally narrow: compare the complete proposal with the approved operation and the required current state.

Do not silently replace those rules with a more elaborate enterprise policy while retaining the same reference labels.

The original exporter produces reviewer-inputs.jsonl and grader-map.jsonl. JSON Lines stores one JSON record per line. The first file contains candidate-visible material; the second retains authoring metadata and expected decisions.

Neither a field named authoritative_records nor a checksum supplied beside a file authenticates a real authority. Those are fixture assumptions and integrity aids within this development exercise.

Define the Pilot’s Deliverable

By the end of this walkthrough, the operator should be able to validate the starter, prepare isolated requests, assign each trial, collect advisory responses through an approved client, and produce a report that preserves missing and invalid outcomes.

The accompanying Reviewer Pilot Companion v0.1 supplies three local operations: check, plan, and score. It does not include a provider adapter. The actual model-call step uses the organization’s approved client, with its request and response records retained separately.

Use a dedicated Python environment, a trusted grading workspace, and a candidate environment that cannot read private reference material. No production infrastructure credentials are needed. Local checks for this companion used Python 3.13.5 and jsonschema 4.26.0 on Linux; other environments require their own validation.

Treat the first run as a development pilot. NIST’s January 2026 announcement of its automated benchmarking draft separates objective definition, execution, and analysis. Completing the execution stage does not settle whether the experiment supports the intended production decision.

Prepare a Trusted Workspace

Extract the original starter and the companion into the same parent directory. Preserve the original archive rather than replacing it with a modified package.

working-directory/
  recursive_trust_benchmark_v0_1/
  rtb-reviewer-pilot-companion/

The following commands use a POSIX shell from the companion directory. Create the environment and install the pinned direct dependency through the approved package source:

cd rtb-reviewer-pilot-companion
python -m venv .venv
. .venv/bin/activate
python -m pip install -r requirements.txt
python -m pip freeze > installed-environment.txt

The requirement pins jsonschema, not the complete transitive dependency tree. Retain the installed environment and use the organization’s normal dependency-locking process before comparative runs. The installation step may access a package repository; the helper’s three operations make no network calls with the supplied, self-contained schemas.

Inspect code before executing it. The package checksum list can detect a mismatch against that list, but replacing both files and checksums defeats that comparison. Preserve the accepted package identity through a separately protected record.

Validate the Cases and Their Projection

Run the original tests and exporter without modifying their behavior:

python -m unittest discover -s ../recursive_trust_benchmark_v0_1/tests -v
python ../recursive_trust_benchmark_v0_1/tools/prepare.py --output ../rtb-prepared-01

Preparation refuses an existing output directory. Use a new directory instead of deleting or overwriting an earlier experiment to make a rerun succeed.

Next, validate the schemas, source records, and exported alignment with the companion:

python reviewer_pilot.py check --starter ../recursive_trust_benchmark_v0_1 --prepared ../rtb-prepared-01
RTB_STARTER=../recursive_trust_benchmark_v0_1 python -m unittest discover -s tests -v

The jsonschema documentation distinguishes checking a schema from validating instances against it. The helper performs both. It also checks that exported inputs and grader records correspond to the same source cases and preserve the original projection.

For the supplied archive, local execution confirmed sixty cases, sixty projected inputs, three schema documents, and a reference balance of twenty per decision. The original six structural tests and eighteen companion tests passed. These are file and helper checks, not model evaluations or independent approval of the reference decisions.

A schema failure should lead to a versioned correction or an investigated packaging problem. Do not remove an inconvenient requirement merely to obtain a green preparation result.

Keep the Candidate Away from the Answer Key

A directory named private is not an access control.

Keep the complete source archive, grader map, expected decisions, and grading code on the trusted side. Send the candidate only the shared review prompt and its assigned projected input. Do not mount the complete repository into an agent workspace.

The same separation used to run coding agents safely in CI/CD applies here: untrusted candidate execution must not inherit the grading environment’s credentials or filesystem access. CI/CD means continuous integration and continuous delivery.

The diagram shows a data-flow boundary, not isolation automatically implemented by the helper. The collector assigns responses to trials independently of what the model says about its identity.

Use a fresh conversation for each trial. Disable browsing, repository access, and tools for this protocol. Also account for application memory and retrieval outside the visible conversation.

Anthropic’s evaluation guidance notes that shared files, caches, and environment state can affect trial outcomes. A fresh chat alone does not demonstrate a clean environment.

The starter’s evaluation identifiers are reproducibly derived from case identifiers. They support correlation, not secrecy. They do not turn publicly exposed development cases into a hidden test set.

Freeze the Assignments Before Collecting Responses

Create a plan for one candidate configuration and one repetition:

python reviewer_pilot.py plan --starter ../recursive_trust_benchmark_v0_1 --prepared ../rtb-prepared-01 --output ../reviewer-a-run-01 --candidate reviewer-a --repetitions 1

This creates sixty trial assignments and makes no model calls. Change the candidate label and output directory for another configuration. More repetitions create more trial slots over the same cases, not more independent scenarios.

The run directory contains a frozen plan, candidate input files, private reference files, the response schema, and an initially empty response directory. Each trial has a separate identifier and collector-assigned filename. Repeated trials retain the original evaluation identifier so the result can be checked against the assigned input.

The helper’s manifest binds local inputs, prompt, schema, and plan by their recorded hashes. It does not verify which provider or model was actually called. Record the actual provider, requested and reported model identifiers, generation settings, client version, and message preparation in collector-controlled run records.

Retain the accepted manifest outside the candidate’s administrative reach. A manifest that the candidate can replace together with its files is not a protected baseline.

Keep Changes Out of an Active Run

Do not edit the prompt halfway through collection, repair a case in place, or switch the model behind the same experiment label.

An intended change creates a new configuration and run. A provider change that cannot be precisely identified is a limitation to record, not a reason to invent an immutable model revision.

The helper detects changes to its frozen files. It cannot prove that an external client sent those exact bytes. Preserve the outgoing messages and correlate them with the assigned trial.

Collect the Verdict Without Improving It

For each plan row, send the shared prompt and one projected input through the approved client. Preserve the returned final text as UTF-8 in the row’s assigned responses/<trial-id>.txt file.

That filename comes from the trusted plan, not from a model-generated path or its echoed evaluation identifier. Use exclusive writes so a later attempt cannot replace the earlier result.

Document how the client extracts final text from a provider response containing multiple content blocks or additional metadata. Preserve the full provider response separately where permitted. Manually copied responses can exercise the workflow, but the manual transfer and its limitations belong in the record.

The original response contract requires evaluation_id, decision, reason_codes, evidence_refs, and explanation. Its decisions remain permit, deny, and hold.

For the initial pilot, use one submitted candidate attempt per slot and no automatic answer-repair retry. Record transport failures separately. Do not strip code fences, replace a missing decision with hold, or ask another model to make the response parse without recording a different protocol.

A production application may legitimately include a repair stage. Evaluate that whole stage as another configuration, retaining the original output, added calls, latency, and final selection rule.

Missing Is Not a Decision

No response artifact means missing to this helper. The collector’s records must distinguish a trial that never started from a timeout, provider error, or capture failure.

An empty response file is invalid. A valid hold is an actual judgment that the available decision evidence does not justify proceeding.

All three conditions may cause a production workflow to pause. They are not the same experimental outcome. Converting them into one category would conceal whether the reviewer recognized uncertainty or failed to return a usable answer.

Validate Meaningful Bindings, Not Only JSON Syntax

Python’s json documentation states that its default decoder accepts repeated object names and keeps the last value. It also accepts nonfinite constants such as NaN unless configured otherwise.

For this pilot, the companion rejects those inputs rather than choosing a favorable interpretation. Consider this deliberately ambiguous decision object, which the pilot rejects:

{"decision": "deny", "decision": "permit"}

It must not become an accepted permit merely because a parser selected the second field.

After parsing, the helper applies the original response schema, checks the evaluation identifier against the assigned input, and requires evidence references to resolve within that input. Additional intake checks reject whitespace-only explanations and reason entries. These are explicit companion protocol choices, not silent changes to the original case labels.

A parsed decision is retained diagnostically even when a later binding check fails. A permit paired with a fabricated reference should remain visible as an attempted permit, while still being excluded from valid findings.

A Resolving Reference Can Still Support the Wrong Argument

The utility does not judge whether an explanation correctly interprets its cited record. A response can cite a genuine record identifier while relying on an untrusted note rather than the authoritative approval.

Inspect that distinction separately. The existing AI context governance pattern provides the production companion for preserving source authority and decision-critical facts.

Keep label agreement, response validity, and explanatory correctness separate. None alone establishes that the system can authorize or execute the action safely.

Account for Every Planned Trial

Stop collector writes and freeze the response directory before scoring. Use a new report directory:

python reviewer_pilot.py score --run ../reviewer-a-run-01 --output ../reviewer-a-report-01

The report includes one record per planned slot, a reference-by-decision matrix, invalid and missing counts, and snapshots of received raw responses. It refuses an existing report directory.

Unexpected response filenames, changed frozen inputs, unsupported file types, and oversized artifacts stop ingestion rather than being silently ignored. The helper’s response limit is one mebibyte per artifact. Preserve rejected artifacts separately and investigate the affected slots.

A useful first check requires no model at all: score a newly created plan before adding response files. The local synthetic check produced this subset of the report:

{
  "planned_trials": 60,
  "status_counts": {
    "valid": 0,
    "invalid": 0,
    "missing": 60
  },
  "agreement_over_planned_trials": 0.0,
  "agreement_over_valid_responses": null,
  "production_authorization": "not_assessed"
}

There is no conditional agreement score because there are no valid responses. The zero over planned slots means no valid label matches were observed; it is not an estimate of an untested model’s accuracy.

A zero process exit code means the report was written. It does not mean the reviewer passed.

Do not connect that exit code directly to production promotion. The report deliberately contains no deployment authorization decision.

Check the Reporter with Deliberately Bad Reviewers

Before assessing a model, verify that the reporting path exposes outputs with known limitations.

The companion tests construct constant permit, deny, and hold responses. Each is structurally valid and cites an existing fixture record. Their explanations explicitly identify them as synthetic controls, not model judgments.

These outcomes were checked locally as helper behavior. They do not compare AI models, providers, people, or infrastructure controls.

The tests also cover wrong identifiers, unknown evidence references, extra fields, duplicate JSON keys, empty responses, altered inputs, and report-overwrite attempts. Eighteen passing tests provide evidence about those cases, not exhaustive security validation of the utility.

Diagnose the Pilot Before Blaming the Model

Treat references as versioned authoring decisions too. If domain review identifies a bad expected answer, preserve the original report, correct the case through a controlled revision, and identify which comparisons need rerunning.

Do not use the candidate’s disagreement alone as proof that either the candidate or the label is wrong.

Turn the Pilot into a Defensible Comparison

Before publishing comparative claims, obtain independent domain review of the relevant policies and labels. Add distinct operational scenarios, not just additional wordings of the same rule. Keep related variants together when designing development and held-out sets.

Compare candidates on matched inputs and declared settings. The companion creates plans but does not schedule providers, interleave execution, or randomize order. Those choices belong in the runner protocol and execution record.

Report agreement over all planned trials alongside agreement conditional on valid responses. The second measure describes the subset that passed intake; it should not hide availability or formatting failures in the first.

Break down permits on deny-labelled cases and permits on hold-labelled cases. Retain invalid attempted permits, missing responses, and reviewer disagreements. Inspect the decisive evidence in material errors rather than presenting only a percentage.

This pilot does not measure task completion, prevented side effects, human readiness, or statistically established model independence. Those require the other benchmark tracks and appropriate observations. Even perfect agreement on all sixty cases would not qualify an executor that bypasses the policy entirely.

Make Findings Change the Next Engineering Step

Use the report to choose work, not to award the reviewer a universal trust label.

An input-alignment failure belongs to evaluation engineering. A malformed-response pattern may require a tested interface change. Misinterpretation of authoritative records belongs to the evaluation rubric and context path. An incorrect reference decision belongs to the domain owners.

When the decision-only pilot is credible, test the actual execution boundary independently. Submit prohibited operations through authorized test interfaces and observe whether the target remains protected. Keep that control-path result separate from the model’s verdict.

OWASP’s agent-security guidance recommends testing after material changes and reviewing attempts to weaken security tests alongside behavior changes. Give the benchmark’s cases, parser, response schema, and grading logic their own change review. The candidate deployment identity should not be able to relax the rules used to approve it.

The next permission granted to an agent should depend on evidence about that permission. A better reviewer may justify better advice before it justifies greater autonomy.

Conclusion

The first useful Recursive Trust Benchmark pilot does not need to produce a leaderboard. It needs to produce a record that another engineer can inspect and challenge.

Keep the original case semantics intact, separate candidate inputs from reference decisions, preserve the response actually received, and account for every planned trial. Treat syntax, label agreement, explanatory support, and execution safety as different claims.

The starter and companion make that initial workflow concrete. They do not resolve the remaining work of domain validation, provider integration, independent control testing, or production acceptance.

The control plane must not grade itself. Neither should the benchmark hide the trials that make its result inconvenient.

Prepare one run, test the reporter with constant and missing responses, then inspect the first real disagreement before calculating a headline score.

External References

• NIST: Towards Best Practices for Automated Benchmark Evaluations
• Anthropic: Demystifying evals for AI agents
• Python: json, JSON encoder and decoder
• jsonschema: Schema Validation
• OWASP: AI Agent Security Cheat Sheet

The post Running the Recursive Trust Benchmark: Your First Reviewer Pilot appeared first on Digital Thought Disruption.