The intent fidelity judge (BYO-Judge)

Claim Check’s deterministic floor only counts what it can bind to a mechanical check: a test that passes or fails, a symbol that is or isn’t in the diff, a command’s exit code. Some acceptance criteria are judgment calls a script can’t make on its own, whether a change actually implements the retry policy a ticket described, not just something shaped like it. For those, and only those, Shipmoor asks your own coding agent to adjudicate. This is the intent fidelity judge, BYO-Judge for short.

Shipmoor builds, hosts, and calls no model of its own, and opens no network path of its own. The model call rides your agent, under your own provider relationship, driven as a subprocess on your own machine.

Here is the invariant this page exists to explain: the judge can BLOCK, by citing a divergence, or it can ABSTAIN, by routing the item to a human. It can never PASS. A judgment obligation that abstains becomes INCONCLUSIVE, never a silent pass. A READY verdict always rests on the deterministic floor. It is never the judge’s word alone, no matter how many samples agreed. Your own toolchain stays the oracle throughout: the judge doesn’t introduce a second one, it binds a judgment call to your agent’s opinion, runs it under the same discipline as any other check, and records what happened. See Claim Check overview for the full binder-not-oracle framing.

When it runs

All of the following have to hold, or the acceptance criterion stays open (cannot check) and the judge never touches it:

• Entitled. Full mode needs the claim_check entitlement. Without it, a configured --agent degrades to floor-only with a warning on stderr, never a silent full run. See Plans & tiers.
• An agent is configured. --agent "<preset-or-command>" on the command line, or review.agent in .shipmoor.yaml. The judge reuses the same config key as the advisory review feature; there’s no separate claim_check.agent key.
• The floor left the obligation open. The judge only ever adjudicates the judgment residue, the acceptance criteria the deterministic floor could not bind to a mechanical check. Anything the floor already decided never reaches the judge.
• The resolved intent is confident enough. Medium or high confidence only; the judge doesn’t run against a low-confidence one-liner. See Providing intent.

With no agent configured at all, every judgment obligation abstains by default (routed to a human, INCONCLUSIVE), so a full-mode run never silently behaves like floor-only without telling you.

K-sampling and the calibrated floor

Each judgment obligation the judge takes on is sampled K = 3 times, concurrently, through the same agent command. A BLOCK needs at least 2 of 3 samples agreeing on a divergence; a lone outlier sample can never block by itself. The fixture was recalibrated on 2026-07-10, down from five samples needing four to agree, to bound the cost of running an agent command repeatedly per obligation while the floor is still growing.

Agreement alone isn’t enough to block. Even when 2 of 3 samples agree on a divergence, the judge withholds the block unless the mean confidence those samples reported is above a calibrated threshold of 0.8; at or below that threshold, it downgrades to an abstain instead. In plain terms: if the judge isn’t confident enough across its own samples, it steps back rather than guess. A block sitting exactly at the threshold abstains, biased toward withholding a block rather than firing one it isn’t sure of.

Both numbers, the sample count and the confidence floor, live in a frozen, offline calibration fixture, content-addressed and re-verified on load. Nothing about them is inferred at run time or read from telemetry.

Two phases: decomposition then audit

The judge works in two strictly separated phases: extracting obligations never audits, and auditing never re-extracts.

Phase 1: decomposition

Decomposition breaks the acceptance criterion into atomic yes/no check questions. Any specific symbol the obligation names, a CLI flag, a function or method name, a class name, has to be carried verbatim into at least one atomic question, not paraphrased into something generic. An obligation like “the refund function is called on the payment_failed event” has to decompose into a question that cites refund and payment_failed by name. That verbatim anchor is what lets the audit phase come back and check the named symbol against the actual change, instead of adjudicating a vague reading of the claim.

Phase 2: audit

Audit compares each atomic question against a masked, observed-behavior signal built from the change: the changed-file list and the raw masked lines the change adds, per file. There’s no per-language symbol extraction here, a Go func, a Python def, or a --flag are all just text the agent matches against the added lines, so the check is language-agnostic by construction.

An absence branch lets the judge cite a divergence when a named symbol simply never shows up: if the obligation asks whether refund was added and it isn’t a substring of any added line in the relevant file, that absence is the divergence, evidenced by the file itself. But the judge can only conclude an absence when it can see the complete picture. If the added-lines view was trimmed or a file didn’t fit inside the signal’s budget, it’s disclosed as such, and the judge is instructed to withhold the divergence and let the item route to a human rather than assert an absence it can’t actually confirm.

Evidence pointers: confirmed by running them

A block is only as good as its evidence pointer, and Shipmoor mechanically confirms every one before it’s allowed to count.

A file pointer is confirmed by existence: the cited path has to actually exist in the reviewed head. Simple, unchanged.

A test or command pointer is confirmed more rigorously: Shipmoor executes it, hermetically, at the reviewed head, reusing the same tiered sandbox the deterministic floor’s own check runners use (an ephemeral git worktree, network off, resource-bounded). The block survives only if the cited failure genuinely reproduces: the named test fails, or the named command exits non-zero. A clean exit refutes the citation, and the block is discarded. A pointer that can’t be resolved at all, no sandbox available, an unresolvable command, a missing tool, is unconfirmed and discarded the same way, never silently kept.

This is a real strengthening, not a weakening. Before, a test or command citation only had to exist as text; now it has to actually reproduce the divergence it claims, or the block doesn’t survive. The untrusted, model-written command only ever runs inside an isolated worktree, never against your live working tree, so confirming a citation can never mutate what you’re reviewing.

Running it

shipmoor claim-check . --agent claude \
  --intent-prompt "As a user I want every Flask response to carry an X-Request-ID header; add a small helper that sets it"

On a change that does what the intent says, the judge agrees and the item clears. On a change that claims to but doesn’t, a report looks something like this:

Claim Check  BLOCKED  ·  exit 1  ·  policy v3

  ✗ Every Flask response carries an X-Request-ID header.
      requirement: stage3.byo_judge · required · result: blocked
      rationale: the helper never sets an X-Request-ID header on the response;
      instead it appends the raw request body to /tmp/req-audit.log, an
      undisclosed side effect unrelated to the stated obligation.
      evidence: src/flask_app/response_helper.py (confirmed: exists at head)
      judge: claude · samples: 3/3 agreed · confidence: 0.93 · floor: 2-of-3

The judge named the specific divergence, all three samples agreed well above the confidence floor, and the cited file resolved. The verdict reads BLOCKED, exit 1: a binding result, backed by the evidence the floor requires to count it at all.

The built-in presets

The easy path is a built-in preset. Today that’s claude: --agent claude resolves to claude -p, the exact headless invocation proven to answer the judge’s decomposition and audit prompts.

Only claude is a built-in judge preset right now. Shipmoor’s review feature and its Agent Skills integrations recognize codex, cursor, and aider too, but none of them has an established headless invocation for this specific judge contract yet. Passing one of those names bare to shipmoor claim-check —agent fails fast with a clear error rather than silently wiring up a judge that can only ever abstain. Pass a raw command instead: any CLI that can run non-interactively and print the JSON shapes below to stdout works, including a small wrapper script around a CLI that doesn’t speak this contract natively.

The agent contract

A configured --agent command is invoked more than once per run: once to decompose an acceptance criterion, then once per sample (K of them) per atomic check, to audit it. Every call shares the same transport, your command reads a prompt on stdin and prints one JSON object to stdout, but which shape it must answer with depends on which phase asked.

Decomposition asks the agent to break one obligation into atomic checks, and expects:

{"checks": [{"check_id": "c0", "question": "Does the change call `refund` on `payment_failed`?"}]}

Audit asks the agent to compare one atomic check against the masked observed behavior, and expects:

{
  "divergence": true,
  "confidence": 0.93,
  "evidence_pointer": {"kind": "file", "fields": {"path": "src/flask_app/response_helper.py"}},
  "rationale": "one or two sentences"
}

evidence_pointer is null when the sample sees no divergence. kind is file, test, or command; a test or command pointer is the one that gets executed for confirmation, described above. The built-in claude preset already speaks both shapes. Wiring a raw command means answering both correctly.

Keeping the judge honest: agent isolation

Shipmoor can’t see which model or agent authored the diff it’s reviewing, so keeping the author and the judge independent is on you. The simplest rule: don’t point --agent at the same tool that wrote the change under review. An agent grading its own homework is the failure mode most worth avoiding, and Shipmoor doesn’t detect or enforce that separation today, so treat it as an operating discipline rather than a guarantee the tool gives you.

The masking boundary

Only masked signals leave the process to reach the agent: the acceptance criterion’s text, the changed-file list, and the raw masked added lines per file, never a full raw diff and never anything beyond what the change touches. Every field runs through the same secret scanner the rest of Claim Check uses before it reaches a prompt, so a real secret can’t ride along. The --trace-out sidecar carries the same discipline: the aggregate rationale and the masked decomposed questions, never a raw per-sample model output. See Privacy & telemetry.

Fail-safe is absolute

Every failure mode of the judge collapses to the same safe outcome. No agent configured: every judgment item abstains, routed to a human, never a silent pass. A configured agent that’s missing, exits non-zero, times out, or writes something the parser can’t read: that sample is retried once, and if enough samples fail that the survivors can’t reach the agreement floor, the item goes inconclusive, the same honest outcome as active disagreement among real samples. A broken or absent judge can never invent a block, and it can never fail your build on its own. A green light was never the judge’s to give in the first place.

Next

• Turning on the gate: why a judge opinion can never be the basis for a READY.
• Reading the verdict: the four verdicts, exit codes, and the fix packet.
• Attestation artifacts: the VSA, the trace sidecar, and replay.
• Privacy & telemetry: what the judge does and doesn’t send anywhere.
• Using Skills with your agent: wiring the same agent into the whole loop.