Evaluation and the Judge

How Arbiter actually scores things: which LLM runs the judgment, how keys flow through, and the four ways to ask for a verdict (one-shot, session, pairwise comparison, calibration) plus A/B judge experiments.

The judge needs a provider credential. Without one, every verdict call fails upstream with 502 judge_failed ("provider returned status 401 Unauthorized").

Providers and keys

The judge routes to anthropic (Messages API) or openai_compat (chat completions against any OpenAI-compatible endpoint), chosen per session or inferred from the model name. Keys are zero-storage: pass one per request in the api_key body field (CLI: --api-key) or the X-LLM-Api-Key header, and it is used transiently, never written to Redis, and masked in logs. Without a per-request key the server falls back to its configured default; if neither exists the call fails with 400 missing_api_key.

One-shot evaluation

Zero setup - criteria, content, optional ground truth:

snug arbiter evaluate \
  --criteria "Rate accuracy and professionalism, 0-1 each" \
  --content '{"agent_response": "Refunds are accepted within 30 days."}' \
  --ground-truth '{"policy": "Refunds are allowed within 30 days."}' \
  --api-key sk-ant-...

A successful response carries evaluation_id, per-criterion scores with reasoning, an aggregate_score, confidence, status, model, and token usage. The optional idempotency_key field deduplicates: a repeat within the TTL (24 hours by default) returns the original evaluation instead of judging again.

Without a credential the same call returns:

{
  "status": 502,
  "msg": "Judge execution failed: provider returned status 401 Unauthorized",
  "error": "judge_failed"
}

Session evaluation

A session evaluates its cached payloads: push content (required) and ground truth (optional), then trigger. Criteria come from the session's pinned rubric; the rubric's aggregation strategy computes the aggregate.

snug arbiter session push-gt --session-id ses_... --file ground-truth.json
snug arbiter session push-content --session-id ses_... --file content.json
snug arbiter session evaluate --session-id ses_... --api-key sk-ant-...
snug arbiter eval list --session-id ses_...

Each completed evaluation bumps the session's running stats and counts against the session's max_evaluations cap (further attempts fail with 400 max_evaluations_reached). session evaluate accepts the same idempotency key mechanism as one-shot.

Judge modes

Set on the session's judge.mode:

• {"kind": "single"} - one judge pass (default).
• {"kind": "multi", "count": N} - N independent passes, per-criterion scores averaged. count must be >= 1 (enforced at session creation).
• {"kind": "cascade", "stages": [{"model": ..., "provider": ...}, ...]} - one pass per stage, scores averaged across stages.

Per-stage or per-session endpoint overrides are HTTPS-only, checked against a server-side host allowlist, and rejected outright while ARBITER_ALLOW_ENDPOINT_OVERRIDE is off (the default) - the error surfaces at evaluation time, not at session creation.

Pairwise comparison

Judge two candidates under the same session criteria and get a winner:

snug arbiter compare --session-id ses_... \
  --content-a '{"reply": "..."}' --content-b '{"reply": "..."}' \
  --api-key sk-ant-...

The response carries both aggregates, both score sets, and "winner": "a" | "b".

Calibration

Submit content with known-good scores; Arbiter runs the judge, measures the deviation, and folds it into cumulative accuracy:

snug arbiter calibrate --session-id ses_... --file calibration.json
snug arbiter calibration --session-id ses_...

With zero samples:

{
  "session_id": "ses_KMLbJMbHZePFDYbyhhGE",
  "total_calibrations": 0,
  "cumulative_accuracy": 1.0,
  "mean_deviation": 0.0,
  "max_deviation": 0.0,
  "trend": "stable"
}

Experiments

A/B-test two judge configurations (requires a platform admin token). Create names two arms; Arbiter materializes a hidden session per arm, and a background worker (30-second tick by default) samples both arms sample_size times and picks a winner:

snug arbiter experiment create --file experiment.json
snug arbiter experiment list --status completed
snug arbiter experiment get --experiment-id exp_...

The worker picked up a created experiment within one tick and, with no provider credential, recorded the failure on the record instead of leaving it stuck:

{
  "experiment_id": "exp_WxJnddGWbjEbhvWNbpZG",
  "status": "failed",
  "results": { "failure_reason": "Judge execution failed: provider returned status 401 Unauthorized" },
  "sample_size": 2
}

On success results holds per-arm mean aggregate, stddev, min/max, token usage, and the winning arm. Non-admin creation fails with 403 insufficient_permissions.

Reference

• Arbiter API - request and response schemas for every operation above
• Back to the Arbiter overview