Arbiter

An LLM-as-a-judge evaluation engine: score any JSON content against free-text criteria or reusable weighted rubrics, run continuous evaluation sessions with ground-truth change detection, compare candidates head-to-head, and close the quality loop with calibration, appeals, admin overrides, and A/B judge experiments. The judge routes to Anthropic or any OpenAI-compatible endpoint, and client API keys pass through per request without being stored. Every example on this page was executed against a live server.

When to reach for it: scoring game rounds against designer-authored rules, judging competition submissions against rubrics, continuous support-conversation quality monitoring, content moderation with explainable per-criterion reasoning, regression-testing LLM output quality.

When not to: decisions that need a human approver belong to Human-in-the-Loop; reaching agreement on a fact across multiple independent reporters belongs to Consensus.

Concepts

• Rubrics are reusable scoring templates: 1-50 weighted criteria, each with a score type (numeric01, numeric0100, numeric_range, boolean, categorical) and an aggregation strategy (weighted_mean, minimum, maximum, custom_formula). Updates auto-increment the version.
• Sessions are stateful evaluation contexts: a judge configuration, an optional rubric (pinned to the rubric version current at session creation), and cached ground-truth and content payloads. Lifecycle: created -> active <-> paused -> completed.
• Evaluations are judged results: per-criterion scores with reasoning, an aggregate score, a confidence value, and token usage. They come from a one-shot call (no setup) or from a session.
• Ground truth is optional reference data the judge compares content against. Pushes are SHA-256 hashed so you can detect change.
• The judge runs on an LLM provider per request; keys, providers, modes, comparison, calibration, and experiments are covered on Evaluation and the judge.

Rubrics

snug arbiter rubric create --file rubric.json
snug arbiter rubric get --rubric-id rub_ArScpbNKvZhjMtxgfMsm
snug arbiter rubric list --search support
snug arbiter rubric update --rubric-id rub_ArScpbNKvZhjMtxgfMsm --file rubric-v2.json
snug arbiter rubric delete --rubric-id rub_ArScpbNKvZhjMtxgfMsm

rubric.json holds the criteria and aggregation:

{
  "name": "docs4-support-quality",
  "description": "Scores support replies for accuracy and tone",
  "criteria": [
    { "name": "accuracy", "description": "Does the reply match the policy in the ground truth?",
      "weight": 0.7, "score_type": "numeric01", "range_min": null, "range_max": null },
    { "name": "tone", "description": "Is the reply professional and empathetic?",
      "weight": 0.3, "score_type": "numeric01", "range_min": null, "range_max": null }
  ],
  "aggregation": "weighted_mean"
}

Create returned "version": 1; an update (even description-only) bumped it to "version": 2.

Sessions

Over HTTP, POST /api/v1/arbiter/sessions returns the envelope:

curl -X POST -H "Authorization: Bearer $SNUG_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name": "docs4-cs-monitor", "rubric_id": "rub_ArScpbNKvZhjMtxgfMsm",
       "judge": {"provider": "anthropic", "mode": {"kind": "single"}}}' \
  http://localhost:4000/api/v1/arbiter/sessions

{
  "status": 201,
  "msg": "Created",
  "data": {
    "session_id": "ses_KMLbJMbHZePFDYbyhhGE",
    "name": "docs4-cs-monitor",
    "status": "created",
    "rubric_id": "rub_ArScpbNKvZhjMtxgfMsm",
    "rubric_version": 2,
    "created_at": "2026-08-28T06:32:11.133873Z"
  }
}

Lifecycle and management from the CLI:

snug arbiter session start --session-id ses_KMLbJMbHZePFDYbyhhGE
snug arbiter session pause --session-id ses_KMLbJMbHZePFDYbyhhGE
snug arbiter session resume --session-id ses_KMLbJMbHZePFDYbyhhGE
snug arbiter session stop --session-id ses_KMLbJMbHZePFDYbyhhGE     # -> completed, terminal
snug arbiter session list --status active
snug arbiter session update --session-id ses_... --file rename.json  # PATCH, partial
snug arbiter session delete --session-id ses_...                     # also deletes its evaluations

Feed a session by pushing ground truth and content, then trigger an evaluation (details). Ground-truth pushes are hashed for change detection:

snug arbiter session push-gt --session-id ses_KMLbJMbHZePFDYbyhhGE --file gt.json

{ "accepted": true, "changed": true,
  "hash": "sha256:389cdcb9...", "previous_hash": null }

The second push of the identical payload returned "changed": false with previous_hash equal to hash.

Results, appeals, and overrides

snug arbiter eval list --session-id ses_... --status completed
snug arbiter eval get --evaluation-id eval_...
snug arbiter eval appeal --evaluation-id eval_... --reason "Score too low"
snug arbiter eval override --evaluation-id eval_... --file scores.json --reason "Correct per policy"
snug arbiter stats --session-id ses_...

An appeal flips the evaluation to appealed status for human follow-up. Override requires a platform admin token: it replaces the scores, recomputes the aggregate, and records who overrode and why, keeping the previous aggregate on the record. A non-admin attempt fails with 403 insufficient_permissions, and appealing a missing evaluation is 404 evaluation_not_found - both reproduced live.

stats returns score statistics (mean, median, stddev, percentiles), a status breakdown, token usage, and drift detection against a rolling baseline; with too few evaluations, drift reports "insufficient evaluations to establish baseline" and alert: false.

Behaviors and gotchas

All reproduced against the live server:

• Lifecycle is enforced - pausing a session still in created fails with 400 invalid_state_transition ("cannot move session from Created to Paused"). Start it first.
• Rubric delete-protection counts only active sessions - the delete failed with 409 rubric_in_use while a referencing session was active, but succeeded once that session was paused. Treat deleting a still-referenced rubric as unsafe regardless of session state.
• Evaluating a session with no pushed content fails with 400 invalid_evaluation ("no content available for session"). Ground truth is optional; content is not.
• multi judge mode requires count >= 1 - {"kind": "multi", "count": 0} is rejected at session creation with 400 invalid_session.
• Per-request judge endpoint overrides are off by default - a session with judge.endpoint set is accepted at creation but evaluation fails with 400 invalid_request ("per-request endpoint override is disabled") until the server enables ARBITER_ALLOW_ENDPOINT_OVERRIDE and allowlists the host.
• Owner isolation - another user reading your session gets 403 insufficient_permissions, and list endpoints only return your own resources.
• Stored but not yet acted on - triggers, result_webhook_url, key_provider_url, and the ground_truth_source/content_source configs are persisted on the session, but no poller runs, no triggers fire, and no result webhooks are delivered yet. Push data explicitly.

Limits and configuration

Default judge model and provider endpoints, judge timeout and retries, the per-session evaluation cap (10000 by default), drift windows, idempotency TTL, the experiment worker, and endpoint-override allowlisting are all set via ARBITER_* variables - see the Arbiter CONFIG reference.

Reference

• Evaluation and the judge - providers, keys, judge modes, one-shot evaluation, comparison, calibration, experiments
• Arbiter API - every endpoint, callable
• Related: Human-in-the-Loop for human judgment, Consensus for fact quorums