Arbiter

Try the interactive example ↓

An LLM-as-a-judge evaluation engine: score any JSON content against free-text criteria or reusable weighted rubrics, run continuous evaluation sessions with ground-truth change detection, compare candidates head-to-head, and close the quality loop with calibration, appeals, admin overrides, and A/B judge experiments. The judge routes to Anthropic or any OpenAI-compatible endpoint, and client API keys pass through per request without being stored.

When to reach for it: scoring game rounds against designer-authored rules, judging competition submissions against rubrics, continuous support-conversation quality monitoring, content moderation with explainable per-criterion reasoning, regression-testing LLM output quality.

When not to: decisions that need a human approver belong to Human-in-the-Loop; reaching agreement on a fact across multiple independent reporters belongs to Consensus.

Concepts

• Rubrics are reusable scoring templates: 1-50 weighted criteria, each with a score type (numeric01, numeric0100, numeric_range, boolean, categorical) and an aggregation strategy (weighted_mean, minimum, maximum, custom_formula). Updates auto-increment the version.
• Sessions are stateful evaluation contexts: a judge configuration, an optional rubric (pinned to the rubric version current at session creation), and cached ground-truth and content payloads. Lifecycle: created -> active <-> paused -> completed.
• Evaluations are judged results: per-criterion scores with reasoning, an aggregate score, a confidence value, and token usage. They come from a one-shot call (no setup) or from a session.
• Ground truth is optional reference data the judge compares content against. Pushes are SHA-256 hashed so you can detect change.
• The judge runs on an LLM provider per request; keys, providers, modes, comparison, calibration, and experiments are covered on Evaluation and the judge.

Rubrics

snug arbiter rubric create --file rubric.json
snug arbiter rubric get --rubric-id 0199e100-0000-7000-8000-a06e9be18dfd
snug arbiter rubric list --query support
snug arbiter rubric update --rubric-id 0199e100-0000-7000-8000-a06e9be18dfd --file rubric-v2.json
snug arbiter rubric delete --rubric-id 0199e100-0000-7000-8000-a06e9be18dfd

rubric.json holds the criteria and aggregation:

{
  "name": "docs4-support-quality",
  "description": "Scores support replies for accuracy and tone",
  "criteria": [
    { "name": "accuracy", "description": "Does the reply match the policy in the ground truth?",
      "weight": 0.7, "score_type": "numeric01", "range_min": null, "range_max": null },
    { "name": "tone", "description": "Is the reply professional and empathetic?",
      "weight": 0.3, "score_type": "numeric01", "range_min": null, "range_max": null }
  ],
  "aggregation": "weighted_mean"
}

Create returned "version": 1; an update (even description-only) bumped it to "version": 2.

Sessions

Over HTTP, POST /api/v1/arbiter/sessions returns the envelope:

curl -X POST -H "Authorization: Bearer $SNUG_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name": "docs4-cs-monitor", "rubric_id": "0199e100-0000-7000-8000-a06e9be18dfd",
       "judge": {"provider": "anthropic", "mode": {"kind": "single"}}}' \
  http://localhost:4000/api/v1/arbiter/sessions

{
  "status": 201,
  "msg": "Created",
  "data": {
    "session_id": "0199e100-0000-7000-8000-ae8403c5b14e",
    "name": "docs4-cs-monitor",
    "status": "created",
    "rubric_id": "0199e100-0000-7000-8000-a06e9be18dfd",
    "rubric_version": 2,
    "created_at": "2026-08-28T06:32:11.133873Z"
  }
}

Lifecycle and management from the CLI:

snug arbiter session start --session-id 0199e100-0000-7000-8000-ae8403c5b14e
snug arbiter session pause --session-id 0199e100-0000-7000-8000-ae8403c5b14e
snug arbiter session resume --session-id 0199e100-0000-7000-8000-ae8403c5b14e
snug arbiter session stop --session-id 0199e100-0000-7000-8000-ae8403c5b14e     # -> completed, terminal
snug arbiter session list --filter 'status[eq]=active'
snug arbiter session update --session-id ses_... --file rename.json  # PATCH, partial
snug arbiter session delete --session-id ses_...                     # also deletes its evaluations

Feed a session by pushing ground truth and content, then trigger an evaluation (details). Ground-truth pushes are hashed for change detection:

snug arbiter session push-gt --session-id 0199e100-0000-7000-8000-ae8403c5b14e --file gt.json

{ "accepted": true, "changed": true,
  "hash": "sha256:389cdcb9...", "previous_hash": null }

The second push of the identical payload returned "changed": false with previous_hash equal to hash.

Results, appeals, and overrides

snug arbiter eval list --session-id ses_... --filter 'status[eq]=completed'
snug arbiter eval get --evaluation-id eval_...
snug arbiter eval appeal --evaluation-id eval_... --reason "Score too low"
snug arbiter eval override --evaluation-id eval_... --file scores.json --reason "Correct per policy"
snug arbiter stats --session-id ses_...

An appeal flips the evaluation to appealed status for human follow-up. Override requires a platform admin token: it replaces the scores, recomputes the aggregate, and records who overrode and why, keeping the previous aggregate on the record. A non-admin attempt fails with 403 insufficient_permissions, and appealing a missing evaluation is 404 evaluation_not_found.

stats returns score statistics (mean, median, stddev, percentiles), a status breakdown and drift detection against a rolling baseline; with too few evaluations, drift reports "insufficient evaluations to establish baseline" and alert: false.

Behaviors and gotchas

• Lifecycle is enforced - pausing a session still in created fails with 400 invalid_state_transition ("cannot move session from Created to Paused"). Start it first.
• Rubric delete-protection counts only active sessions - the delete failed with 409 rubric_in_use while a referencing session was active, but succeeded once that session was paused. Treat deleting a still-referenced rubric as unsafe regardless of session state.
• Evaluating a session with no pushed content fails with 400 invalid_evaluation ("no content available for session"). Ground truth is optional; content is not.
• multi judge mode requires count >= 1 - {"kind": "multi", "count": 0} is rejected at session creation with 400 invalid_session.
• Per-request judge endpoint overrides are off by default - a session with judge.endpoint set is accepted at creation but evaluation fails with 400 invalid_request ("per-request endpoint override is disabled") until the server enables ARBITER_ALLOW_ENDPOINT_OVERRIDE and allowlists the host.
• Owner isolation - another user reading your session gets 403 insufficient_permissions, and list endpoints only return your own resources.
• Stored but not yet acted on - triggers, result_webhook_url, key_provider_url, and the ground_truth_source/content_source configs are persisted on the session, but no poller runs, no triggers fire, and no result webhooks are delivered yet. Push data explicitly.

Persistence

Rubrics, sessions, pushed content and ground truth, evaluations, calibration samples, experiments and retry claims live in PostgreSQL. Session lists do not load pushed payloads. Evaluation receipts and session counters commit together; experiment creation commits both arm sessions and the experiment together. External judge calls run outside database transactions.

Limits and configuration

Default judge model and provider endpoints, judge timeout and retries, the per-session evaluation cap (10000 by default), drift windows, idempotency TTL, the experiment worker, and endpoint-override allowlisting are all set via ARBITER_* variables.

Reference

• Evaluation and the judge - providers, keys, judge modes, one-shot evaluation, comparison, calibration, experiments
• Arbiter API - every endpoint, callable
• Related: Human-in-the-Loop for human judgment, Consensus for fact quorums