Reliability

What happens when a webhook fails, and the levers you have: retry budgets, backoff, the dead letter queue, redrive, and queue stats. Everything below

Retry policy

Each queue manifest can carry a retry_policy; any field left unset falls back to the global default:

• max_attempts - delivery attempts before the job fails (default 3)
• base_delay_ms - first retry delay (default 2000)
• multiplier - backoff factor per retry (default 2.0)
• max_delay_ms - backoff cap (default 300000, i.e. 5 minutes)

snug job-queues create docs4-emails --enable-dlq --max-attempts 2 --base-delay-ms 500
snug job-queues update jq_fvXKFGTtWYGa --max-attempts 5 --base-delay-ms 2000

Retries hold no worker slot - a failed job is re-parked in the schedule with its backoff delay and picked up again by the scheduler, so it shows as scheduled between attempts (which is why jobs cancel-all can catch it).

The failure arc, observed

A job scheduled into that 2-attempt queue against an endpoint returning HTTP 500 ran twice and finished like this (snug jobs info, trimmed):

{
  "id": "job_MApgwfcqfspu",
  "status": "failed",
  "attempts": 2,
  "error": "Max retries exceeded"
}

Because the queue has a DLQ (--enable-dlq, or dlq_config over HTTP), the job was also dead-lettered. Without a DLQ the terminal failed record is all you get, until retention prunes it.

Inspecting the dead letter queue

snug job-queues dlq jq_fvXKFGTtWYGa --limit 20

{
  "queue": "docs4-emails",
  "entries": [
    {
      "entry_id": "1787898571468-0",
      "job_id": "job_MApgwfcqfspu",
      "error": "HTTP 500 Internal Server Error",
      "dead_lettered_at": 1787898571,
      "job": { "id": "job_MApgwfcqfspu", "status": "failed", "attempts": 2, "...": "..." }
    }
  ],
  "total": 1
}

Each entry records the final delivery error and timestamp, plus the full job document while it is still retained - after cleanup prunes the job, the entry remains but job is null, and such entries can no longer be redriven.

Redrive

Redrive replays dead-lettered jobs back onto the schedule - all of them, or a selected few, optionally with a delay:

snug job-queues dlq-redrive jq_fvXKFGTtWYGa
snug job-queues dlq-redrive jq_fvXKFGTtWYGa --job-id job_MApgwfcqfspu --delay 30

{ "redriven_count": 1, "skipped_job_ids": [] }

Verified end to end: after the failing endpoint was fixed, redrive re-scheduled the job and it completed - with attempts: 1, because redrive resets the attempt counter and grants a fresh retry budget. skipped_job_ids lists entries that could not be replayed (job pruned, or not yours). A successful redrive removes the entry from the DLQ.

What the webhook receives

The delivery is a POST with the job payload JSON-serialized into the message field of an event envelope - not the raw payload. Captured by a live receiver:

{
  "id": "jobqueue_cf5f9b72-...",
  "type": "test.webhook",
  "message": "{\"template\":\"welcome\",\"to\":\"user@example.com\"}",
  "timestamp": 1787898570,
  "created": 1787898570
}

Headers carry the metadata: snug-signature: t=<unix>,v1=<HMAC-SHA256>, x-delivery-attempt, x-webhook-id, x-webhook-event, x-request-id, and x-webhook-timestamp. The signing secret comes from webhook.secret on the schedule request, or is auto-generated (whsec_...) and visible in snug jobs info under webhook_config. Deliveries time out after 30 seconds; only a 2xx response counts as success.

Watching a queue

snug --output json queues stats docs4-emails

[
  {
    "queue": "docs4-emails",
    "scheduled": 1,
    "processing": 1,
    "completed_last_hour": 3,
    "failed_last_hour": 1,
    "oldest_job_age_seconds": 0,
    "avg_processing_time_ms": null,
    "success_rate": 0.75
  }
]

scheduled is the exact current queue depth. The other counters are incrementally maintained and reset an hour after the last activity - treat processing and the hourly counters as approximate monitoring signals, not exact accounting. success_rate is completed over completed-plus-failed for that window; a queue with no recent activity reports 1.0. There is no DLQ size here - use snug job-queues dlq. The CLI always prints an array, even for a single queue; the underlying endpoint (GET /api/v1/queues/{queue_name}/stats) returns one object.

Pausing is the other operational lever: snug job-queues pause flips the manifest to active: false and delivery stops while intake continues, so a misbehaving downstream can be paused, fixed, and resumed without losing jobs

• the exact arc verified on the main page.