Distributed Locks

Each {namespace}/{key} has one lock slot: one owner at a time, a TTL that auto-releases if the holder crashes, and a fencing token for downstream safety.

Acquire

snug --output json distributed-state acquire-lock -n docs4-jobs -k nightly-report -o worker-a -t 60

{
  "acquired_at": "2026-08-28T04:33:32.065175Z",
  "expires_at": "2026-08-28T04:34:32.065175Z",
  "fence_token": 1,
  "key": "nightly-report",
  "lock_id": "lock_XXvDUxusTSXY",
  "metadata": null,
  "namespace": "docs4-jobs",
  "owner_id": "worker-a",
  "ttl_seconds": 60
}

owner_id is an identity you choose (instance ID, hostname) - it is not the authenticated user. Keep the returned lock_id: extend and release require both. TTL defaults to 30 seconds when omitted. Optional -m '{"job":"nightly"}' attaches JSON metadata, echoed back on status.

Contention: fail fast or wait

By default (-w 0) a held lock rejects the acquire immediately. The 409 envelope is unusual - it carries data alongside the error code, telling you who holds it and when to come back:

{
  "status": 409,
  "msg": "Lock already held by 'worker-a'",
  "error": "lock_already_held",
  "data": {
    "current_owner": "worker-a",
    "expires_at": "2026-08-28T04:34:32.065175Z",
    "retry_after_ms": 59980
  }
}

With -w <ms> the acquire blocks until the lock frees or the wait runs out (408 lock_timeout). Verified end to end: worker-a held a 2-second lock, worker-c called with -w 5000, blocked 2.0 seconds, and acquired the moment the TTL expired:

snug distributed-state acquire-lock -n docs4-jobs -k nightly-report -o worker-a -t 2
snug distributed-state acquire-lock -n docs4-jobs -k nightly-report -o worker-c -t 30 -w 5000

That is also the crash story: a holder that dies without releasing blocks nobody for longer than its TTL.

Holding: status and renewal

snug --output json distributed-state lock-status -n docs4-jobs -k nightly-report
snug distributed-state extend-lock -n docs4-jobs -k nightly-report \
  -l lock_XXvDUxusTSXY -o worker-a -t 120

Status shows is_locked plus holder details and time_remaining_ms (all null when unlocked). For work that outlives the TTL, keep the TTL short and renew with extend-lock before expiry rather than acquiring with a huge TTL - a crash then frees the lock quickly. Extending resets expires_at and leaves fence_token unchanged.

Release, and the ownership guards

snug distributed-state release-lock -n docs4-jobs -k nightly-report \
  -l lock_XXvDUxusTSXY -o worker-a

Extend and release require the matching pair, and each mismatch fails distinctly:

wrong owner_id - 403 lock_not_owned
wrong lock_id - 409 lock_id_mismatch
no lock held (already released or expired) - 404 lock_not_found

These guards are cooperative, not access control: owner_id and lock_id are strings any caller in the project could present. They prevent accidents - a stale worker renewing a lock it lost - not malice.

Fence tokens

fence_token is a monotonic counter per lock key, incremented on every successful acquire - including a re-acquire after the previous holder's TTL expired. Four successive acquires on one key returned tokens 1, 2, 3, 4, with the expiry takeover included.

Use it when the locked work touches an external system: pass the token along with each write, and have the downstream reject tokens lower than the highest it has seen. A holder that lost its lock to expiry (and thinks it still holds it) then carries a stale token and is rejected, instead of corrupting the successor's work. Extending preserves the token, so renewal does not invalidate in-flight writes.

Reference

Endpoints and exact request fields: Distributed Locking API.