Liveness

Heartbeat monitoring for machines: devices, services, game servers, and DePIN nodes post periodic heartbeats, and the service derives alive/dead status, uptime percentages, network-latency trends, and fleet-wide queries from the resulting time series. Status changes push out over WebSocket and webhooks.

When to reach for it: IoT fleet monitoring, "is this worker still running", per-device uptime SLAs, client-to-server latency charts, alerting when a silent node comes back online.

When not to: "who is active in this room or document right now", with idle states and occupancy counts, belongs to Presence; whether the Snug API itself is up is Health.

Concepts

• An entity is anything that reports - a free-form entity_id of up to 128 characters, created implicitly by its first heartbeat. There is no registration step.
• Alive is computed, not stored. An entity is alive when its last heartbeat falls within heartbeat_interval_seconds + grace_period_seconds (30 + 60 = 90 seconds by default). Stop heartbeating and it flips to dead on its own; nothing has to mark it.
• Two time series per entity. Every heartbeat appends to the heartbeat series. A heartbeat carrying client_timestamp_ms also appends server_time - client_time to a separate latency series, the source of every latency figure. All analytics are derived from these two on read.
• Metadata is replaced, labels are inherited - both optional on a heartbeat, but they behave differently on omission (see below).
• Entities are tenant-wide. Any token in the tenant can read, heartbeat, or delete any entity; there is no per-entity ownership.

Sending heartbeats

snug liveness send -i device-001 -m '{"ip":"10.0.0.1","fw":"2.3.1"}' -L region=us-east
snug liveness send -i device-001                      # the steady-state call

The CLI stamps the local send time into client_timestamp_ms automatically, so every CLI heartbeat produces a latency sample. Over HTTP that field is yours to supply; omit it and you get heartbeats without latency tracking:

curl -X POST -H "Authorization: Bearer $SNUG_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"entity_id":"docs4-sensor-3","metadata":{"role":"gateway"},"labels":{"region":"eu-west"}}' \
  http://localhost:4000/api/v1/liveness

{
  "status": 200,
  "msg": "OK",
  "data": { "entity_id": "docs4-sensor-3", "alive": true, "server_timestamp_ms": 1788007658330 }
}

Null fields are omitted rather than sent as null, which is why client_timestamp_ms and network_delay_ms are absent above. When present, network_delay_ms is a raw clock difference including drift, so a client running ahead of the server produces a negative value.

An entity_id must be non-empty, at most 128 characters, and free of control characters and of leading or trailing whitespace; violations are 400 BAD_REQUEST with the reason in msg.

Reading status

snug --output json liveness status -i device-001

{
  "entity_id": "docs4-sensor-1",
  "alive": true,
  "last_seen_server_ms": 1788007754203,
  "last_seen_client_ms": 1788007754197,
  "network_delay_ms": 6,
  "latency_ms": 6.0,
  "labels": { "region": "us-east" },
  "summary": { "uptime_1h": 5.72, "uptime_24h": 0.24, "avg_latency_1h_ms": 5.44 }
}

Status never 404s. An entity that has never reported returns {"entity_id": "...", "alive": false, "labels": {}} with everything else omitted, so treat a missing last_seen_server_ms as "never seen" rather than an error. summary is a convenience view of the four analytics endpoints, which have their own page: Uptime and analytics. Read it before trusting a low number - a healthy entity provisioned an hour ago really does report uptime_24h: 0.24.

Metadata versus labels

Verified with three consecutive heartbeats on one entity:

Metadata is the latest snapshot and is dropped by any heartbeat that omits it, so resend it every beat to keep it visible. Labels are sticky: an empty set inherits the previous one, a non-empty one replaces it wholesale.

Querying the fleet

snug liveness query --alive false                 # everything that went silent
snug liveness query --uptime-below 50 -r 300      # under 50% over the last 5 minutes

{
  "entities": [
    { "entity_id": "docs4-sensor-2", "alive": false, "last_seen_ms": 1788007596189, "uptime": 0.1 }
  ],
  "total_matched": 1
}

There is no pagination object and no offset or cursor: limit truncates an entity-id-sorted list while total_matched reports the full match count (verified - -l 1 returned one entity with total_matched: 4). There is no label filter either, so labels are for display, not for slicing the fleet. The uptime column uses --range, 86400 seconds by default, which is why a young entity reports near-zero uptime until you narrow the window.

Real-time updates

Every heartbeat broadcasts a liveness_update frame on the platform's general socket, /ws/liveness. Auth modes, scopes, limits, and the rest of the frame protocol are covered once in WebSocket; the liveness-specific part is what a subscribe frame gets you. Captured live after {"type":"subscribe","entity_id":"docs4-sensor-1"}:

{
  "type": "liveness_update",
  "entity_id": "docs4-sensor-1",
  "alive": true,
  "server_timestamp": 1788007736157,
  "client_timestamp": 1788007736153,
  "network_delay_ms": 4,
  "metadata": { "fw": "2.3.1" },
  "last_seen": 1788007736157
}

Subscribing without an entity_id receives updates for every entity in the tenant (verified: two entities beat, both frames arrived). Frames fire on every heartbeat, not only on state changes, so a 1-second heartbeat interval is a 1-second frame rate per subscriber. The frame's timestamp fields are server_timestamp / client_timestamp / last_seen, not the _ms-suffixed names the HTTP responses use.

Status-change webhooks

A dead-to-alive transition emits an entity.status_changed event. Verified end to end against a local listener - the first-ever heartbeat on a new entity fired, a second heartbeat four seconds later did not:

{
  "id": "evt_LgBuqwjNgueL",
  "type": "entity.status_changed",
  "entity_id": "docs4-hook-a",
  "previous_status": "dead",
  "current_status": "alive",
  "last_liveness": 1788007792,
  "metadata": null,
  "created": 1788007792
}

There is no alive-to-dead event. Death is not an action the server takes, only a conclusion a reader draws from the last-seen timestamp, so alerting on a disappearance means polling query --alive false on your own schedule. The one background worker refreshes a Prometheus active-entity gauge; it emits nothing. Subscribe with Webhooks: snug webhooks create -u https://example.com/hook -e entity.status_changed.

Deleting an entity

snug liveness delete -i device-001

DELETE /api/v1/liveness/{entity_id} answers 204 and removes the metadata document, both time series, and the fleet-index entry; a second delete is 404 NOT_FOUND. It is irreversible and, unlike most destructive snug commands, takes no --force flag and asks no confirmation. Deletion does not stop an entity: a later heartbeat recreates it with empty history.

Limits and configuration

Retention (7 days), the 30-second heartbeat interval and 60-second grace period, the 500-entity query cap, the 1000-sample history cap, and the Prometheus active-entity gauge worker are all compile-time defaults - this service reads no environment variables, as the Liveness CONFIG reference records. Moving the grace window means changing LivenessConfig, so build heartbeat intervals around 90 seconds rather than expecting to tune it.

Reference

• Liveness API - every endpoint, callable
• Related: Presence for who is active in a context, WebSocket for the socket carrying liveness_update, Webhooks for entity.status_changed delivery, Geo for locating the entities you monitor