Context
The host services (provisioning, ingestion) and the broker need liveness monitoring that pages admins when something breaks. The critical design rule: the alerting path must not depend on the component it monitors. An "ingestion → DB is down" alert that itself writes to the DB dies with it. So urgent alerts go out-of-band, independent of the DB (and ideally the broker too).
Pattern: a dead man's switch — alert on the absence of an expected signal, which also catches silent crashes, not just loud errors. Note this is monitoring + alerting, NOT failover (no automatic switch to a backup).
Scope
A small watchdog container with no DB dependency that detects broker/service outages and notifies admins out-of-band.
Two liveness concerns (handled differently)
- Service liveness — each service publishes a periodic heartbeat to e.g.
repacss/services/<name>/status; watchdog subscribes, missing heartbeat for N seconds → alert. Reuses the existing broker.
- Broker liveness — CANNOT be checked over MQTT (if the broker is dead there is no MQTT). Watchdog must probe the broker out-of-band via a direct TCP connect to
mosquitto:1883.
So the watchdog uses three independent paths (MQTT subscribe + direct broker probe + outbound alert channel) so no single failure silences it.
Two alert tiers
| Alert |
Urgency |
Path |
DB? |
| Broker / service down |
urgent |
out-of-band (webhook / SMTP) |
no — must survive DB/broker loss |
| Unknown-device 24h summary (README) |
low |
DB-backed, rate-limited |
yes |
Behavior details
- Debounce: don't alert on the first missed beat. Wait a grace period, alert once on the transition to DOWN, and send a RECOVERY notice on return. (Same logic as the ESP32's 2000 ms heartbeat timeout, one layer up.)
- Alert channel: SMTP or Slack/Discord webhook — pick one, keep it DB- and broker-independent.
Acceptance criteria
- Killing a service container triggers exactly one DOWN alert after the grace period, and one RECOVERY alert when it returns.
- Killing the broker triggers a DOWN alert via the out-of-band path (not via MQTT).
- No alerting path touches the database.
Future work (not dev-blocking). Relates to #8 (device-side reconnect) and the README unknown-device alerting spec.
Context
The host services (provisioning, ingestion) and the broker need liveness monitoring that pages admins when something breaks. The critical design rule: the alerting path must not depend on the component it monitors. An "ingestion → DB is down" alert that itself writes to the DB dies with it. So urgent alerts go out-of-band, independent of the DB (and ideally the broker too).
Pattern: a dead man's switch — alert on the absence of an expected signal, which also catches silent crashes, not just loud errors. Note this is monitoring + alerting, NOT failover (no automatic switch to a backup).
Scope
A small watchdog container with no DB dependency that detects broker/service outages and notifies admins out-of-band.
Two liveness concerns (handled differently)
repacss/services/<name>/status; watchdog subscribes, missing heartbeat for N seconds → alert. Reuses the existing broker.mosquitto:1883.So the watchdog uses three independent paths (MQTT subscribe + direct broker probe + outbound alert channel) so no single failure silences it.
Two alert tiers
Behavior details
Acceptance criteria
Future work (not dev-blocking). Relates to #8 (device-side reconnect) and the README unknown-device alerting spec.