Impact
Store-derived metrics can remain stale precisely when processing is stalled, a queue is paused, claims fail, or the store is unavailable. Operations documentation also recommends meters that do not exist.
Evidence
ThreadmillMetrics.refresh() runs at construction, explicit host calls, and a throttled completion interceptor. With no completions, queue depth, state counts, heartbeat age, and refresh errors do not update. recordClaimLatency is not called by the engine. There are no orphan-reclaim, claim-failure, or rejected-write meters despite operations guidance.
Acceptance criteria
- Refresh store gauges on a bounded periodic/pull path independent of job completion.
- Instrument claim latency/failure, orphan reclaim, and rejected writes at their actual engine/store boundaries.
- Define stale/error behavior during store outages.
- Prevent unbounded queue-tag cardinality.
- Add tests for stalled processing, paused queues, newly appearing queues, and store failures.
Impact
Store-derived metrics can remain stale precisely when processing is stalled, a queue is paused, claims fail, or the store is unavailable. Operations documentation also recommends meters that do not exist.
Evidence
ThreadmillMetrics.refresh()runs at construction, explicit host calls, and a throttled completion interceptor. With no completions, queue depth, state counts, heartbeat age, and refresh errors do not update.recordClaimLatencyis not called by the engine. There are no orphan-reclaim, claim-failure, or rejected-write meters despite operations guidance.Acceptance criteria