feat(runner): add pull-dispatch coordination layer - #55
Conversation
There was a problem hiding this comment.
Pull request overview
This PR introduces a new runner/ coordination layer for the pull-based dispatch runner side, adding backend registration + heartbeat plumbing and a thread-safe active-job snapshot mechanism, with unit tests validating backoff, auth fail-fast, request shapes, and config defaults.
Changes:
- Added
BackendClient,RunnerIdentity, andRunnerConfigfor runner↔backend registration/heartbeat HTTP calls. - Added
register_with_backoff(infinite retry with capped backoff; 401 fail-fast) andHeartbeatThread(interruptible stop; consecutive-401 fail-fast). - Added
ActiveJobTrackerplus a comprehensive unit test suite for the new runner coordination components.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| runner/init.py | Adds package docstring for the new coordination layer (currently inaccurate given existing legacy runner code in the same package). |
| runner/active_jobs.py | Implements thread-safe active job tracking with snapshot and idempotent remove. |
| runner/client.py | Adds backend HTTP client + dataclasses for identity/config parsing and API error modeling. |
| runner/config.py | Introduces new env-driven config for the pull-based runner API (comment currently misleading vs existing legacy BACKEND_API). |
| runner/heartbeat.py | Implements a daemon heartbeat thread with consecutive-401 fail-fast behavior. |
| runner/registration.py | Adds registration retry loop with capped exponential backoff and injectable sleep for tests. |
| tests/test_active_jobs.py | Unit tests for ActiveJobTracker behavior and thread-safety smoke test. |
| tests/test_runner_client.py | Unit tests for client request shapes, URL normalization, error mapping, and config defaults parsing. |
| tests/test_runner_heartbeat.py | Unit tests for heartbeat behavior, auth failure counting/reset, stop responsiveness, and interval selection. |
| tests/test_runner_registration.py | Unit tests for backoff schedule, retry behavior, and 401 fail-fast behavior. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
codex: 我看完了，有一個地方建議 merge 前先修：
建議改成用固定的 另有一個不擋 merge 的小地方： 我在本機跑過這次新增的 31 個單元測試，全部通過。 |
BackendClient (register/heartbeat), register_with_backoff (1-2-4-8-16-30s, 401 no-retry), HeartbeatThread (interval from register config, carries active_job_ids, 2x consecutive 401 fail-fast via on_fatal) and ActiveJobTracker. Dark slice - nothing wires into main.py yet; unit tests run without docker. Normal-OJ/Normal-OJ#68 (sandbox slice 2/4); spec section 10/13, ADR-0004
2c6f7db to
c0a205b
Compare
Subtract the time the beat itself took from the next wait, so a slow or timed-out request (up to 10s) cannot push the next beat past the 30s lease TTL. Without this, one timeout almost always dropped the lease and forced a needless reclaim. Addresses review feedback.
BACKEND_API is not replaced yet; it still serves the old push path until the keystone slice. Addresses review feedback.
|
Claude: Heartbeat cadence fixed in 491eabc — the wait now subtracts the time the beat took (time.monotonic, clamped at 0), so a slow or timed-out request no longer pushes the next beat past the 30s lease TTL. Added two unit tests: a 10s beat shrinks the following wait to 5s, and a beat longer than the interval waits 0. On the test-double dedup: leaving it as is for now; will extract a shared helper if slice 3 (#69) needs a third copy. |
Normal-OJ/Normal-OJ#68（sandbox slice 2/4）
新協調層
runner/，pull-based dispatch 的 runner 側基礎。Dark PR——main.py尚未接線，無任何呼叫者；不修改既有檔案。內容
runner/client.py—BackendClient（register/heartbeat，契約見 spec §7.1–7.2）、RunnerIdentity（memory-only，tokenrepr=False，ADR-0004）、RunnerConfig（parse register 回應的 config，缺 key 落 15/3/8 預設）runner/registration.py—register_with_backoff：backoff 1→2→4→8→16→30s（之後維持 30、無限重試）；401 不重試直接 raiserunner/heartbeat.py—HeartbeatThread：interval 取自 register config（預設 15s）、body 夾帶ActiveJobTracker快照、連續 2 次 401 →on_fatal（非 401 結果歸零計數）、Event-based 可即時stop()runner/active_jobs.py—ActiveJobTracker：thread-safe add/remove（冪等）/snapshot/lenrunner/config.py—BACKEND_URL、RUNNER_REGISTRATION_TOKEN、RUNNER_NAME（未設時落 hostname）等；不動舊BACKEND_API測試
31 個單元測試，無 docker/網路依賴（注入 fake session）：backoff 序列逐值驗證、401 fail-fast／歸零矩陣、request shape（URL/Bearer header/body）、config 預設值、tracker 冪等與 thread-safety、token 不出現在 repr。
備註
runner/→executor/改名）再 merge 本 PR，縮短runner/新舊檔案混居期；兩者無 git 衝突（本 PR 純新增檔案），順序僅影響可讀性。main.py接線與 Flask 移除為 keystone slice。docs/specs/pull-based-job-dispatch.md§10、§13；ADR-0004。