feat(runner): main entrypoint with SIGTERM drain, remove Flask (keystone)

[as535364]

closes Normal-OJ/Normal-OJ#70

Keystone slice (4/4) of the pull-based dispatch rework (spec §10, §15.3). The Sandbox becomes a pure pull-mode runner process: no Flask, no inbound HTTP.

What changed

main.py (new entrypoint)

• Registers with backoff (401 → exit 1, per ADR-0004; retry sleep is interruptible so docker stop during registration does not hang).
• Starts dispatcher, result sender, poller, heartbeat.
• SIGTERM/SIGINT drain, in this order: stop+join poller → abort unstarted jobs with reason=drain (not counted against attempts) → wait for in-flight jobs → stop dispatcher → stop sender (it exits only once the result queue is empty, so every queued complete/abort is reported) → stop heartbeat last (leases must stay renewed until the final sends).
• Heartbeat 2×401 fail-fast: exit 1 with no drain — identity has evaporated, sends would 401 anyway; compose restart re-registers. The dispatcher thread is daemon so this path actually terminates the process (caught in review: a non-daemon dispatcher would keep the interpreter alive forever and the restart policy would never fire).

dispatcher

• handle() is now atomic (issue comment item 1): all validation happens before any state mutation, and the task queue is unbounded so put cannot fail halfway. No partial-enqueue orphans, no rollback needed.
• Capacity gate / task dimension (issue comment item 2): rather than making the gate inspect remaining task slots, the task-level bound is removed entirely. Rationale: the poller cannot know a job's case count before claiming it, so a slot-aware gate can never fully guarantee the claimed job fits — the "claimed but cannot enqueue, attempts wasted" path would only narrow. With the poller as the queue's sole producer, task volume is already capped by max_concurrent_jobs × per-job case count; the bound only produced spurious queue.Full. Unbounding kills the path completely.
• Job completion pushes through an injected on_complete callback into the result queue (sender owns reporting, retries, dir cleanup, tracker removal) instead of the old direct PUT with SANDBOX_TOKEN. The result is enqueued before release() so a concurrent drain can never observe an empty dispatcher while a result is still un-queued.
• Removed the dispatcher-level 300s is_timed_out: it dropped queued tasks without resolving them, which in pull mode leaves the job in the tracker forever — heartbeat renews its lease indefinitely and the submission is stuck permanently (also wedges drain). Per-case runtime is already hard-bounded by the executor layer (compile 20s; execute docker wait 5×time_limit → JE).
• Crashes the executor layer does not wrap (e.g. docker APIError, unreadable config) now resolve the case/compile as JE, keeping the drain invariant "in-flight jobs always finish" and preventing permanently-leased jobs.

Removal / unification

• Deleted app.py, gunicorn.conf.py; dropped flask + gunicorn from requirements; Dockerfile CMD is python main.py.
• BACKEND_API unified into BACKEND_URL (single definition in dispatcher/config.py, reused by runner/config.py).
• SANDBOX_TOKEN is deliberately NOT deleted: the legacy testdata channel (/problem/<id>/meta|testdata|checksum ?token=) still authenticates with it. Backend keystone backend: keystone — switch submit/rejudge to enqueue, remove push path (slice 6/6) Normal-OJ#66 deletes mongo/sandbox.py, which those endpoints depend on — the replacement auth is an open coordination point (commented on #66); a small sandbox follow-up will adopt whatever it defines.

Tests

• New: tests/test_main.py (drain ordering, fatal/registration exit paths, interruptible registration sleep); dispatcher tests for unbounded enqueue, handle() atomicity, drain_unstarted(), on_complete wiring, compile JE fallback, daemon flag.
• Full suite locally: 100 passed; test_c_tle / test_non_strict_diff fail only for lack of a local docker daemon (CI runs them). yapf . -rd clean.
• Independent verifier pass: drain/start race stress (40 randomized trials, 0 invariant violations), lock-ordering check, atomicity probes.

Deployment

Deploys together with the backend keystone (Normal-OJ/Normal-OJ#66) per spec §15.3; the docker stop drain smoke test happens there (Normal-OJ/Normal-OJ#71 sets stop_grace_period / restart policy).

Review follow-ups (e309f91)

Codex found a drain-correctness bug and a verification gap; both are fixed.

Drain no longer burns attempts, and no longer waits forever. A job claimed from the backend but not yet dispatched is invisible to drain_unstarted() (it isn't in dispatcher.result yet), so only the poller can return it — and it used to return it as prep_failed, which counts against the submission's attempts. A rolling restart could therefore push healthy submissions toward JE, exactly what spec §7.5/§12 say drain must not do. The poller now checks its stop event at the top of every prep attempt and on the exhausted path (stop can land during the final attempt, after the loop's own check), returning the claim as abort(drain). The prep backoff waits on the stop event, so it is interruptible. Dispatch failures stay prep_failed on purpose: a bad payload has to converge rather than bounce to the next runner.

Bounded prep I/O. The legacy testdata channel had no timeouts at all — three requests.get calls and a redis lock with no blocking timeout — so poller.join() during a drain could hang until docker's SIGKILL. Now: HTTP (connect 5s, read 30s), lock blocking_timeout=30, redis pool socket_timeout=10 / socket_connect_timeout=5.

A drain budget, because per-request timeouts aren't enough. requests' read timeout applies per socket read, not to total duration, so a slow-drip response can extend a single request indefinitely. drain() now shares one deadline (DRAIN_TIMEOUT_SEC = 540, under the docker stop --time=600 the deployment uses) across every join and the in-flight wait loop, logging an error for anything that outlives it and leaving that work to lease expiry and reclaim. Waiting forever only hands the decision to SIGKILL and throws away results already collected.

Real-thread drain tests. tests/test_drain_integration.py runs main.drain() against the real poller, sender, heartbeat and dispatcher, faking only the backend client, the container executor and local cleanup. It covers an in-flight job completing through the real sender, a claimed-but-never-started job returned as drain (made deterministic with MAX_CONTAINER_SIZE=1 plus a blocking stub executor, not timing), and a claim blocked in prep when stop lands. install_signal_handlers is now exercised with a real SIGTERM. Both tests were mutation-checked: removing the poller's stop guard turns them red with prep_failed. Because the drain is now bounded, a hanging-drain regression fails the suite in ~22s instead of hanging CI forever (it previously did).

Deployment gap found along the way: compose sets no stop_grace_period, so it inherits docker's 10s default and a correct drain would still be SIGKILLed. Filed on Normal-OJ/Normal-OJ#71 and added to spec §14.

Follow-up 2 (b381b18): nothing starts judging after shutdown

A second review round found the remaining hole in the same area. A claim whose prep happens to succeed after stop() still gets dispatched, and the dispatcher's run loop is alive throughout the first half of the drain — so it could mark that job started in the gap between main.drain() joining the poller and calling drain_unstarted(). The job would begin judging after shutdown, be abandoned at process exit, and come back via reclaim with its claim attempt already spent instead of being requeued attempt-neutrally. Checking the stop event in the poller before dispatching cannot close this: the gap is on the dispatcher's side.

drain_unstarted() now latches the dispatcher (accepting = False) and scans in one lock hold, and the run loop takes that same lock before marking a job started, so nothing can move from unstarted to started in between. Jobs already in flight are exempt — they must keep consuming their remaining cases, or a multi-case job could never complete and the drain would wait out its whole budget and lose results it had already computed.

Tests: two at the dispatcher level (after the latch the run loop starts nothing new; after the latch an in-flight job still runs its remaining cases) and the requested end-to-end regression (prep blocked, stop lands, prep then succeeds, poller dispatches anyway — the job comes back as abort(drain) and the executor is never called). All mutation-checked. Worth noting the first version of the latch test was vacuous: it waited 0.5s while the run loop sleeps a full second on an empty queue, so removing the latch still passed. It now waits for the run loop to actually consume the task.

Also closed a gap an independent verification pass found: the shared drain budget had no test. join(timeout=remaining()) could have been changed to join(timeout=timeout_sec) — handing each component a fresh full budget, stretching a slow drain to 4x and past the stop grace period — with the whole suite still green. There is now a test asserting the granted timeouts strictly decrease.

Known limits, recorded rather than fixed here: the 540s budget is not provably enough in the worst case (8 concurrent jobs × many cases, or a sender retrying against a dead backend, can each exceed it on paper). Exceeding it breaks no spec invariant — the job stays in Redis and is reclaimed after its lease expires — but the drain's attempt-neutral property is lost for whatever did not get reported in time, and in-flight containers are left behind for manual cleanup. Tuning that is a deployment question for Normal-OJ/Normal-OJ#71 together with stop_grace_period.

Follow-up 3 (e31196c, d95e72d): the latch was still in the wrong place

The previous round put the latch inside drain_unstarted(), which runs after the poller join — so it did nothing for the case it was meant to fix. The poller can finish its prep and dispatch during that join, and until the latch is set the run loop is free to start that job. stop_accepting() is now its own step between poller.stop() and the join, with the scan still after it so every late dispatch is visible. handle() publishes a job's state and its tasks under state_lock too, so a scan holding that lock never sees half a job.

Worth being explicit about the testing, because the first attempt was misleading: the earlier end-to-end test only passed because the dispatcher's run loop sleeps a full second on an empty queue and never woke up during the test at all. I tried to make the interleaving deterministic — feeding the loop junk tasks to keep it awake, then widening the dispatch-to-poller-exit window to 300ms — and it still passed with the latch moved back to the wrong place. Whether the run loop wins those microseconds is scheduling, not something a test can force without injecting scheduling points. So the guarantees are pinned in layers instead, both mutation-checked: the required call order in tests/test_main.py (deterministic, with fakes) and the latch's actual effect on the run loop in tests/test_dispatcher.py (including that in-flight jobs still get to finish their remaining cases — blocking those would leave a multi-case job unable to complete). The end-to-end test stays as a path check, with a comment saying plainly what it does not pin.

tests/test_sigterm_process.py closes the last verification gap: it runs main.py as a real subprocess against a stub backend, points it at a dead redis so the claimed job sits in prep, sends a real SIGTERM into that window, and asserts the backend saw abort(reason='drain'), saw no complete, and that the process exited 0 by itself. No main.drain() call anywhere in it — the signal handler drives the whole thing. 1.6s, no docker, no new dependencies. Mutation-checked in both directions (report the claim as prep_failed → fails; skip the drain on SIGTERM → fails). Real docker stop with stop_grace_period still belongs to the deployment smoke test in Normal-OJ/Normal-OJ#71, but "process gets SIGTERM, drains, exits" is now checked on every CI run.

Follow-up 4 (95890ec, 5ca0d71): latch first, and guards for what was only commented

The latch was still one scheduling point late: poller.stop() only sets an event, and stop_accepting() took the lock on the next line, so a prep completing in between could dispatch and have the run loop start the job while the dispatcher was still accepting. Chasing the window one statement at a time was the wrong approach — the invariant is "once a drain begins, nothing starts judging", so the latch is now the drain's first act, before the poller is even asked to stop. The window that remains between the latch and poller.stop() is harmless by construction: a job claimed there cannot start, and the scan hands it back.

The process-level SIGTERM test also claimed more than it proved. Waiting for the backend to serve next-job only shows the response was written, not that the child received it or entered prep, so the signal could land earlier and the test would pass on a different path. It now waits for the child to connect to a stub redis — something only ensure_testdata does — and drops that connection after signalling. Verified against the child's own log: prep attempt 1/3 is in flight when the signal arrives.

I then ran an adversarial audit over the whole drain looking for a fifth hole. It found none live, but it did find that several invariants with explanatory comments had no test at all — each of these mutations passed the entire suite: stopping the dispatcher before the in-flight wait (which strands a multi-case job, burns the whole budget and discards results already computed), the sender dropping its queue on stop(), on_job_complete releasing before enqueueing, and the JE fallback in create_container. All four now have guards, each verified to fail without the behaviour.

It also surfaced a real design gap: every wait drew from the budget first-come-first-served, so a poller stuck on a slow-drip response could spend all 540s and leave the sender joined with nothing — completes already computed, and the drain aborts just queued, unsent. Waiting for the poller is the least valuable thing the drain does, so it now takes a capped share and the in-flight wait keeps a reserve for reporting. If the poller does outlive its share it can still dispatch, so the scan runs a second time before the sender stops.

Two smaller things: create_container now looks up its per-job lock defensively like compile() already did (a case finishing after its job was released — a drain, or the same job_id handled again after a reclaim — raised outside the JE guard and killed the worker without resolving the case, hanging the job for the whole drain); and the end-to-end test named after the prep-succeeds-after-stop case now says plainly that it passes with or without the latch, because it does — tests/test_dispatcher.py is what pins the latch.

[as535364]

codex: Found 1 drain correctness issue and 1 verification gap. Please resolve both conversations before merge.

[as535364]

codex: [P1] generation token 在 dequeue 後才從 self.result[job_id] 取得，仍綁得太晚：Compile/Execute queue entry 只有 job_id/task_id/case_id。若 G1 尚有 queued task 時同 job_id 被 release/re-handle 成 G2，G1 entry 會在這裡直接拿到 G2 ctx，冒充現任 generation；可能重複執行，兩代 task shape 不同時 tasks[_job.task_id] 還會 IndexError 殺掉 dispatcher。started 也只按 job_id，G1 曾 started 會讓尚未真正開工的 G2 在 latch 後被當成 in-flight。請在 handle() enqueue 時把 generation 綁進每個 queue entry，dequeue 時以 identity 丟棄 stale entry，並讓 started 等狀態 generation-aware。具名 JobContext 也能避免目前 result/submission_ids/started/compile state 的 data clump 繼續漏綁。

[as535364]

Claude: 你說得對，上一輪只綁了寫回那一端，dequeue 還是按 job_id 撈 ctx，冒充成立；started 共用也成立。fe41f02 照你的建議把 generation 做成具名 JobContext。

meta、case 結果、compile 結果、started 旗標、submission_id 全收進 context，按 job_id 的平行字典（compile_results、submission_ids、started set）整組刪除——data clump 不在了，就沒有「漏綁」這個類別。handle() 入列時把 context 綁進每個 Compile/Execute entry；dequeue 只用 entry 自己的 token 做 identity 檢查，完全不按 job_id 取狀態。task 索引用的也是 entry 自己那代的 meta，所以兩代 task shape 不同的 IndexError 一併結構性消失。started 是 per-generation 旗標，G1 開過工不會讓 G2 在 latch 後被當成 in-flight。

測試：stale entry 排在 fresh entry 前面餵真 run loop，斷言 executor 只跑一次、現任那代正常判完。mutation：dequeue 改回按 job_id 撈，測試紅。驗證時另外構造了你說的 IndexError 情境（G1 三個 task、G2 一個 task、G1 的 task_id=2 殘留 entry），entry 被丟棄、run loop 活著。

一個要坦白的相鄰縫：job dir 還不是 generation-scoped。supersede 時新一代的 prepare 會 rmtree 再解壓，而舊一代的 container 或 compile 可能還在用同一目錄。outcome 帳面不受影響（identity 檢查把舊代全擋掉），但檔案層確實共用。這是 prep／executor 層的既有行為，這個 PR 不展開，先記在這裡。

[as535364]

codex: [P1] compile_results 仍只以 job_id 儲存，而且 handle() 發布新 generation 時沒有清除或換代。若 G1 已有 AC/CE、同 job_id re-handle 成 G2，G2 的 Execute（包含殘留的 G1 queue entry）會在新 compile 完成前直接讀到 G1 結果：可能跳過新 compile、沿用 stale CE，或在新 binary 尚未產生時執行並提早以錯誤結果完成。舊/new Compile entry 也可能同時被當成 G2 啟動。請把 compile result 納入 generation-scoped context，並配合 queue entry token 丟棄 stale Compile/Execute；僅在 worker 寫回時檢查 ctx 無法阻止讀取既有 stale state。

[as535364]

Claude: 修了，fe41f02。compile result 收進 JobContext（每一代從 None 起始），run loop 的 requeue 條件和 worker 的讀取都走自己那代的 ctx。跨代讀取結構上不存在——那個字典沒有了，handle() 也不需要「清除或換代」任何東西，新代天生是乾淨的。

「G2 的 Execute 在 G2 compile 完成前不會跑」有測試釘住：G1 帶著已寫好的 AC compile 結果被 supersede，先餵 G2 的 Execute 再餵 Compile，Execute 一直等到自己那代的 compile 寫回才執行。殘留的 G1 Compile/Execute entry 則在 dequeue 被 identity 檢查丟棄，不會被當成 G2 啟動。

順手修了一個驗證時抓到的相鄰瑕疵：create_container 先把 lang 覆寫成 'c11' 這類字串才呼叫 extract_compile_result，而 compile_need 是對 Language enum 比較，字串永遠不成立，所以那個防禦性的 CE fallback 其實永遠打不到。現在傳原本的 enum，fallback 名實相符（它仍然只是防禦——run loop 本來就 gate 在 compile_result 非 None 才派 Execute）。

[as535364]

codex: [P1] 這個 tail 把 len(tracker)==0 當成「所有已知 claim 都已回報」，但 ActiveJobTracker 是 job_id 的 set，而 spec 同時允許同 runner 以相同 job_id 再 claim 新 generation。G2 的 add(job_id) 對既有 G1 是 no-op，任一代 outcome 的 remove(job_id) 就會把另一代也一起消掉：heartbeat 停止替仍 active 的 generation 續租、capacity 提早釋放，drain 也可能誤判 tracker empty 後 stop sender，留下另一代 outcome。請讓 tracker 內部追蹤 claim generation/refcount；heartbeat snapshot 可以再去重為 job_id，但 drain/reporting 與容量判斷不能用 lossily-collapsed set。也請補同 job_id 兩代交錯 add/remove 的 regression test。

[as535364]

Claude: 改了，fe41f02。tracker 從 job_id set 改成 per-claim refcount：add／remove 各對應一次 claim 的取得與了結，len() 計 claim 數給容量閘和 reporting tail，snapshot() 去重成 job_id 給 heartbeat（一個 job 一份租）。你要的兩代交錯 regression 補在兩層：test_active_jobs 的交錯 add/remove，以及 test_main 的 reporting tail 版本——set 語意下第一代的 remove 會讓尾段在一次 sleep 後就停，測試紅。

refcount 帶出一個你沒點名、但不一起關就會換一種方式壞的帳目洞：被 supersede 的舊 claim 永遠不會再有 outcome（它那代的 queue entry 和 worker 全被 identity 檢查擋掉），沒有人會 remove 它的 count——容量永久漏一格，下次 drain 的尾段也會為它空等到 deadline。所以 handle() 現在以回傳值告知「這次 dispatch supersede 了前一代」，poller 在那一刻歸還舊 claim 的 count，維持每個 claim 恰好一次 add、一次 remove。

平衡性靠的不變量是「ctx 還在 result ⟺ 該代還沒 enqueue 過 outcome」——on_complete 入列和 release 在同一個 state_lock 臨界區，所以 handle() 持鎖時看到 ctx 在，就保證舊代不會再冒出第二次 remove。驗證時把所有 add/remove 路徑各走了一遍（complete 的 2xx/409/404、400 轉 abort、abort 送不出去、cleanup 炸掉、supersede 全流程 trace），每條都恰好一次。mutation 三個方向都紅：tracker 退回 set 語意、poller 不歸還、handle 不回報。