feat: on-demand materialization (nudge) for recurring tasks

[hemju]

Closes #108.

What this adds

Scheduler.nudgeRecurring(taskName) (Spring: JobScheduler.nudgeRecurring) — on-demand materialization for registered recurring tasks, so outbox-style pollers can run wake-driven with a slow self-healing backstop schedule instead of polling every few seconds.

Design (as resolved in the issue discussion)

Durable flag consumed by the materializer — never a bypass lane. A nudge is one write of nudge_requested_at on the task's schedule state. The maintenance master's existing per-task recurring tick observes it alongside next_run_at (zero extra steady-state store cost), materializes one instance through the normal machinery, and clears the cell with a compare-and-clear on the observed timestamp. That single mechanism carries all four behavioral requirements:

• Run-after-wake: the pile-up guard defers a nudge while an instance is in flight; the flag survives to the tick after that instance terminates, so the in-flight run — which may have read the work table before the nudge's triggering write committed — never counts as satisfying it. A nudge accepted between the materializer's read and its clear has a newer timestamp and survives (the client-go workqueue rule). Ordering is materialize-then-clear: a crash between the two costs one extra run, never a lost one.
• Coalescing: one cell per task — a burst overwrites one value and produces at most the current run plus one follow-up (Temporal's BufferOne semantics). A tick that materializes a due scheduled fire also satisfies an observed nudge.
• Durability over signaling: there is no transient signal at all — acceptance from any node is a store write; the maintenance tick (default 1 s) bounds latency. Nothing can be dropped.
• Schedule non-interference: next_run_at is never touched by a nudge. Cron keeps its wall-clock grid; an interval's phase is preserved (every-6h task last fired 06:00, nudged 07:00 → next regular run stays 12:00). Nudged instances carry no CRON_FIRE_TIME_META (they represent no schedule tick).

Edges: unknown task → IllegalArgumentException; disabled task → IllegalStateException (explicit pause wins); an enabled-flip clears any pending nudge (consistent with #106's re-enable-does-not-catch-up); a nudge racing task removal cannot resurrect schedule state (Redis: task-existence check in the same Lua script; Postgres: FK).

Clobber safety: the nudge cell is written only by the two new store ops. upsertCronTaskState deliberately preserves it on every backend — Postgres leaves the column out of the upsert, Redis's DEL+HSET overwrite script carries the field across, in-memory merges — so re-registrations, materializer bookkeeping, and dashboard edits cannot clobber a concurrently accepted nudge.

Spring transaction modes (threadmill.spring.enqueue-mode):

• after_commit (default): validation fails fast at call time; the write fires in afterCommit, a rollback discards it. The residual crash window is covered by the backstop schedule (explicit non-goal in the issue).
• join_transaction: the nudge write rides the caller's SQL transaction via the store's PostgresTransactionBoundary — committed with the work row, no crash window. Documented caveat: it holds the schedule-state row lock until the caller commits, so hot-path producers should prefer after_commit.
• immediate: direct write.

Producer-burst protection: NudgeCoalescer single-flights nudge writes per task per JVM. Callers arriving during an in-flight write share one follow-up write that starts after they arrived — joining the in-flight write would be the classic single-flight-without-rerun bug (its commit can predate a joiner's own triggering commit). Failures propagate to every covered caller, so acceptance is never reported for a write that did not commit. Bypassed in join_transaction mode, where coalescing across open transactions would bind one caller's nudge to another's commit.

Observability: all three materialization paths now stamp threadmill.cron.origin = schedule / nudge / manual; JobExecutionContext.cronOrigin() reads it. The dashboard's triggerRecurring stays the separate operator force lane, unchanged.

Storage

• Postgres: additive migration V5__cron_state_nudge.sql (nudge_requested_at TIMESTAMPTZ, deliberately unindexed so nudge writes stay HOT-eligible). Accept is one guarded UPDATE … FROM statement on the hot path.
• Redis: a field in the schedule-state hash; accept and compare-and-clear are single Lua scripts; the state-upsert script preserves the field across its overwrite.
• New JobStore SPI: requestCronNudge(name, at) → ACCEPTED | UNKNOWN_TASK | DISABLED, clearCronNudge(name, observed). CronTaskScheduleState gains a read-only nudgeRequestedAt component. TracingJobStore and ForwardingJobStore updated.

Tests

Every acceptance bullet from the issue is a deterministic test (durable state + explicit materializer.tick(now) — no timing-sensitive signal path exists):

• Contract suite (identical on in-memory, Postgres, Redis): nudgeRoundTripsAndSurvivesBlanketStateUpserts, nudgeOnUnknownTaskIsRejectedWithoutResurrectingState, nudgeOnDisabledTaskIsRejected, clearCronNudgeOnlyClearsTheObservedValue.
• SchedulingTest: nudgeMaterializesPromptlyWithoutTouchingTheSchedule, nudgeDuringAnInFlightRunProducesExactlyOneFollowUpAfterCompletion (burst during in-flight → exactly one follow-up), nudgeCoalescesIntoADueScheduledFire, nudgeInstanceTakesThePileUpGuardSoAScheduledFireWaits, nudgeUnknownTaskFailsLoudly, nudgeDisabledTaskFailsLoudlyAndDoesNotRun, reEnablingATaskClearsAPendingNudgeFromBeforeThePause.
• NudgeCoalescerTest: burst-against-in-flight collapses to one follow-up write; failure propagates and the slot recovers.
• Spring: nudgeInsideTransactionTakesEffectOnlyAfterCommit, nudgeInsideRolledBackTransactionIsDiscarded, nudgeValidationFailsFastInsideTheTransactionNotAtCommit, and (Testcontainers, real Postgres) nudgeJoinsTheCallerTransactionAndRollsBackWithIt.
• Dashboard: manualTriggerStampsTheManualOriginMarker, updateRecurringEnabledFlipClearsAPendingNudge.
• Migration-count regression tests updated for V5.

Docs: new "Nudging a recurring task (wake-driven pollers)" section in docs/transactions.md, docs/handlers.md (cronOrigin()), docs/postgres-schema.md, Spring README, CHANGELOG, and AGENTS.md (vocabulary, design decision, regression-matrix rows).

Full ./gradlew check green, including the Testcontainers-backed Postgres and Redis suites.

[hemju]

Deep review of all 31 changed files at 23fa69a, including the issue design, all three stores, core scheduling, Spring transaction modes, dashboard/API, metrics, tests, and docs.

I would not merge this revision yet. The main blockers are:

• the timestamp-based compare-and-clear token can erase a newer accepted nudge;
• materialization can consume a nudge using a task definition captured before its mutex;
• enabled flips and task lifecycle changes are not linearizable/crash-safe;
• PostgreSQL's slow path can poison a joined host transaction;
• the promised dashboard + metrics origin visibility is not implemented;
• the in-memory implementation can return ACCEPTED and then lose the request during delete/re-register ABA.

Important follow-ups are failure atomicity versus the hard coalescing guarantee, coalescer leader starvation, and the breaking JobStore SPI addition.

CI and the affected local suites are green, but the problematic paths are race/failure interleavings not exercised by the current tests. Suggested deterministic regressions: same persisted token around read → materialize → second nudge → clear; task edit between list and mutex; nudge versus enable/disable/delete/re-register; PostgreSQL no-state deletion under join_transaction; failure injection after insert/state/clear; real cross-client Postgres/Redis execution; dashboard/Micrometer origin visibility; and sustained coalescer arrivals.

[hemju]

[P1] Use a unique generation, not the timestamp, as the CAS identity. This assumes every later accepted nudge has a different/newer stored timestamp. Scheduler uses Instant.now(), Redis truncates it to epoch milliseconds, PostgreSQL has finite timestamp resolution, and in-memory accepts identical caller-supplied instants. If nudge B is accepted with the same stored value after materialization but before clearCronNudge, this clear erases B even though the inserted job predates it, violating run-after-wake. Keep requestedAt for observability, but compare-and-clear a store-generated monotonic revision or collision-free nonce; add a same-token contract regression.

[hemju]

Fixed — the CAS identity is now a store-generated nudge_revision: strictly monotonic, advanced on every acceptance, never reset (a clear removes only the pending flag), so no two acceptances ever share an identity even at identical timestamps. clearCronNudge(name, observedRevision) compares the revision; nudge_requested_at is retained for observability only. Postgres generates it via COALESCE(nudge_revision, 0) + 1 in the single conditional statement, Redis via HINCRBY in the accept script, in-memory under the lifecycle lock. Same-token contract regression added on all three backends: nudgeAcceptancesWithIdenticalTimestampsAreDistinguishable (same-instant double accept; the first observation's clear must not erase the second).

[hemju]

[P1] Reload the task after acquiring its mutex. tick() snapshots each CronTask before tickOne takes the per-task mutex. Definition A can therefore be listed, definition/state B can be committed under the mutex, and a nudge for B can be accepted before this line executes with the stale A object. This inserts A's handler/payload and then clears B's request. Reload findCronTask, recheck enabled, and validate/repair the timing fingerprint after locking and before materialization.

[hemju]

Fixed — tickOneLocked now reloads findCronTask under the mutex and rechecks enabled before any materialization (nudge or scheduled), so a stale listed snapshot can never be inserted nor consume a nudge made against the new definition. The reload happens only when a materialization is imminent, keeping idle ticks at one state read per task. Deterministic regressions simulate the exact interleaving with a stale-listing ForwardingJobStore: materializerReloadsTheDefinitionUnderTheTaskMutexBeforeActing (fresh handler materialized, not the listed one) and materializerRechecksEnabledUnderTheTaskMutexBeforeActing. One scope call: I did not add fingerprint validation/repair here — a mismatched fingerprint is the pre-existing crashed-edit signature and choosing skip-vs-recompute changes scheduled-fire behavior beyond this PR; happy to take that as a follow-up issue.

[hemju]

[P1] Make the enabled flip and nudge reset atomic/crash-safe. A nudge can commit after existingState was read but before disable commits, so this only clears the older observation. In the other direction, a crash after upsertCronTask(enabled=true) but before this clear/recompute leaves stale pre-pause demand executable; retry sees the task as already enabled and loses the flip information. This needs a store-level transition or persisted definition generation. For re-enable, a safe ordering is to clear/recompute while the task is still disabled and enable it last.

[hemju]

Fixed via crash-safe ordering rather than a new store transition. Re-enable now clears the pending nudge and writes the recomputed state WHILE THE TASK IS STILL DISABLED, and flips enabled last — a crash anywhere before the final task write leaves the task disabled, so the retry re-detects the flip and repeats; stale pre-pause demand can never become executable. Disable persists the disabled definition first; a crash before the clear leaves the nudge parked on a disabled task, which the materializer's new under-mutex enabled recheck refuses to run and the re-enable path clears. The nudge-after-state-read race in the disable direction resolves the same way (parked, unrunnable, cleared before re-enable); in the enable direction a post-read nudge can only have been accepted once the task was already enabled, which makes it legitimate new demand. I chose ordering over a persisted definition-generation because it needs no SPI growth — if you want lifecycle generations as a general mechanism, I'd rather design that in its own issue.

[hemju]

[P1] Determine the flip from state read inside the mutex. existing is loaded before withTaskMutex. Two concurrent enable requests can both read disabled; the first enables, a producer's nudge is accepted, and the second later enters the mutex and clears that legitimate post-enable request because its snapshot is stale. Reload/build the effective task and decide the flip inside the mutex, ideally through the same atomic transition used by Scheduler.

[hemju]

Fixed — updateRecurring now loads the existing definition, builds the effective task, reads prior state, and decides the flip entirely INSIDE withTaskMutex (only a fail-fast not-found probe remains outside). Two concurrent enable requests serialize on the mutex and the second sees enabled=true, so it no longer misclassifies itself as a flip and cannot clear a legitimate post-enable nudge. The same enable-last / disable-first ordering as Scheduler.upsertCron applies (see the sibling thread).

[hemju]

[P1] Do not translate this FK violation without restoring the transaction. PostgreSQL has already marked the transaction aborted when 23503 is caught. Under join_transaction, this is the caller's host connection, so returning UNKNOWN_TASK leaves unrelated business writes doomed to roll back. The preceding SELECT enabled can also race a disable before the insert. Prefer error-free conditional SQL (INSERT … SELECT … WHERE enabled) or locking; if an exception remains necessary, use a savepoint and roll back to it before translating.

[hemju]

Fixed — the accept is now one error-free conditional statement: INSERT INTO threadmill_cron_task_state (…) SELECT t.name, ?, 1 FROM threadmill_cron_tasks t WHERE t.name = ? AND t.enabled ON CONFLICT (task_name) DO UPDATE SET …. Sourcing the insert from the parent row itself makes an FK violation unreachable, and the ON CONFLICT arm is the plain-UPDATE hot path (columns still unindexed, HOT-eligible), so no exception is ever caught inside a joined host transaction. Zero rows → a follow-up SELECT enabled decides UNKNOWN vs DISABLED; if a racing flip re-enabled the task between the two statements, the conditional write is retried (bounded at 3) instead of misreporting. The disable-races-insert window is gone with it — the enabled check and the write are one statement. Covered by the join-mode Testcontainers test plus the contract suite on real Postgres.

[hemju]

[P1] This observability claim is not implemented under normal configuration. The PR stamps metadata, but threadmill-metrics still uses untagged processed/failed/timing meters; JobSummary has no origin field; and secure-default dashboard reads redact metadata entirely. The UI only dumps raw metadata for privileged detail reads. Please add a non-sensitive origin field/badge plus a bounded-cardinality origin=schedule|nudge|manual|none metric dimension, with API/UI/Micrometer tests, or narrow this claim.

[hemju]

Implemented rather than narrowed, with one honest exception. Added: JobSummary.cronOrigin — a closed three-value field deliberately visible on redacted read-level views (nothing outside schedule|nudge|manual passes the filter, so arbitrary metadata can never leak through it; test redactedJobSummariesStillCarryTheCronOrigin) — and Micrometer threadmill.jobs.recurring.runs{origin=schedule|nudge|manual|other} with unrecognized values clamped to other (test recurringRunsCounterTagsTheTriggerOriginWithBoundedCardinality). The RecurringTaskView also now carries the pending nudgeRequestedAt/nudgeRevision via the embedded schedule state. The CHANGELOG states exactly this and notes the shipped React UI does not render the new field yet — I kept the npm-built UI out of this PR deliberately; say the word if you want the badge included here.

[hemju]

[P1] This name-only recheck does not prevent delete/re-register ABA. deleteCronTask removes the task and state from separate maps. A deletion can remove the old task, a same-name replacement can be registered, this request can write its nudge, and the old deletion can then remove that state; containsKey sees the replacement and returns ACCEPTED even though the request was lost. Make task/state lifecycle operations linearizable with one lock or generation-tagged entries.

[hemju]

Fixed — cron task/state lifecycle mutations (task upsert, delete, state upsert, nudge accept, nudge clear) are now linearized under a single cronLifecycleLock; the name-only recheck/undo is gone. The existence+enabled check and the nudge write are one atomic step, so the delete/re-register ABA can no longer strand an ACCEPTED nudge — and, incidentally, a slow delete can no longer erase a same-name re-registration's fresh state either (the same pre-existing ABA, one level up). Reads stay lock-free on the concurrent maps; these are control-plane operations, so the lock adds no data-path contention.

[hemju]

[P2] The public at-most coalescing bound is stronger than these failure semantics. Job insert, state upsert, and compare-clear are independent durable operations. A crash/ambiguous outage after insert leaves the nudge pending, so recovery can insert another job; repeated partial failures can exceed ‘current run plus one follow-up.’ Either combine insertion, state advancement, and generation clearing into an atomic store operation, or qualify the bound as failure-free/best-effort under Threadmill's at-least-once model.

[hemju]

Qualified everywhere rather than made atomic: RecurringMaterializer class Javadoc (new "Nudge failure semantics" paragraph), Scheduler.nudgeRecurring Javadoc, docs/transactions.md, CHANGELOG, and AGENTS.md now state the bound is failure-free — a crash or outage between the follow-up's insert and the request's clear produces an extra run, never a lost one, consistent with at-least-once. Plus a deterministic failure-injection regression: aFailedNudgeClearProducesAnExtraRunNeverALostOne (clear throws once → nudge stays pending → exactly one extra run after the instance terminates, then the retried clear lands). I kept the three independent writes: a combined atomic materialize-and-advance-and-clear store operation would be significant SPI surface to close a window whose worst case is one duplicate run of an already-required-idempotent handler.

[hemju]

[P2] Sustained traffic can retain the original caller indefinitely. The first caller synchronously drives every promoted generation; as long as each store round trip collects another follow-up, drive never returns, even though this caller's own future completed after the first write. That can retain an application request or Spring afterCommit thread until a lull. Transfer leadership to a waiter or decouple the serialized driver from producer threads, and add a continuous-arrival fairness test. Also note that the current scope is per Scheduler instance, not per JVM.

[hemju]

Fixed — drive now executes exactly one generation on the calling thread; a promoted follow-up is handed to a dedicated virtual thread (threadmill-nudge-coalescer), so no application request or Spring afterCommit callback is ever retained past its own covering write, no matter how sustained the arrivals. Fairness regression added: theFirstCallerIsNotRetainedToDriveFollowUpGenerations — the first caller joins successfully while generation 2's write is still latched open and a generation-3 joiner is already queued. The Javadoc now states the per-scheduler-instance (not per-JVM) scope explicitly and why that only loosens the write-rate bound, never correctness.

[hemju]

[P2] This is a breaking public SPI addition. Existing third-party JobStore implementations no longer compile until both methods are added, and old binaries can fail with AbstractMethodError when nudging. If that is intentional for the pre-1.0 release, mark it explicitly as breaking and provide migration guidance; otherwise consider a versioned capability/default compatibility strategy.

[hemju]

Documented as explicitly breaking — the CHANGELOG now carries a dedicated "Breaking (SPI)" entry with the failure mode for stale binaries (AbstractMethodError on nudge) and the implementation contract for store authors: atomic existence+enabled check on accept, store-generated monotonic never-reset revision, revision-compare-and-clear that leaves the revision untouched, upsertCronTaskState preservation of both nudge fields, and no state resurrection for deleted tasks — all pinned by the shared contract suite. I deliberately kept the methods abstract rather than defaulted: a throwing/no-op default would compile everywhere and then silently break run-after-wake on third-party stores at runtime, which is strictly worse than a loud compile error pre-1.0.

[hemju]

Review round 1 addressed in f6960c1 — all ten findings were valid; each thread has a detailed reply. The load-bearing changes: compare-and-clear now uses a store-generated never-reset nudge_revision instead of the timestamp; the materializer reloads the definition under the task mutex before acting; enabled flips are crash-safe by ordering (re-enable flips enabled last); the Postgres accept is a single error-free INSERT…SELECT…ON CONFLICT (no FK catch inside a joined host transaction); in-memory cron lifecycle is linearized under one lock; origin visibility is actually implemented (JobSummary.cronOrigin on redacted reads + threadmill.jobs.recurring.runs{origin=…}); the coalescer never retains a caller past its own write; the failure-free coalescing bound and the breaking SPI addition are documented as such.

Every suggested deterministic regression is in: same-persisted-token clear race, task-edit/disable between list and mutex (stale-listing store), nudge vs enable/disable/delete/re-register, clear-failure injection, dashboard + Micrometer origin visibility, and sustained coalescer arrivals — plus the join-transaction rollback test on real Postgres. Full ./gradlew check green including the Testcontainers suites.

[hemju]

6h dual-backend endurance: passed

nudge-pump scenario, 6h × 50 jobs/s × 4 producers × 3 nodes, node churn every 10 min, against real PostgreSQL 18 and Redis 7 (AOF, noeviction) in parallel.

	Postgres	Redis
Verdict / exit	passed / 0	passed / 0
Enqueued / succeeded	1,510,426 / 1,531,722	1,552,970 / 1,574,346
Failed	4	0
e2e p99	45 ms	43 ms
Failed invariants	—	—

All three invariants (atLeastOnce, nudgeRunAfterWake, outboxDrainedByLaterRun) passed on both backends.

Feature-specific numbers: 755,219 → 21,296 nudge-to-pump-run on Postgres (35:1) and 776,491 → 21,376 on Redis (36:1), i.e. coalescing held under sustained load and master handover. Exactly 36 schedule-origin runs per backend — the 10-minute backstop firing precisely on grid across 6 hours while nudges carried all the real work, which is the wake-driven shape doing what it claims.

Wall clock was 360 min with zero execution gaps >60 s on either backend, so this was an uninterrupted 6 hours (the earlier attempt died with the laptop battery and is not comparable). 35 churn cycles completed. The 4 Postgres failure events are the documented shutdown-release shape (attempts: 0, final: false, all retried, atLeastOnce clean) from a churn cycle.

Round-2 review fixes (074379d) are included in this run.