feat(schedule): narrow recurring overlap windows

[hemju]

Closes #110.

Threadmill is at-least-once by design, but two of the three windows in which a recurring task can briefly run twice were engineering artifacts rather than distributed-systems necessities. Both are closed here; the third is documented as unclosable.

The issue analysis and the reporter's reply settle the shape: item 3 as proposed, item 2 via a cheaper mechanism than the one proposed, item 1 declined on the merits with a regression test taken from it.

Exclusive recurring tasks (item 3)

CronTask gains an exclusive flag, surfaced as @Recurring(exclusive = true) and as an exclusive parameter on Scheduler.defineRecurring. Every materialized instance — scheduled, caught-up, or triggered by hand from the dashboard — is claimed under the derived key recurring:<name> in ConcurrencyMode.EXCLUSIVE, so the store refuses to admit a second instance while one is processing.

@Job
@Recurring(interval = "PT1M", exclusive = true)
public class NightlySweep implements JobAction { … }

The key is derived, never user-supplied, so the recurring: namespace cannot collide with an application's own keys; names past the 256-UTF-8-byte cap truncate on a code-point boundary with a stable hash suffix.

The reason this is worth more than a materializer-side check: it moves the guarantee from what the maintenance leader decides to materialize to claim-time admission enforced by every store on every node. That is what lets it also cover a dashboard manual trigger racing a scheduled instance, and the retry-handoff window below.

It does not close the lease-expiry reclaim window — reclaim releases the concurrency slot as part of the terminal failure save. Documented on the feature, on the annotation, and in docs/transactions.md.

Retry-handoff window (item 2)

JobState.FAILED.isTerminal() is deliberately false — a retry may follow — but the pile-up guard treated every FAILED as finished, so a materializer tick landing between the failure save and RetryInterceptor's reschedule save could create a fresh instance beside a retrying one.

A FAILED instance now blocks while its retry budget is not provably spent and the failure is younger than a 5s handoff grace. Both halves are load-bearing, as the reporter argued: the budget test keeps the common retry-exhausted failure from delaying the next run at all, and the age bound keeps an instance that is terminal under a per-exception-type policy — which the guard cannot read, since those policies live on the interceptor — from blocking its task until recoverStrandedFailures reaches it.

The issue proposed making FAILED → SCHEDULED atomic instead. Rejected: it turns JobInterceptor.onProcessingFailed into a decision-returning hook (breaking SPI) and requires legalizing PROCESSING → SCHEDULED plus non-terminal concurrency-slot release in all three backends — to close a window the guard closes in one file.

Shutdown requeue (item 1)

Declined as specified; the premise does not hold. close() drains via workerPool.shutdown() + awaitTermination(grace) before shutdownNow(), and the requeue is emitted from recordFailure on the handler's own worker thread, strictly after handler.run returned or threw — no peer can claim the job while user code is still executing. The reporter confirmed this was inferred from a comment rather than observed.

That ordering was implicit in where recordFailure is called. It is now pinned by ProcessingNodeTest.shutdownRequeueIsNeverPublishedWhileTheHandlerIsStillRunning, which asserts at the exact moment of the SCHEDULED write that the handler has left run — plus a positive check that the write was actually observed, so the test cannot pass vacuously.

Storage

• Postgres: additive V6__cron_task_exclusive.sql (BOOLEAN NOT NULL DEFAULT FALSE). Renumbered from V5 in the merge — v0.2.0 shipped V5__cron_state_nudge.sql.
• Redis: a cron-task hash field with the existing overwrite-on-re-upsert semantics, so an override-less re-registration clears the flag.
• In-memory: persists the record directly.
• DashboardPayloads.CronTaskView gains an exclusive component.

Tests

• AbstractJobStoreContractTest.cronTaskExclusiveRoundTripsAndClearsOnReUpsert — runs against all three backends.
• SchedulingTest: exclusiveRecurringInstancesAreSerializedByClaimTimeAdmission, nonExclusiveRecurringInstancesCarryNoConcurrencyKey, plus the three guard cases (failedInstanceAwaitingItsRetryBlocksTheNextMaterialization, retryExhaustedFailedInstanceDoesNotBlockTheNextMaterialization, failedInstanceStopsBlockingOnceTheRetryHandoffGraceElapses).
• CronTaskTest.derivedConcurrencyKeyFitsTheKeyCapWithoutSplittingSurrogatePairs.
• DashboardApiServiceTest.updateRecurringPreservesTheExclusiveFlag + manualTriggerOfAnExclusiveTaskCarriesTheDerivedConcurrencyKey.
• ThreadmillAutoConfigurationTest.exclusiveRecurringHandlerLandsOnTheRegisteredCronTask.
• ProcessingNodeTest.shutdownRequeueIsNeverPublishedWhileTheHandlerIsStillRunning.

Two traps worth recording, both now in AGENTS.md:

• A guard written as if (!current.isTerminal()) return true; swallows the FAILED case entirely, because FAILED.isTerminal() is false. Check == FAILED first.
• Spring has two recurring registration paths — the namespaced one via taskFor + reconcileRecurring, and the un-namespaced one calling Scheduler.defineRecurring directly in ThreadmillRecurringRegistrar.registerAll. The first cut wired only the former; the auto-config test caught it. Any new CronTask field must go through both.

Docs

docs/transactions.md names all three overlap windows, states plainly that lease-expiry reclaim is not closable by any job system, and recommends fencing at the consumer's data (compare-and-set transitions, unique constraints, claim columns). docs/concurrency.md documents exclusive recurring tasks including the reclaim limitation.

./gradlew check green on the merge result, including the Postgres and Redis Testcontainers suites.