[P1.34] Parallel-execution barrier timeout is load-sensitive — flakes under full-suite runs

[frankbria]

Surfaced while implementing #895 (PR #974). Not caused by that change.

Problem

tests/core/test_conductor.py::TestParallelExecution::test_parallel_independent_tasks_run_concurrently fails intermittently under a loaded full-suite run, and passes in isolation.

Observed in a full CI-gate run (uv run pytest tests/ --ignore=tests/e2e -m "not lifecycle", 4372 passed / 5 failed, 4h12m wall clock). Re-running the test alone: passes.

Mechanism. The test proves real concurrency with a barrier rather than sleeps:

barrier = threading.Barrier(n, timeout=15)   # n = 3 independent tasks

def mock_execute(ws, tid, batch_id=None, **kwargs):
    ...
    barrier.wait()      # blocks until all N task bodies are in-flight together
    return "COMPLETED"

A serial executor can never get N bodies to the barrier at once, so it trips the timeout — that is the whole point, and it is a good design. But the budget is a fixed 15 seconds, and it is spent on the machine's ability to schedule N worker threads, not on the code under test. During a 4-hour run with heavy CPU/IO contention, all N workers may not be simultaneously scheduled inside 15s → BrokenBarrierError → false red.

So the failure mode is "the box was busy", not "execution was serial".

Why this matters

A flake in the concurrency-proof test is especially costly: red here reads as "parallel execution is broken", which is alarming enough to stop work and investigate. During #895 it consumed a diagnosis cycle before being cleared as load-induced.

Constraint — do NOT weaken the assertion

This test was deliberately hardened in #773. Its docstring records that the previous version asserted only completion and "would pass green even if execution were fully serial (the exact no-op-assertion this batch targets)", and that sleep-based overlap detection was rejected on purpose as timing-dependent.

Any fix must keep the barrier proof intact. Removing the barrier, dropping assert len(threads_seen) == n, or reverting to a completion-only assertion re-introduces the #773 defect and is not an acceptable fix.

Suggested direction

Raise/parameterise the scheduling budget without weakening the proof — e.g. a longer default timeout, or one scaled by an env var so CI and loaded local runs get more headroom. The barrier still fails closed for genuinely serial execution; it just stops failing for a busy scheduler. Distinguishing BrokenBarrierError (scheduling) from a wrong-result assertion in the failure message would also make future reds self-explaining.

Evidence

• tests/core/test_conductor.py — TestParallelExecution::test_parallel_independent_tasks_run_concurrently, threading.Barrier(n, timeout=15)
• Full-suite run: this test among 5 failures; passes standalone in ~3.9s

Acceptance criteria

• The test no longer fails under a loaded full-suite run (verify by running the full non-e2e gate on a contended machine)
• The concurrency proof is preserved: a deliberately serialised executor (e.g. max_parallel=1) still fails the test
• assert len(threads_seen) == n and the barrier remain — no reversion to completion-only assertions ([P3.2] Test-trust cleanup: re-enable/fix stale-skipped suites and no-op assertions — batch fix #773)
• A scheduling timeout failure reports a message distinguishing it from a genuine serial-execution failure

---

Atomic by construction: one developer, one focused session.

[frankbria]

Independently re-observed while implementing #975 (PR #977), with additional data that narrows the cause.

It is not caused by PR #977. Sampled TestParallelExecution 4× per branch, sequentially, with nothing else running:

Branch	Result
main	4/4 runs — 7 passed
fix/975-isolate-tests-ambient-workspace	3/4 runs — 7 passed; 1 run — 1 failed

Counts alone are ambiguous at that sample size, but the code path settles it: the conductor resolves its DB via get_db_connection(workspace) → Workspace.db_path → self.state_dir / STATE_DB_NAME (core/workspace.py:57-60). state_dir is a stored dataclass field, resolved once when the fixture builds the workspace — before the test body. PR #977's guard wraps _get_state_dir, which the conductor never calls during execution, so it adds zero syscalls inside the barrier window.

Corroborating detail: a different test in the class (test_parallel_handles_failure) failed on a separate isolated run while the barrier test passed. A deterministic interaction would fail the same test consistently; load-induced flake moves around.

Probable upstream cause — see #979. The contention this test loses to is measurable: /proc/pressure/io reports full avg300=65.95 during a full local run (every runnable task stalled on I/O 66% of the time), driven by per-test SQLite schema init with WAL fsyncs. #979 has a measured fix path — moving pytest's basetemp to tmpfs made an identical 397-test subset go from 598.89s to 16.21s.

Suggestion: re-measure this issue after #979 lands. The 15s budget may stop being tight once the machine is not I/O-starved, which would be a better outcome than redesigning the timeout.

[frankbria]

Second instance of the same root cause, seen while working #898: tests/core/test_config_reload_integration.py::TestConfigReloadLifecycle::test_full_reload_cycle also fails under sustained full-suite load and passes in isolation.

It starts a ConfigFileWatcher(poll_interval_s=0.1) and waits a bounded time for it to notice an AGENTS.md write — the same shape as the parallel barrier here: a short wall-clock deadline that holds on CI (4m38s total) but not after an hour of fsync-saturated local IO (#979).

Both failed in ~70-minute local runs and passed in isolation immediately after (verified twice). Worth fixing together — whatever deadline/synchronisation approach lands for the barrier likely applies to the watcher too.