[P2.18] Full test suite takes ~4h locally vs 4m38s on CI — per-test SQLite fsync

[frankbria]

Surfaced while implementing #975 (PR #977). This is why #975 survived: nobody runs the full suite locally when it takes four hours, so a permanently-red local suite went unnoticed.

Problem

The full non-e2e suite runs in 4m38s on CI and takes roughly 4 hours on a WSL2 developer machine — a ~47× gap on identical code and an identical command:

uv run pytest tests/ --ignore=tests/e2e -m "not lifecycle"

Measured on this repo at d84b1fc:

Scope	CI (backend-tests)	Local (WSL2)
Full non-e2e suite (4,421 tests)	4m38s	~4h (extrapolated; a 3h run reached 89%)
tests/core + 8 dirs (396 tests)	—	9m58s
tests/ui tests/unit tests/workspace tests/contract (697 tests)	—	52m49s

A gap that size is not "WSL2 is slower." It is a specific pathology.

Root cause: fsync, not CPU

During a full run the pytest process sits in D state (uninterruptible I/O wait) with 30 seconds of CPU across 62 minutes of wall time, and /proc/pressure/io reports:

full avg300=65.95     # every runnable task stalled on I/O 66% of the time

The driver is per-test workspace creation. create_or_load_workspace → _init_database creates ~20 tables plus indexes, and _open_db sets PRAGMA journal_mode = WAL (itself a write). Every commit fsyncs. WSL2's ext4-on-VHD makes each fsync enormously more expensive than on a GitHub runner's disk, and the suite does this once per test.

Evidence: two controlled measurements

Same tests, same machine, same commit — the only variable is whether pytest's temp dirs live on real disk or on tmpfs (/dev/shm, RAM-backed, where fsync is a no-op):

Scope	/tmp (real disk)	--basetemp=/dev/shm/...	Speedup
tests/core/test_workspace.py (23 tests)	166.21s	1.17s	142×
tests/core/test_workspace.py + test_worktree* + 8 dirs (397 tests)	598.89s	16.21s	37×

Identical pass/fail results in both configurations. That isolates the cost to fsync-on-disk with no ambiguity.

Extrapolating the 37× figure, the full suite would land in the 5–10 minute range locally — in line with CI.

Why this matters

• It hides real bugs. [P1.33] Isolate tests from the developer's ambient .codeframe/ workspace (cwd walk-up) #975 was four permanently-red tests on every developer machine that CI could not see. A four-hour suite means nobody runs it, so nobody noticed.
• It taxes every change. Pre-PR verification for [P1.33] Isolate tests from the developer's ambient .codeframe/ workspace (cwd walk-up) #975 consumed most of a working day of wall clock, and one 3-hour run had to be killed and restarted after hitting a timeout at 89%.
• It manufactures flakes. Sustained 66% I/O stall is what pushes the fixed 15s threading.Barrier deadline in [P1.34] Parallel-execution barrier timeout is load-sensitive — flakes under full-suite runs #976 over the edge. Fixing this likely fixes that too.

Reproduction

# Slow path (default)
uv run pytest tests/core/test_workspace.py -q          # ~166s

# Fast path (tmpfs)
mkdir -p /dev/shm/pytest-cf
uv run pytest tests/core/test_workspace.py -q --basetemp=/dev/shm/pytest-cf   # ~1.2s

Suggested fixes

1. Immediate, zero code change — document/enable a tmpfs basetemp.
A make test-fast target, a documented env var, or an opt-in addopts entry. Caveats worth stating: /dev/shm is RAM-backed (7.9G here — the run must fit), it is Linux-specific, and it should stay opt-in rather than becoming the default, since running on a real filesystem is what CI does and is the more faithful environment.

Note this composes correctly with the #975 ambient-workspace guard: pytest_configure already registers --basetemp as an isolated root, so a tmpfs basetemp does not trip it. Verified.

2. Structural, helps CI too — stop rebuilding the schema per test.
Build the workspace DB once per session into a template file, then shutil.copy it per test. A file copy is dramatically cheaper than ~20 CREATE TABLEs plus indexes plus fsyncs, and it cuts CI time as well as local time. This is the real fix; option 1 is a workaround.

3. Consider, with care — relax durability for test DBs only.
PRAGMA synchronous=OFF / journal_mode=MEMORY would remove the fsync cost directly, but it requires an env-gated switch inside core/workspace._open_db, i.e. touching production code to serve tests. Only worth doing if option 2 proves insufficient, and it must not weaken durability on any non-test path.

Acceptance criteria

• Full non-e2e suite completes locally in a time comparable to CI (target: under ~15 minutes on a WSL2 dev machine)
• Pass/fail results are identical to the current slow configuration — no tests skipped or weakened to hit the number
• If the fix is opt-in (option 1), it is discoverable: documented in CLAUDE.md / contributor docs, not tribal knowledge
• Production durability semantics are unchanged on every non-test path
• Re-check [P1.34] Parallel-execution barrier timeout is load-sensitive — flakes under full-suite runs #976 afterwards — the barrier flake may disappear once I/O pressure drops