fix(sandbox): publish the L2 archive without rename, for a blob store

[pedrofrxncx]

The shared store for the L2 cache (#5351) is going to be S3 via the mountpoint CSI driver, not EFS — that driver is already installed on eks-serverless with IAM scoped to every bucket, so it needs no new addon, and it's ~13× cheaper per GB.

Mountpoint has no rename. Publish wrote <key>.tmp.<pid> and renamed, so as written L2 could never populate.

Change

Write straight to the final key.

Nothing is lost by dropping the rename: an object becomes visible only once its upload completes, so a publisher killed mid-write leaves no readable object — exactly the guarantee the rename was buying. The comment says so inline, and says what a POSIX-backed store would need instead, because there a partial file is visible to a concurrent reader.

The read-back verification stays, and matters more now: it's the only thing between a malformed archive and a permanently broken key, since publish no-ops once the key exists and every node would then keep missing forever with nothing to repair it.

mkdir of the key prefix becomes best-effort — a blob store has no real directories, and writing the key is what creates the prefix.

What it costs

The read-back is now a network read of the archive rather than a local one. The alternative was staging the archive in the pod first, which would spend the tenant's /app quota on a file that can be several hundred MB — and a full workdir breaks their sandbox. A slower publish is the better trade: it runs after the boot is healthy, off the critical path.

Restore is untouched

Its staging-then-rename is on the pod's local disk, not the store, and has to stay. I dropped that import while removing the store-side one, and the ~9 MB round-trip test from #5351 caught it immediately (restore skipped: rename is not defined) — which is a decent argument for that test having been worth writing.

Also unaffected

utimes on restore was already best-effort (.catch(() => {})), so it degrades silently on mountpoint. That does mean the mtime-touch no longer protects an in-use archive from the TTL sweep, which is why the bucket's lifecycle rule is set to 14 days — comfortably longer than L1's 7-day TTL, so a lockfile still in use is re-published before its object expires. That's in the terraform PR.

pruneRemoteGoldens still works: mountpoint supports prefix listing and synthesises mtime from LastModified. Deletes need allow-delete, which only the publisher mount has — on the tenant's read-only mount the prune is a harmless no-op.

Testing

bun test packages/sandbox/daemon/setup/ → 98 pass. knip clean, lint 0 errors.

Still not validated against a real mountpoint mount — that needs the bucket, which is the terraform PR. What's verified here is that the publish/restore round trip still holds on a POSIX path, including the large-tree backpressure case.

---

Summary by cubic

Fixes L2 archive publishing for the S3 mountpoint store by writing directly to the final key, unblocking cache population on eks-serverless. Safety is kept via read-back verification; no temp+rename is used.

• Bug Fixes
  • Write archive to the final key (publishRemoteGolden); drop temp+rename unsupported on mountpoint.
  • Read back the just-published archive; delete and abort if it fails to parse to avoid poisoning the key.
  • Make prefix mkdir best-effort; blob stores create the prefix on write.
  • Restore remains unchanged (stage then rename on local disk).
  • Trade-off: read-back is now a network read, but publish runs post-boot off the critical path.

^{Written for commit acbd7d2. Summary will update on new commits.}

[pedrofrxncx]

Settled the one mountpoint limitation that could have made this not work at all.

Mountpoint does not support random writes. If zstd -o <path> seeked back to patch its frame header, the publish would fail on the mount no matter what else was right. Tested rather than assumed, using the fact that a FIFO cannot be seeked:

$ mkfifo out.fifo && (cat out.fifo > captured.tar.zst) &
$ tar -cf - -C src node_modules | zstd -3 -T0 -q -o out.fifo
zstd -o FIFO: OK   → output is append-only, no seek
captured: 3001824 bytes

So the write is a pure sequential append, which is exactly what mountpoint does support. No code change needed for it.

What is still unproven, and why it fails safe. The read-back verification now reads through the mount immediately after closing the object, so it depends on mountpoint's metadata cache serving the just-written key. I can't test that without a real mount.

If it turns out flaky, the failure direction is the safe one: a spurious read-back failure deletes a good archive and the next boot re-publishes it. The result is a wasted publish, never a poisoned key — which is the opposite of the failure this verification exists to prevent. So a caching surprise costs throughput, not correctness.

Real gate remains: apply the terraform in stg, mount it, publish once, restore once.