Skip to content

feat(lattice): add durable runtime stores - #4369

Open
mgazza wants to merge 1 commit into
feat/lattice-atomic-autoconfig-materializerfrom
feat/lattice-durable-runtime-stores
Open

feat(lattice): add durable runtime stores#4369
mgazza wants to merge 1 commit into
feat/lattice-atomic-autoconfig-materializerfrom
feat/lattice-durable-runtime-stores

Conversation

@mgazza

@mgazza mgazza commented Jul 29, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • add exact filesystem CAS stores for fragment adapter state and compiled Lattice publications
  • add a secret-free materializer checkpoint store with a separately durable monotonic fence allocator
  • add a target-owned opaque journal with bounded bytes, revision/fence checks, and payload-hiding diagnostics
  • use a strict versioned/tagged allowlist codec with integrity, type, size, depth, node and container bounds
  • provide thread/process exclusion, private filesystem invariants, atomic replace, file/parent fsync, and exact post-replace readback

Scope

This persistence-only tranche is stacked on #4359.

It does not implement WholeConfigTarget, construct or drain the compiler, register a runtime component, touch COMPONENT_LIST, mutate live args/configuration, write apps.yaml or secrets.yaml, or wire any provider. The two new files have no production registration.

Validation

  • complete targeted matrix: 125/125 on Python 3.9 and 125/125 on Python 3.14
  • new focused durable-store suite: 6/6 per interpreter
  • all existing autoconfig tests: 53/53 per interpreter
  • existing fragment adapter suite: 25/25 per interpreter
  • existing compiled publication suite: 19/19 per interpreter
  • existing atomic materializer suite: 22/22 per interpreter
  • targeted pre-commit: whitespace, EOF, Ruff, Black and CSpell
  • git diff --cached --check
  • GitNexus staged scope: MEDIUM, 72 symbols, four new internal persistence flows; no production compiler/runtime flow
  • final independent correctness review: APPROVE

Safety invariants

  • existing HIGH-reach persistence protocols are not modified
  • missing state is distinct from corrupt/torn/oversized state
  • every decoded object is reconstructed only from an explicit enum/dataclass allowlist and revalidates domain invariants
  • bool, integer and float encodings are non-interchangeable
  • CAS compares exact canonical state under a process/thread lock
  • a fence is durably advanced before it is returned
  • record and lock files must be current-euid, mode 0600, single-link regular files in a non-symlink, current-euid, non-group/world-writable directory
  • zero/short writes, file fsync, parent fsync, replace ambiguity and exact-readback failures fail closed; restart can recover an exact landed replacement
  • coordinator state retains digests and bounded opaque handles, not raw feedback/config secrets
  • opaque target bytes are filesystem-protected rather than encrypted; their later schema remains target-owned
  • persisted compiled publications are revalidated against compiler-equivalent target, topology,
    projection, readiness, field and provenance invariants before use
  • provider-declared non-monotonic projection order is preserved while slot uniqueness and bounds
    remain enforced

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant