Skip to content

fix: fence artifact namespaces across restart - #456

Open
wchwawa wants to merge 2 commits into
mainfrom
fix/object-namespace-fence
Open

fix: fence artifact namespaces across restart#456
wchwawa wants to merge 2 commits into
mainfrom
fix/object-namespace-fence

Conversation

@wchwawa

@wchwawa wchwawa commented Aug 15, 2026

Copy link
Copy Markdown
Member

Related Issue

Fixes #450

Summary

  • Bind each metadata root to one immutable, provider-neutral object namespace across etcd routing, Holt fences, recovery records, server bootstrap, CLI, SDK, and Python clients.
  • Fail closed before metadata mutation when a healthy S3/RustFS endpoint points at the wrong bucket or prefix.
  • Preserve exact legacy fence and outbox bytes behind an explicit, one-way adoption command, and keep temporary object-read failures retryable and redacted.
  • Add independent real-etcd/RustFS recovery CI covering wrong-prefix rejection, E2 kill/retry, outage recovery, and a deterministic materials workflow.

The root cause was not RustFS or etcd health. Metadata ownership and artifact storage had no shared durable identity, so a restart could combine a valid metadata root with a different but healthy object prefix. This change makes that identity part of the authority contract at every mutation and recovery boundary.

Startup now requires a namespace-marker read. Existing roots require explicit operator-confirmed adoption, roots owned by one shard must use one namespace, and the workspace RPC protocol advances to v3.

Scope

  • This PR changes one logical boundary only.
  • No unrelated refactor, benchmark, metadata model, Holt layout, object-store, agent interface, or docs change is mixed in.
  • The linked issue describes the user-visible problem, design decision, or maintenance task this PR resolves.
  • Any breaking change is intentional and documented.
  • No compatibility shim, deprecated alias, or forwarding wrapper was added without a removal condition.

Code Contract (Code Changes Only)

  • Not applicable; this is a docs/config-only change.
  • Package boundaries follow docs/development/code_contract.md.
  • Shared helpers reuse the standard library or existing repository helpers. New generic helper modules are domain-neutral and tested.
  • File names and file placement follow the code contract.
  • New types, interfaces, structs, fields, and functions use domain-specific names.
  • New errors are in the owning package's errors.rs and carry stable error kinds when crossing package boundaries.
  • New metrics/stats are owned by the package that reports or serves them.
  • Metadata changes document durability, atomicity, revision-reference lifetime, snapshot/commit/event retention, GC epochs, and fail-closed recovery boundaries.
  • Placement or routing changes preserve persisted RootId -> LogicalShardId affinity, one active writer per shard, owner-epoch fencing, and explicit shard-local atomicity; no split root or cross-shard transaction is implied.
  • Agent-interface changes keep the transport-free facade and schemas in nokv-agent, routing and workflows in nokv-client, and the concrete backend plus MCP wiring in the nokv CLI.

Claims and Evidence

  • User-facing documentation distinguishes current, experimental, and planned capabilities.
  • Security claims do not treat a Workbench root or Agent scope as tenant authentication or authorization.
  • Performance claims state the topology, comparison boundary, cache state, run count, and raw-evidence location.
  • Benchmark-only behavior does not alter product semantics.

Validation

  • cargo fmt --all -- --check: pass
  • cargo clippy --workspace --all-targets -- -D warnings: pass
  • cargo test --target-dir /private/tmp/nokv-object-namespace-fence-target -q --workspace: pass
  • Workbench, recovery-gate, object-namespace-gate, and release-script contract tests: 55 pass
  • Real etcd E2 kill/retry before and after the local fence: pass; both recover at epoch 2 and never consume epoch 3
  • Real etcd + digest-pinned RustFS: wrong-prefix rejection, owner SIGKILL/reopen, temporary outage recovery, and materials structure/relaxation round trip pass
  • GitHub Actions object-namespace-recovery: pass on a clean Ubuntu runner with retained terminal evidence
  • GitHub Actions nokv-workspace: pass, including the independent real-etcd E2 fence gate and RustFS volume cleanup
  • GitHub Actions DCO, Workbench contract, and multi-architecture Docker image: pass

The first clean-runner attempt exposed a Linux-only harness bug before NoKV started: RustFS runs as non-root UID/GID 10001:10001, while the host bind directory was root-owned. A focused RED test now requires a Docker-managed volume and fail-fast container-exit diagnostics. The corrected gate passed without changing product recovery semantics.

Contributor Sign-off

  • Every commit in this PR includes a DCO Signed-off-by trailer.

Bind every root to one immutable provider-neutral object namespace across etcd routing, Holt fences, recovery records, CLI, SDK, and Python clients. Preserve legacy fence and outbox bytes behind an explicit one-way adoption path.

Keep temporary S3 read failures retryable and redacted, and add independent real-etcd/RustFS gates for wrong-prefix rejection, kill/reopen epochs, outage recovery, and a deterministic materials workflow.

Signed-off-by: wchwawa <wch19961116@gmail.com>
@feichai0017 feichai0017 moved this to In Progress in NoKV Delivery Aug 15, 2026
Signed-off-by: wchwawa <wch19961116@gmail.com>
@wchwawa
wchwawa marked this pull request as ready for review August 15, 2026 13:26
@wchwawa
wchwawa requested a review from feichai0017 August 15, 2026 13:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Standalone local-WAL deployment cannot restart after first owner exits (--metadata-reopen is always rejected as successor)

2 participants