Skip to content

perf(storage): prototype compact PieceV2 storage - #292

Draft
Kubuxu wants to merge 11 commits into
mainfrom
feat/optimized-add-pieces-2
Draft

perf(storage): prototype compact PieceV2 storage#292
Kubuxu wants to merge 11 commits into
mainfrom
feat/optimized-add-pieces-2

Conversation

@Kubuxu

@Kubuxu Kubuxu commented Jul 28, 2026

Copy link
Copy Markdown
Collaborator

This is largely LLM code. I haven't yet fully reviewed it myself, so some LLM-isms might be there.
Published because results are encouraging.
I will do a detailed review before anyone else.


Summary

Draft implementation of the compact two-slot PieceV2 storage prototype, related to #286.

New compact-state datasets store pieces in a contiguous per-dataset array:

struct PieceV2 {
    bytes32 root;
    uint256 metadata;
}

metadata packs padding, tree height, leaf count, and the Fenwick partial sum into one storage slot.

Changes

  • Hardened canonical PieceCIDv2 validation and decoded (padding, height, root) in one pass.
  • Added compact PieceV2 storage while retaining the existing declarations for physical layout safety.
  • Moved piece addition, proof verification, lookup, getters, pagination, CID search, scheduling, removal, and cleanup to compact storage.
  • Preserved piece IDs, public ABI behavior, events, and listener callback arguments for compact-state datasets.
  • Added coverage for packing bounds, Fenwick semantics, proofs, pagination, removal, cleanup, and raw storage reclamation.
  • Measured steady-state addPieces storage activity for 1, 4, 16, and 32-piece batches.

Storage measurements

Each scenario creates a fresh dataset, adds one seed piece before recording, then records the measured addPieces call.

Batch EVM-slot reads EVM-slot writes KAMT touched KAMT modified Newly occupied slots
1 10 → 12 7 → 4 11 → 7 6 → 3 5 → 2
4 21 → 25 28 → 13 24 → 7 19 → 3 20 → 8
16 69 → 85 112 → 49 72 → 8 67 → 4 80 → 32
32 133 → 165 224 → 97 137 → 9 132 → 5 160 → 64

Values are legacy → compact.

Findings

  • Storage footprint: consistently 60% lower5N → 2N newly occupied slots.
  • EVM-slot writes: reduced from 7N to 3N + 1; reduction grows from 42.9% at one piece to 56.7% at 32 pieces.
  • KAMT locality: compact contiguous records substantially reduce object activity as batches grow:
    • 4 pieces: 24 → 7 touched, 19 → 3 modified.
    • 16 pieces: 72 → 8 touched, 67 → 4 modified.
    • 32 pieces: 137 → 9 touched, 132 → 5 modified.
  • EVM-slot reads: increase by 19.0–24.1%. These are repeated reads of already-accessed values, which are cached; they do not correspond to a comparable increase in KAMT object activity.
  • Scaling: the compact layout modifies two shared objects plus roughly one compact-data object per 16 pieces, subject to alignment. The legacy representation modifies approximately four piece-data objects per piece.

At 32 pieces, the compact layout occupies 64 slots instead of 160, touches 9 KAMT objects instead of 137, and modifies 5 instead of 132.

Compatibility note

This is a forward-only prototype. Backwards compatibility for datasets created before the compact representation was not considered.

Keep this PR as draft; it is not ready to merge or deploy as an upgrade.

Verification

make test
209 passed, 0 failed

Kubuxu added 11 commits July 28, 2026 15:51
Record storage reads, writes, KAMT object locality, and persistent slot growth for additions to an existing data set. This establishes the baseline for the compact piece representation work.

Signed-off-by: Jakub Sztandera <oss@kubuxu.com>
Document the compact PieceV2 design and split the prototype into sequential, agent-ready implementation digests with explicit invariants, acceptance criteria, verification, and handoff requirements.

Signed-off-by: Jakub Sztandera <oss@kubuxu.com>
Reject ambiguous and malformed CID encodings before they can be stored or used for proof verification.

Signed-off-by: Jakub Sztandera <oss@kubuxu.com>
Append the two-slot compact piece mapping and centralize bounded metadata packing before production paths adopt the new storage.

Signed-off-by: Jakub Sztandera <oss@kubuxu.com>
Store newly added pieces in compact slots with their final Fenwick sums, while batching the dataset leaf-count update.\n\nThis removes legacy addition writes ahead of migrating readers and removers.

Signed-off-by: Jakub Sztandera <oss@kubuxu.com>
Serve piece getters, pagination, CID search, and deletion scheduling from compact metadata while preserving canonical CID output and mapping-style getter defaults.

Signed-off-by: Jakub Sztandera <oss@kubuxu.com>
Adapt behavioral and raw-storage tests to compact piece records, including Fenwick sums and cleanup reclamation.

Signed-off-by: Jakub Sztandera <oss@kubuxu.com>
Run generation after layout cleanup so make cannot evaluate generated targets before their files are removed.

Signed-off-by: Jakub Sztandera <oss@kubuxu.com>
Measure isolated 1, 4, 16, and 32-piece additions and lock the observed storage activity. Document the compact baseline, slot reductions, and KAMT alignment behavior.

Signed-off-by: Jakub Sztandera <oss@kubuxu.com>
Signed-off-by: Jakub Sztandera <oss@kubuxu.com>
Keep the consolidated measurement comparison and align the four-piece KAMT baseline with the revised measurement method.

Signed-off-by: Jakub Sztandera <oss@kubuxu.com>
@FilOzzy FilOzzy added this to FOC Jul 28, 2026
@github-project-automation github-project-automation Bot moved this to 📌 Triage in FOC Jul 28, 2026
@Kubuxu Kubuxu self-assigned this Jul 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: 📌 Triage
Status: No status

Development

Successfully merging this pull request may close these issues.

2 participants