perf(storage): prototype compact PieceV2 storage - #292
Draft
Kubuxu wants to merge 11 commits into
Draft
Conversation
Record storage reads, writes, KAMT object locality, and persistent slot growth for additions to an existing data set. This establishes the baseline for the compact piece representation work. Signed-off-by: Jakub Sztandera <oss@kubuxu.com>
Document the compact PieceV2 design and split the prototype into sequential, agent-ready implementation digests with explicit invariants, acceptance criteria, verification, and handoff requirements. Signed-off-by: Jakub Sztandera <oss@kubuxu.com>
Reject ambiguous and malformed CID encodings before they can be stored or used for proof verification. Signed-off-by: Jakub Sztandera <oss@kubuxu.com>
Append the two-slot compact piece mapping and centralize bounded metadata packing before production paths adopt the new storage. Signed-off-by: Jakub Sztandera <oss@kubuxu.com>
Store newly added pieces in compact slots with their final Fenwick sums, while batching the dataset leaf-count update.\n\nThis removes legacy addition writes ahead of migrating readers and removers. Signed-off-by: Jakub Sztandera <oss@kubuxu.com>
Serve piece getters, pagination, CID search, and deletion scheduling from compact metadata while preserving canonical CID output and mapping-style getter defaults. Signed-off-by: Jakub Sztandera <oss@kubuxu.com>
Adapt behavioral and raw-storage tests to compact piece records, including Fenwick sums and cleanup reclamation. Signed-off-by: Jakub Sztandera <oss@kubuxu.com>
Run generation after layout cleanup so make cannot evaluate generated targets before their files are removed. Signed-off-by: Jakub Sztandera <oss@kubuxu.com>
Measure isolated 1, 4, 16, and 32-piece additions and lock the observed storage activity. Document the compact baseline, slot reductions, and KAMT alignment behavior. Signed-off-by: Jakub Sztandera <oss@kubuxu.com>
Signed-off-by: Jakub Sztandera <oss@kubuxu.com>
Keep the consolidated measurement comparison and align the four-piece KAMT baseline with the revised measurement method. Signed-off-by: Jakub Sztandera <oss@kubuxu.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is largely LLM code. I haven't yet fully reviewed it myself, so some LLM-isms might be there.
Published because results are encouraging.
I will do a detailed review before anyone else.
Summary
Draft implementation of the compact two-slot
PieceV2storage prototype, related to #286.New compact-state datasets store pieces in a contiguous per-dataset array:
metadatapacks padding, tree height, leaf count, and the Fenwick partial sum into one storage slot.Changes
PieceCIDv2validation and decoded(padding, height, root)in one pass.PieceV2storage while retaining the existing declarations for physical layout safety.addPiecesstorage activity for 1, 4, 16, and 32-piece batches.Storage measurements
Each scenario creates a fresh dataset, adds one seed piece before recording, then records the measured
addPiecescall.Values are
legacy → compact.Findings
5N → 2Nnewly occupied slots.7Nto3N + 1; reduction grows from 42.9% at one piece to 56.7% at 32 pieces.At 32 pieces, the compact layout occupies 64 slots instead of 160, touches 9 KAMT objects instead of 137, and modifies 5 instead of 132.
Compatibility note
This is a forward-only prototype. Backwards compatibility for datasets created before the compact representation was not considered.
Keep this PR as draft; it is not ready to merge or deploy as an upgrade.
Verification