CompactProof: Deduplicate values by skunert · Pull Request #233 · paritytech/trie

skunert · 2026-07-07T15:12:17Z

Summary

encode_compact re-emits a shared detached value once per referencing node, causing proof size to grow with reference count instead of value count. Adds opt-in encode_compact_skip_duplicate_values to dedupe on encode. decode_compact_from_iter (and decode_compact) now always handle both old-style and deduplicated proofs correctly. Also fixes an off-by-one in decode_compact_from_iter's returned item count when the last node has an attached value.

Compatibility

encode_compact / decode_compact_from_iter behavior unchanged — new functions are additive.
Deduplicated proofs decode correctly with this crate's decoder, and with an unpatched old decoder into a hash-keyed MemoryDB (verified against 0.31.0) — but not into a PrefixedKey DB, which needs the value re-inserted per prefix.
Old decoder + hash-keyed DB: refcount for the shared value is undercounted (1 instead of N). Reads still work (get/contains only check rc > 0); This matters if you later do balanced insert/remove against that decoded DB.
decode_compact's item count is now correct when the last node has an attached value — previously it was off by one in that case.

encode_compact emits a detached value node once per referencing trie node, so a value shared by N keys is sent N times (paritytech/polkadot-sdk#12565). Add encode_compact_skip_duplicate_values to emit each distinct value once; repeats are emitted unmodified and stay decodable by existing hash-keyed decoders. For prefixed databases, decode_compact_from_iter_with_known_values re-inserts deduplicated values at every referencing position.

A trailing attached value node was not counted in the returned used-item count, so decoding concatenated encodings at that offset re-read the value bytes as a node.

trie_codec_proof only ran ExtensionLayout (inline values), so value detachment and deduplication were never fuzzed. Round-trip the deduplicating encoding into hash-keyed and prefixed databases, add a hashed-value target with heavily shared values, and a deterministic smoke test so the assertions run in CI without libFuzzer. Also fix the fuzz crate's stale memory-db path dependency version.

An old decoder inserts a deduplicated value once, so its refcount in a hash-keyed database understates the referencing nodes; a consumer consolidating removals could drop a still-referenced value. The re-inserting decoder restores exact parity. Assert full database equality (entries and refcounts, hash-keyed and prefixed) between the plain and deduplicated encodings.

bkchr

You can do the de-duplication on the node level and not just on the value level.

bkchr · 2026-07-09T10:45:25Z

+/// [`decode_compact_from_iter_with_known_values`]).
+pub fn encode_compact_skip_duplicate_values<L>(
+	db: &TrieDB<L>,
+	seen_value_hashes: &mut BTreeSet<Vec<u8>>,


Does someone requires this after the encoding?

We could use it for example here to deduplicate across child and main trie. Whether its really necessary to deduplicate across main and child is not super clear to me, but I can see use cases for this.

bkchr · 2026-07-09T10:45:51Z

+/// `seen_value_hashes` collects the emitted value hashes. Pass `&mut Default::default()` for a
+/// standalone encoding, or thread the same set across concatenated encodings (the matching decode
+/// calls must then thread a known-values map through
+/// [`decode_compact_from_iter_with_known_values`]).


Suggested change

/// `seen_value_hashes` collects the emitted value hashes. Pass `&mut Default::default()` for a

/// standalone encoding, or thread the same set across concatenated encodings (the matching decode

/// calls must then thread a known-values map through

/// [`decode_compact_from_iter_with_known_values`]).

/// `seen_value_hashes` collects the emitted value hashes.

Ty claude :P

What is the use case for "concatenated encodings"? Do we actually need to share the set across calls, or can get rid of this argument?

Vendor the compact-proof decoder from the released trie-db 0.31.0 verbatim into the test crate as a frozen snapshot of deployed decoder behavior, and assert that encodings produced by encode_compact_skip_duplicate_values decode with it into a hash-keyed database with every entry readable. This pins the backward compatibility the deduplicating encoder relies on, instead of leaving it as an argument in documentation.

encode_compact_skip_duplicates now emits each distinct trie node once, not just each detached value. A later occurrence of an already-emitted subtree keeps a plain hash reference (like any reference outside the partial trie), so the encoding never grows. The decoder threads a known-items map and re-inserts skipped subtrees at every position, reconstructing the same database as an un-deduplicated encoding.

skunert · 2026-07-13T10:34:44Z

You can do the de-duplication on the node level and not just on the value level.

We now also deduplicate nodes

bkchr · 2026-07-14T09:31:14Z

+					// `omit_children` bit stays unset, keeping a plain hash reference. The root is
+					// never skipped, so each encoding stays individually decodable when
+					// `seen_hashes` is threaded across successive encodings.
+					if !stack.is_empty() && seen_hashes.contains(node_hash.as_ref()) {


I don't get why you check here if stack is not empty?

This is needed to decide whether we are at the root of a child trie for example. If the root would be in seen_hashes, the encoding for the child trie would be empty.

bkchr · 2026-07-14T09:36:24Z

+		},
+		_ => 0,
+	};
+	for index in 0..entry.children.len() {


Why are you not directly iterating the entry.children?

Because we use it to access the bitmask

bkchr · 2026-07-14T09:36:49Z

+		if entry.hash_ref_children & (1u16 << index) == 0 {
+			continue
+		}


Suggested change

if entry.hash_ref_children & (1u16 << index) == 0 {

continue

}

This is the exact same check as below?

See comment here

bkchr · 2026-07-14T09:37:19Z

 	children: Vec<Option<ChildReference<C::HashOut>>>,
+	/// Bit mask of children kept as plain hash references, which may point at subtrees
+	/// deduplicated into an earlier occurrence (see [`encode_compact_skip_duplicates`]).
+	hash_ref_children: u16,


Suggested change

hash_ref_children: u16,

has_ref_children: bool,

And then just set it to true if there is one.

This is a bitmask that marks which children came on the wire as hash. We insert the deduplicated subtrees towards the end and iterate all the children. But at that point it is not possible to distinguish between items that came as detached value and items that where always Hash.

The ones that came as detached value where already inserted into the db, so we should not insert them again to keep the ref count in the db correct.

bkchr · 2026-07-14T09:52:59Z

+			if hash.is_none() {
+				// The node's detached value may have been deduplicated into an earlier occurrence
+				// (see `encode_compact_skip_duplicates`); re-insert it here too so `db` matches an
+				// un-deduplicated encoding. A miss means the value is not in the encoding, which is


If the hash is none and the the value is missing, it means the db is missing values and we should return an error?

What does value missing mean for you here?

If the hash is none that means that there was no detached node in the compact encoding. So we just have the hash.

If that hash belongs to a node that was deduplicated and we have seen it earlier, we need to reinsert to maintain refcount.

if the hash belongs to a node that we have not yet seen, that means the node is just not referenced in the proof, which should also be fine.

bkchr · 2026-07-14T09:53:26Z

+pub fn decode_compact_from_iter_with_known_items<'a, L, DB, I>(
+	db: &mut DB,
+	encoded: I,
+	known_items: &mut BTreeMap<Vec<u8>, DBValue>,


Do we actually need to supply this?

https://github.com/paritytech/trie/pull/233/changes#r3581312671

bkchr · 2026-07-14T09:54:16Z

+	DB: HashDB<L::Hash, DBValue>,
+	I: IntoIterator<Item = &'a [u8]>,
+{
+	decode_compact_from_iter_with_known_items::<L, DB, I>(db, encoded, &mut BTreeMap::new())


This method is also doing the same reconstruction of the database with the same counts. So, there is no real difference between these two?

The difference is just backwards compat. The previous method existed before and does not break anything. with_known_items is only needed if you want to pass through the known_items, which we could do in the scenario outlined here.

skunert · 2026-07-15T12:43:38Z

@cheme Would be super nice if you could support us with a review here :)

skunert and others added 5 commits July 7, 2026 16:41

trie-db: count attached values in decode_compact's used-items result

37afef5

A trailing attached value node was not counted in the returned used-item count, so decoding concatenated encodings at that offset re-read the value bytes as a node.

Shorten comments

75728c1

skunert requested a review from bkchr July 7, 2026 15:12

bkchr reviewed Jul 9, 2026

View reviewed changes

skunert added 3 commits July 13, 2026 12:25

Shorten comment

29d7b10

bkchr reviewed Jul 14, 2026

View reviewed changes

Comment thread trie-db/src/iterator.rs Outdated

bkchr reviewed Jul 14, 2026

View reviewed changes

Nits

f5362f1

skunert requested a review from cheme July 15, 2026 12:42

	if entry.hash_ref_children & (1u16 << index) == 0 {
	continue
	}

Uh oh!

Conversation

skunert commented Jul 7, 2026

Summary

Compatibility

Uh oh!

bkchr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

skunert commented Jul 13, 2026

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

skunert commented Jul 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants