feat(mumbling): add read-only Mumbling bitmap decoder and PFOR codec - #1750
Draft
nssalian wants to merge 1 commit into
Draft
feat(mumbling): add read-only Mumbling bitmap decoder and PFOR codec#1750nssalian wants to merge 1 commit into
nssalian wants to merge 1 commit into
Conversation
nssalian
force-pushed
the
mumbling-impl-go
branch
from
August 10, 2026 05:27
16b3a7d to
e305815
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
Adds a Go implementation of the Mumbling compressed bitmap (the V4 manifest deletion-vector format,
apache/iceberg#16747) to iceberg-go.
mumbling.Bitmap-New,IsSet,Cardinality. Lazy decode, sparse and dense containers, spec caps enforced(<= 8,192 containers, <= 2,097,152 set bits).
Checked against
decoder reads them and the Go encoder reproduces the same bytes. (Stored as TSV, one line per case.)
New/IsSetreturn a typed error, never panic.PFOR on vs off - descriptor-array size
Descriptor-array size with PFOR on vs off across container-distribution shapes, for the open question of whether
PFOR earns its complexity. The sizes match the Java encoder's; only the input distributions are synthetic.
wholeSavedis the fraction of the whole bitmap PFOR removes: ~16% for uniform-sparse (many containers, samesmall delete count), ~1-3% for dense/random-sparse/mixed, ~0% for tiny bitmaps where the 3-byte chunk header
dominates. Which regime real manifest DVs fall into is for the write path to settle.
Reproduce:
go test ./mumbling/ -run TestPFORSizeAnalysis -v.Benchmarks
PFOR vs raw copy for a 1,024-value descriptor array (Apple M4 Max,
-benchtime=2s -count=5):PFOR encode is ~40x a raw byte copy and decode ~4x, but both are microseconds for 1,024 descriptors, so codec
speed is not the deciding factor - size is.
Reproduce:
go test ./mumbling/ -run '^$' -bench . -benchmem -benchtime=2s -count=5.Scope
Read path plus the PFOR codec. No V4 manifest-DV consumer exists in iceberg-go yet, so this is format validation
in a second language, not a wired feature.