parser: columnar Arrow decode for parquet, and --file input#3151
Draft
jameswinegar wants to merge 1 commit into
Draft
parser: columnar Arrow decode for parquet, and --file input#3151jameswinegar wants to merge 1 commit into
jameswinegar wants to merge 1 commit into
Conversation
Replace the row-at-a-time parquet record API (RowIter + to_json_value) with parquet::arrow's columnar ParquetRecordBatchReader plus a converter that reproduces the record API's JSON formatting byte-for-byte: decimals and timestamps as strings, binary as base64, non-finite floats as null, and nested structs/lists/maps. INT96 and INT64 nanosecond timestamps both decode to Arrow Timestamp(Nanosecond); a per-column plan built from the parquet leaf schema renders INT96 as the record API's millisecond string and INT64 nanoseconds as a raw integer. Arrow types outside a verified allow-list fall back to the record API unchanged. Correctness is covered by a test asserting equality with the record API across nulls, nesting, logical types, and all four compression codecs, plus INT96 (including pre-epoch) and nanosecond cases. Also wire up the parser's --file flag end to end: when set, the file is opened directly as a seekable input instead of buffering stdin to a temporary file, and go/parser gains a ParseFile entry point that passes it. Seekable formats can then read footers/row groups without a full pre-read. A local benchmark (not included) showed the columnar decode is meaningfully faster across column shapes, with the largest gains on string-heavy and few-column tables.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Replaces the parquet parser's row-at-a-time record API (
RowIter+to_json_value) withparquet::arrow's columnarParquetRecordBatchReaderplus a converter that reproduces the record API's JSON output byte-for-byte. Also finishes wiring the parser's--fileseekable-input flag end to end.Decode (byte-for-byte compatible)
Timestamp(Nanosecond)but the record API renders them differently (INT96 → millisecond string, INT64 nanos → raw integer). A per-column plan built from the parquet leaf schema disambiguates; INT96's millisecond value is reconstructed to matchInt96::to_millis()across the full nanosecond range, including pre-epoch.Correctness is covered by a test asserting equality with the record API across nulls, nesting, logical types, and all four compression codecs (uncompressed/snappy/gzip/zstd), plus dedicated INT96 (including a pre-epoch value) and nanosecond cases.
--fileinputWhen
--file <path>is set, the parser opens the file directly as a seekable input instead of buffering stdin into a temporary file; seekable formats (parquet, Excel) then read footers/row groups without a full pre-read.go/parsergains aParseFileentry point that passes it. A test asserts--fileoutput matches piped-stdin output across parquet/csv/json/jsonl/avro.Performance
Ran a local benchmark (not included in this PR) comparing the columnar decode to the record API across column shapes. Preliminary bare-decode results: roughly +12% to +100% depending on shape — largest on string-heavy and few-column tables, smallest on format-bound (timestamp) and very-wide tables. No shape regressed.
Notes
arrowfeature to theparquetdep, plusarrow-array/-buffer/-schema/half.--fileflag (default-= stdin, unchanged). The mirrored parser invocation in the connectors repo can pass--filefor locally-staged inputs.Draft — opening for early review.