Skip to content

parser: columnar Arrow decode for parquet, and --file input#3151

Draft
jameswinegar wants to merge 1 commit into
estuary:masterfrom
jameswinegar:claude/parquet-throughput-improvements-91c0e0
Draft

parser: columnar Arrow decode for parquet, and --file input#3151
jameswinegar wants to merge 1 commit into
estuary:masterfrom
jameswinegar:claude/parquet-throughput-improvements-91c0e0

Conversation

@jameswinegar

Copy link
Copy Markdown

What

Replaces the parquet parser's row-at-a-time record API (RowIter + to_json_value) with parquet::arrow's columnar ParquetRecordBatchReader plus a converter that reproduces the record API's JSON output byte-for-byte. Also finishes wiring the parser's --file seekable-input flag end to end.

Decode (byte-for-byte compatible)

  • Columnar decode, one JSON object per row, formatting matched to the record API exactly: decimals and timestamps as strings, binary as base64, non-finite floats as null, nested structs/lists/maps.
  • INT96 and INT64-nanosecond timestamps both decode to Arrow Timestamp(Nanosecond) but the record API renders them differently (INT96 → millisecond string, INT64 nanos → raw integer). A per-column plan built from the parquet leaf schema disambiguates; INT96's millisecond value is reconstructed to match Int96::to_millis() across the full nanosecond range, including pre-epoch.
  • Arrow types outside a verified allow-list fall back to the record API unchanged (defensive; no type the reader produces today hits it).

Correctness is covered by a test asserting equality with the record API across nulls, nesting, logical types, and all four compression codecs (uncompressed/snappy/gzip/zstd), plus dedicated INT96 (including a pre-epoch value) and nanosecond cases.

--file input

When --file <path> is set, the parser opens the file directly as a seekable input instead of buffering stdin into a temporary file; seekable formats (parquet, Excel) then read footers/row groups without a full pre-read. go/parser gains a ParseFile entry point that passes it. A test asserts --file output matches piped-stdin output across parquet/csv/json/jsonl/avro.

Performance

Ran a local benchmark (not included in this PR) comparing the columnar decode to the record API across column shapes. Preliminary bare-decode results: roughly +12% to +100% depending on shape — largest on string-heavy and few-column tables, smallest on format-bound (timestamp) and very-wide tables. No shape regressed.

Notes

  • Adds the arrow feature to the parquet dep, plus arrow-array/-buffer/-schema/half.
  • CLI contract: new optional --file flag (default - = stdin, unchanged). The mirrored parser invocation in the connectors repo can pass --file for locally-staged inputs.

Draft — opening for early review.

Replace the row-at-a-time parquet record API (RowIter + to_json_value)
with parquet::arrow's columnar ParquetRecordBatchReader plus a converter
that reproduces the record API's JSON formatting byte-for-byte: decimals
and timestamps as strings, binary as base64, non-finite floats as null,
and nested structs/lists/maps.

INT96 and INT64 nanosecond timestamps both decode to Arrow
Timestamp(Nanosecond); a per-column plan built from the parquet leaf
schema renders INT96 as the record API's millisecond string and INT64
nanoseconds as a raw integer. Arrow types outside a verified allow-list
fall back to the record API unchanged.

Correctness is covered by a test asserting equality with the record API
across nulls, nesting, logical types, and all four compression codecs,
plus INT96 (including pre-epoch) and nanosecond cases.

Also wire up the parser's --file flag end to end: when set, the file is
opened directly as a seekable input instead of buffering stdin to a
temporary file, and go/parser gains a ParseFile entry point that passes
it. Seekable formats can then read footers/row groups without a full
pre-read.

A local benchmark (not included) showed the columnar decode is
meaningfully faster across column shapes, with the largest gains on
string-heavy and few-column tables.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant