feat: add v3 geometry and geography support - #880
Open
wgtmac wants to merge 1 commit into
Open
Conversation
wgtmac
commented
Aug 8, 2026
wgtmac
commented
Member
- align geospatial type semantics and schema JSON with the v3 spec
- add bound encoding and Java-compatible intersection evaluation
- represent WKB as Arrow binary and support Avro byte round trips
- generate and validate Parquet geospatial logical types
- preserve count metrics while omitting geospatial byte-order bounds
- align geospatial type semantics and schema JSON with the v3 spec - add bound encoding and Java-compatible intersection evaluation - represent WKB as Arrow binary and support Avro byte round trips - generate and validate Parquet geospatial logical types - preserve count metrics while omitting geospatial byte-order bounds
There was a problem hiding this comment.
Pull request overview
Adds Iceberg v3 geospatial (geometry/geography) support across type semantics, JSON schema strings, and file format integrations (Arrow, Avro, Parquet), including bound encoding/intersection utilities.
Changes:
- Update Geometry/Geography type parameter defaults and equality/string semantics to match v3 expectations.
- Add geospatial bounds encoding + intersection evaluation utilities and corresponding tests.
- Enable round-tripping of geospatial WKB through Arrow (binary), Avro (bytes), and Parquet (geospatial logical types), plus metrics behavior updates.
Reviewed changes
Copilot reviewed 23 out of 23 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| src/iceberg/type.h | Default CRS/algorithm storage for geospatial primitive types. |
| src/iceberg/type.cc | Updated geospatial constructors, equality, and string representations. |
| src/iceberg/test/visit_type_test.cc | Update expected primitive type string representations. |
| src/iceberg/test/type_test.cc | Extend geospatial type tests; add bounds/intersection test coverage. |
| src/iceberg/test/schema_json_test.cc | Align schema JSON strings with v3; accept bare geospatial type names. |
| src/iceberg/test/parquet_test.cc | Add Parquet round-trip test for WKB stored as binary. |
| src/iceberg/test/parquet_schema_test.cc | Add Parquet geospatial logical type conversion/projection tests. |
| src/iceberg/test/parquet_metrics_test.cc | Ensure geospatial metrics exclude bounds while preserving counts. |
| src/iceberg/test/avro_test.cc | Add Avro writer test treating geospatial values as opaque bytes. |
| src/iceberg/test/avro_schema_test.cc | Ensure geospatial types map/project as Avro bytes. |
| src/iceberg/test/arrow_test.cc | Map geometry/geography to Arrow binary and adjust unsupported list. |
| src/iceberg/schema_internal.cc | Enable Arrow schema conversion for geometry/geography as binary. |
| src/iceberg/parquet/parquet_writer.cc | Switch Parquet writer to Iceberg-driven Parquet schema generation. |
| src/iceberg/parquet/parquet_schema_util.cc | Implement Parquet schema generation + geospatial compatibility validation. |
| src/iceberg/parquet/parquet_schema_util_internal.h | Expose Iceberg→Parquet schema conversion API. |
| src/iceberg/parquet/parquet_metrics.cc | Skip bounds collection for geometry/geography metrics. |
| src/iceberg/meson.build | Add geospatial sources/headers to Meson build/install. |
| src/iceberg/geospatial.h | New public API for bounds/box encoding and intersection checks. |
| src/iceberg/geospatial.cc | Implement geospatial bound/box encoding and intersection logic. |
| src/iceberg/CMakeLists.txt | Add geospatial.cc to CMake sources. |
| src/iceberg/avro/avro_schema_util.cc | Write geospatial types as Avro bytes nodes. |
| src/iceberg/avro/avro_direct_decoder.cc | Decode geospatial values via the Avro-bytes path. |
| src/iceberg/avro/avro_data_util.cc | Append geospatial values via the Avro-bytes path. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Member
Author
|
@zhjwpku @manuzhang PTAL :) |
zhjwpku
approved these changes
Aug 9, 2026
| case TypeId::kVariant: | ||
| return NotSupported("Cannot write Iceberg variant type to Parquet"); | ||
| default: | ||
| return InvalidSchema("Expected nested Iceberg type, got {}", type); |
Collaborator
There was a problem hiding this comment.
nit: Variant is not a nested type in spec, maybe we can use unreachable here.
manuzhang
reviewed
Aug 9, 2026
| if (descr.physical_type() != ::parquet::Type::BYTE_ARRAY) { | ||
| return InvalidSchema("Iceberg type {} requires Parquet BYTE_ARRAY", expected_type); | ||
| } | ||
| if (expected_type.type_id() == TypeId::kGeometry) { |
Member
There was a problem hiding this comment.
This (and the following) validates only whether the logical type is GEOMETRY or GEOGRAPHY; it does not compare the logical-type parameters with the expected Iceberg type. As a result, the reader currently accepts cases such as:
geometry(EPSG:3857) projected as geometry(OGC:CRS84)
geography(OGC:CRS84, spherical) projected as geography(EPSG:4326, karney)
Because WKB does not encode the CRS or edge-interpolation algorithm, this can silently interpret the same bytes using different semantics.
Member
|
cc @huan233usc who implemented geometry and geography support in iceberg java |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.