feat(iceberg): add AWS Glue Data Catalog + S3 Tables catalog modes - #1451
Open
aaj3f wants to merge 4 commits into
Open
feat(iceberg): add AWS Glue Data Catalog + S3 Tables catalog modes#1451aaj3f wants to merge 4 commits into
aaj3f wants to merge 4 commits into
Conversation
Two native-AWS-SDK catalog modes for the Iceberg/R2RML graph-source path, at parity with REST (Polaris/Snowflake) and Direct: fluree iceberg map gs --mode glue --table db.tbl --region us-east-1 --r2rml m.ttl fluree iceberg map gs --mode s3tables --table ns.tbl --table-bucket-arn arn:... --r2rml m.ttl Both resolve the table's metadata_location via the native AWS SDK (aws-sdk-glue GetTable / aws-sdk-s3tables GetTableMetadataLocation) and read metadata/manifests /data from S3 with the ambient credential chain, reusing the existing direct.rs reader + from_default_chain. No aws-sigv4, no request signing, no REST prefix, no vended-credential parsing. Rationale: for a normally S3-authorized principal, neither Glue DC (IAM mode) nor S3 Tables vends credentials, so ambient reads suffice; and staying on the AWS SDK keeps consistent signing/transport paths rather than a hand-rolled REST+SigV4 client. Validated against a live Glue + S3 Tables harness (offline config tests + exact-count SPARQL e2e incl. multi-table joins and a dangling-FK check). - fluree-db-iceberg: GlueSdkCatalogClient + S3TablesSdkCatalogClient (catalog/aws_sdk.rs); CatalogConfig::Glue/S3Tables (serde glue/s3tables) — variant ungated, SDK impl aws-gated. read_glue example; offline config tests. - fluree-db-api: CatalogMode::Glue/S3Tables + to_iceberg_gs_config + two r2rml scan-path arms mirroring Direct. - fluree-db-cli / fluree-db-server: --mode glue|s3tables (+ --region, --catalog-id, --table-bucket-arn) on `iceberg map`, local and remote. - docs: graph-sources + CLI reference. Follow-ups: Lake-Formation-governed (vended-cred) Glue, and catalog browse for Glue/S3Tables.
aaj3f
force-pushed
the
feat/iceberg-glue-sigv4
branch
from
July 7, 2026 16:24
7c83c7c to
d777a13
Compare
…query snapshot pin Before this, only map/query/list/info/drop worked for glue/s3tables; the read-only onboarding/introspection surface hard-rejected them, and the query path lacked the REST arm's per-query snapshot pin + catalog caches. Onboarding/introspection (catalog browse, table preview, R2RML auto-generate, mapping validate): - API: `rest_catalog_client` -> `catalog_client`, returning a boxed `SendCatalogClient` per mode (RestCatalogClient / GlueSdkCatalogClient / S3TablesSdkCatalogClient; Direct still rejected — no catalog to browse). Preview fetches + parses table metadata from S3 via `metadata_location` when the catalog returns no inline metadata (the SDK loadTable does), reusing the scan-path reader. generate/validate call preview per table, so they unblock transitively — no edits needed there. - Server: `IcebergConnectionRequest` gains region/catalog_id/table_bucket_arn; `build_iceberg_connection` gains glue/s3tables arms. Query-time (correctness + perf): - glue/s3tables scans now resolve loadTable through the same per-query snapshot pin + cross-query loadTable cache as REST (`sdk_resolve_load_table`), so every scan in a query reads one pinned Iceberg snapshot (was: independent per-scan SDK loads could observe different snapshots if the table committed mid-query), and the SDK client is built only on a genuine cache miss (was: rebuilt + a fresh GetTable/GetTableMetadataLocation per scan). Validated live: browse + preview against Glue (`examples/glue_onboarding.rs`); SPARQL e2e counts unchanged (300/60000/6/20000). Offline: iceberg_catalog + server route tests pass.
…3Tables Captures the PR's acceptance criteria as assertions (previously only exercised by hand-run examples), matching the repo's it_gcs_sdk_reads.rs live-test convention. - fluree-db-iceberg/tests/it_glue_sdk_reads.rs: SDK catalog load_table -> ambient S3 read -> scan row count, for Glue and S3 Tables. Env-gated (FLUREE_GLUE_IT_* / FLUREE_S3TABLES_IT_*); skipped when unset, so no live dependency in CI. Asserts an exact row count when FLUREE_*_EXPECT_ROWS is set. - fluree-db-api/tests/it_glue_live.rs: catalog browse + table preview via the public API, asserting browse lists the database and preview returns a schema. - catalog/aws_sdk.rs: factor the Glue metadata_location extraction into a pure fn with offline unit tests (present + missing-key error) — CI-level coverage of the one bit of the Glue read path that previously had none. Verified live against the sandbox harness: Glue dim_store=300, S3 Tables dim_geography=200, browse (3 namespaces / 11 tables), preview (8 columns, v2).
aaj3f
force-pushed
the
feat/iceberg-glue-sigv4
branch
from
July 7, 2026 17:59
4acab4e to
dbbd901
Compare
…EST loadTable VendedCredentials only read vended S3 creds from the loadTable response's legacy top-level `config` map, never the standardized top-level `storage-credentials` array (apache/iceberg #10722) that S3 Tables / Lake-Formation-vended Glue / newer Iceberg REST catalogs return. Such a response yielded `credentials = None`, so the scan fell through to ambient creds and 403'd on the backing bucket — a permissions error masquerading as a parse gap. - `VendedCredentials::from_load_table_response(response, metadata_location)`: checks `storage-credentials` FIRST (spec-normative precedence), picking the longest prefix that yields *usable* static creds (a remote-signing-only entry carries no `s3.*` keys and is skipped), then falls back to the flat `config` map. - Factor the two duplicated REST loadTable parse blocks (the `CatalogClient` (?Send) and `SendCatalogClient` impls) into one shared `parse_load_table_response` — preserves the inline `metadata` + `config` passthrough; fixes once, kills the duplication. - Offline fixtures: precedence, longest-usable-prefix, non-matching-prefix -> config fallback, legacy config, malformed entries, none. Not hit by #1451's SDK-native AWS path (ambient by design); benefits spec-compliant REST vending catalogs (Polaris/Unity/Snowflake). The remaining follow-up (SDK-native Lake-Formation vended creds for Glue) is captured as an audit-hardened spec, deferred to a stacked PR.
bplatz
reviewed
Jul 7, 2026
| async fn list_tables(&self, namespace: &str) -> Result<Vec<String>> { | ||
| let out = self | ||
| .client | ||
| .get_tables() |
Contributor
There was a problem hiding this comment.
These four list ops (get_databases/get_tables here, and list_namespaces/list_tables in the S3 Tables client) only read the first page. All are paginated — Glue defaults to 100 results + next_token, S3 Tables returns a continuation token — so browse silently truncates any catalog with >100 databases/tables. Loop on the token or use .into_paginator().send().
bplatz
reviewed
Jul 7, 2026
| Ok(match self { | ||
| SdkCatalog::Glue { region, catalog_id } => Box::new( | ||
| fluree_db_iceberg::catalog::GlueSdkCatalogClient::new( | ||
| region.as_deref(), |
Contributor
There was a problem hiding this comment.
The catalog client is built with only the mode region, but the S3 storage in the same scan arm (and the browse path in catalog_client) falls back region → io.s3_region. If a user sets only io.s3_region, GetTable resolves via the default-chain region while S3 reads use io.s3_region — mismatched, and it contradicts the CatalogConfig::Glue.region doc ("falls back to ... io.s3_region"). Thread io.s3_region into SdkCatalog so the fallback matches.
bplatz
reviewed
Jul 7, 2026
| } | ||
| } | ||
|
|
||
| // Fall back to the legacy flat top-level `config` map. |
Contributor
There was a problem hiding this comment.
Edge case in this fallback: if the only prefix-matching storage-credentials entry is remote-signing-only (no static s3.* keys) and a legacy top-level config carries static creds, we skip the matching entry and fall back to config's static creds — which can contradict the catalog's intent to remote-sign that prefix. Rare shape, likely fine, but worth a note on the precedence.
bplatz
approved these changes
Jul 7, 2026
bplatz
left a comment
bplatz
left a comment
Contributor
There was a problem hiding this comment.
Several comments inline, approving so you don't have to wait after reviewing them.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds two native-AWS-SDK catalog modes to the Iceberg/R2RML graph-source path — AWS Glue Data Catalog (
--mode glue) and AWS S3 Tables (--mode s3tables) — at full parity with the existing REST (Polaris / Snowflake Horizon) and Direct modes.Approach
Both modes resolve the table's
metadata_locationthrough the native AWS SDK (aws-sdk-glueGetTable/aws-sdk-s3tablesGetTableMetadataLocation), then read metadata/manifests/data from S3 with the ambient credential chain — reusing the existingdirect.rsreader andfrom_default_chain. Noaws-sigv4, no request signing, no REST prefix.Why SDK-native rather than REST + SigV4: empirically, for a normally S3-authorized principal, neither Glue DC (IAM mode) nor S3 Tables vends credentials, so ambient reads suffice — and staying on the AWS SDK keeps consistent signing/transport paths (prefer the SDK / override its HTTP client via
aws-smithy-httprather than roll alternatives) instead of a hand-rolled REST+SigV4 client.Changes
GlueSdkCatalogClient+S3TablesSdkCatalogClient(catalog/aws_sdk.rs);CatalogConfig::Glue/S3Tables(serde"glue"/"s3tables", variant ungated / SDK implaws-gated); twor2rml.rsscan arms;CatalogMode+ CLI + servermap;aws-sdk-glue/aws-sdk-s3tablesdeps;read_glueexample.catalog_client()dispatches to the SDK clients; preview fetches table metadata from S3 when the catalog returns no inline metadata; server request struct + dispatch widened. Query-time: glue/s3tables scans use the same per-query snapshot pin + cross-queryloadTablecache as REST (sdk_resolve_load_table).it_glue_sdk_reads.rs,it_glue_live.rs) + a CI-levelmetadata_location-extraction unit test.storage-credentialsparsing fix (was a listed follow-up):VendedCredentials::from_load_table_responsenow parses the standardized top-levelstorage-credentialsarray (apache/iceberg #10722) with spec-normative precedence over the legacy flatconfigmap (longest-usable-prefix; remote-signing-only entries skipped), and the two duplicated RESTloadTableparse sites are factored into one sharedparse_load_table_response. This was a latent bug that would 403 any REST catalog vending viastorage-credentials. Not hit by this PR's SDK-native path (ambient by design); benefits spec-compliant REST vending catalogs (Polaris/Unity/future Snowflake).Validation
iceberg_catalog+ server-route + api-lib (689) + iceberg (202, incl.storage-credentialsfixtures) suites pass;fmt/clippyclean.read_glueexact counts (Glue 500 / 60,000, S3 Tables 200); full SPARQL e2e over Glue exact (300 stores / 60,000 order-line⋈product / 6 categories / 20,000 order⋈customer) incl. multi-table joins + a dangling-FK check; browse + preview live; the env-gatedit_glue_*tests assert Glue 300 / S3 Tables 200 / browse / preview.Follow-up
VendedS3Pathscope enforcement — produced by this PR's adversarial audit. S3 Tables LF-vending is deferred within that issue (managed storage needs the Glue federation integration).