feat(iceberg): refreshable Google metadata-server catalog auth - #1567
Open
christophediprima wants to merge 2 commits into
Open
feat(iceberg): refreshable Google metadata-server catalog auth#1567christophediprima wants to merge 2 commits into
christophediprima wants to merge 2 commits into
Conversation
A static bearer for a Google Iceberg REST catalog (BigLake) is a short-lived
OAuth access token: it expires after ~1h and `BearerTokenAuth` cannot renew it,
so a long-running materialization tracking worker starts returning 401s.
Add `AuthConfig::GoogleMetadata` + `GoogleMetadataAuth`, which mints and
auto-refreshes tokens from the GCE/GKE instance metadata server (Workload
Identity), mirroring the existing `OAuth2ClientCredentials` cache+refresh — the
jittered-expiry `CachedToken` is now shared via `auth::token`. Wire it through
`{Iceberg,R2rml}CreateConfig::with_auth_google_metadata` and the `/iceberg/map`
`auth_google_metadata` field.
The GCS reader's storage HMAC keys are unaffected (static, non-expiring). The
metadata server is only reachable on GCE/GKE; local runs keep using a static
`auth_bearer`.
Extracted from fluree#1422 as part 1 of the split proposed in review. Two adaptations
were needed against current main, neither visible in the diff:
- `AuthConfig::hydrate` gained a `GoogleMetadata` arm. Spelled out rather than
folded into a wildcard, so that match keeps failing compilation when a new
variant appears — `GoogleMetadata` carries no `ConfigValue`, so there is no
secret reference to resolve.
- `IcebergCreateConfig` has since been refactored into `connection` +
`table_identifier` with the auth builders living on `IcebergConnectionConfig`,
so the `R2rmlCreateConfig` delegator needed a forwarding method alongside the
existing `with_auth_bearer`.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`test_every_auth_config_variant_redacts_on_emit` matches `AuthConfig` with no
wildcard arm on purpose: a new variant is meant to stop the test compiling
until someone has made an explicit decision about whether it carries secret
material. Adding `GoogleMetadata` tripped exactly that guard, so `clippy` and
`test` both failed to build `fluree-db-api`'s lib test target.
The decision it was asking for: GoogleMetadata carries no secret material and
earns no allowlist entry. Both of its fields are plain `String` rather than
`ConfigValue`, so there is no secret-capable field to seed with a sentinel —
the credential is minted per call from the instance metadata server and lives
only in the in-memory `CachedToken`, which is never serialized and whose
hand-written `Debug` prints `***`.
Adds the variant to the fixture so it goes through the redaction assertions
alongside the others, covers it in the canary match, and records the reasoning
where the next person to add a variant will read it.
Only `fmt` passed before this, because it is the one gating job that never
compiles. Verified with the flags CI actually uses — `--all-features` matters
here, since the canary is `#[cfg(feature = "iceberg")]`:
cargo clippy --all --all-features --all-targets # exit 0
cargo test -p fluree-db-iceberg --all-features # 272 passed
cargo test -p fluree-db-api --all-features --lib # 904 passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A static bearer token for a Google Iceberg REST catalog (BigLake) is a short-lived
OAuth access token. It expires after roughly an hour, and
BearerTokenAuthhas no way torenew it — so any long-running consumer of that catalog starts returning 401s and stays
broken until someone re-registers the graph source with a fresh token.
This adds
AuthConfig::GoogleMetadata/GoogleMetadataAuth, which mints andauto-refreshes tokens from the GCE/GKE instance metadata server (Workload Identity).
What changes
fluree-db-iceberg/src/auth/google_metadata.rs— the provider. Fetches from themetadata server with the required
Metadata-Flavor: Googleheader, optional scopes(defaults to
cloud-platform), and the same cache-plus-refresh behaviourOAuth2ClientCredentialsalready has.fluree-db-iceberg/src/auth/token.rs— the jittered-expiryCachedTokenis lifted outof
oauth2.rsso both providers share one implementation rather than growing a secondcopy.
oauth2.rsshrinks by 42 lines as a result; behaviour is unchanged.{Iceberg,R2rml}CreateConfig::with_auth_google_metadata(scopes)and anauth_google_metadatafield onPOST /v1/fluree/iceberg/map.docs/api/endpoints.mdanddocs/graph-sources/iceberg.md.Scope notes
non-expiring; only the catalog leg had the expiry problem.
auth_bearer, which is why the tests below drive the provider against a stub serverrather than the real endpoint.
Tests — two in
google_metadata.rs: one asserts the flavor header, the scope stringand that a second call is served from cache rather than refetched; one asserts a metadata
failure surfaces rather than silently yielding no header.
Why it is worth having on its own: this is a correctness fix for anyone pointing
Fluree at BigLake, independent of anything else in #1422 — the expiring-token failure is
real today and there is no workaround short of periodically re-registering the source.