Skip to content

[Rust] Add support to fetch schema from UC for dynamic proto - #704

Open
andrijast-db wants to merge 4 commits into
mainfrom
andrijast-db/effort/rust-dynamic-proto-fetch-from-uc
Open

[Rust] Add support to fetch schema from UC for dynamic proto#704
andrijast-db wants to merge 4 commits into
mainfrom
andrijast-db/effort/rust-dynamic-proto-fetch-from-uc

Conversation

@andrijast-db

@andrijast-db andrijast-db commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

What changes are proposed in this pull request?

Follow-up to #465, closing #521. Adds the ability to fetch a table's schema from Unity Catalog and use it as a dynamic-proto descriptor, so the descriptor no longer has to be built column-by-column in code.

New public API (all additive):

  • uc_schema module — fetch_message_descriptor(...) mints an OAuth token, reads GET /api/2.1/unity-catalog/tables/{name}, and resolves it to a MessageDescriptor; fetch_table_schema(...) returns the raw schema.
  • ZerobusSdk::fetch_message_descriptor(table, client_id, client_secret) — wrapper using the SDK's unity_catalog_url.
  • ZerobusError::SchemaFetchError { message, retryable }.

let descriptor = sdk.fetch_message_descriptor("catalog.schema.table", client_id, client_secret).await?;
let stream = sdk.stream_builder().table("catalog.schema.table")
.oauth(client_id, client_secret).dynamic_proto(descriptor).build().await?;

The fetch is a separate step feeding the existing .dynamic_proto(...) — rather than a new builder method that fetches inside build() — so .dynamic_proto() stays the single format selector, network I/O stays visible, and one descriptor can be reused across streams. It mints a plain all-apis token (the ingestion token DefaultTokenFactory produces is rejected by the UC REST API), bounds response body size, classifies transport/5xx/429 as retryable, and rejects credential-bearing URLs. No FFI/wrapper changes.

How is this tested?

Full cargo test --workspace passing; clippy (--all --all-features -D warnings) and fmt clean. Unit tests cover validation, endpoint/URL handling, and descriptor conversion; 20 integration tests run against a loopback HTTP mock covering the happy path, request shapes, and error classification (4xx/5xx/429, unparseable/oversized body, dropped connection). Not tested against a live workspace.

@andrijast-db
andrijast-db force-pushed the andrijast-db/effort/rust-dynamic-proto-fetch-from-uc branch from ba4fc6b to 7593a4b Compare August 10, 2026 15:02
Comment thread rust/sdk/src/uc_schema.rs Outdated
Comment thread rust/sdk/src/errors.rs Outdated
Comment thread rust/sdk/src/uc_schema.rs Outdated
Comment thread rust/sdk/src/uc_schema.rs Outdated
@andrijast-db
andrijast-db force-pushed the andrijast-db/effort/rust-dynamic-proto-fetch-from-uc branch from 8d4f9de to 3f7dfc8 Compare August 17, 2026 13:20
@andrijast-db
andrijast-db force-pushed the andrijast-db/effort/rust-dynamic-proto-fetch-from-uc branch from 8f9f80a to cc9a56c Compare August 18, 2026 13:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants