perf(iceberg): loadTable-METADATA cache — zero-REST first-ask (q031 72s→~6s arc) + F19 residual by aaj3f · Pull Request #1503 · fluree/db

aaj3f · 2026-07-15T06:17:31Z

loadTable-METADATA cache — persisted credential-free pointer + lazy storage (+ F19 residual)

Stacked on perf/r2rml-q031-refprune (#1502). Closes the isolated first-ask ~2–3s loadTable REST/OAuth cost that F20/PR-4d left on q031/q016 (on top of the shared fact-scan/materialize floor). 4 commits, +906/−10.

The win (deterministic gate — GREEN, both runs)

Warm-disk / cold-memory, fresh process per query. The COUNTERS are the deterministic gate; walls vary with cache warmth (two runs shown — pre- and post- the pin fix, which perturbed nothing):

q	pointer cache ON	OFF (old eager path)
q031	load 0 / oauth 0 / 5000 rows / 5.9–6.5s	load 2 / oauth 1 / 5000 / 10.3–14.9s
q016	load 0 / oauth 0 / 5000 rows / 17.0–25.6s	load 2 / oauth 1 / 5000 / 20.6–28.5s

When a query's metadata (slice-2), scan files (slice-2), and every Parquet file (DiskArtifactCache) are all disk-resident, it now issues zero loadTable GET / OAuth / S3. OFF is byte-identical to the old path; row parity holds. q031's first-ask is 72s → 5.9s across this arc (F20 + this). Full-corpus baseline: 54/54 oracle 0-mismatch, 44/51 ok ≤3s (corpus-order loads stay amortized, so full-run walls track #1502 — this closes the per-query first-ask the amortized baseline hid).

Mechanism

(a) Persisted credential-free pointer (disk_catalog_cache.rs): a lt_key → metadata_location entry (same Envelope + CACHE_FORMAT_VERSION discipline). Payload is the location + snapshot_ms + cached_at_ms ONLY — no creds/token/config. Switch FLUREE_ICEBERG_LOADTABLE_PTR_CACHE; TTL FLUREE_ICEBERG_LOADTABLE_PTR_TTL_SECS (300s, 0=off).

(b) LazyS3Storage (new lazy_storage.rs): impls SendIcebergStorage over a tokio OnceCell + a deferred builder. The pointer rung in load_table_context serves metadata_location (pointer) + metadata (caches) with no S3, and defers storage; the reader only forces the builder on a real S3 miss. The eager REST client is built up front — construction does no network; the OAuth token exchange rides the first catalog op (the GET), so deferring the GET zeroes both counters (the gate's oauth_token.n=0 enforces this forever). The 'static builder captures owned Arc handles (the provider holds fluree: &'a, which can't cross the Parquet tokio::spawn) and runs resolve_rest_load_and_storage — the SAME session-pin ladder as the eager path (no duplicate; full slice-1 reuse incl. the cross-query put). LazyS3Storage is Clone via an Arc-shared cell (SendParquetReader requires S: Clone); clones share one build.

Security (creds never persisted)

Hermetic pointer_persists_no_credential_bytes asserts the on-disk pointer has exactly {metadata_location, snapshot_ms, cached_at_ms} and no credential/token/secret/access_key/session/sig bytes. Vended creds are fetched in-memory, per process, only when the lazy handle is forced (a real S3 miss).

Snapshot-pin correctness (review fix, commit `6750286`)

The pointer rung consults session.pinned_metadata_location BEFORE the disk pointer (a prior touch's pin wins), and registers a location-only pin on a hit — so a query that touches the same table twice reads ONE snapshot even if the pointer's TTL expires between touches. Kept in a separate location_pins map so cached_load_table never serves a credential-less entry (which would build ambient-cred storage); the eager path reloads only creds and converges on the pinned location. Hermetic location_pin_keeps_query_on_one_snapshot_across_pointer_ttl_expiry.

TTL staleness semantics

The pointer (unlike content-addressed entries) CAN go stale on a table commit, so it carries a TTL = the max cross-process snapshot staleness for a latest-snapshot read — the same tradeoff class the existing 60s in-memory cross-query cache accepts, extended deliberately, consistent with the disk-cache-is-steady-state ruling. A stale pointer never returns WRONG data for the snapshot it names; it can only lag a newer snapshot within the TTL.

as_of_t forward-guard + wiring obligation

Iceberg snapshot time-travel is deliberately unimplemented on the scan path (grep r2rml-as-of-t), so the caller passes min_snapshot_ms=None (latest). The guard (pointer_is_usable) + hermetic pointer_ttl_and_as_of_rider exist; doc-21 §4 + the register carry the obligation: whoever implements time-travel MUST thread the requested snapshot ms into get_metadata_location.

F19 residual (ride-along, commit `ebcab75`)

with_graph_ref now clones the R2RML parent-memo Arc (its key is store-disambiguated (graph_source_id, parent_tm, cols, as_of_t), unlike const_sid_cache which stays fresh for the store switch). Closes the F19 residual; hermetic parent_memo_survives_with_graph_ref_rebuild. No corpus query exercises SERVICE/multi-source R2RML, so gate-neutral.

Tests

lazy_storage 3/3 (deferred-not-invoked, 16-concurrent-forces→1-build, no-panic error path); disk_catalog_cache 8/8; catalog_session 7/7; full -p fluree-db-api --lib 765/0; it_graph_source_r2rml 53/53. All fluree-db-api test targets compile.

CI note: base is perf/r2rml-q031-refprune, not main, so repo CI does not fire (stacked-PR convention).

aaj3f · 2026-07-15T11:42:29Z

Heads-up from the sibling branch feat/external-s3-iceberg (BYO-IAM engine work for #1500, forked at the same base eee643572 / #1502): there is a known semantic interaction to reconcile when these two land, beyond the expected r2rml.rs textual conflicts.

That branch adds §2 fail-closed credential handling: in the REST arm, vended_credentials == true + load_response.credentials == None now errors (err:catalog/CredentialsNotVended, 403) instead of silently downgrading to ambient AWS credentials. The decision matrix lives in one shared decide_credential_source() used by both the scan and preview paths.

If this PR's zero-REST first-ask path synthesizes/reconstructs a loadTable response with credentials: None for a vended-configured source, the composed branches would 403 every cached first-ask (loud, not silent — but broken). Whoever rebases second should make the cached path either (a) carry an explicit "credentials come later / from cache" marker through the decision, or (b) join behind the same decision matrix the way the existing per-query pin and 60s loadTable caches do (they preserve credentials verbatim, which is what keeps the §2 guard from false-firing on today's cache hits).

Related, in your favor: that branch deliberately kept the disk metadata cache layer as the fallback for responses carrying metadata: None (only inline-vending REAL loadTable responses bypass it), so a zero-REST path still has its cross-process metadata source.

…sh-collision guard) The disk catalog cache names entries by a 64-bit DefaultHasher of the key (metadata_location / lt_key), so two distinct keys can collide onto one filename and silently serve another table's payload — a strictly worse failure than the content-addressed entries' wrong-snapshot, and #1503's pointer entry keys by table. Store the FULL key in the Envelope and verify equality on read: a mismatch is a clean miss, so hash quality is irrelevant to correctness. Bump CACHE_FORMAT_VERSION 1->2 (old keyless entries miss cleanly via the existing version check). Forced-collision hermetic test. Covers the metadata/scanfiles/ countstats entries; the #1503 pointer rides the same Envelope + version.

…isted) + F19 with_graph_ref ride-along Design-first sketch (STOP for lead review, no engine code). Persistent credential-free metadata_location pointer + lazy storage/creds so a warm-disk first-ask issues zero loadTable GET / OAuth / S3 — closing q031's + q016's ~2-3s isolated-first-ask residual on top of the shared fact-scan/materialize floor. AJ hard constraint: vended creds NEVER persisted; fetched in-memory per-process only on an actual S3 byte-fetch. Rides the deferred F19 with_graph_ref parent-memo clone + its hermetic test. Open questions for the lead in section 9.

…(store-disambiguated key) with_graph_ref switches to a different graph's store, so it re-creates const_sid_cache fresh (its key is the const IRI alone, store-implicit). But the R2RML parent-lookup memo key carries graph_source_id + as_of_t, so a lookup can never be served for the wrong source — sharing it is sound, and lets PR-8b's query-scoped memo survive a correlated inner-join rebuilt across a with_graph_ref boundary (SERVICE / multi-source default R2RML), the one path with_active_graph's clone did not cover. Hermetic pr8b::parent_memo_survives_with_graph_ref_rebuild proves the DIM parent is scanned once across 5 with_graph_ref rebuilds. Closes the F19 residual; no corpus query exercises that path, so it ships gate-neutral.

…ointer + lazy storage Closes q031's/q016's isolated-first-ask ~2-3s loadTable residual (on top of the shared fact-scan/materialize floor). Two coupled parts: (a) A persisted, credential-free lt_key->metadata_location pointer in disk_catalog_cache.rs (same Envelope+version discipline; POINTER STRING + snapshot_ms + cached_at_ms ONLY — no creds/token/config, asserted by pointer_persists_no_credential_bytes). TTL FLUREE_ICEBERG_LOADTABLE_PTR_TTL_SECS (300s, 0=off) bounds cross-process snapshot staleness; switch FLUREE_ICEBERG_LOADTABLE_PTR_CACHE. An as_of_t forward-guard (pointer_is_usable) refuses a pointer older than a requested min_snapshot_ms — inert until Iceberg time-travel lands (caller passes None; grep r2rml-as-of-t). (b) LazyS3Storage (new lazy_storage.rs) impls SendIcebergStorage over a tokio OnceCell + deferred builder. When the pointer + metadata (slice-2) + parquet (DiskArtifactCache) all hit disk, no read reaches it, so the builder never runs => zero loadTable GET / OAuth. The eager REST client is built up front (NO network: the OAuth token exchange rides the first catalog op, the GET) so only the GET is deferred, zeroing both gate counters. The provider holds fluree:&'a which can't cross the Parquet tokio::spawn, so the 'static builder captures owned Arc handles and runs resolve_rest_load_and_storage — the SAME session-pin ladder the eager path uses (no duplicate). LazyS3Storage is Clone (Arc-shared cell) because SendParquetReader needs S: Clone; clones share one build. Switch off = eager (byte-identical). Lib + R2RML integration suites green (lazy_storage 3/3 incl. concurrent-force-dedup + no-panic error path; pointer 8/8; it_graph_source_r2rml 53/53). Rides the F19 with_graph_ref fix (ebcab75). Deterministic gate (load_table.n=0 AND oauth_token.n=0 disk-warm) + baseline next.

Review catch: the loadTable-metadata pointer rung consulted the disk pointer BEFORE any session pin and served metadata WITHOUT registering a pin. A query that touches the same table twice (correlated re-loads — the reason the pin machinery exists) whose disk-pointer TTL expires BETWEEN the touches could read two different snapshots: touch 1 serves pointer-loc1 (no pin), touch 2 misses the expired pointer, falls to the eager path, and a fresh GET returns loc2 (table advanced). Silent-wrong-results, and strictly worse than the pre-change eager ladder (which consulted the pin first on every call). Fix: (a) the rung consults session.pinned_metadata_location BEFORE the disk pointer — a prior touch's pin wins unconditionally; (b) on a rung hit it registers a location-only pin (new IcebergCatalogSession::pin_metadata_location + a separate location_pins map) so every later touch resolves the SAME snapshot even after TTL expiry. Kept separate from load_tables so cached_load_table never serves a credential-less entry (which would build ambient-cred storage); the eager path reloads only creds and its existing pin-override converges on the pinned location. Hermetic location_pin_keeps_query_on_one_snapshot_across_pointer_ttl_expiry proves first-writer-wins + the cached_load_table non-poisoning. catalog_session 7/7, it_graph_source_r2rml 53/53. Pin-register is in-memory only — does not perturb the disk-warm 0/0 gate counters.

…ilder symptom doc, F19 memo-key rationale, fmt - lazy_storage.rs: |res| res.is_ok() -> Result::is_ok (workspace-denied redundant_closure_for_method_calls in test code). - r2rml.rs: name send_parquet.rs's tokio::spawn as the site a future '&a narrowing refactor would fail at, so the '\''static owned-Arc capture reads as load-bearing. - operator.rs: mirror the store-disambiguation rationale at R2rmlParentMemoKey so the F19 with_graph_ref memo-share (why graph_source_id can't be dropped) is documented where an editor would remove the component. - cargo fmt (own-delta hunks). C2 pointer hardening (full lt_key stored in the Envelope + verified on read) is delivered by get/put_metadata_location adopting slice-2's (key, suffix) read/write signatures during the restack onto the #1491 CACHE_FORMAT_VERSION=2 chain.

…nter scope) q029 closes its burndown row at 2.61s in-protocol (F17 + cache stack); cold isolated first-ask ~57s-class (S3-fetch floor). The F17 3-rep baseline's 5 perf violations (q002/q004/q024/q030/q043) are attributed and are NOT F17: all five are UNION/BIND-free so F17's switch-gated code is inert, and it is NOT cache eviction (fetch_bytes≈0; artifact cache 90MB, 5.7x under the 512MiB cap). The wall is the loadTable catalog GET (~1.5s vs ~0.4s at bless — network-path). Filed as F21: the #1503 loadTable-pointer cache is TTL-window-bounded (300s + prune-on-read), so it spares the GET only within ~5min of a table's last touch (only 1 .pointer.json survived on disk). Mechanism question — TTL-window vs per-table/keying gap — deferred to a design sketch. q043's 6-GET edw:name fan-out is noted as an F20 widening candidate (class-constrained lang-tag star). doc-21 gets the dated TTL scope correction to its "warm first-ask = zero GET" claim.

…g note, SWITCHES inventory, fmt - union.rs: add the coupling cross-reference the review asked for at the budget-met lever — count (1)'s soundness rests entirely on the forward/absorb classification in Operator::set_row_budget's doc, so an author who forwards a budget through an order-SENSITIVE operator breaks this lever, not the forwarding site. Documented at the dangerous end. - SWITCHES.md: the standing switch-inventory ask — one page, one row per virtual perf lever (default / mechanism / PR / OFF-restores), the falsy convention, the loadTable TTL governing-bound note (#1503) + F21 pointer, and the corpus-side timeout semantics (timeout_s is CI headroom, not the bar). - cargo fmt (union.rs own-delta hunks). The FLUREE_R2RML_UNION_BUDGET naming nit is addressed by the SWITCHES.md scope row (it gates BIND forwarding too) rather than a rename. The scan early-stop test-seam suggestion is declined on binary-purity grounds (see review response).

…e-METADATA cache (#1500 x #1503) The restack onto the pointer-rung/lazy-storage chain applies textually clean but is semantically wrong in four ways: #1503 did not replace the two sites this branch hardens, it CLONED them into a deferred copy that none of the guards reached. Fixed by unifying, not by patching copies. - rest_session_storage(): ONE helper holding the §2 fail-closed decision, the #1498 session reuse, and the vended/ambient construction. The eager arm and the deferred LazyS3Storage builder now share it. Previously the builder's copy had no §2 guard at all, so any lazy-forced storage build silently downgraded a vended-configured source to the process's ambient AWS identity — the exact fail-open §2 exists to close, resurrected. - hydrated_auth_provider(): pairs hydrate() with create_provider_arc() so a client-construction site cannot ship un-hydrated. #1503's pointer rung added a second create_provider_arc that §3's hydrate never reached, so a ConfigValue::SecretRef hard-errored via resolve()'s fail-closed arm exactly when the pointer cache was warm — i.e. rotation broke on the fast path. Hydration still runs strictly AFTER rest_client_cache_key in both arms. - Typed errors now round-trip the builder's IcebergError channel (new IcebergError::CatalogCredentialsNotVended + a storage_query_error lift) instead of being flattened by a blanket to_string(), which had destroyed the 403 wire codes (err:storage/AccessDenied, err:catalog/CredentialsNotVended) on the lazy path. - §1 now seeds the disk metadata layer from the inline copy. #1503's rung serves a zero-REST resolve only when metadata_from_caches hits, and a fresh process's in-memory cache is empty — so the pre-restack §1 would have starved the rung for exactly the catalogs that vend inline metadata (Snowflake), reviving the ~1-3s loadTable GET with the regression visible only in the cache gate's load_table.n. Seeding from the free inline copy arms the rung one touch EARLIER than #1503 alone. - §4: CACHE_FORMAT_VERSION stays at #1503's 2 (a payload change); our filename re-key needs no bump and is documented as orthogonal. path()'s key param generalized for the lt_key space; the pointer layer gets an explicit scope verdict (catalog-gated + already per-graph-source). Tests: fail-closed on the shared helper, explicit ambient opt-in still resolves, typed errors survive the builder channel, inline metadata seeds both caches. All were invisible to both suites before: ours predates the rung, #1503's predates SecretRef/fail-closed/typed errors.

aaj3f · 2026-07-21T20:29:03Z

Consolidating at maintainer request. #1503 (the loadTable-METADATA cache — a credential-free pointer + lazy storage for a zero-REST first-ask) is carried forward in #1507 (the R2RML/Iceberg virtual-dataset performance program), which brings the #1475–#1507 lineage forward as a single reviewable unit. Its verification of record is the program's corpus benchmark, published as C2-bench-wave1.md (native == virtual, 0 hash mismatches). The pre-consolidation branch tip is preserved at tag archive/pre-refactor-2026-07-21/perf_r2rml-loadtable-meta-cache — no history was rewritten. Closing in favor of #1507; discussion continues there.

aaj3f force-pushed the perf/r2rml-q031-refprune branch from eee6435 to c38d412 Compare July 15, 2026 17:17

aaj3f added 5 commits July 15, 2026 13:18

aaj3f force-pushed the perf/r2rml-loadtable-meta-cache branch from 6750286 to 6d9fe55 Compare July 15, 2026 17:34

aaj3f mentioned this pull request Jul 15, 2026

Typed storage errors are flattened on the disk-cache-fill Parquet route (500 instead of 403 err:storage/AccessDenied) #1510

Open

aaj3f closed this Jul 21, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

perf(iceberg): loadTable-METADATA cache — zero-REST first-ask (q031 72s→~6s arc) + F19 residual - #1503

perf(iceberg): loadTable-METADATA cache — zero-REST first-ask (q031 72s→~6s arc) + F19 residual#1503
aaj3f wants to merge 5 commits into
perf/r2rml-q031-refprunefrom
perf/r2rml-loadtable-meta-cache

aaj3f commented Jul 15, 2026

Uh oh!

aaj3f commented Jul 15, 2026

Uh oh!

aaj3f commented Jul 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

aaj3f commented Jul 15, 2026

loadTable-METADATA cache — persisted credential-free pointer + lazy storage (+ F19 residual)

The win (deterministic gate — GREEN, both runs)

Mechanism

Security (creds never persisted)

Snapshot-pin correctness (review fix, commit 6750286)

TTL staleness semantics

as_of_t forward-guard + wiring obligation

F19 residual (ride-along, commit ebcab75)

Tests

Uh oh!

aaj3f commented Jul 15, 2026

Uh oh!

aaj3f commented Jul 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Snapshot-pin correctness (review fix, commit `6750286`)

F19 residual (ride-along, commit `ebcab75`)