feat(virtual): serverless Lambda-usability + BYO-IAM — admission widenings, capacity guards, typed memory budget (consolidates #1505 + #1514) - #1514
Merged
Conversation
) Local-mode 'fluree info <gs>' and 'fluree iceberg info <gs>' printed the stored graph-source config verbatim, including catalog.auth.client_secret (a live Snowflake PAT). Route all four CLI print sites (local + remote defense-in-depth) through the existing fail-closed allowlist redactor, now re-exported as fluree_db_api::redact_graph_source_config. The issue's suggested type-level redacting Serialize is unimplementable: persistence requires lossless config serialization (CI-locked in ledger_info.rs 'storage serialization must be lossless' + server/CLI round-trip tests). Display-layer redaction via the shared allowlist is the correct seam; SecretRef-at-rest (#1500 §3) removes the secret from storage entirely.
…m S3 (#1500 §1) The REST catalog's loadTable response carries the parsed table metadata inline (LoadTableResponse.metadata — Snowflake Horizon/Polaris vend it), but the query path never read it: every cold table touch paid an S3 GET + parse for bytes the catalog already handed us (~85-98ms/table). Extract the resolution chain into resolve_table_metadata() with order: in-memory cache -> inline loadTable copy -> disk catalog cache -> S3 GET. The inline branch seeds the in-memory cache (cache-reconstructed loads carry metadata: None and hit it), writes NO disk-cache entry (that layer exists for processes never handed the bytes — Direct mode keeps it), and emits NO r2rml.read_metadata span (must fire only on a real S3 read; the live gate asserts n=0 for inline-vending catalogs). Fallback chain is unchanged for Direct mode and catalogs that omit inline metadata. Tests: inline_metadata_short_circuits_disk_and_s3 (panic-stub storage + panic-on-build disk cache prove neither is touched; mutation-verified), no_inline_metadata_falls_back_to_storage_read (exactly one storage read).
…1500 §4) Two hedges that make mechanism B (per-graph-source sts:AssumeRole) a localized edit instead of a format migration, and fix a latent full-miss: - path() now hashes the metadata_location with xxh64 (spec-defined, already a workspace dep) instead of std DefaultHasher, whose algorithm is not guaranteed across toolchains — a routine toolchain bump would silently re-key the entire cache dir. Golden test pins the value. - Filenames gain a scope segment ({hash}.{scope}.{suffix}.json), constant "shared" today. When B lands the scope must become a stable fingerprint of the catalog-auth scope (what rest_clients is keyed on) and NEVER the vended credential (rotates every loadTable => 0% hit rate). Documented at CACHE_SCOPE. - CACHE_FORMAT_VERSION 1->2; v1 orphans keep old names, are never opened, and are evicted oldest-first by the existing MAX_CACHE_BYTES mtime prune.
…owngrade (#1500 §5+§2) S3 AccessDenied previously surfaced as HTTP 400 INVALID_QUERY with the reason buried in an opaque string, forcing hosts to regex-classify and misfire (an admin missing a bucket grant gets told to rotate their PAT). And a REST source configured for vended credentials whose catalog vended none silently downgraded to the process's ambient AWS identity — a fail-open that granted a catalog-scoped source the shared role's entire S3 reach. - IcebergError::StorageAccessDenied {bucket,key,region,message}, classified at the 5 SDK send sites via error code + HTTP 403 (pure helper is unit-testable without SDK types). Display carries the s3:ListBucket caveat: no permission and object-moved are indistinguishable at the API. - QueryError/ApiError gain StorageAccessDenied + CatalogCredentialsNotVended; both map to HTTP 403 with wire codes err:storage/AccessDenied and err:catalog/CredentialsNotVended (fluree-vocab), on direct AND Query-wrapped paths. - decide_credential_source(): single tested decision matrix shared by the scan and preview paths. (true,None,REST) fails closed; ambient only via explicit vended_credentials=false; Direct mode never fails closed; cached loadTable entries preserve credentials verbatim so the guard cannot false-fire on cache hits. - build_preview_storage now enforces the same policy, so wizard preview/generate cannot green-light a credential path queries won't use. - Catalog-side 401->refresh-retry (rest.rs) untouched.
…ver injection (#1500 §3) The engine half of the credential-rotation fix (fluree/solo#782): configs can now carry an opaque secret *reference* instead of a baked literal, resolved at client-build time by a host-injected resolver. - ConfigValue::SecretRef { secret_ref } — declared BETWEEN Literal and Dynamic: untagged serde tries variants in order and Dynamic's all-default fields match any object, so a later position would silently swallow the ref (ordering canary test pins this). Wire shape {"secret_ref":"<opaque>"}; db never parses the ref. resolve() on a SecretRef fails closed with an actionable error (OSS/CLI contexts). - #[async_trait] SecretResolver + SecretResolveError{Denied,NotFound, Unavailable}: host implements authorization internally; db stays tenant-agnostic. Injected via Fluree::with_secret_resolver (cheap clone — Fluree is now Clone) or FlureeBuilder::with_secret_resolver. - AuthConfig::hydrate() rewrites SecretRef→Literal at the four create_provider_arc sites; create_provider_arc stays sync. On the scan path hydration sits INSIDE the rest_clients cache-miss arm, strictly AFTER rest_client_cache_key fingerprints the RAW reference-bearing config — hydrating first would re-key the client cache on every rotation (resolver call + OAuth exchange per query instead of once per 900s TTL). secret_ref_fingerprint_is_rotation_stable pins this. - Wrapper-boundary hydration (hydrate_conn) for browse/preview/sample/ ephemeral/generate/validate; connection-test errors before any network I/O so solo's connection_tested gate catches missing resolvers. - Builders (frozen cross-repo names) at all three levels: with_auth_oauth2_client_secret_ref / with_auth_bearer_token_ref on IcebergConnectionConfig, IcebergCreateConfig, R2rmlCreateConfig. - Display redactor preserves strict {"secret_ref":"<string>"} objects under secret keys (a reference is not a secret — same policy as env-var names); every other shape stays redacted (canary extended).
The Direct branch of load_table_context rebuilt S3IcebergStorage (from_default_chain: env -> ~/.aws -> IMDS/ECS network legs, plus a fresh connection pool) on every call, and did so even when its own metadata-location cache hit — the REST branch has used the query session's storage cache all along. Mirror it: direct_session_storage() checks session.cached_storage first and store_storage's after building; construction now sits below the 2s metadata-location cache check in both arms. Direct never calls store_load_table (no vended credentials to rotate), so the cached client is never invalidated mid-query — first build wins for the whole query. Tests: direct_session_storage_reuses_arc_across_calls (Arc::ptr_eq across two calls, distinct per table key) and a session-layer guard that cached storage persists absent a fresh loadTable.
…s gate - format/sparql.rs: semicolons on the () write_node branches (semicolon_if_nothing_returned, deny-level in this crate) - catalog/rest.rs: drop with_catalog_semaphore — cfg(test) helper whose consuming test was removed on the perf line (zero callers) - scan/topk.rs: vec! -> arrays in test fixtures (useless_vec) All pre-existing on the perf-line base; fixed here because the branch gate is cargo clippy --workspace --all-features --all-targets -D warnings at the pinned 1.96.1 toolchain.
…engine's own credential path (#1500 §6) Solo's wizard needs to prove at configure time that the stack can read a DATA file — not merely reach the catalog — and it must do so through the exact credential + storage construction queries use, or Test goes green over a broken feature. - verify_storage_access(conn, table) + Fluree::verify_iceberg_storage_access (hydrate_conn first — SecretRef-aware): loadTable honoring io.vended_credentials -> build_preview_storage (§2 fail-closed fires before any probe) -> manifest list + manifests read (proves metadata/ prefix) -> HeadObject on the first data file (proves data/ prefix). Prefix-scoped grants that cover only one prefix fail here exactly as queries would. - StorageAccessReport: credential_source/metadata_location/ data_files_listed/probed_data_file(_bytes)/data_probe_skipped/skip_reason (wire shape pinned by a golden JSON test). Empty or snapshotless table reports the data probe as skipped, not failed. - POST /v1/fluree/iceberg/catalog/verify mirroring /preview (admin-gated); response body is the report. - All probe reads map through storage_api_error: a denial surfaces as 403 err:storage/AccessDenied naming bucket/key. - Direct mode rejected identically to preview (no REST catalog to query); like preview, requires the catalog to vend inline metadata.
Pre-existing on the perf line; params mirror the exec-one CLI surface 1:1. Last lint blocking the workspace clippy -D warnings gate, which is now clean at the pinned 1.96.1 toolchain.
…allowlist Adversarial-review finding S1: the doc comments claimed 'allowlist redaction that FAILS CLOSED', but redact_graph_source_config redacts only values under SECRET_CONFIG_KEYS (a denylist) — an unrecognized secret-bearing field prints verbatim. All current secret leaves ARE covered (no live leak); the fail-closed property applies to unparseable input only. Say so accurately: a maintainer must add new secret fields to SECRET_CONFIG_KEYS.
…e-METADATA cache (#1500 x #1503) The restack onto the pointer-rung/lazy-storage chain applies textually clean but is semantically wrong in four ways: #1503 did not replace the two sites this branch hardens, it CLONED them into a deferred copy that none of the guards reached. Fixed by unifying, not by patching copies. - rest_session_storage(): ONE helper holding the §2 fail-closed decision, the #1498 session reuse, and the vended/ambient construction. The eager arm and the deferred LazyS3Storage builder now share it. Previously the builder's copy had no §2 guard at all, so any lazy-forced storage build silently downgraded a vended-configured source to the process's ambient AWS identity — the exact fail-open §2 exists to close, resurrected. - hydrated_auth_provider(): pairs hydrate() with create_provider_arc() so a client-construction site cannot ship un-hydrated. #1503's pointer rung added a second create_provider_arc that §3's hydrate never reached, so a ConfigValue::SecretRef hard-errored via resolve()'s fail-closed arm exactly when the pointer cache was warm — i.e. rotation broke on the fast path. Hydration still runs strictly AFTER rest_client_cache_key in both arms. - Typed errors now round-trip the builder's IcebergError channel (new IcebergError::CatalogCredentialsNotVended + a storage_query_error lift) instead of being flattened by a blanket to_string(), which had destroyed the 403 wire codes (err:storage/AccessDenied, err:catalog/CredentialsNotVended) on the lazy path. - §1 now seeds the disk metadata layer from the inline copy. #1503's rung serves a zero-REST resolve only when metadata_from_caches hits, and a fresh process's in-memory cache is empty — so the pre-restack §1 would have starved the rung for exactly the catalogs that vend inline metadata (Snowflake), reviving the ~1-3s loadTable GET with the regression visible only in the cache gate's load_table.n. Seeding from the free inline copy arms the rung one touch EARLIER than #1503 alone. - §4: CACHE_FORMAT_VERSION stays at #1503's 2 (a payload change); our filename re-key needs no bump and is documented as orthogonal. path()'s key param generalized for the lt_key space; the pointer layer gets an explicit scope verdict (catalog-gated + already per-graph-source). Tests: fail-closed on the shared helper, explicit ambient opt-in still resolves, typed errors survive the builder channel, inline metadata seeds both caches. All were invisible to both suites before: ours predates the rung, #1503's predates SecretRef/fail-closed/typed errors.
…mbers Component 1 of the lambda-usability epic. Every chat SPARQL is executed as a FROM <ledger> dataset query (the single-view path rejects SPARQL), so it routes through DatasetOperator, which inherited the no-op set_row_budget/set_topk — a bare LIMIT never reached the R2RML scan and the whole table was materialized. Mirror GraphOperator's wrapper forwarding: record the directive and thread it into each member's inner subtree at build time in open(). Switch-gated FLUREE_R2RML_DATASET_BUDGET (default on; OFF byte-identical to pre-T1.3). Hermetics: forwarding to member (budget + top-k), no-directive baseline, Sort/Distinct absorb-boundary on the dataset path. fluree-db-query lib 1265 green.
…tadata path Component 2 of the lambda-usability epic. get_data_model / any /info over a virtual dataset ran native full-stats (assemble_full_stats), which federates into a full-table scan of every backing Iceberg table — the deployed 900s runaway. A metadata-only graph-source path already exists (build_graph_source_info: counts from Iceberg manifests, classes/properties from the mapping, NDV/flakes null) but was reached only on the ledger-not-found fallback. Tightened predicate (team-lead refinement): route to the metadata path iff the id resolves to an EMPTY genesis shell (ledger.t()==0, cheaply read off the head, no stats assembly) AND a graph source is registered. This fixes the deployed empty-shell+GS runaway while keeping a real native ledger, a non-GS id, and the exotic hybrid (t>0 + graph sources) on the complete native path — no under-report, no tradeoff to document. GS lookup runs only for empty shells; a lookup failure falls through to native (fail-safe). Consumers null-safe (data_model.rs reads count/total_instances present; MCP formatter guards NDV). Hermetic: serve_virtual_stats_from_metadata truth table (empty+GS / native / hybrid). fluree-db-api lib 796 green (iceberg).
…ain loops Component 3 of the lambda-usability epic. The main streaming scan already polls cancellation per batch (operator.rs next_batch — the load-bearing poll for a large fact-table scan), so the deployed many-file FACT cancels already bite near the deadline. This closes the drain helpers that materialized a whole scan WITHOUT a checkpoint: collect_stream (parent-dimension lookup), collect_scan_capped (cached inner), and the three fused-aggregate scan drains. A deadline/abort now stops those mid-sweep instead of after the full drain. Ripple-free: threaded the existing ctx into the two collect helpers (their only callers are in build_progress, which has ctx) — NO scan_table signature change, so the ~13 provider mocks are untouched. Cheap: check_cancelled is a relaxed atomic read. Residual (documented): a single-file table's monolithic decode is one stream item, so the checkpoint fires only after that file decodes — bounded to one file (deployed single-file DIMs are <=~31s, under the 55s deadline) and split into row-group yield points by T2.3. Hermetics: collect_stream / collect_scan_capped return Cancelled up front on a pre-cancelled query. fluree-db-query lib 1267 green.
…-file scans Component 4 of the lambda-usability epic. A single-file table pins the scan's file-level concurrency to 1 (min(cores, files)), so its whole decode runs on ONE core while the other vCPUs sit idle — DIM_CUSTOMER (1.7M rows / 1 file) at ~56k rows/s/core. Decode the file's row groups across multiple blocking tasks instead. Division formula (team-lead ruling): the scan grants each file per_file_rowgroup_concurrency = max(1, available_parallelism / file_concurrency), where file_concurrency = iceberg_scan_concurrency(num_files) (the buffer_unordered width). 1 file on 6 cores -> 6 per file; 3 files -> 2 each; 64 files -> 1 = no oversubscription. decode_large_file reads the row-group count once, splits 0..n into that many contiguous slices, and decodes each via decode_batches_arrow with a row-group subset; row groups are independently decodable so the union == a sequential decode (order-independent for a scan). Declines: switch FLUREE_ICEBERG_ROWGROUP_PARALLELISM off; a bounded max_rows peek (decline v — a small budget wants only the first row group); a single row group; the top-k path (reads bound-ordered files sequentially — keeps the default concurrency 1). No scan_table signature change, so the scan mocks are untouched. Residual: the per-slice tasks each re-read the small footer (N+1 footer GETs); bounded and cheap relative to the decode saved. Memory: N concurrent row-group decodes are resident at once (bounded by concurrency x row-group size). Hermetics: subset [0]++[1] == full decode (parity) + empty-subset no-op; split_row_groups partition. fluree-db-iceberg 199 green; fluree-db-api 796 green.
…erification)
The fused-aggregate machinery (grouped COUNT/SUM/AVG single-table + fact-dim join)
already exists (PR-6/F22); the deployed rollups likely DECLINE it because
detect_fused_r2rml_aggregate requires a Pattern::Graph{Iri} wrapper the dataset
path (the C1 root) may not present. Reframes slice 1 from 'build new' to 'admit
the dataset-path shape', names the one EXPLAIN that decides it, and lists the
adversarial hooks. For lambda-audit's review before slice-1 impl.
…ts, mechanism, soundness)
The fused-aggregate fold is COMPLETE (single-table + fact-dim join, resolve_at_open
+ resolve_join_at_open); slice 1 = lift the FIRST decline the deployed grouped-COUNT
shapes hit (detect:338 Pattern::Graph{Iri} on the dataset path, most likely). Adds
the decline receipts (file:line), the FK-key count+join mechanism from the IR, the
soundness invariants (null FK, #1490 dup-key, filter/DISTINCT decline, multi-key is
ALREADY supported), hermetics, predicted effect, switch reuse, and dataset-path
corpus members.
… to slice 2 Tracing the dataset path: maybe_wrap IS called (dataset_query.rs:127) and resolve_as_graph_source SETS graph_source_id (fluree_ext.rs:663) -> the Pattern::Graph wrapper is present -> detect:338 passes; and the dataset execution reaches the fused-enabled build_operator_tree (runner.rs:466) -> detect runs (no bypass). So the decline is likely SHAPE-specific, not path/wrapper: family-A/C carry SUM/AVG (slice 2), and family-B COUNT-only may already fuse. Re-points the EXPLAIN to check family-B (already fuses?) vs family-C/SUM (declines at SUM/AVG = the real lever).
…ed star_constraints lambda-audit O1 (ship-blocker + a CURRENT correctness bug): the fused aggregate fold has ZERO star_constraints handling (grep = 0), but the rewrite folds a constant-object member (?c ex:isCurrent true) into the R2RML pattern's star_constraints, and the single-pattern gate then admits it — so a fused COUNT over a flagged star silently IGNORES the constraint and OVER-COUNTS (the normal scan applies it, operator.rs:524/1719). Any such query that reaches the fused path today over-counts. Guard (fold_over_star_constraints): decline the fuse when ANY participating R2RML pattern carries star_constraints — broadened per the team-lead to EVERY map (the aggregated fact AND every joined dim, both the single-table resolve_at_open and the fact-dim resolve_join_at_open; the q029 eventType family carries fact-side constant-object constraints). Slice 1.5 will teach the fold to APPLY the constraint and lift this for that shape. TDD tripwire: fold_over_star_constraints_declines_a_constrained_pattern (red under an unguarded fuse, green with the guard; also traps a future lift). fluree-db-query lib 1268 green.
… + O2 mixed-dataset guard The fused R2RML aggregate fold already exists, but the single-table path (resolve_at_open) declined any GROUP BY key on an un-annotated column (group_kind(None)), so the deployed family-A rollups — single-DIM COUNT and single-table COUNT+SUM grouped by a natural string attribute (INDUSTRY, ORDER_CHANNEL, ...) — fully materialized on BOTH the GRAPH and dataset (FROM) paths. A live SF01 discriminator confirmed: detect fires on both paths (the Pattern::Graph wrapper IS present on the dataset path), and the decline is the resolve_at_open group_kind(None) gate, not the detect gate. Family-C SUM fact-dim joins (single- and 2-hop) already fuse; slice-2 needs no new fold. Slice-1: - default an un-annotated group-key column to xsd:string on the single-table path, mirroring the join path's unwrap_or(xsd::STRING); admits family-A. The q022-sentinel over-count the old decline guarded against is now caught earlier by the star_constraints guard (O1), so widening is sound. - O2 mixed-dataset guard (dataset_is_single_source): admit the fused path only when the dataset resolves to exactly one distinct source ledger (the deployed FROM <gs> lists the graph source as both a default and a named graph — one ledger). A native member or a second graph source declines (the single-scan fold would under-count vs the DatasetOperator union). Gates both the single-table and join paths. - hermetic o2_single_source_admits_deployed_shape_declines_mixed. Switches unchanged (FLUREE_FUSED_R2RML_AGG / _AGG_JOIN); off = byte-identical materialize. fluree-db-query lib 1269 green.
…ting family-A fusion + the over-count trap The corpus was GRAPH-path only (fluree.graph). Add the deployed solo shape — a `FROM <graph-source>` dataset query through DatasetOperator — so the C1 dataset-path budget forwarding AND the C5 dataset-path fused-aggregate admission are corpus-gated forever. - exec.rs: exec_once now routes a query whose text carries a top-level `FROM <…>` clause through fluree.query_from() (auto-applies with_r2rml under the iceberg feature); everything else stays on the graph path. is_from_query detects the clause (comment lines ignored). - corpus.rs: QueryDef gains an `exec_path` (Graph|From, default Graph via #[serde(default)] so no existing entry changes); count assertion 59 -> 61. - q060 (family-A, 2-key single-DIM COUNT: Store by channel,storeType) — parity member: the fused answer must equal the materialized oracle. - q061 (family-B over-count trap: Customer isCurrent by segment) — the isCurrent flag folds into star_constraints; the fused fold has no constraint handling, so it MUST decline (O1) and return the CONSTRAINED current-only count. A silent fuse would over-count (all customers) and fail this member. Oracles blessed from a fusion-OFF (materialized) native run — an independent ground truth, so the gate proves fused==materialized rather than fused==fused. Gate on virtual-sf01 (fusion ON): 0 hash mismatches, 0 perf violations — q060 fuses and matches (15 groups), q061 declines and returns the constrained count (3 segments). vbench bin tests 33 green; scoped clippy clean.
…g/IRI group key Slice-1's `unwrap_or(xsd::STRING)` default for an un-annotated group-key column is a PLAIN-LITERAL assumption. A column whose object map is language-tagged (rdf:langString) or IRI-/blank-node-typed materializes a term the fused fold would emit as an xsd:string literal — a different datatype / language / term-type than the generic materialize path — so the grouped key would disagree with the materialized answer. `scalar_column_for_var` discards the object map's `language` and `term_type`, so add `group_key_col_is_plain_literal` and decline any non-plain-literal group key on the single-table path. Decline-only: it never widens admission past the plain-literal case, so the deployed corpus (whose dim group keys are all plain string columns) is unchanged and the 3cf7dc4 full-corpus gate stays valid. The join path carries the same `unwrap_or(xsd::STRING)` default in pre-existing base code (out of this PR's scope) — flagged as a follow-up. Hermetic q2_group_key_plain_literal_gate covers plain / typed-literal (admit) vs lang / IRI / blank-node (decline).
…tic, comment credits both mechanisms Adversarial-pass follow-ups on slice-1: - Q1 (O2 residual): the mixed-dataset dedup keyed on ledger_id alone would collapse the SAME ledger at two different to_t snapshots, or under two different policy enforcers, into one view and wrongly admit, while the materialize path unions both. Extend the dedup key to the full (ledger_id, to_t, policy) view tuple — GraphRef exposes to_t and policy_enforcer; policy identity is by Arc pointer (distinct instances read as distinct views, decline-only). Hermetic now covers same-ledger/different-to_t and different-policy declines. - Parity hermetic default_string_group_key_lexical_parity: locks the None->xsd:string default's parity-by-construction — a String column reads its raw value as GKey::Str (an xsd:string literal via the same dt_sid fallback the materialize LiteralEncoder uses for an un-annotated column) and NULL reads as GKey::Null (validity-drop == BGP exclusion). The int/decimal/null-decimal cases stay locked by key_at_reads_integer_group_key_from_decimal; the 2-key case by corpus q060. - :1341 comment now credits BOTH retirement mechanisms — the star_constraints guard (O1) for the flag over-count, and LiteralEncoder xsd:string-fallback parity (operator.rs:1637/1643) for the default's group-key term — and scopes the "natural string mapping" claim to plain-literal columns (Q2 gates the rest). fluree-db-query lib 1270 green; scoped clippy clean.
…3 (family-A ORDER BY/OFFSET) Two more FROM <gs> members, per the review riders: - q062 (family-C, fact-dim SUM: InventorySnapshot on-hand by store) — gates the already-fused join fold on the deployed dataset-path shape (the join fold predates this PR but was only ever corpus-exercised via fluree.graph, never FROM). - q063 (family-A COUNT with ORDER BY DESC(count) + OFFSET 2) — exercises the sort and offset wrapper around the fused single-table aggregate on the query_from path; total order (count desc, industry asc) keeps the OFFSET-truncated set deterministic so hash_gate=full holds. Oracles blessed from a fusion-OFF materialized native run. Isolated gate on virtual (fusion ON) at the final head: 0 hash mismatches, 0 perf violations — q062 fuses via the join path (500 rows), q063 fuses through the wrapper (8 rows). Corpus count 61 -> 63; vbench bin tests 33 green; scoped clippy clean. Note: the Q2 lang/IRI sentinel is NOT a corpus member — the enterprise mapping has no rr:termType-IRI or rr:language column to group by (no such shape exists in the data, so no materialized oracle). The q2_group_key_plain_literal_gate unit hermetic is its gate.
…n path The single-table Q2 gate (977bde9) left the join path's identical unwrap_or(xsd::STRING) default for its terminal-dim group key unguarded — the same latent parity-break class in the pre-existing (base) join fold. Wire the shared group_key_col_is_plain_literal helper into resolve_join_at_open's group-key resolution so a lang-tagged or IRI-/blank-node-typed dim key declines symmetrically with the single-table path. Decline-only: no deployed dim key is lang/IRI (store name, region, category are plain string columns), so prod behavior is unchanged — validated live: family-C members q008 (2-hop), q032 (single-hop), q062 (FROM) still fuse and match their materialized oracles (0 hash mismatches, 0 perf violations). The shared-predicate hermetic q2_group_key_plain_literal_gate now documents both-path coverage. fluree-db-query lib 1270 green; scoped clippy clean.
aaj3f
marked this pull request as ready for review
July 17, 2026 13:48
class_fusion_is_safe refused to fuse the rdf:type into the star whenever the star's base predicate also appeared on ANOTHER class's TriplesMap — even when the class's OWN map co-locates the predicate and the other map has disjoint subjects. That refusal emitted a 2-pattern shape (star + a separate class-only scan) that failed the fused single-pattern gate and materialized. It is the round-2 deployed timeout for every group-by on a shared predicate: ex:category is a plain column on both Product and SupportTicket, so grouping either class by category declined and materialized the fact (lambda-audit's two-scan telemetry: an un-fused bare subject-key scan alongside the aggregate scan). Fix in fuse_class_if_safe: when strong class_fusion_is_safe fails but (i) at least one class-declaring map co-locates the base predicate AND (ii) every non-class map is subject-template DISJOINT from the class maps (wildcard_class_fusion_is_safe), fuse to a single scan (class_filter) instead of the weaker 2-scan class_prune_hint. Sound: resolution pins the one (class, predicate) map, and disjointness guarantees no dropped predicate map shares a class-instance subject (no lost bindings). Vertical partition (no class map has the predicate) still refuses; strong fusion (every predicate map is a class map) is unchanged. Hermetics: class_fusion_shared_category_across_two_classes (the ex:category shape), class_fusion_when_shared_predicate_disjoint_colocated (now fuses, was hint), class_prune_hint_refused_under_vertical_partition (still refused). Corpus q064 (Product-by-category, FROM path, shared edw:category on Product+SupportTicket): fused == materialized oracle (confirmed against a fusion-OFF native run), 0 hash mismatches. The single-pattern fused-aggregate O1/Q2 guards are unchanged — a shared-predicate+flag star still carries star_constraints and declines at O1. fluree-db-query lib 1271 green; corpus 63->64; scoped clippy clean.
…flag slice-1.5) The fused fact⋈dim join declined on ANY folded constant-object constraint (star_constraints) — the blanket O1 guard — so a dim flag like `?prod ex:isCurrent true` forced the whole join to materialize the fact. It is the round-2 deployed timeout for the two headline shapes (#1/#2: OrderLine/SupportTicket ⋈ Product with a dim isCurrent flag). E2 replaces the blanket decline with constraint APPLICATION: - dim-side constraints are enforced while building the FK→GKey map — a dim row failing its flag is skipped, so its join key never enters the map and fact rows probing it drop (inner-join + constraint semantics); - fact-side constraints are enforced per fact row in the fold (next_batch); - resolve_star_constraint_checks resolves each (predicate, constant) to a scalar column PredicateObjectMap and DECLINES (falls back to materialize) any constraint that is a RefObjectMap object or absent — never a silent over-count; - the constraint columns are projected into the respective scans; the equality reuses the normal scan's exact primitives (materialize_object_from_batch + object_column_is_numeric + rdf_term_eq_object_constant_cached, now pub(crate)), so the fused constrained count is byte-parity with the materialized answer. The single-table O1 decline is unchanged (slice-1.75). The COUNT(*) manifest shortcut gains an explicit fact_constraints guard (belt-and-suspenders; the join path always groups, so it never coincided). Hermetics: e2_resolve_star_constraint_checks_scalar_vs_ref (scalar admits, ref / missing declines), e2_row_satisfies_boolean_flag (isCurrent true/false/null under the normal-scan equality). Corpus q065 (Order⋈Customer, dim isCurrent flag, group by segment, FROM path): fused == materialized CONSTRAINED oracle, 0 hash mismatches — the join-path over-count trap. Live: fusion-OFF materialize 52s → fusion-ON E2 0.45s (~115x), same constrained count. fluree-db-query lib 1273 green; corpus 64->65; scoped clippy clean.
Pure formatting — canonicalizes the O2/Q2/parity/E2 test blocks under the env rustfmt (standard max-width reflows: multi-line arrays, wrapped calls). No logic change; the E1+E2 engine + hermetics are identical to 818bcf0.
…te fold Round-3 forensics: heavy multi-FACT chat queries OOM'd at the 10 GB Lambda max after 344s/818s of SILENT, uncancelled post-scan work — the deadline is polled in the scan-drain loops but the two operators the query runs after scanning have NO cancellation poll: GroupAggregateOperator (the streamable-aggregate fold) and HashJoinOperator (whose build side drains FULLY in open()). So an over-budget fold ran minutes to an OOM instead of a clean 180s timeout. Thread ctx.check_cancelled()? at batch granularity into the four uncancelled loops: GroupAggregate's general (:903) and partitioned (:830) drains, and HashJoin's build_table drain (:667) and probe drive loop (:768). Batch-granularity only — the per-row folds between polls are bounded by one batch. Capacity protection: it makes the OOM class fail fast with a clean typed Cancelled at the deadline instead of burning a 10 GB container for ~13 minutes. (The sub-180s OOMs that die before the deadline need the memory-budget guard — R3-B, separate.) Hermetics: pre-cancelled ctx -> HashJoin open() aborts Cancelled mid-build; GroupAggregate aborts Cancelled mid-fold. fluree-db-query lib 1275 green; scoped clippy clean.
…egate fold Round-3 forensics: the sub-180s OOMs (156s/163s/170s) fill the 10 GB Lambda max BEFORE the deadline fires, so R3-A cancellation can't save them — they need a memory bound. There is NO spill or bound anywhere in the join/aggregate pipeline. Add a typed pre-OOM guard (QueryError::MemoryBudgetExceeded — NOT a panic, and 507 Insufficient Storage at the API boundary, distinct from the 408 timeout so the caller degrades on it specifically): - HashJoin: the build side buffers the ENTIRE build input in open() (the confirmed round-3 OOM driver — a multi-FACT join). Account it per build batch and abort typed once the accumulated bytes cross the budget. - GroupAggregate: guard the groups map size per batch (a high-cardinality GROUP BY). SUM/COUNT rollups have few groups so it never trips for them. - Budget = env FLUREE_QUERY_MEMORY_BUDGET_BYTES (0 disables) else ~78% of the detected container/system limit (cgroup v2 memory.max -> v1 -> /proc/meminfo), else an 8 GiB absolute fallback. On the 10240 MB query Lambda -> ~8 GiB, aborting ~2 GiB before the hard OOM. Accounting is approximate (batch-granularity, rows x cols x a conservative per-binding estimate) — over-counting only aborts a query already near OOM. Capacity protection: with R3-A it makes the OOM class fail fast and typed (timeout OR budget-exceeded) instead of a raw Runtime.OutOfMemory 502 that burns a 10 GB container. The make-it-succeed levers (2-FACT rollup fusion, LIMIT-through-join, semi-join) are the parked follow-ups. Hermetics: tiny budget -> HashJoin build + GroupAggregate fold abort typed; default budget -> corpus unchanged (small data never approaches it). fluree-db-query lib 1277 green; fluree-db-api builds; scoped clippy clean.
Fold the separate R3-A cancellation polls and R3-B per-operator memory guards into ONE cooperative checkpoint, so every buffering post-scan operator polls the same mechanism for both the deadline AND the memory budget. Supersedes the separate framing in 2ef9a25 / 3c6a31f. Mechanism (fluree-db-core cancellation.rs): the shared QueryCancellation handle now also carries a query-scoped `allocated: AtomicUsize` usage counter plus an optional `memory_limit` ceiling — pure mechanism, core never enforces. Every derived per-graph ExecutionContext already clones this handle, so the counter is automatically shared across all of a query's operators: one budget covers a hash-join build AND a downstream GROUP BY fold (a fused query's 36M interior map is under the same budget from day one). New: record_alloc / allocated_bytes / set_memory_limit / memory_limit. Policy (fluree-db-query context.rs): ExecutionContext::checkpoint() checks cancellation FIRST (an external timeout / RSS-watchdog signal wins → Cancelled/408), then the budget (allocated_bytes > budget → MemoryBudgetExceeded/507, distinct so a caller tells "over memory" from "over time"). Budget = the handle's pinned ceiling if set, else query_memory_budget_bytes() (env override; else 78% of the detected cgroup v2->v1->/proc/meminfo limit; else 8 GiB; 0 disables). record_alloc / mem_used also exposed on the context. Sites (7 checkpoints, 6 with record_alloc): hash_join build_table + probe drive; group_aggregate general + partitioned drains; fused_aggregate main fold + terminal-dim + interior-dim map builds. Non-buffering scan drains keep plain check_cancelled (no retained memory to guard). mem_used_bytes emitted on the group_aggregate spans as growth-curve telemetry. Layering with solo's deployed RSS watchdog: engine budget is the first line (typed, fires at 78% before the 85% RSS trip); the watchdog is the generic-cancel backstop. The typed MemoryBudgetExceeded stays distinguishable from the watchdog's external Cancelled at the API surface. Capacity protection, no behavioral change on in-budget queries. The make-it-succeed levers (2-FACT rollup fusion, LIMIT-through-join, semi-join) remain the parked follow-ups. Verify: 4 R3 hermetics green (pre-cancelled -> Cancelled mid-build/ mid-fold; 1-byte handle ceiling -> typed MemoryBudgetExceeded from HashJoin build AND GroupAggregate accumulation); 3 new core mechanism unit tests; fluree-db-query lib 1277 + fluree-db-core lib 702 green; fluree-db-api builds; scoped clippy clean on both crates.
…e BGP + datatype coercion
A constant-object key equality only pushed as an Iceberg scan filter when it was a
pattern's LONE base object_constant. The moment the subject carried a second
predicate (e.g. `?ol ex:orderLineKey "1"; ex:order ?ord`), the key equality
partitioned into star_constraints, which build_scan_filters never consulted and
materialize_batch enforced only residually — so the whole FACT was READ and
filtered post-scan. The round-3b point-lookup fanned a 156 M-row 2-FACT scan to OOM.
PRIMARY: push each SCALAR star_constraint as a scan filter under the same
exactly-one-scalar-pushdown-column soundness gate as the base object_constant
(shared via push_scalar_eq_filter). A constant key equality now prunes the scan
even alongside other predicates. IRI/decimal/double constraints stay
operator-enforced only. The residual check still runs post-push (the push is a
conservative file-prune optimization; materialize_batch remains the authority, and
star_constraints stay in topk_residual_filter_present).
SECONDARY (datatype coercion, hard soundness bar): the residual compares a
Scalar(Str) constant LEXICALLY against the column's rendering, so a pushed prune is
admissible only when its match-set PROVABLY covers the residual's. coerce a string
literal to Int against a declared xsd-integer column ONLY when it round-trips
canonically (n.to_string() == s ⇒ lexical-eq ⟺ integer-eq); non-canonical forms
("01", "1.0", "+1", " 1", non-numeric) DECLINE the push (residual stays correct).
A string on a string column pushes as-is (lexicographic prune); values already
carrying a pushable, residual-matched type push as-is.
Does NOT fix the 071cd59f wildcard-crawl shape:
PREFIX ex: <https://example.org/enterprise-byo/>
SELECT ?p ?o WHERE {
?ol a ex:OrderLine ;
ex:orderLineKey "1" .
?ol ?p ?o .
} LIMIT 20
There the constant is a separate object_constant scan joined to an UNCONSTRAINED
wildcard scan (the OOM driver), so no star_constraint reaches the wildcard. That
requires the fold-onto-wildcard rewrite (attach the same-subject const-object as a
star_constraint on the co-located wildcard so this pushdown then prunes it) — filed
as W4-1b, blueprint in lambda-round3-engine-analysis.md, rides on this PR.
Hermetics: coercion refutation matrix (canonical coerces; "01"/"1.0"/"+1"/" 1"/
non-numeric decline; string/decimal columns push as-is), push-helper resolution +
soundness gate (single-column pushes, duplicate-predicate and non-column decline),
and end-to-end build_scan_filters proving a folded star_constraint now pushes.
fluree-db-query lib 1280 green; scoped clippy clean. No corpus run — the wave-4-head
gate (after W4-1b + W4-2) covers parity; nothing deploys mid-wave.
resolve_join_at_open resolved EVERY GROUP BY key on the terminal dim, so a fact-column key (#7's `GROUP BY yearNum (date dim), shipMethod (fact)`) failed scalar_column_for_var(terminal_dim, shipMethod) and declined → 156s materialized fold → deadline. Route each key to its single source instead: fact-column keys read inline from the fact scan, dim-attribute keys from the FK→GKey map, composed per SPARQL order. - KeySource plan + assemble_group_key interleave fact-inline and dim-resolved positions; group_cols stays one-per-position so the output binding() is unchanged. Both existing paths are special cases (all-Fact = single-table, all-Dim = today's join, byte-identical). - Gate Q1 (route_group_key_sources): a key's source must be EXACTLY ONE participating pattern — 0 (unbound), ≥2 (cross-source value-equality the fold cannot enforce), or an interior-dim source all decline (v1 admits fact or the terminal dim only). No fact-wins tiebreak. - NULL in ANY key position drops the fact row (BGP unbound-object semantics): assemble_group_key's per-position Null-drop is the robust guarantee, with validity_cols (a fact group key is a fact object var) as the secondary drop; dim keys drop via the map-miss. Symmetric across sources. - Q2 plain-literal gate applied per key on ITS OWN source pattern/TM (fact AND dim). O1/E2 star_constraints, #1490 dup-key decline, and the memory/cancel checkpoints are untouched. Empty-dim-subset (all-fact keys over a join) degenerates the FK→GKey map to a join-existence set. - Hermetics: route_group_key_sources (mixed both orders, cross-source/interior/ unbound declines, all-fact); assemble_group_key (interleave both orders, fact-null + dim-null drops, no-resolver defensive drop, and the empty-dim-subset existence-only slice `Some(&[])`). - Corpus q066 (FACT_SHIPMENT mixed COUNT via FROM: string fact key + integer dim key; COUNT-only because SHIP_COST is xsd:double and the f64 SUM is summation-order dependent) + q067 (FACT_SUPPORT_TICKET mixed COUNT + SUM over the xsd:integer csatScore — exact i128, hash-deterministic — exercising the mixed key path together with a value fold). Oracles blessed at the wave-4 corpus gate. Riders folded per review: - W4-3 DEFENSIVE cancellation: values.rs ValuesOperator now polls check_cancelled per batch, bounding the O(input_rows × value_rows) match for a large VALUES product. This is capacity protection, NOT the round-3b #9 fix — #9's timeout is the un-lowered full scan (the VALUES→IN-set lowering is the real fix, next-slate); this poll only makes the execution-side product cancellable. - W4-1: pin "-0" (parses to 0, canonical "0" ≠ "-0" → declines) and a >i64 overflow string (parse Err → declines) in the coercion refutation hermetic.
…d crawl wildcard
071cd59f (`?ol a OrderLine ; orderLineKey "1" . ?ol ?p ?o`, the agent "show me
everything about entity X" idiom) full-scanned 120M FACT_ORDER_LINE: the
const-object `orderLineKey "1"` was emitted as a STANDALONE joined key scan while
the crawl wildcard scanned unconstrained. W4-1's pushdown could not engage because
the constant was a separate scan, not a constraint on the wildcard. Fold the
const-object members onto the co-located crawl wildcard as star_constraints, so
W4-1's build_scan_filters pushes the scalar equality as an Iceberg scan filter and
prunes the crawl scan to the matching subject(s). Semantically identical (the
star_constraint is the same existence filter the join was).
REWRITE (ordering fix a): DEFER the fold until AFTER the class-fusion loop.
Attaching star_constraints in the star loop would make is_standalone_wildcard false
→ try_fuse_wildcard_class SKIPS the class prune → the wildcard fans out to ALL
TriplesMaps (worse than the OOM). A class_filter-tolerant finder (is_crawl_wildcard)
re-locates the now-class-fused wildcard and attaches the constraints, so the two
mutations (class prune + constraint fold) compose. Guard: fold only when var_members
is empty AND exactly one co-located crawl wildcard exists AND every const member is
foldable; else standalone (no regression). Defensive standalone re-emit if the
wildcard is gone.
EXECUTION (ship-blocker caught + upheld via lambda-audit-2's second pass): a folded
wildcard has {predicate_var: Some, predicate_filter: None, star_bindings: [],
star_constraints: non-empty}, so has_star_members()==true routed it AWAY from the
wildcard `?p`/`?o` emission loop into the fixed-predicate star machinery — which
emitted a subject-only row (?p/?o UNBOUND) and read only the constraint column/POM
(the production shape would return ZERO triples). A wildcard (predicate_var.is_some())
is now treated as a true wildcard at all three sites — projection (all columns),
filtered_poms (all POMs), and materialize_batch (skips the fixed-star branch) — and
its star_constraints are applied as a SUBJECT-level existence pre-check before the POM
loop (shared helper row_passes_star_constraints, dedup'd from the fixed-star branch).
So a folded wildcard emits every (p,o) for a subject that satisfies the constraint,
and drops a subject that does not — the same existence semantics as the standalone
join it replaced. (My earlier "gate items i/iii/iv sound" claim was WRONG for item
(i): the fixed-star materialize never reaches the wildcard emission.)
- Hermetics: REWRITE plan — w4_1b_const_object_folds_onto_crawl_wildcard_composing_
with_class + is_crawl_wildcard tolerance. EXECUTION —
w4_1b_folded_wildcard_emits_po_rows_under_star_constraint asserts the actual (p,o)
BINDINGS (satisfied → the SAME triples the unfolded wildcard emits: ?p=storeKey∧?o
AND ?p=rdf:type∧?o=Store, per lambda-audit-2; violated → 0 rows). 1286 lib tests
green (fixed-predicate star path — q061/q065 shapes — unchanged).
- Corpus q068 (orderline detail crawl, FROM path): the folded-crawl shape class NO
member covered — const-object edw:lineNumber 1 folded onto `?ol ?p ?o`. Row-count
parity (native == virtual, rows_only) catches the ship-blocker's 0-triples
regression end-to-end (which is why it nearly shipped); its ex:order/product ref
POMs also drive the F-16 parent-lookup telemetry. Oracle blessed at the gate.
W4-1b prunes the FACT_ORDER_LINE crawl scan. Whether the ex:order RefObjectMap
parent-lookup ALSO bounds to the referenced Order (vs. building over the full
FACT_ORDER) is measured live via q068 at the wave-4 gate; if the full parent scan
survives the child prune, F-16 (#16) carries that parent-lookup-bounding follow-up.
…d crawl + wildcard TM-prune guard The W4-1b fold sets star_constraints on the crawl wildcard, which flipped has_star_members() and thereby (catch #10) DISABLED the trust_fk_refs FK-template ref shortcut by construction — the folded crawl paid a full parent-table scan (FACT_ORDER 36M) for every RefObjectMap POM that the plain crawl renders scan-free. This was F-16's residual, realized by W4-1b's own mechanism. - ref_template_shortcut (operator.rs): re-enabled for a folded wildcard ONLY when every folded constraint is ObjectConstant::Scalar (catch #11: an IRI constraint on a ref predicate would be satisfied by the templated render of a DANGLING FK — the shortcut skips exactly the parent scan that would have dropped that subject = over-match). Decimal/Double constants are deliberately conservative-excluded (sound parent-scan path; only plain scalars qualify). Helper folded_wildcard_all_scalar + unit gate test covering scalar/iri/mixed/star/plain/fixed shapes. - star_required_preds TM-prune: now fixed-predicate-star-only (predicate_var.is_none()). The all-preds prune argument does not transfer to a wildcard, which emits per-(p,o) across co-subject maps — pruning a map lacking the folded constraint predicate would drop its rows for vertically-partitioned subjects (F10-class mappings). - harness: admit the mixed_group_keys feature tag (q066/q067 re-tagged with it; q066 joins the smoke subset for tag coverage); corpus meta-tests updated for the 68-member corpus incl. q068 rows-only. F-16 closes pending the live q068 telemetry (expect ref render via FK-template, zero FACT_ORDER files planned).
…lar-folded crawl Extracts the trust_fk_refs admission into ref_template_shortcut_enabled (non-behavioral: the parent-lookup build now calls the named fn) and pins the fire condition the corpus structurally cannot exercise: SPARQL corpus queries run via query_from where trust_fk_refs is always false, so a shortcut regression is perf-silent to q068 (the flag-true-vs-path-taken seam, catch #10's class). f16_ref_template_shortcut_fires_for_scalar_folded asserts: admission true for a scalar-folded crawl under trust + build_ref_shortcut Some for the single-col templated ref (together = the build inserts the shortcut, no parent scan); admission false for Iri-folded (catch #11 dangling-FK over-match hazard — sound parent-scan path), for trust-off (the chat path), and for fixed-predicate stars; plain wildcards unchanged.
aaj3f
changed the base branch from
feat/external-s3-iceberg
to
perf/r2rml-f17-union-budget
July 21, 2026 20:26
aaj3f
changed the title
This was referenced Jul 21, 2026
bplatz
requested changes
Jul 26, 2026
bplatz
left a comment
bplatz
left a comment
Contributor
There was a problem hiding this comment.
Two issues inline, one of them a sequencing constraint rather than a defect in this diff.
The engine-wide surface I checked holds up: all 7 checkpoint() sites are batch-granular (hot-loop purity respected), checkpoint() is cancel-first so an external cancel can't be masked as a 507, query_memory_budget_bytes is OnceLock-cached so there's no per-call cgroup read, DatasetOperator concatenates without dedup (so C1's per-member budget is sound, and the distinct_above_dataset_absorbs_budget test pins the boundary), per-member top-k is sound, and W4-1's coerce_scalar_for_pushdown gate is genuinely tight — n.to_string() == *s rejects "01"/"+1"/"1.0", out-of-i64 xsd:integer declines rather than wrapping, and every decline direction is toward the residual.
| /// on a query already near its ceiling. No-op on a disabled handle. | ||
| #[inline] | ||
| pub fn record_alloc(&self, bytes: usize) { | ||
| if let Some(inner) = &self.inner { |
Contributor
There was a problem hiding this comment.
Definite issue — the monotonic counter over-counts operators that are rebuilt, not just ones that are sequentially freed.
The doc frames the conservatism as summing sequentially-freed operators, which is the benign case. The one that isn't benign is re-execution: GraphOperator builds a fresh inner subplan per parent batch (graph.rs:357, :536), and the batched-OPTIONAL path rebuilds similarly. If a hash join sits in that inner subplan, every rebuild re-records its full build table while the previous one has already been dropped — so the counter grows linearly in parent batches while actual peak memory stays flat at one build.
With N driving batches the counter reports ~N× the true peak. On a Lambda-sized container (budget = 78% of, say, 1 GiB) a correlated GRAPH query that never exceeds a few hundred MB can trip a typed MemoryBudgetExceeded → 507. That's a false abort on a legitimate query, and it lands on native queries too — hash_join and group_aggregate are shared operators and the guard is default-on.
This is different from the disclosed scoping note. "Cumulative across sequential operators" is bounded by plan size; cumulative across re-executions is bounded by data size, which is the thing the budget is supposed to be measuring.
Cheapest fix that keeps the mechanism: have the correlated rebuild sites reset or scope the counter per inner execution, rather than making it fully per-operator. Worth at least confirming whether a rebuild-heavy corpus member exists — I don't think the 68-member gate covers this shape, since it'd show up as a spurious 507 rather than a hash mismatch.
Contributor
Author
There was a problem hiding this comment.
Confirmed — and reproduced before fixing: a hermetic test with two sequential inner builds on one cancellation handle showed the second build falsely aborting at the summed estimate against a budget the true peak never approached. Your distinction (cumulative-across-plan is bounded by plan size; cumulative-across-re-executions is bounded by data size) is exactly right and is now quoted in the fix's invariant comments. Fixed in 5b33104 at the rebuild boundary: the counter is snapshotted before each correlated inner build and the delta released after the inner drains, at all three rebuild sites — execute_in_graph (per-row), execute_in_graph_batched, and the batched-OPTIONAL driver — using the release primitive ported byte-identically from the audit line (which forbids releasing live persistent builds; the snapshot-delta pattern satisfies it by construction). The repro is now a permanent regression test (d1_rebuild_boundary_release_no_false_abort, with a negative control). On your corpus question — confirmed, and stronger than framed: no harness sets the budget env at all, so the 68-member gate ran at the ~78%-of-container default and could not have tripped a spurious 507 on any shape; 0 spurious aborts was structurally not evidence.
| for (i, p) in chain.iter().enumerate() { | ||
| if Self::predicate_for_var(p, *gv).is_some() { | ||
| if src.is_some() { | ||
| return None; // ≥2 sources: cross-source equality |
Contributor
There was a problem hiding this comment.
Definite issue (sequencing) — this PR widens admission into a fold that still has an open under-count.
The cross-source decline rule here is right, and the W4-2 per-key resolution reads correctly. The problem is what it widens into: insert_dim_gkeys is unchanged on this branch, and it still treats an equal-value duplicate dim join-key as harmless. It isn't — the map holds one entry and is probed once per fact row, while the reference pipeline emits one row per matching dim row, so COUNT(*) comes back short. I've flagged that on #1507.
C5, E1, E2 and W4-2 together admit substantially more shapes to the fused path — shared-predicate group keys, join+flag rollups with constraints applied, mixed fact+dim keys. Each of those is a new shape reaching a fold whose dim-side duplicate handling is known-wrong, so this PR increases the blast radius of that bug rather than being independent of it.
Not asking for a fix here — asking that this land after the insert_dim_gkeys fix on #1507, and that the widened shapes get re-checked against a dim with a non-PK join key once it's in. The 68-member SF01 gate can't catch it: unique PKs mean the duplicate case never arises in the corpus, which is exactly why 0 hash mismatches isn't evidence on this point.
Contributor
Author
There was a problem hiding this comment.
Agreed on the sequencing, now moot in the good direction: the insert_dim_gkeys fix landed three rungs earlier on #1507 (0b4269a) and this branch has merged it up (5b59664), so the widenings here sit on top of the decline rather than the bug. New regression tests exercise a NON-unique dim join key against the widened admission (join_fold_declines_on_non_pk_dim_join_key (control: unique key fuses; non-PK duplicate declines to the generic pipeline)) — with one precision on scope: of the widened shapes, W4-2 mixed fact+dim keys and dim-side E2 lower through the join fold and could reach insert_dim_gkeys; E1 shared-predicate, C5, and fact-only E2 lower through the single-scan fold and structurally cannot. The tests target the former; the boundary is now stated in the test module so the distinction survives.
| // inherently un-LIMIT-able, every build row is needed for correctness — | ||
| // surfaces a typed `MemoryBudgetExceeded` BEFORE the runtime OOMs. | ||
| // Batch-granularity, approximate (rows × cols × est/binding). | ||
| ctx.record_alloc(batch.len() * ncols * crate::context::BINDING_EST_BYTES); |
Contributor
There was a problem hiding this comment.
Worth a look — ncols here is the build schema width, so the estimate scales with declared columns rather than retained ones.
BINDING_EST_BYTES is a flat 64 regardless of what the binding actually holds, which the body already acknowledges under-counts long IRIs and GROUP_CONCAT state. Combined with ncols being schema width, the per-batch estimate can be off in both directions at once on a wide, string-heavy build — under on the strings, over on the narrow columns.
That's fine for the stated purpose (catching a runaway before the OOM), and I wouldn't hold the PR for it. Worth knowing when the 507s start arriving, though: the number in MemoryBudgetExceeded { used_bytes } isn't close enough to real bytes to debug against, so it'd help if the error or the span made clear it's an estimate.
Contributor
Author
There was a problem hiding this comment.
Fixed in 5b33104 at the labeling level you suggested: the abort message now reads "used ~N B (estimated)", and the span key is renamed to mem_used_bytes_est (in the process it also got properly declared — it was previously recorded but unregistered). The field name stays for API stability. The underlying estimate remains schema-width × flat per-binding cost as you describe; making it byte-accurate is deliberately out of scope for a guard whose job is catching runaways before the OOM.
…lated boundary (D1) + label the budget estimate (W1) D1 (false 507 on rebuilt operators): the query memory counter was add-only, so a correlated GraphOperator / batched-OPTIONAL that rebuilds its inner subplan once per parent row/batch summed each rebuild's build-table charge (hash-join / GROUP BY / fused-dim) into the shared counter while true peak memory stayed at one build. With N driving rows the counter reported ~Nx the true peak and false-aborted a legitimate correlated query with a typed MemoryBudgetExceeded (507) on a small (Lambda-sized) ceiling. - Forward-port the release primitive verbatim from cfd773d: QueryCancellation::release (saturating fetch_update decrement) + its invariant doc, plus the ExecutionContext::release wrapper. Byte-identical to that rung so a later merge-up auto-resolves. (Its scan-window caller in r2rml/operator.rs is NOT ported — that belongs to the other rung.) - Release at the rebuild BOUNDARY, never inside a live build: snapshot mem_used() before each correlated inner opens and release exactly the delta after it drains and closes, at both GraphOperator sites (execute_in_graph per-row, execute_in_graph_batched per-batch) and the batched-OPTIONAL rebuild driver (OptionalOperator::build_batch). The whole inner (build tables included) is provably dropped there, so the delta is precisely its transient charge; execution on one handle is sequential so nothing races the delta. An early ?-exit skips the release (over-counts, the safe direction) on a path that aborts the query anyway. - Regression test hash_join::tests::d1_rebuild_boundary_release_no_false_abort: two independent builds on one handle under a budget admitting one build's peak but not the two-build sum complete only with the boundary release (a negative-control assertion proves the un-released sum would exceed the budget). W1 (estimate honesty): MemoryBudgetExceeded.used_bytes is rows x declared-schema cols x a flat per-binding estimate, not measured bytes. Make that explicit at both surfaces the reviewer reads: append "(estimated)" to the Display and document the field (name kept to avoid public-API churn); rename the GROUP BY telemetry span field to mem_used_bytes_est and declare it so the record is no longer a silent no-op. No span allowlist/registry references the key, so nothing else needs updating.
Merge the PR base up to pick up B1 (fused_aggregate declines any duplicate dim join key, 0b4269a — the fix D2 depends on) plus B2+W3 disk-cache delete-manifest safety and f17's absorbed main-merge (DEC-004 consolidation, 7e949f1). Almost everything auto-merged, including the D1 cancellation.rs release primitive (byte-identical port, zero-conflict) and the D1/W1 context.rs / group_aggregate.rs / optional.rs hunks (disjoint regions). Two files conflicted, both in the iceberg disk-cache layer, and both provably reconciled: - fluree-db-api/src/graph_source/disk_catalog_cache.rs: taken from f17 wholesale. Provably lossless: f17's version is byte-identical to family-c dab3b99 (which inherited this branch's own #1500 §4 "stable key hash" rework and layered B2 on top), and diffing this branch's version against dab3b99 shows only the B2 additions (has_delete_manifests, CACHE_FORMAT_VERSION 2->3, F-AUD-1 guard tests) with none of the §4 content removed. Same origin, one direction of supersession. - fluree-db-api/src/graph_source/r2rml.rs: UNION of the test modules — this branch's §1 (#1500) inline-loadTable-metadata tests kept first, f17's servable_scan_files / delete-manifest (F-AUD-1) tests appended. f17's servable_scan_files production hunks auto-merged outside the conflict; verified they gate BOTH scan-file cache-hit arms (in-memory + disk), a delete-bearing entry falling through to a guarded re-plan, and that CACHE_FORMAT_VERSION appears once and equals 3.
…widened-shape regression) The #1514 review (D2) flagged that this PR widens fused-aggregate admission into the JOIN fold (W4-2 mixed fact+dim GROUP BY keys, dim-side E2), which reaches insert_dim_gkeys — whose equal-value-duplicate under-count B1 (0b4269a, merged from the f17 base) has now closed by declining ANY duplicate parent join key. B1's dim_dup_join_key_always_declines guards the primitive. This adds the missing other half: join_fold_declines_on_non_pk_dim_join_key drives resolve_join_at_open (reached only by the widened join shapes) with a real R2RML mapping + terminal-dim table provider, and asserts the CALLER propagates the decline to Ok(None) — falling back to the sound materialize path — when the dim has a non-PK (repeating) join key. A unique-key control run FUSES (Ok(Some)), so a broken fixture fails the control rather than passing vacuously; SF01's unique PKs mean the corpus can never raise this. The test doc records the boundary: E1, C5, and fact-only E2 take the single-scan fold (resolve_at_open, scan_table n=1) and never build a FK->GKey map, so the widened-shape concern does not reach them (route_group_key_sources_mixed_and_declines covers routing). Also refresh the two resolve_join_at_open caller comments that still said "CONFLICTING duplicate": post-B1 the decline fires on ANY duplicate dim join key.
Contributor
Author
|
Both definite items and the worth-a-look are addressed: the rebuild over-count is reproduced, fixed at the rebuild boundary with the audit line's release primitive, and pinned by a regression test (5b33104); the widening now lands after the dup-key decline via the merge-up (5b59664) with non-unique-key regression tests for the shapes that can actually reach the join fold; and the budget abort is labeled as an estimate. This PR gets no repo CI on its stacked base, so the local gate run is recorded in the PR's verification appendix: core 714/0, query 1340/0, api 2772/0 at 5b33104, re-run at 44af118. |
bplatz
approved these changes
Jul 28, 2026
bplatz
left a comment
bplatz
left a comment
Contributor
There was a problem hiding this comment.
All three items resolved at 44af118, verified locally:
- D1: fixed at the rebuild boundary as suggested —
QueryCancellation::release(saturating, with the persistent-build invariant doc) + snapshot-delta at all three sites (execute_in_graphper-row,execute_in_graph_batched, batched-OPTIONALbuild_batch). I re-derived the soundness independently: everyrecord_allocsite in the crate lives inside a correlated inner tree (hash-join build, GROUP BY maps, fused-dim maps) and the outer drain structures never charge, so the released delta is exactly the dropped inner's transient charge — and the pattern composes for nested rebuilds. Early?-exit skips the release, which only over-counts on an aborting path.cancellation.rsdiffs empty against the audit line. The regression test's negative control proves the un-released sum would have tripped the pinned budget, so it can't pass vacuously. Also confirmed the corpus point: no harness sets the budget env, so the 68-member gate structurally couldn't surface this. - D2: satisfied structurally — base is the f17 branch and 5b59664 brings in the
insert_dim_gkeysdecline, so the widenings cannot land ahead of it.join_fold_declines_on_non_pk_dim_join_keycovers the re-check with a unique-key control that fuses, and the stated join-fold/single-scan boundary (W4-2 + dim-side E2 reachinsert_dim_gkeys; E1/C5/fact-only E2 cannot) matches the code. - W: "(estimated)" in the Display, field doc'd as approximate with the name kept stable, span key renamed
mem_used_bytes_est— and properly declared, fixing that the record was previously a silent no-op.
Re-verified: cargo test -p fluree-db-query → 1293 passed; core cancellation tests incl. the two new release tests pass; D1/D2 regression tests pass individually. Stacked base gets no repo CI, consistent with the verification appendix.
Reconcile graph-source secret redaction + CLI /info display refactor: - ledger_info.rs: union redact_json_secrets — keep lambda's ConfigValue::SecretRef preservation (is_secret_ref_object, checked first) AND main's #1534 fail-closed container handling (redact_secret_subtree / SECRET_REF_SAFE_KEYS). - cli iceberg.rs & info.rs: adopt main's unified graph_source_display module; drop the now-orphaned inline print/redact helpers and their tests (coverage moved to graph_source_display.rs).
hash_join.rs and r2rml/operator.rs carried rustfmt drift from the feat/lambda-usability line that never gated on CI (base was not main). Now that PR #1514 targets main, CI fmt fires; format them clean. Whitespace-only, no behavior change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
feat(virtual): serverless Lambda-usability + BYO-IAM — admission widenings, capacity guards, typed memory budget (consolidates #1505 + #1514)
This is one of five surviving pull requests in the consolidated virtual-dataset review program, which collapses the #1475–#1529 lineage into five reviewable units at the maintainers' request. #1514 makes R2RML/Iceberg virtual-dataset chat queries usable in the serverless (Solo / AWS Lambda) scenario and folds in the BYO-IAM (operator-owned S3) engine work; it carries the program's single most important native call-out, the R3 typed memory budget. It is based on #1507 (so its own diff is exactly #1505 + #1514), and reviewers rely on the pasted gate record rather than CI. Reading order across the program: #1520 (standalone MoR correctness fix) → #1507 (performance program) → #1514 (this PR) → #1528 (big-Iceberg-audit implementation) → #1529 (native-twin materialize builder). The outward-facing audit record is on the
docs/virtual-dataset-audit-2026-07branch — the serverless envelope this epic targets is in C1-constraint-envelope.md and the memory-budget finding the R3 change protects against is in V2-membudget-verification.md.Original PR body
Scope expanded by consolidation; the tip PR's own change (the Lambda-usability epic) is the narrative below, and it folds in the BYO-IAM engine work (#1505) — rolled up under "Consolidated scope."
Makes Iceberg/R2RML virtual-dataset chat queries "reasonably usable" in the solo/Lambda scenario. Grounded in the Lambda-reality assessment (
docs/audit/2026-07-lambda-reality/) and two rounds of post-deploy forensics (~/fluree/db/lambda-round2-*.md,lambda-round3*-*.md). Three waves: C1-C5 (the original epic), E1-E2 (round-2 fusion admission), R3-A/B (round-3 capacity protection).Component checklist
C1 —
DatasetOperatorforwards theLIMITbudget / top-k to members. Every chat SPARQL is aFROM <ledger>dataset query; the dataset operator inherited the no-op budget hooks — aLIMIT 20never reached the R2RML scan (observed live: 8M-row full scan). SwitchFLUREE_R2RML_DATASET_BUDGET.C2 — manifest-backed info-stats routing. The 900s
get_data_modelrunaway. Live-verified fixed: 4s.C3 — mid-scan cancellation in the R2RML drain helpers. Live-verified: scan-bound queries cancel cleanly at the deadline.
C4 — intra-file row-group parallelism for single/few-file tables. Switch
FLUREE_ICEBERG_ROWGROUP_PARALLELISM.C5 — fused-aggregate admission for the deployed rollup shapes (adversarially gated): plain-literal group-key default matching materialization by construction, guarded by O1 (
star_constraintsdecline), O2 (single-source-view dataset guard with(ledger, to_t, policy)dedup), Q2 (lang/IRI group-key decline, both fold paths).E1 — shared group-key predicate fusion (round-2: the
GROUP BY category2-scan timeouts). Fuse when a class-declaring map co-locates the predicate and all other maps are subject-template-disjoint (structurally provable). Vertical-partition/overlap cases still refuse.E2 — join+flag rollups fuse with constraints APPLIED (round-2: the
isCurrent truejoin timeouts). Dim-side constraints filter during FK-map build; fact-side ride the fold; anything unresolvable declines to materialize (enumeration adversarially verified total). Live: 52s → 0.45s with the identical constrained count.R3-A/B — unified
ctx.checkpoint(): cancellation + typed memory budget in the join/aggregate phases (round-3: OOMs at 344s/818s ran 2-4.5× PAST the deadline in post-scan phases with zero polls — 813s of log silence on the worst; OOMs at 156-170s died BEFORE any deadline, where cancellation can't help). One mechanism: a query-scoped memory counter lives on the shared cancellation handle (every operator of a query reaches the SAME counter — a join build and a downstream fold share one budget), andcheckpoint()= cancellation check first, then budget — wired at the 7 buffering sites (hash-join build/probe, both aggregate folds, and the fused-fold's map builds, so any future interior-FACT map is covered from day one). The ~85 non-buffering scan drains keep plain cancellation checks deliberately. TypedMemoryBudgetExceeded→ 507, distinct from the 408 timeout; an external cancel (deadline / RSS watchdog) can never be masked as a 507 (cancel-first ordering, adversarially verified). Core crate carries pure mechanism (never enforces); enforcement lives only in the query engine. Also makes solo's deployed RSS memory-watchdog bite inside the fold — in the round-3b window, 3 hard OOMs slipped past that watchdog exactly where these checkpoints were missing. Knobs:FLUREE_QUERY_MEMORY_BUDGET_BYTESoverrides;0disables; default = ~78% of the detected container/system memory (cgroup v2 → v1 → /proc/meminfo; 8GiB fallback) — proportional to the machine, not a fixed ceiling. There is no engine-imposed time limit anywhere: these are polls only; deadlines are armed solely by callers.W4-1 — constant-key scan-filter pushdown + datatype coercion (round-3b: a point lookup full-scanned 120M+36M rows into a hard OOM). A constant-object key equality now pushes as a scan filter even when the pattern carries other predicates (previously only a LONE constant pushed; any second predicate demoted it to a post-read residual). String literals coerce to the column's type ONLY on canonical round-trip (lexical-eq ⟺ typed-eq;
"01"/"1.0"/non-numeric decline the push — residual stays the sole authority, no over-prune possible).W4-2 — mixed fact+dim GROUP BY keys fuse (round-3b
#7: a 36M two-hop rollup grouped by dim-year + fact-shipMethod declined → 156s materialize → deadline). Group keys now resolve per-key to their binding pattern (fact column or terminal-dim attribute; a var bound in ≥2 patterns is a cross-source equality and DECLINES — adversarially derived rule). Live: the corpus analogues run at ~110ms fused.W4-1b — const-object folds onto the co-located crawl wildcard (the deployed detail-crawl shape:
?x key "v" . ?x ?p ?o). The key constant and the wildcard previously ran as SEPARATE JOINED SCANS — the constant never pruned the crawl. The fold attaches it as a subject-level constraint; the executor treats a variable-predicate wildcard as a true wildcard at all sites (a mid-review ship-blocker: the first cut produced an INEXECUTABLE plan returning subject-only rows — caught by the layered review, fixed, and pinned by an execution hermetic asserting bound?p/?oplus corpus sentinel q068).F-16 amendment — scalar-gated
trust_fk_refsrestore + wildcard TM-prune guard. W4-1b's constraint fold had disabled the FK-template ref shortcut by construction (catch#10) and would have TM-pruned vertically-partitioned wildcard subjects (latent wrong-results). The shortcut is restored for folded wildcards ONLY when every folded constraint is scalar — an IRI constraint would let a DANGLING FK's templated render falsely satisfy the existence check (catch#11), so non-scalar keeps the sound parent-scan path.What this changes for the deployed workload (honest attribution)
category, flagged join rollups).query_fromwheretrust_fk_refsis OFF — parent scans for ref rendering predate this PR there and remain (bounded by the R3 guards; F-16 stays open for parent-lookup bounding on that path). The browse-crawl API path (trust_fk_refsON) gets the full win: pruned child + scan-free ref render via the restored scalar-gated shortcut, evidenced by the integration hermetic (landing pre-merge) since no SPARQL corpus query can exercise the trust path.Verification
33a7d540binclusive — both the separate-mechanism head and the unified refactor gated identically, zero behavioral change). New members validate every deployed failure shape: q060-q065 (budget/flag/join/sort/shared-predicate/join+flag paths), oracles blessed from fusion-OFF runs.Not in this PR (follow-ups, evidence-backed)
trust_fk_refs).query_frompath (bound the lookup to the child-pruned FK set, or a solo-side decision to enabletrust_fk_refsfor chat crawls — a product call on browse semantics).#9, diagnosis complete): VALUES → IN-set scan filter riding the parked ref-value-set backend; the diagnosed query would then fuse end-to-end.HashJoininherits the no-op and drains its build fully); bounded/spilling aggregation (large, whole-cloth).?ol ex:orderLineKey "1") that failed to bind as a scan constraint and full-scanned 120M+36M rows.star_constraintsapplication);SELECT DISTINCTfolds; solo-side companions (seesolo-round2-brief.md, incl. differentiating the RSS-watchdog 507 body).Base:
feat/external-s3-iceberg.Consolidated scope
The diff of this survivor is exactly #1505 followed by #1514. #1505's pre-consolidation tip is preserved at
archive/pre-refactor-2026-07-21/feat_external-s3-icebergand #1514's atarchive/pre-refactor-2026-07-21/feat_lambda-usability.loadTablemetadata (no S3 GET for handed bytes), fail-closed vended→ambient credential handling with typed 403s (err:storage/AccessDenied,err:catalog/CredentialsNotVended), aConfigValue::SecretRef+ async hydration path for credential rotation, session-cached S3 clients in Direct mode, and a config-timeverify_storage_accessprobe. Closes #1500 / #1497 / #1498. Its review-critical commit reconciles four latent semantic breaks that a clean rebase onto the loadTable-metadata cache would otherwise have shipped silently, by unifying the credential-decision and hydration seams into single call sites.trust_fk_refsamendment.Verification of record
CI does not fire on a stacked base, so the verification of record is the in-body gate record of the two constituents; both were built and measured against a standing Snowflake + AWS rig.
For #1514: the full 65-member live corpus gated at every wave head with 0 hash mismatches, including the R3 unified-refactor head
33a7d540b(the separate-mechanism and unified heads gate identically — zero behavioral change), and the wave-4 68-member live SF01 gate reported 0 hash mismatches. The only perf-gate violations in any run fall inside the documented loadTable-GET catalog-latency set (q002/q004/q022/q024/q030/q043), which are non-fused, known-cause, and network-attributed rather than engine regressions. For #1505: an independent adversarial review over the full diff returned SHIP with zero blockers — it recomputed the disk-cache golden hash, byte-diffed the legacy error strings, verified resolved secrets are never persisted / fingerprinted / logged, and confirmed the 401→refresh path is untouched; its one should-fix (a redactor doc-comment) landed in the same branch. The live rig verified the vended and BYO-IAM (zero-identity-permission role, bucket-policy-only access) paths end-to-end, both AWS AccessDenied message variants surviving verbatim to the CLI and the 403 JSON, and the zero-REST composition holding under BYO-IAM.The serverless constraint envelope this epic targets — the deployed cap, heartbeat, and deadline, and the query-class × wall matrix — is documented in C1-constraint-envelope.md. The memory-budget blind spot that R3 closes (the scan-path and post-scan buffering that could grow to ~4.4–11 GB unaccounted) is verified independently in V2-membudget-verification.md, and the per-core throughput ceiling that frames the "capacity protection, not throughput" posture is recomputed in RT2-redteam-empirics.md.
Native-impact appendix
This survivor carries the program's single most consequential native call-out. It is foregrounded here.
ctx.checkpoint()(cancellation check first, then budget) is wired at seven engine-wide buffering sites — the hash-join build and probe, both aggregate folds (group_aggregate), and the fused-fold map builds — with a query-scoped memory counter living on the shared cancellation handle, so every operator of a query reaches the same counter (a join build and a downstream fold share one budget). A native query performing a large hash-join or group-aggregate now trips a typedQueryError::MemoryBudgetExceeded→ 507 (distinct from a 408 timeout) at roughly 78% of the detected container/system memory (cgroup v2 → v1 →/proc/meminfo, with an 8 GiB fallback) instead of OOMing the process. This is real, intended capacity protection, not a silent change. Cancel-first ordering guarantees an external cancel or deadline can never be masked as a 507. Knobs:FLUREE_QUERY_MEMORY_BUDGET_BYTESoverrides the budget and=0disables it; there is no engine-imposed time limit anywhere — the checkpoints are polls only, and deadlines are armed solely by callers. Verified: the full 65-member live corpus gated 0 hash mismatches at every wave head, including the R3 unified-refactor head33a7d540b. Honest scope (kept from the adversarial passes): per-row byte estimates under-count large heap-strings, so "over-count-safe" is scoped to small-binding builds like the deployed DW workload; the counter is query-lifetime-cumulative (also over-count-safe); and an RSS-level watchdog remains the backstop for untracked scan-stage buffers. The full mechanism — the seven buffering sites, the 507 response code distinct from the 408 timeout, the cancel-first ordering, and the knobs — is set out per the original feat(virtual): serverless Lambda-usability + BYO-IAM — admission widenings, capacity guards, typed memory budget (consolidates #1505 + #1514) #1514 PR record (feat(virtual): serverless Lambda-usability + BYO-IAM — admission widenings, capacity guards, typed memory budget (consolidates #1505 + #1514) #1514), in its R3-A/B component-checklist item, and the memory-budget blind spot it closes is verified independently in V2-membudget-verification.md.DatasetOperatorLIMIT-budget / top-k forwarding — a second generic-operator budget extension, living here.DatasetOperatoris a generic operator on everyFROM <ledger>query, so this is a distinctset_row_budgetextension from the F17 UNION/BIND forwarding in perf(virtual): R2RML/Iceberg virtual-dataset performance program — corpus, pruning, fused aggregation, budget forwarding (consolidates #1475–#1507) #1507 and the OPTIONAL forwarding in perf(virtual): big-Iceberg-audit implementation — correctness/capacity guards, coverage widenings, harness re-bless, browse-parity, family-C (consolidates the audit program + #1525) #1528. It is inert without a LIMIT budget, and it is switch-gated byFLUREE_R2RML_DATASET_BUDGET. It is called out here so the generic-operator budget story is complete across the program, and it is documented per the original feat(virtual): serverless Lambda-usability + BYO-IAM — admission widenings, capacity guards, typed memory budget (consolidates #1505 + #1514) #1514 PR record (feat(virtual): serverless Lambda-usability + BYO-IAM — admission widenings, capacity guards, typed memory budget (consolidates #1505 + #1514) #1514), as component-checklist item C1.Confirmed negatives across the span (git-checked): no
arrow/datafusion/parquetdependency bump, and no.github/workflowsorrust-toolchainedits.