perf(r2rml): fuse GROUP-BY rollups over FK joins into the aggregate fold (PR-6) - #1490
Closed
aaj3f wants to merge 2 commits into
Closed
perf(r2rml): fuse GROUP-BY rollups over FK joins into the aggregate fold (PR-6)#1490aaj3f wants to merge 2 commits into
aaj3f wants to merge 2 commits into
Conversation
Extend FusedR2rmlAggregateOperator to fold COUNT/SUM/AVG rollups directly from the fact column batches over a fact→dim FK chain, resolving GROUP BY dimension attributes through a memoized per-hop lookup instead of materializing 180K+ join bindings and hash-joining them. 6a (0/1-hop): admit fact+one-dim at the resolve_at_open join-rejection site; classify via joins_via with a cycle-guard; single-column FK; group keys as scalar dim attributes (un-annotated column defaults to xsd:string, R2RML natural mapping, JOIN PATH ONLY — the single-table path stays byte-identical; see F12); HAVING applied by a wrapping HavingOperator (admitted only when its lifted aggregate is projected); dangling/NULL-FK row-drop matches the inner join. q032 ~70s→~2.6s, q025 fused with HAVING. 6b (k-hop): order_chain orders a linear fact→dim1→…→dimk chain (single ref-join per hop; declines branch/cycle/merge/disconnected); the group-key resolver is composed from the terminal dim back to the fact so a dangling FK at any hop folds into one fact-row drop. q008/q009 (2-hop) ~45-50s→~600ms. Switches standardized on r2rml::env_switch_enabled (FLUREE_FUSED_R2RML_AGG and FLUREE_FUSED_R2RML_AGG_JOIN). Validated: 1227 unit tests; native corpus 54/54 0 hash mismatch; every target rollup hash-parity vs the native oracle; q022 single-table sentinel byte-identical.
…invariant + fmt The fused-aggregate FK-chain lookup asserted "a dim join key is unique" in a comment but nothing checked it. A dimension with a duplicate join-key mapping to DIFFERENT group attributes silently under-counts vs the generic pipeline (the reference semantics emits both bindings — the dim subject carries two attribute triples, so a fact row lands in two groups — while the single-value probe keeps one). Reachable via #1450's unverified subject-keys / name-based FK inference / hand-written mappings; SF01 dims have unique PKs so the corpus can't catch it. Extract a shared insert_dim_gkeys helper used by both the terminal scan and every interior hop: a CONFLICTING duplicate (same key, different group-keys) declines the fused plan (Ok(None) → byte-identical generic pipeline, the operator's decline contract); equal-value duplicates are harmless and kept. Unit-tested. Design doc §5.5 records the checked invariant. cargo fmt clears this PR's own fmt debt.
aaj3f
added a commit
that referenced
this pull request
Jul 17, 2026
…ts, mechanism, soundness)
The fused-aggregate fold is COMPLETE (single-table + fact-dim join, resolve_at_open
+ resolve_join_at_open); slice 1 = lift the FIRST decline the deployed grouped-COUNT
shapes hit (detect:338 Pattern::Graph{Iri} on the dataset path, most likely). Adds
the decline receipts (file:line), the FK-key count+join mechanism from the IR, the
soundness invariants (null FK, #1490 dup-key, filter/DISTINCT decline, multi-key is
ALREADY supported), hermetics, predicted effect, switch reuse, and dataset-path
corpus members.
aaj3f
added a commit
that referenced
this pull request
Jul 18, 2026
resolve_join_at_open resolved EVERY GROUP BY key on the terminal dim, so a fact-column key (#7's `GROUP BY yearNum (date dim), shipMethod (fact)`) failed scalar_column_for_var(terminal_dim, shipMethod) and declined → 156s materialized fold → deadline. Route each key to its single source instead: fact-column keys read inline from the fact scan, dim-attribute keys from the FK→GKey map, composed per SPARQL order. - KeySource plan + assemble_group_key interleave fact-inline and dim-resolved positions; group_cols stays one-per-position so the output binding() is unchanged. Both existing paths are special cases (all-Fact = single-table, all-Dim = today's join, byte-identical). - Gate Q1 (route_group_key_sources): a key's source must be EXACTLY ONE participating pattern — 0 (unbound), ≥2 (cross-source value-equality the fold cannot enforce), or an interior-dim source all decline (v1 admits fact or the terminal dim only). No fact-wins tiebreak. - NULL in ANY key position drops the fact row (BGP unbound-object semantics): assemble_group_key's per-position Null-drop is the robust guarantee, with validity_cols (a fact group key is a fact object var) as the secondary drop; dim keys drop via the map-miss. Symmetric across sources. - Q2 plain-literal gate applied per key on ITS OWN source pattern/TM (fact AND dim). O1/E2 star_constraints, #1490 dup-key decline, and the memory/cancel checkpoints are untouched. Empty-dim-subset (all-fact keys over a join) degenerates the FK→GKey map to a join-existence set. - Hermetics: route_group_key_sources (mixed both orders, cross-source/interior/ unbound declines, all-fact); assemble_group_key (interleave both orders, fact-null + dim-null drops, no-resolver defensive drop, and the empty-dim-subset existence-only slice `Some(&[])`). - Corpus q066 (FACT_SHIPMENT mixed COUNT via FROM: string fact key + integer dim key; COUNT-only because SHIP_COST is xsd:double and the f64 SUM is summation-order dependent) + q067 (FACT_SUPPORT_TICKET mixed COUNT + SUM over the xsd:integer csatScore — exact i128, hash-deterministic — exercising the mixed key path together with a value fold). Oracles blessed at the wave-4 corpus gate. Riders folded per review: - W4-3 DEFENSIVE cancellation: values.rs ValuesOperator now polls check_cancelled per batch, bounding the O(input_rows × value_rows) match for a large VALUES product. This is capacity protection, NOT the round-3b #9 fix — #9's timeout is the un-lowered full scan (the VALUES→IN-set lowering is the real fix, next-slate); this poll only makes the execution-side product cancellable. - W4-1: pin "-0" (parses to 0, canonical "0" ≠ "-0" → declines) and a >i64 overflow string (parse Err → declines) in the coercion refutation hermetic.
Contributor
Author
|
Consolidating at maintainer request. #1490 (PR-6 — fused GROUP-BY rollups over FK joins) is carried forward in #1507 (the R2RML/Iceberg virtual-dataset performance program), which brings the #1475–#1507 lineage forward as a single reviewable unit. Its verification of record is the program's corpus benchmark, published as C2-bench-wave1.md (native == virtual, 0 hash mismatches). The pre-consolidation branch tip is preserved at tag |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Roadmap PR-6 (
docs/audit/2026-07-virtual-dataset-perf/ROADMAP.md; design + attribution in10-pr6-rollup-fusedagg.md): the last big slow-tail class. With the decode wall gone (#1482), the dominant real-world BI shape —GROUP BYa dimension attribute +COUNT/SUM/AVGover a fact⋈dim join — was operator-bound: the generic pipeline materialized 180K+ join bindings just to fold them into a handful of groups.What it does
Extends
FusedR2rmlAggregateOperatorto fold aggregates directly from the fact column batches over a fact→dim FK chain, resolving GROUP-BY dimension attributes through per-hop lookups (one small scan per dim, PR-4-style memoization) instead of materializing bindings. Two phases in one commit (an intermediate buildable split wasn't reconstructible after the in-place chain rewrite; revertability is via the sub-switch):resolve_at_openjoin-rejection site; group keys resolved via a siblingbuild_attr_lookup(FK → attribute value, not FK → subject IRI). Un-annotated string group keys default toxsd:string(R2RML natural mapping) — join-path only (extending it single-table exposed latent bug F12, filed in the register, unreachable today).order_chainorders leaves into a linear fact→dim₁→…→dimₖ chain — single-column ref-join per hop, no branch, no merge, cycles decline (tested). Dangling/NULL FK at any hop ⇒ that fact row drops from the rollup (R2RML no-triple semantics, same validity gate as materialization).HavingOperatorabove the fused op, like ORDER BY/LIMIT today) when its lifted aggregates ⊆ projected aggregates; rejection case tested.Kill switch
FLUREE_FUSED_R2RML_AGG_JOIN(default on, standardenv_switch_enabledfamily) gates only the join admission — off ⇒ the proven single-table fused path and the generic join pipeline, byte-identical. Also migratesFLUREE_FUSED_R2RML_AGGto the switch family (fixes the exact-"0"/"false" parsing inconsistency documented in the WP1 audit).Live results (SF01, hash == native oracle on every query)
Validation
Post-restack onto the review-updated base (
67daa5686): query suite 1227/1227 (new: chain-order/cycle-decline, HAVING admit/reject, string-key default, decimal-key regression); api suite full green; native corpus 54/54, 0 hash mismatches; W3C suite green (36/36 register groups, 0 failures); fused-vs-generic A/B verified per query with decline-path debug attribution (the 6a gate initially showed no win — root cause was agroup-key-datatype-unsupporteddecline on un-annotated string keys, found by instrumenting the decline path, fixed, re-validated). BSBM native budgets deferred to an idle-machine run before merge-to-main (changes are R2RML-graph-source-gated; native corpus already gates the native path).Dropped from this PR: the F9 CURIE-parity rider — its one-line formatter fix proved a no-op (root cause is a namespace-map gap: virtual snapshots never register the R2RML mapping's vocab prefixes, so the compactor has nothing to compact against). Re-scoped in the findings register; stays in the tail queue.
Stacks on #1487 (PR-4b) ← #1485 ← #1484 ← #1482 ← #1478 ← #1476 ← #1475 ← #1450.
Admission contract (precise, matches the code)
Admitted. A
GRAPH { … }over an R2RML graph source feeding a GROUP-BY (or implicit) aggregation of multisetCOUNT/SUM/AVG, whose WHERE lowers to a linearfact → dim₁ → … → dimₖforeign-key chain.order_chainrequires exactly one root (the fact), one ref-join per hop, and a full single-visit walk — branch, merge (diamond), cycle, and disconnected shapes decline. Each hop is a single-column FK. GROUP-BY keys are scalar attribute columns on the terminal dimension (an un-annotated column takes R2RML's naturalxsd:string, join path only). Aggregates fold from fact columns / fact decimal expressions. HAVING is applied by a wrappingHavingOperator, admitted only when its lifted aggregate is in the SELECT projection.Excluded → generic pipeline (byte-identical with the sub-switch off): a FILTER over the join; branch/merge/cycle joins; composite FK; IRI/ref group keys or keys off a non-terminal dim; aggregates over a dim column; DISTINCT aggregates; the single-table path with an un-annotated string key (kept byte-identical — see F12); and a non-lowered sub-scope (
PropertyPath/Subquery), which declines so the generic GRAPH path raises the loud unsupported-subscope error.Row semantics. A dangling or NULL FK at any hop, or a NULL terminal group attribute, drops the fact row — matching the R2RML/inner-join row-drop exactly.