perf(r2rml): admit the same-subject STAR to the batched OPTIONAL (PR-4c, q016 — last corpus DNF) - #1493
Closed
aaj3f wants to merge 2 commits into
Closed
perf(r2rml): admit the same-subject STAR to the batched OPTIONAL (PR-4c, q016 — last corpus DNF)#1493aaj3f wants to merge 2 commits into
aaj3f wants to merge 2 commits into
Conversation
…4c, q016)
q016 (`?o a Order ; orderId ?oid . OPTIONAL { ?sh edw:order ?o ; edw:shipStatus ?st }`
LIMIT 5000) DNF'd at 180s: its OPTIONAL inner is a same-subject STAR on ?sh
(FactShipment) whose correlation var ?o is the OBJECT of a ref member, so PR-4b's
narrow admission (a subject-driven single-object leaf — scalar POM / single-valued
RefObjectMap, exactly q050) declined it → the per-row `OptionalBuilder::build`
rebuild → ~445-517 FactShipment scans.
This widens `inner_pattern_is_hash_join_safe` to also admit a same-subject star
(`r2rml_star_is_hash_join_safe`), so q016 takes the batched hash-left-join: the
inner is seeded with the batch's distinct correlation tuples and hash-partitioned
back to each required row, instead of a fresh operator tree per required row.
Measured: q016 DNF@180s → ok, 5000 correct rows, ~39s hot. scan_table 517 → 182,
attributed 2 FACT_ORDER (outer, collapsed) + 180 FACT_SHIPMENT (inner). The inner
count is NOT scan-once: the batched path drives the seed in WINDOWS and re-scans
the inner per window — the SAME mechanism the shipped PR-4b already has (q050: 92
inner scans), just on a larger seed. That per-window re-scan is filed as F14 (a
PR-4d follow-up, uniform with shipped behavior), NOT a blocker. PR-4c ships the
DNF→ok + outer collapse + parity + the differential.
Sound to admit — the two facts:
- The correlation set is COMPLETE for a star. `R2rmlPattern::referenced_vars`
exhaustively destructures the struct (a new field is a compile error) and
surfaces every star-member object var — the P1 audit, landed + tested as
PR-4b's precursor. So a star's members (?o, ?st) are in the partition-key
input; no silent-drop mis-partition. `star_constraints` (constant-object
existence filters) and `class_prune_hint` carry no var.
- Cartesian multiplicity ≡ per-row for LEFT-JOIN. A correlation matching several
member rows yields a cartesian of inner solutions; every one attaches to the
required row, and the per-row path produces the SAME cartesian — the same leaf
materialization runs both paths. It is NOT the excluded row-multiplying
subquery (an R2RML leaf has no internal ops).
The object-side correlation needs no code change: `corr_var_only_triple_object` is
Triple-only, so an R2RML object corr var is SEEDED BOUND (sound — the scan
restricts to the value via the operator FILTER; only forgoes a subject-probe that
doesn't apply to an object). Kept EXCLUDED (each a distinct, separately-testable
shape q016 doesn't need): type_var (multi-class cartesian), a wildcard
predicate_var, a bound/constant subject. New sub-switch
`FLUREE_R2RML_BATCHED_OPTIONAL_STAR` (default on within the PR-4b
`FLUREE_R2RML_BATCHED_OPTIONAL` / `FLUREE_OPTIONAL_HASH_JOIN` family) so the star
widening reverts independently. No change to `referenced_vars` or the
seed/partition core.
Tests: `batched_equals_per_row_on_object_correlated_star` — ONE mock (Shipment→Order
ref + shipStatus scalar), FOUR required orders asserted row-for-row batched≡per-row:
order/1 matched by 2 shipments (the CARTESIAN), order/2 by 0 (the LEFT-JOIN miss),
order/3 by 1 valid, order/4 by 1 with a NULL shipStatus (the null star member
drops the star-conjunction row — both paths agree). `r2rml_star_admission` pins the
admitted/excluded shapes. PR-4b's within-scope tests still pass; full query lib
1231/0. See docs/audit/2026-07-virtual-dataset-perf/13-pr4c-optional-star.md.
aaj3f
force-pushed
the
perf/r2rml-pr8b-innerjoin-memo
branch
from
July 15, 2026 16:47
8fe47ae to
a934563
Compare
…me a base member One doc line on r2rml_star_is_hash_join_safe: admission qualifies on subject var + non-empty star_bindings, NOT on a base member; completeness comes from referenced_vars, not the star's shape. Forecloses the question for the next widening (#1493 review). No behavior change.
aaj3f
force-pushed
the
perf/r2rml-pr4c-optional-star
branch
from
July 15, 2026 16:48
b1cb988 to
edcd07b
Compare
aaj3f
added a commit
that referenced
this pull request
Jul 21, 2026
…tion) Author the merge of origin/main (bbf92e8) into the #1507 f17 head (c948f0a) so the consolidated perf-program PR is mergeable and CI-runnable. main advanced 122 commits past the merge-base (a49ac3f). CONFLICT: fluree-db-query/src/optional.rs (the only content conflict; Cargo.lock and 7 other overlapping files auto-merged and were spot-checked). Both sides independently extended the SAME batched-OPTIONAL hash-join admission lane, so `inner_pattern_is_hash_join_safe` was defined twice: - HEAD (lineage / PR-4b #1487, PR-4c #1493, PR-4d): admits R2RML leaves (`Pattern::R2rml` arm) via r2rml_leaf_is_hash_join_safe / r2rml_star_is_hash_join_safe behind FLUREE_R2RML_BATCHED_OPTIONAL[_STAR], plus seed-coalescing (supports_seed_coalescing). - main (Cypher edge-annotation work, 57d9a0d / 4bc0c6c / f77c259): admits single-source `Pattern::DefaultGraphSource` chains and adds the AnnotationValueOptionalBuilder / AnnotationSidecarMaps that answer a value-only reified-edge OPTIONAL by draining the three f:reifies* predicates once instead of per row. RESOLUTION (DEC-004 F2 — preserve BOTH native semantics): * Merged the two `inner_pattern_is_hash_join_safe` bodies into one match carrying BOTH arms — the R2rml arm (from HEAD) and the DefaultGraphSource arm (from main). The variants are disjoint, so neither side's admission changes. This is required, not cosmetic: the auto-merged build_batch (lines ~1226/1237) already relies on the DefaultGraphSource arm (main's dataset guard) AND the R2rml arm (HEAD), and HEAD's supports_seed_coalescing calls the same predicate. * Kept HEAD's R2RML switches + r2rml_leaf/star helpers verbatim. * Kept main's AnnotationValueOptionalBuilder / AnnotationSidecarMaps block verbatim. * Tests: kept BOTH sides' additions — HEAD's PR-4d seed-coalescing differential (pr4d_pull_coalesced_required_*, pr4d_multi_batch_driver_*) and main's annotation_sidecar_maps_preserve_multi_target_values. Auto-merge semantic gap fixed: fluree-db-query/src/r2rml/fused_aggregate.rs (one of the 8 overlapping files) — main added a `cypher_vocab` field to `query::Query`; the lineage's two PR-6 HAVING tests construct Query with exhaustive struct literals, so they needed `cypher_vocab: None` added. This is filling the field the merged struct now requires, not a semantics change. Interaction note (sound, kill-switchable): the merged predicate now makes a single-source DefaultGraphSource inner eligible for PR-4d seed-coalescing (main never had that path). Coalescing is inner-shape-agnostic and byte-identical for any hash-join-safe inner, is double-guarded (the generic build_batch still applies the multi-ledger + dataset+DefaultGraphSource guard), and reverts via FLUREE_R2RML_OPTIONAL_SEED_COALESCE. Gates at this head: cargo check --workspace OK; cargo test -p fluree-db-query 1312/0; cargo test -p fluree-db-api 2871/0; rustfmt --check clean on the resolved + edited files; scoped clippy -p fluree-db-query --no-deps clean.
Contributor
Author
|
Consolidating at maintainer request. #1493 (PR-4c — same-subject star admission to the batched OPTIONAL) is carried forward in #1507 (the R2RML/Iceberg virtual-dataset performance program), which brings the #1475–#1507 lineage forward as a single reviewable unit. Its verification of record is the program's corpus benchmark, published as C2-bench-wave1.md (native == virtual, 0 hash mismatches). The pre-consolidation branch tip is preserved at tag |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR-4c — admit the same-subject STAR to the batched OPTIONAL (q016, the last corpus DNF)
Stacked on PR-8b (
perf/r2rml-pr8b-innerjoin-memo).q016 (
?o a Order ; orderId ?oid . OPTIONAL { ?sh edw:order ?o ; edw:shipStatus ?st }LIMIT 5000) DNF'd at 180 s. Its OPTIONAL inner is a same-subject STAR on?sh(FactShipment) whose correlation var?ois the OBJECT of a ref member, so PR-4b's narrow admission (a subject-driven single-object leaf — exactly q050) declined it → the per-rowOptionalBuilder::buildrebuild → ~445–517 FactShipment scans.This widens
inner_pattern_is_hash_join_safeto also admit a same-subject star (r2rml_star_is_hash_join_safe), so q016 takes the batched hash-left-join.Sound to admit (design doc §a–b)
R2rmlPattern::referenced_varsexhaustively destructures the struct (a new field is a compile error) and surfaces every star-member object var — the P1 audit, already landed + tested as PR-4b's precursor. So a star's?o/?stare in the partition-key input; no mis-partition.star_constraints/class_prune_hintcarry no var. No change toreferenced_varsor the seed/partition core.?ois seeded BOUND (corr_var_only_triple_objectis Triple-only) — sound, only de-optimized. Kept EXCLUDED:type_var, wildcardpredicate_var, bound subject. Sub-switchFLUREE_R2RML_BATCHED_OPTIONAL_STAR(default on within the PR-4b family).Result + the scan-count attribution
scan_tableAn initial gate criterion expected the inner to collapse to single-digit scans. It didn't — and the attribution (an env-gated per-scan table tag) showed why: the outer driving side IS collapsed (2 scans), but the batched hash-left-join drives the seed in windows and re-scans the inner table per window. That is not a PR-4c regression — the shipped PR-4b does the same (q050: 92 inner scans, 9.3 s, shipped on DNF-flip + parity). The single-digit criterion was calibrated against an idealized scan-once model, not the shipped sibling's baseline; the resolution was checking that baseline rather than forcing either number. Filed as F14 (
04-findings-register.md): batched R2RML OPTIONAL inner re-scans the main table per seed window — affects both PR-4b and PR-4c; fix class = consume the whole seed in one inner scan + in-memory hash-join (a PR-4d candidate; prize q016 ~39 s→seconds, q0509.3 s→1 s). A properly-scoped follow-up, not a blocker — uniform with what already shipped.Gate
batched_equals_per_row_on_object_correlated_star: ONE mock (Shipment→Order ref + shipStatus scalar), FOUR required orders asserted row-for-row batched≡per-row — the CARTESIAN (order matched by 2 shipments), the LEFT-JOIN miss (0), a valid single (1), and a NULL star member (the star-conjunction drops the row — both paths agree).r2rml_star_admissionpins admitted/excluded shapes. PR-4b tests still pass; full query lib 1231/0.FLUREE_R2RML_BATCHED_OPTIONAL_STAR=0⇒ q050 byte-identical (PR-4b unaffected).rows_onlyhash-gate) + native corpus 54/54, 0 hash mismatches + W3C + suites green. (The q050 nativeSLOWflag is F13 stale-baseline drift, already base-A/B-exonerated.)Design note:
docs/audit/2026-07-virtual-dataset-perf/13-pr4c-optional-star.md. This closes the corpus — the last DNF.