perf(r2rml): query-scoped parent-lookup memo — q031 inner-join re-scan (PR-8b) - #1492
Closed
aaj3f wants to merge 3 commits into
Closed
perf(r2rml): query-scoped parent-lookup memo — q031 inner-join re-scan (PR-8b)#1492aaj3f wants to merge 3 commits into
aaj3f wants to merge 3 commits into
Conversation
aaj3f
force-pushed
the
perf/r2rml-pr8-cold-floor
branch
from
July 15, 2026 16:32
c4a9b79 to
2bcc258
Compare
…PR-8b, q031)
q031 (`?inv … edw:product ?p . ?p edw:name ?pn . FILTER(?oh<?rp)` LIMIT 5000)
DNF'd at 180s: it built the DimProduct RefObjectMap parent lookup and rebuilt it
~1,305 times — one per driving batch — re-scanning the parent every time
(scan_table.n = 1306 = 1 fact scan + 1305 parent re-scans; ~102ms of
loadTable-context + scan_plan each). PR-4 memoized this lookup, but only in a
PER-OPERATOR field: an inner join with an interposed NON-PUSHABLE FILTER + LIMIT
rebuilds the whole R2RML operator per driving batch (operator.rs — "correlated …
rebuilds the whole operator … resetting any cache"), so the per-operator cache
never outlives one batch. PR-4b covers only the OPTIONAL batched hash-join, so
neither seam caught an inner join.
This extends the memo to a QUERY-SCOPED lifetime that survives the rebuild: a new
`r2rml_parent_memo: Arc<Mutex<…>>` on `ExecutionContext` (mirroring the existing
`const_sid_cache`), consulted in `build_progress` AFTER PR-4's per-operator cache
(kept unchanged) and populated alongside it; a ctx hit also seeds the per-operator
cache so later batches of the same operator instance take the fast, lock-free
path. Same content, same `FLUREE_R2RML_PARENT_MEMO` switch — only the lifetime is
extended, valid because a lookup's content is fixed by its key at a stable
`as_of_t` (PR-4's own invariant, carried to query scope).
The wider (cross-operator, cross-source) share adds two guards:
- KEY carries graph_source_id + as_of_t: `(graph_source_id, parent_tm_iri,
sorted_join_cols, as_of_t)` — two sources holding a same-named parent table
can't cross-pollute, and two snapshots can't alias. (PR-4's per-operator key
stays the 2-tuple; an operator is single-source.)
- TOTAL-ROWS BOUND: `try_insert` refuses an entry that would push the memo past
`parent_memo_total_cap_rows()` (default 2× the materialize window,
`FLUREE_R2RML_PARENT_MEMO_TOTAL_WINDOWS`), on top of PR-4's per-entry
≤1-window guard (the q015 fact-as-parent case). A refused insert falls through
to a per-batch rebuild, so a many-parent query can't grow the cache unbounded.
Tests (r2rml::operator::tests::pr8b): parent_lookup_survives_operator_rebuild — 5
FRESH operators against one ctx ⇒ the DIM parent is scanned ONCE (memo on) vs 5
(off) — the q031 seam, distinct from PR-4's within-operator test which still
passes; and parent_memo_isolated_by_graph_source — same table, gs:A/gs:B ⇒ 2
scans (no cross-source pollution). Full query lib suite 1229/0.
See docs/audit/2026-07-virtual-dataset-perf/12-pr8b-innerjoin-memo.md. The F8
resolution-fan-out co-factor (load_table.n=7 for a 2-table query) is a separate
PR-3-widening issue, out of scope here.
… machine state (PR-8b ctx rider)
…rent memo R2rmlParentMemoInner::try_insert refuses an insert that would push the memo past its total-rows cap (the caller then rebuilds per-batch for that key) — the one untested branch of the new type. Add a hermetic test passing the cap directly (env-hermetic) covering: fits-under-cap, idempotent-same-key, refused-over-cap, fits-exactly, and refused-at-cap.
aaj3f
force-pushed
the
perf/r2rml-pr8b-innerjoin-memo
branch
from
July 15, 2026 16:47
8fe47ae to
a934563
Compare
Contributor
Author
|
Consolidating at maintainer request. #1492 (PR-8b — the query-scoped parent-lookup memo for the inner-join re-scan) is carried forward in #1507 (the R2RML/Iceberg virtual-dataset performance program), which brings the #1475–#1507 lineage forward as a single reviewable unit. Its verification of record is the program's corpus benchmark, published as C2-bench-wave1.md (native == virtual, 0 hash mismatches). The pre-consolidation branch tip is preserved at tag |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR-8b — query-scoped parent-lookup memo (q031 inner-join re-scan)
Stacked on PR-8 (
perf/r2rml-pr8-cold-floor, #1491).q031 (
?inv … edw:product ?p . ?p edw:name ?pn . FILTER(?oh<?rp)LIMIT 5000)DNF'd at 180 s: it built the DimProduct RefObjectMap parent lookup and rebuilt it
~1,305 times — one per driving batch — re-scanning the parent each time
(
scan_table.n = 1306 = 1 fact scan + 1305 re-scans). PR-4 memoized this lookupbut only in a per-operator field, and an inner join with an interposed
non-pushable
FILTER + LIMITrebuilds the whole R2RML operator per drivingbatch, resetting that cache; PR-4b covers only the OPTIONAL hash-join, so
neither seam caught an inner join.
Fix (a cache-lifetime extension, one commit): hoist the parent-lookup memo to
a query-scoped
r2rml_parent_memoonExecutionContext(mirroringconst_sid_cache), consulted inbuild_progressafter PR-4's per-operatorcache (kept unchanged) and populated alongside it; a ctx hit seeds the
per-operator cache too. Same content, same
FLUREE_R2RML_PARENT_MEMOswitch —only the lifetime is extended, valid because a lookup's content is fixed by its
key at a stable
as_of_t(PR-4's own invariant, carried to query scope).The wider (cross-operator, cross-source) share adds two guards:
(graph_source_id, parent_tm_iri, sorted_join_cols, as_of_t)— nocross-source pollution (two sources' same-named parent), no cross-snapshot alias.
try_insertrefuses an entry pastparent_memo_total_cap_rows()(default 2× the materialize window,FLUREE_R2RML_PARENT_MEMO_TOTAL_WINDOWS), on top of PR-4's per-entry ≤1-windowguard (the q015 fact-as-parent case); a refusal falls through to a per-batch
rebuild, so a many-parent query can't grow the cache unbounded.
Result
scan_table.nGate
r2rml::operator::tests::pr8b):parent_lookup_survives_operator_rebuild(5 fresh operators / one ctx ⇒ parent scanned once — the q031 seam; off ⇒ 5),
parent_memo_isolated_by_graph_source(same table, two gs ⇒ 2 scans). PR-4'swithin-operator test still passes. Full query lib suite 1229/0.
0 hash mismatches.
(same queries on
c4a9b799e, without this PR'sExecutionContextfield)reproduced the ratios within noise (base q034 1.83× / q050 2.93×), so the
slowdown pre-dates PR-8b. This PR adds one
Arc<Mutex<…>>allocation per query(ctx is built once per query,
runner.rs:786), which cannot account for a+190 ms delta — environmental (baseline drift).
class) need a re-bless on a quiet machine before the next gating cycle —
chronic drift is now demonstrated twice (q009/q010 era + now), and stale
baselines produce recurring false alarms that cost A/B runs to dismiss.
Design note:
docs/audit/2026-07-virtual-dataset-perf/12-pr8b-innerjoin-memo.md.The F8 resolution-fan-out co-factor is a separate PR-3-widening issue, out of scope.