Skip to content

perf(r2rml): query-scoped parent-lookup memo — q031 inner-join re-scan (PR-8b) - #1492

Closed
aaj3f wants to merge 3 commits into
perf/r2rml-pr8-cold-floorfrom
perf/r2rml-pr8b-innerjoin-memo
Closed

perf(r2rml): query-scoped parent-lookup memo — q031 inner-join re-scan (PR-8b)#1492
aaj3f wants to merge 3 commits into
perf/r2rml-pr8-cold-floorfrom
perf/r2rml-pr8b-innerjoin-memo

Conversation

@aaj3f

@aaj3f aaj3f commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

PR-8b — query-scoped parent-lookup memo (q031 inner-join re-scan)

Stacked on PR-8 (perf/r2rml-pr8-cold-floor, #1491).

q031 (?inv … edw:product ?p . ?p edw:name ?pn . FILTER(?oh<?rp) LIMIT 5000)
DNF'd at 180 s: it built the DimProduct RefObjectMap parent lookup and rebuilt it
~1,305 times
— one per driving batch — re-scanning the parent each time
(scan_table.n = 1306 = 1 fact scan + 1305 re-scans). PR-4 memoized this lookup
but only in a per-operator field, and an inner join with an interposed
non-pushable FILTER + LIMIT rebuilds the whole R2RML operator per driving
batch
, resetting that cache; PR-4b covers only the OPTIONAL hash-join, so
neither seam caught an inner join.

Fix (a cache-lifetime extension, one commit): hoist the parent-lookup memo to
a query-scoped r2rml_parent_memo on ExecutionContext (mirroring
const_sid_cache), consulted in build_progress after PR-4's per-operator
cache (kept unchanged) and populated alongside it; a ctx hit seeds the
per-operator cache too. Same content, same FLUREE_R2RML_PARENT_MEMO switch —
only the lifetime is extended, valid because a lookup's content is fixed by its
key at a stable as_of_t (PR-4's own invariant, carried to query scope).

The wider (cross-operator, cross-source) share adds two guards:

  • Key = (graph_source_id, parent_tm_iri, sorted_join_cols, as_of_t) — no
    cross-source pollution (two sources' same-named parent), no cross-snapshot alias.
  • Total-rows boundtry_insert refuses an entry past
    parent_memo_total_cap_rows() (default 2× the materialize window,
    FLUREE_R2RML_PARENT_MEMO_TOTAL_WINDOWS), on top of PR-4's per-entry ≤1-window
    guard (the q015 fact-as-parent case); a refusal falls through to a per-batch
    rebuild, so a many-parent query can't grow the cache unbounded.

Result

before after
q031 DNF @ 180 s ok — 65 s cold-ish (5000 rows, correct), 188 ms hot
q031 scan_table.n 1306 ~2 (1 fact + 1 DimProduct lookup)

Gate

  • Tests (r2rml::operator::tests::pr8b): parent_lookup_survives_operator_rebuild
    (5 fresh operators / one ctx ⇒ parent scanned once — the q031 seam; off ⇒ 5),
    parent_memo_isolated_by_graph_source (same table, two gs ⇒ 2 scans). PR-4's
    within-operator test still passes. Full query lib suite 1229/0.
  • W3C SPARQL suite green · suites green.
  • Corpus: q031 oracle + q015 no-OOM + q008/q050 sentinels clean; native 54/54,
    0 hash mismatches.
  • Perf note: the native compare flagged q034 (1.84×) / q050 (2.98×). A base A/B
    (same queries on c4a9b799e, without this PR's ExecutionContext field)
    reproduced the ratios within noise (base q034 1.83× / q050 2.93×), so the
    slowdown pre-dates PR-8b. This PR adds one Arc<Mutex<…>> allocation per query
    (ctx is built once per query, runner.rs:786), which cannot account for a
    +190 ms delta — environmental (baseline drift).
  • Follow-up: the native perf baselines for these micro-queries (q034/q050
    class) need a re-bless on a quiet machine before the next gating cycle —
    chronic drift is now demonstrated twice (q009/q010 era + now), and stale
    baselines produce recurring false alarms that cost A/B runs to dismiss.

Design note: docs/audit/2026-07-virtual-dataset-perf/12-pr8b-innerjoin-memo.md.
The F8 resolution-fan-out co-factor is a separate PR-3-widening issue, out of scope.

@aaj3f
aaj3f force-pushed the perf/r2rml-pr8-cold-floor branch from c4a9b79 to 2bcc258 Compare July 15, 2026 16:32
aaj3f added 3 commits July 15, 2026 12:33
…PR-8b, q031)

q031 (`?inv … edw:product ?p . ?p edw:name ?pn . FILTER(?oh<?rp)` LIMIT 5000)
DNF'd at 180s: it built the DimProduct RefObjectMap parent lookup and rebuilt it
~1,305 times — one per driving batch — re-scanning the parent every time
(scan_table.n = 1306 = 1 fact scan + 1305 parent re-scans; ~102ms of
loadTable-context + scan_plan each). PR-4 memoized this lookup, but only in a
PER-OPERATOR field: an inner join with an interposed NON-PUSHABLE FILTER + LIMIT
rebuilds the whole R2RML operator per driving batch (operator.rs — "correlated …
rebuilds the whole operator … resetting any cache"), so the per-operator cache
never outlives one batch. PR-4b covers only the OPTIONAL batched hash-join, so
neither seam caught an inner join.

This extends the memo to a QUERY-SCOPED lifetime that survives the rebuild: a new
`r2rml_parent_memo: Arc<Mutex<…>>` on `ExecutionContext` (mirroring the existing
`const_sid_cache`), consulted in `build_progress` AFTER PR-4's per-operator cache
(kept unchanged) and populated alongside it; a ctx hit also seeds the per-operator
cache so later batches of the same operator instance take the fast, lock-free
path. Same content, same `FLUREE_R2RML_PARENT_MEMO` switch — only the lifetime is
extended, valid because a lookup's content is fixed by its key at a stable
`as_of_t` (PR-4's own invariant, carried to query scope).

The wider (cross-operator, cross-source) share adds two guards:
  - KEY carries graph_source_id + as_of_t: `(graph_source_id, parent_tm_iri,
    sorted_join_cols, as_of_t)` — two sources holding a same-named parent table
    can't cross-pollute, and two snapshots can't alias. (PR-4's per-operator key
    stays the 2-tuple; an operator is single-source.)
  - TOTAL-ROWS BOUND: `try_insert` refuses an entry that would push the memo past
    `parent_memo_total_cap_rows()` (default 2× the materialize window,
    `FLUREE_R2RML_PARENT_MEMO_TOTAL_WINDOWS`), on top of PR-4's per-entry
    ≤1-window guard (the q015 fact-as-parent case). A refused insert falls through
    to a per-batch rebuild, so a many-parent query can't grow the cache unbounded.

Tests (r2rml::operator::tests::pr8b): parent_lookup_survives_operator_rebuild — 5
FRESH operators against one ctx ⇒ the DIM parent is scanned ONCE (memo on) vs 5
(off) — the q031 seam, distinct from PR-4's within-operator test which still
passes; and parent_memo_isolated_by_graph_source — same table, gs:A/gs:B ⇒ 2
scans (no cross-source pollution). Full query lib suite 1229/0.

See docs/audit/2026-07-virtual-dataset-perf/12-pr8b-innerjoin-memo.md. The F8
resolution-fan-out co-factor (load_table.n=7 for a 2-table query) is a separate
PR-3-widening issue, out of scope here.
…rent memo

R2rmlParentMemoInner::try_insert refuses an insert that would push the memo past
its total-rows cap (the caller then rebuilds per-batch for that key) — the one
untested branch of the new type. Add a hermetic test passing the cap directly
(env-hermetic) covering: fits-under-cap, idempotent-same-key, refused-over-cap,
fits-exactly, and refused-at-cap.
@aaj3f

aaj3f commented Jul 21, 2026

Copy link
Copy Markdown
Contributor Author

Consolidating at maintainer request. #1492 (PR-8b — the query-scoped parent-lookup memo for the inner-join re-scan) is carried forward in #1507 (the R2RML/Iceberg virtual-dataset performance program), which brings the #1475#1507 lineage forward as a single reviewable unit. Its verification of record is the program's corpus benchmark, published as C2-bench-wave1.md (native == virtual, 0 hash mismatches). The pre-consolidation branch tip is preserved at tag archive/pre-refactor-2026-07-21/perf_r2rml-pr8b-innerjoin-memo — no history was rewritten. Closing in favor of #1507; discussion continues there.

@aaj3f aaj3f closed this Jul 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant