perf(r2rml): scan-side top-k for ORDER BY DESC LIMIT (PR-5, cold 37s→3.1s)

[aaj3f]

What

PR-5 (the final roadmap item): scan-side top-k for ORDER BY DESC(<scan col>) LIMIT k over a single-table R2RML/Iceberg scan. The scan reads files in upper_bound(sort_col)-DESC order, keeps a running k-th bound, and stops once no unread file can beat it — reading ~k files instead of the whole table.

Closing gate (DW_SF01, 7670 files per fact table)

q046	wall	files_pruned / selected	hash
ON (hot)	136ms	7660 / 10	== oracle
OFF (hot)	406ms	0 / 7670	== oracle (byte-identical revert)
COLD	3129ms	7660	== oracle

Cold ~37s → 3.1s (~12×), files_pruned 7660/7670 = 99.87% — the headline. ON is faster than OFF even hot. ORDER-BY-class parity sweep (q005/q012 declined via fallback, q046 identical), native 54/54, and the W3C ORDER BY/LIMIT suite: 0 hash mismatch, 0 perf violations.

Honest scope: q046 is the only corpus query this helps (raw-column top-k over a fact table), and the win is cold (warm q046 was already ~400ms). PR-7's numeric stats paid the hard cost; this is the harvest on the canonical "top N by measure" shape.

How

The SortOperator offers a TopK(primary DESC var, k = LIMIT + OFFSET) directive to the scan below (mirroring row_budget); GraphOperator threads it into its per-parent inner subplan (where a virtual query's scan lives), and ProjectOperator forwards it. The api scan reads files in bound-DESC order with a size-(k+offset) heap and early-stops; a TOPK_SEQUENTIAL_CAP fallback to the parallel reader means an ineffective prune is never slower than the full scan. sort.rs::new_topk above stays the sole authority for the exact (compound) order + LIMIT — the scan only skips files it proves cannot contribute, so results are byte-identical to full sort.

Invariants

• Strict superset: the scan streams every row of every non-pruned file; the authoritative sort above re-selects the exact top-k → results identical to full sort.
• Heap fed post-filter qualifying rows only: resolve_topk_directive DECLINES the pushdown when the scan carries a RESIDUAL filter (consumed_filter / object_constant / subject_constant / star_constraints) the operator enforces after emit — feeding a superset would ride the k-th bound too high and silently drop qualifying rows. Pushed scan filters are applied by the reader (the emitted batch is already post-filter), so they compose fine.
• No pruned result is ever cached: a top-k scan bypasses the per-operator scan cache (cacheable && self.topk.is_none()), keyed on the STORED directive — so a declined-topk scan runs FULL and merely skips the cache, never caching a subset that a later full scan of the same (table, projection) could replay.
• k = LIMIT + OFFSET (single-owner arithmetic) so the scan retains the rows the sort's OFFSET then skips.
• DESC only: ASC declines to fallback — SPARQL orders unbound values first in ASC, so a null-bearing file could never be pruned.

Kill switch

FLUREE_R2RML_TOPK_PUSHDOWN (default on); off ⇒ no directive ⇒ today's full-materialize top-k.

Tests

iceberg core (10): TopKBound bound/tie/NaN, plan_topk_read / batch_sort_values, read-loop simulations (prune + k-exceeds-non-null + all-null-file). query: topk_declines_on_residual_filter (all four residual shapes + pure-scan positive), topk_scan_bypasses_scan_cache (cache-poison — a topk scan re-reads each batch vs a cached full scan), topk_directive_carries_limit_plus_offset (k = LIMIT+OFFSET). All green; clippy/fmt clean; the full -p fluree-db-api --features iceberg suite is green.

Note on observability: the topk branch emits an iceberg.scan_plan span for the harness counters; on the cold miss arm the planner also emits one (the pre-topk plan), so files_selected double-counts there while files_pruned stays authoritative — the gate ratio is files_pruned / total_files.

Stacked on perf/r2rml-pr7-numeric-stats. The F13 native-perf re-bless rides on this branch. Closes the virtual-dataset perf roadmap.

[aaj3f]

Consolidating at maintainer request. #1495 (PR-5 — scan-side top-k for ORDER BY DESC + LIMIT) is carried forward in #1507 (the R2RML/Iceberg virtual-dataset performance program), which brings the #1475–#1507 lineage forward as a single reviewable unit. Its verification of record is the program's corpus benchmark, published as C2-bench-wave1.md (native == virtual, 0 hash mismatches). The pre-consolidation branch tip is preserved at tag archive/pre-refactor-2026-07-21/perf_r2rml-pr5-scan-topk — no history was rewritten. Closing in favor of #1507; discussion continues there.