test(bench): re-bless at the audit head + resilience hermetics + SWITCHES.md regeneration (audit Tier-1/hygiene) - #1523
Closed
aaj3f wants to merge 9 commits into
Closed
test(bench): re-bless at the audit head + resilience hermetics + SWITCHES.md regeneration (audit Tier-1/hygiene)#1523aaj3f wants to merge 9 commits into
aaj3f wants to merge 9 commits into
Conversation
The committed virtual perf baseline was 62% gate-blind: 28 members sat pinned at exactly 180000ms — DNFs (deadline-capped walls) blessed AS the timeout, so a now-fast query could regress ~1740x (to 216s) before the 20%-over gate tripped. Root cause: write_perf blessed r.wall_ms regardless of status. Guard it with a pure is_dnf_wall(status, wall_ms, timeout_ms): a terminal Status::Dnf, or (belt-and-suspenders) any wall at/over the query's deadline, blesses as no-baseline (must-fix) — the matching wall field is cleared to None so a timeout cap can never masquerade as a budget, and a query that regressed to a DNF loses any stale fast wall. Corpus supplies each deadline; DNF counters still recorded. Two unit tests: the predicate, and the end-to-end no-baseline + stale-clear.
E1 (the ex:category shared-predicate class collapse) shipped unswitched on the audited line — fuse_class_if_safe called wildcard_class_fusion_is_safe directly with no kill switch, so an operator could not revert it in the field (A2 D2). Add a dedicated FLUREE_R2RML_SHARED_PREDICATE_FUSION (default on). OFF does not re-materialize: control falls through to the weaker pre-E1 class_prune_hint (star + separate class scan, still correct), exactly the pre-E1 behavior. Kept distinct from FLUREE_R2RML_CRAWL_CLASS_FUSION (the browse-crawl wildcard path) so the two mechanisms are not conflated. The existing ON-path test still passes; the OFF path targets an already-tested fallback. W4-1/W4-1b are documented exemptions in SWITCHES.md (coarse-revertible via PREDICATE_PUSHDOWN; a dedicated switch is invasive — they share the baseline pushdown apply site).
The two existing C3 tests cancel up front on an empty stream — they prove the first checkpoint fires but not that a drain loop is stopped mid-way. Add a test where a 5-batch stream cancels the query while producing its first batch: the loop must consume exactly one batch and then return Cancelled at the next checkpoint, leaving four unread (poll count 1, not 5). This is the actual 'cancellation stops a drain loop' property the audit flagged as uncovered. Coverage note: R3-B (r3b_scan_window/parent_build_budget_aborts_typed), PR-8 429/Retry-After (retries_on_429_then_succeeds + backoff-bound, wiremock), and C2 /info routing (#1522 info_member_routing/merge_virtual/mor_approximate) are already covered by #1521/#1522 — this closes the one remaining C3 gap.
…(F-AUD-19) SWITCHES.md documented ~19 of the ~40 on-path FLUREE_* switches and omitted PR-8 (catalog concurrency/retries/backoff/disk-cache/parallel-catalog), PR-2 (footer-from-cache/scan-concurrency), C1 (dataset-budget), C4 (rowgroup-parallelism), R3-B (memory-budget-bytes), and the #1520/#1521/#1522 additions — an operator could not revert those from the doc. Regenerated by enumerating every FLUREE_* read across query/iceberg/api and verifying each row against code (A2 is the map; code is truth). Adds the ~8 A2 omissions + the 9 new switches (incl. the inverted-polarity ALLOW_MOR_DELETES escape hatch and the new SHARED_PREDICATE_FUSION), sections for correctness/safety/capacity and baseline levers, the W4-1/W4-1b unswitched exemptions, and fixes the CRAWL_CLASS_FUSION name drift (A2 D2). Format: name | default | gates | OFF restores | introducing PR.
The live virtual re-bless surfaced 4 members (q038, q056/q057/q059) aborting with MemoryBudgetExceeded (a Status::Error 507, ~8.6-9.4GB) under #1521's default-on FLUREE_SCAN_MEM_ACCOUNTING — its cumulative-no-decrement counter conflates total-rows-streamed with resident memory and false-aborts long bounded-window streams (q038 completes in 52.5s with accounting off; it completed in C2). The never-bless-a-DNF guard did NOT catch these: an error is not a Status::Dnf and its wall (time-to-abort) is under the deadline, so write_perf would have pinned the abort time as a perf budget. Generalize is_dnf_wall -> is_unblessable_wall: only a Status::Ok run finishing inside its deadline yields a blessable wall; DNF, Error, and ExpectedError all bless as no-baseline (must-fix). Tests extended with the error and expected-error cases.
…budget The drift set (q002/q004/q022/q024/q030/q043) is network-flappy Snowflake catalog latency, not engine (C2 §3/§4 measured up to ~3.8x run-to-run swing). Give each a 300% per-query override so a catalog-latency spike (backed by the compare auto-rerun) does not flap the gate, while a gross (>4x) regression still trips.
…8, item 2a) Full-corpus native-sf01 run (77 members, 3 reps, local ledger, no creds) at the audit head. Native perf baseline goes 54 -> 77 entries with ZERO timeout caps (the never-bless-a-non-completion guard held; native has no DNFs). The 74 correctness oracles all reproduced exactly (3 expected-virtual-error skips) — no native regression. q077 (doubly-constrained COUNT) = 65.9s native release, a native-planner cost class.
…gate record (F-AUD-18, item 2b) The committed virtual perf baseline was 62% gate-blind (33 members pinned at 180000ms — DNF caps blessed as budgets, blessed_from 7d77218). Re-blessed from a full live virtual-sf01 corpus run, spliced with a post-#1521-fix subset re-run: vbench compare --run <merged> --gate -> 77 records, 0 hash mismatches, 0 perf violations, exit 0. Now 77 entries, ZERO at 180000; the only no-baselines are the 3 expected-errors (q013/q034/q051) + 3 memory-abort members not in the re-run subset (q056/q057/q059). q038 blessed at its real 58.2s wall (the #1521 window-release fix confirmed live — no abort). Splice methodology + memory-abort finding + q038-fusion-PARTIAL diagnosis + new-member wall table recorded in docs/audit-impl/cov-scan-gates.md; the three PAT-scrubbed JSONLs ship in audit-2026-07/data/.
…ot 33) + drift-residual note R-1523 nonblocking: the stale virtual-sf01.json had 28 members at hot=180000ms + 2 at 120000ms (q044/q050), not 33 (an earlier grep over-counted string occurrences). Also document the accepted drift-override residual: the 300% budgets on the 6 catalog-dominated drift members tolerate a sub-4x engine regression there (hash arm unaffected) — a deliberate anti-flap trade-off.
Contributor
Author
|
Consolidating at maintainer request. #1523 (the harness re-bless, resilience hermetics, and switch-registry regeneration) is carried forward in #1528 (the big-Iceberg-audit implementation), which brings the audit program forward as a single reviewable unit. Its verification of record is recorded under R-1523 in pr-reviews-impl.md (verdict SHIP; the gate can no longer pin a timeout or an abort as a budget) — and it is the PR that produced the whole-stack live gate of 77 records / 0 hash mismatches / 0 perf violations. The pre-consolidation branch tip is preserved at tag |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR-HARNESS — re-bless at the audit head + resilience hermetics + SWITCHES.md regeneration
Final PR (4 of 4) of the big-iceberg-audit implementation phase (DEC-001). Restores
harness honesty: the perf baseline is re-blessed at the true integrated head (the live
verification of record), the one uncovered resilience path gets a regression test, every
mechanism has a documented kill switch, and the vbench bless path can no longer pin a
timeout — or an abort — as a budget.
Leaf on
perf/audit-coverage(#1522) →perf/audit-mem-guards(#1521) →perf/audit-tier012(merge-up #1508) → the audit line. Links:audit-2026-07/00-MASTER-AUDIT.md,audit-2026-07/decisions/DEC-001-pr-bundling.md,audit-2026-07/C2-bench-wave1.md.Forest map — what this closes
is_unblessable_wall) + full-corpus re-bless (native + LIVE virtual, spliced)FLUREE_R2RML_SHARED_PREDICATE_FUSION); full SWITCHES.md regeneration; document W4-1/W4-1b exemptionscount_shortcut_eligiblepredicate + decline test — landed upstream as3ece996fd; my redundant commit dropped on rebaseTHE LIVE GATE — verification of record for the whole stack
vbench compare --run <merged> --gate→ 77 records, 0 hash mismatch(es), 0 perfviolation(s), exit 0. Correctness passes end-to-end at the final integrated head
(#1521 mem-fix + #1522 coverage + this PR). The drift-set 300% overrides absorbed the
loadTable-GET network variance. Accepted residual: those 4×-baseline (300%-over)
budgets on the 6 catalog-dominated drift members (q002/q004/q022/q024/q030/q043) mean a
sub-4× engine regression there won't trip the perf arm (the hash/correctness arm is
unaffected) — a documented anti-flap trade-off, re-narrowable if the catalog path stabilizes.
New / changed member walls (live virtual-sf01, hot median):
q077 = 64.1 s virtual vs 65.9 s native — the walls CONVERGE (both planner-bound;
supersedes the earlier "virtual is the fast path" phrasing, which compared against a 133 s
dev-build native).
The re-bless record (F-AUD-18)
is_unblessable_wall(status, wall_ms, timeout_ms)blesses a perf wall ONLY for aStatus::Okrun finishing inside its deadline; a DNF (timeout cap), an Error (abort time),or an ExpectedError all bless as no baseline / must-fix. Baseline coverage:
7d77218e2with 28 pinned at 180000ms (+2 at 120000ms, q044/q050) + 14 missing; now 77 entries, ZERO at 180000/120000. Six honest no-baselines: q013/q034/q051 (expected-virtual-error) + q056/q057/q059 (see finding).Splice methodology (honest, documented): the live re-bless spans two heads because the
#1521 abort fix cascaded after the full run. Blessed baseline = 76 valid records from the
full 77-query run at pre-fix
73a7694bf+ a post-fix single-flight re-run of theaccounting-sensitive subset {q038, q014, q069, q073, q075, q077} at
cd3779480. The 71non-subset records are valid at the final head (the #1521 fix changes only memory
ACCOUNTING, affecting exactly the false-aborting members). All three PAT-scrubbed JSONLs
ship in
audit-2026-07/data/.FINDING — memory-abort false-positives (F-AUD-3 / #1521), now FIXED
The pre-fix full run aborted 4 members (q038, q056, q057, q059) with
MemoryBudgetExceeded(~8.6–9.4 GB, 8 GiB macOS-fallback budget) — false positives of #1521's
cumulative-no-decrement counter (summed total-rows-streamed, not resident memory, on long
bounded-window streams). MEASURED: q038 completes in 52.5 s with
FLUREE_SCAN_MEM_ACCOUNTING=off. impl-mem's window-scoped release fix (cfd773d75) landsin the cascade; the post-fix subset re-run confirms q038 completes (58.2 s, no abort).
q056/q057/q059 (exploration-wildcard family) were not in the re-run subset → they remain
no-baseline (a follow-up re-run would bless them; a known heavy whole-warehouse family
regardless). The bless guard was broadened this PR so an abort time cannot be pinned as a
budget.
q038 fusion — PARTIAL, not CLOSED (F-AUD-8 ladder refinement)
q038 uses the constant-object TRIPLE form (not a SPARQL FILTER), identical to the fusing
q077/q022, yet still materializes (58 s) — 9b did NOT admit its exact shape. Discriminator:
ungrouped-vs-grouped (q022 GROUPED fuses at 87 ms; q038 UNGROUPED
COUNT(*)declines),matching the empty-GROUP-BY cost guard
filter.is_some() && group_by.is_empty()(
fused_aggregate.rs:441) — which fires only if the direct-path constraint arrives as aresidual FILTER, not a folded
star_constraint. The ladder's q038 "886× payoff" istherefore PARTIAL: q077-class + grouped constant-object COUNTs fuse; q038's exact
ungrouped direct-path single-constraint form does not yet. A #1522 follow-up.
Resilience hermetics (F-AUD-20)
F-AUD-20 was largely closed by the implementation PRs; PR-HARNESS adds the one genuine gap:
collect_stream_stops_a_drain_loop_mid_sweep— a 5-batch stream cancelled while producingits first batch must stop after consuming exactly one (poll count 1, not 5). The two prior
C3 tests only cancelled up front on an empty stream. Already covered (verified present +
passing): R3-B scan-window + parent-build typed aborts + shared-ceiling division (#1521);
PR-8 429/Retry-After retry loop + backoff bound (wiremock, #1491); C2 /info empty-shell +
member-routing + hybrid-merge + MoR-flag (#1522).
SWITCHES.md regeneration (F-AUD-19)
Regenerated from every
FLUREE_*read across query/iceberg/api, verified against code(~40 switches, was ~19). Adds the ~8 A2 omissions (DATASET_BUDGET, PARALLEL_CATALOG,
CATALOG_DISK_CACHE, CATALOG_CONCURRENCY/MAX_RETRIES/BACKOFF_BASE_MS, FOOTER_FROM_CACHE,
ROWGROUP_PARALLELISM, QUERY_MEMORY_BUDGET_BYTES, INFO_COUNT_BUDGET_MS) + the 9 new switches,
sections for correctness/safety/capacity and baseline levers, and fixes the
CRAWL_CLASS_FUSIONname drift (A2 D2). The inverted-polarityALLOW_MOR_DELETESescapehatch is called out.
E1 / W4-1 / W4-1b decision (F-AUD-19)
fn directly. Retro-switched with a dedicated
FLUREE_R2RML_SHARED_PREDICATE_FUSION(default on); OFF falls through to the pre-E1
class_prune_hint(still correct).FLUREE_ICEBERG_PREDICATE_PUSHDOWN(coarse-revertible today). Adedicated fine-grained switch is invasive — they share the baseline pushdown apply site.
Documented as exemptions in SWITCHES.md §7 (mechanism, coarse field-revert,
commit-revert). The mandate favors minimal changes.
Gates
fmt (clean on all changed files; the sole
--checkdiff is the pre-existinghash_join.rs:1070/1120base drift, inherited and deliberately left per db-verify-gotchas);clippy
-p fluree-bench-virtual -p fluree-db-query --all-targets --no-deps= 0;cargo test -p fluree-db-query/-p fluree-bench-virtual --bins/-p fluree-db-api --features icebergall pass; the LIVE corpus gate above. Full record indocs/audit-impl/cov-scan-gates.md. CI does not fire on this branch (base ≠ main); thisbody + the gate doc are the reproduction of record.
Residuals (for follow-up, not this PR)
73a7694bf(the splice kept the full-run meta); the gate doc records both source SHAs.