perf(virtual): big-Iceberg-audit implementation — correctness/capacity guards, coverage widenings, harness re-bless, browse-parity, family-C (consolidates the audit program + #1525) - #1528
Merged
Conversation
First facet of the governance-model tooling (fluree model). Users
declare intent; the CLI compiles it to stored policies and transacts
them as ordinary data — no bespoke server API, so it works identically
against local ledgers, fluree-db-server, and hosted stacks.
fluree model access enable <dataset> --profile read|write|intake
--entity <class-iri> [--property <iri>...] [--allow-shared]
[--class-iri <iri>] [--dry-run] [--remote <r>]
fluree model access show <dataset>
Compilation: a policy class (the assignment unit grants/tokens carry),
a view policy (f:onClass — exact for reads), a property-whitelist
modify policy (f:onProperty + f:allow — class-targeted modify cannot
cover new-subject inserts and f:query evaluates pre-state), and a
declarative fm:AccessProfile node recording the intent so enable is
idempotent (upsert) and future sync/verify re-derive from it.
Property derivation: explicit → SHACL shape → observed data →
fail-closed. Uniqueness partition: properties used by other classes
are excluded by default with the blast radius disclosed
('also used by: Invoice'), included only with --allow-shared; every
run prints an exactness verdict (class-exact vs property-approximate)
and flags the rdf:type creation caveat.
Unit tests cover compilation shapes per profile, the parse gates, and
the IRI validation; verified end-to-end in local mode (enable → show →
identity+policy-class insert allowed → non-whitelisted write denied →
shared-property partition + disclosure).
…tacks Completes the enable flow end-to-end: after the policies land in the ledger (data plane), --space merges the compiled policy class into the space's grant on the dataset via the stack's grants API — the one system-plane touch in the compiler, since grants are router-owned invariants (scope validation, membership checks). - Reads existing grants first and merges classes (never clobbers); keeps the existing access level, upgrading read → write with a printed notice only when the profile requires writes. - Policies-then-grant ordering is deliberate: a partial failure leaves unused policies (harmless), never a grant naming absent classes. - Root URL derived from the remote's data-plane URL (…/v1/fluree); clear errors when the remote isn't a hosted stack or has no token. - --space without --remote is a usage error. Verified against a live stack: enable … --space wrote 4 policies + attached the grant in one command; a minted space app token then carried leads2 with the compiled class, a whitelisted insert passed, and a non-whitelisted write was rejected at the ledger.
Terminology only: user-facing output now prints 'Allowed: N properties', identifiers use allowed/allow_list, docs say property allow-list. No behavior change.
Second facet of the governance-model tooling. The entity shape is the
model's single source of truth: the property list is written ONCE here
and everything else derives — model access enable now prefers
shape-derivation for its allow-lists, transaction-time validation
enforces the shape (the shapes-exist heuristic activates reject mode),
and SDK types / form fields can be generated from it later.
fluree model entity define <dataset> --entity <iri> [--label <l>]
--property "<iri> [string|integer|decimal|boolean|date|datetime|iri]
[required] [in[v1,v2,...]]" ...
fluree model entity show <dataset>
Compiles to an rdfs:Class node + a sh:NodeShape targeting it, with
stable property-shape ids ({entity}/shape/{localname}) so re-define is
an idempotent upsert. The define output states plainly that validation
becomes ACTIVE (existing data is not retro-validated; fluree validate
reports) and prints the model access enable synergy hint.
Shared ledger-IO helpers (resolve_mode/query/upsert) promoted from
access.rs to model/mod.rs.
Verified locally end-to-end: define (no seed data) → access enable
reports 'Derivation: shacl-shape' (class-exact, 3 properties) → a
shape-violating insert is rejected by SHACL with a violation report →
a conforming insert through the compiled policy class commits → both
show views render.
Third tier of the access compiler: a SPARQL property-path subset
(sequence / and inverse ^ of angle-bracketed absolute IRIs), anchored
identity → path → subject, expanded into f:query where-patterns on the
view policy (f:query replaces f:allow — they are alternative decision
modes). The raw path is recorded as fm:connected on the profile node.
--connected "^<https://x/owner>" # I see what I own
--connected "<https://x/memberOf>/^<https://x/team>"
# I see entities whose team I'm in
v1 scope: --profile read only. Relationship gates evaluate against
pre-transaction state, so gating writes would deny every create (the
connection triples are not yet visible); staged-state evaluation at
the engine lifts this later. Alternatives (|) and transitive (+/*)
paths are rejected with a clear message until f:query supports them.
Verified live in local mode: two users on different teams, one enable
with --connected, and each user's query returns only their own team's
entities — graph-native row-level security from a single flag.
…nd hierarchy - 'model access verify' recompiles every stored fm:AccessProfile from its intent node and semantically compares against the policies in the ledger (compact/full form tolerant). Hand-edited allow-lists, flipped f:allow, retargeted classes, and missing/unexpected policies report as DRIFT with a nonzero exit for CI. - 'model access sync' re-derives profiles whose property surface came from a source that can change (SHACL shape, observed data) and recompiles on drift; explicit lists are never rewritten. The compiler now records fm:derivation and fm:allowShared on the intent node so sync honors the author's original choices; nodes written before these fields (or by other front ends) read as explicit — the safe default. - 'model class define/show' — rdfs:subClassOf hierarchy as governed data, with the entailment blast-radius note; show excludes compiler-minted policy classes from the domain vocabulary. Verified live against a hosted stack (verify caught a hand-widened allow-list; enable repaired it) and locally (shape grew -> sync applied '+score' -> verify clean).
Proves the storage form governance tooling uses for relationship gates: a stored, classed view policy whose f:query is one triple over an @path term (the author's path expression survives verbatim in the artifact), selected via policy-class with a bind-only identity. Two identities, one gate, each sees only entities connected via ex:memberOf/^ex:team.
…caffolder The engine's f:create/f:update/f:delete verbs (with pre∪post class targeting and class-scoped rdf:type matching) make thin class policies exact, so the entire workaround layer is deleted: no property allow-list derivation, no uniqueness partition / --allow-shared, no fm: intent vocabulary, no sync, no drift-verify (1,461 → ~700 lines). Profiles map directly: read → view; write → view + create/update/delete; intake → create-only. --property remains as optional column narrowing (f:onProperty conjunction). --connected now stores the SPARQL path VERBATIM via the engine's @path context term (readable + reversible in the artifact; show extracts it back). model entity define gains --closed (sh:closed + rdf:type carve-out): validation owns the property surface, policy owns state transitions. Contract proven by integration test scaffolder_write_profile_grants_class_ownership_only: the scaffolder's byte-exact output, stored + selected via policy-class with bind-only identity, allows create/update of the class and denies minting or editing other classes. @path gate test switched to the CLI's angle-bracket absolute-IRI storage form.
The incremental Phase 3b ref-edge re-attribution keyed batch-local class membership on base_subject_classes alone. A subject typed for the FIRST time in a batch has no base entry (no rdf:type hits in the base index), so both Case A (outgoing) and Case B (inbound) edges resolved it through the external base-index lookup, found no classes, and dropped the +1 side of the move — refs indexed before the type arrived stayed unattributed forever. Re-types (class -> class) were unaffected, which is why the #1266 tests passed. Key the batch-local branch on either map (base OR net) so first-time types resolve their net classes from the in-batch membership. This also skips the now-pointless external rdf:type lookups for those endpoints. Repro: insert refs to untyped targets, index, assert the targets' rdf:type in a later transaction, index incrementally -> ref-classes stayed empty ({} attributed) while types.@id counted the refs. Full reindex was the only recovery. Adds typed-later variants (subject, object, both endpoints) alongside the existing re-type tests.
Property-merge upsert could leave a stale f:allow: true on a policy node
when re-running `model access enable --connected`; the policy loader
gives f:allow precedence over f:query, so the new relationship gate was
silently disabled and the policy stayed allow-all. Install compiled
artifacts through one update transaction that wildcard-deletes every
OWNED node id (delete-if-exists via optional patterns) and inserts the
fresh compilation.
- access: owns {class}/view and {class}/write on every run, so gating
switches are exact and profile switches on the same class revoke the
node the new profile no longer emits
- entity: owns the shape and its property-shape children (a dropped
sh:closed/sh:minCount/sh:in no longer survives a re-run); the entity
class node is shared authorship with `model class define` and stays
additive
- class show: recognize policy classes by the extra @type on
f:AccessPolicy nodes (the fm:AccessProfile intent vocabulary no longer
exists); correct the --subclass-of re-run hint (replaces, not extends)
- integration test pins the CLI-emitted replace transaction end-to-end:
an allow-all read profile switched to a --connected gate must narrow
Add docs/cli/model.md covering the three governance facets (access profiles, SHACL entity shapes, class hierarchy), their compile-to-data architecture, and the atomic replace semantics of re-running enable and define. Index it in docs/SUMMARY.md and the CLI command table.
The access command's target restriction compiles to f:onClass, but the flag was named --entity — which reads as subject-level targeting (f:onSubject) to RDF users, and clashed with `model class define --class` taking the SAME iri for the same concept. Rename: - --entity → --class (the governed class; what data) - --class-iri → --policy-class (the assignment unit grants/tokens carry; how a request selects its policy set — not a data restriction) `model entity define --entity` is unchanged: there the flag names the entity facet's own concept. Internal names and enable's output follow (Policy class: line), and the docs spell out the three class-shaped roles: --class restricts data, --property narrows columns, the policy class selects the policy set.
…lopes An update with neither an insert nor a delete clause can never change data, yet it previously parsed to an empty Txn, committed zero flakes, and returned HTTP 200 with a real commit envelope. A Cypher write posted with Content-Type: application/json fell through to this path and looked committed while writing nothing. - parse_update now rejects bodies without an insert or delete clause, mirroring the existing empty-insert/upsert guards. Explicit "delete": [] keeps its documented no-op semantics. - The server transact choke point detects a Cypher envelope sent with a JSON Content-Type and returns a 400 pointing at application/cypher.
explain_cypher existed in fluree-db-api (sharing the SPARQL/JSON-LD plan
machinery) but nothing routed to it. Now:
- explain_cypher accepts the $param map and substitutes before lowering,
so the reported plan matches what execution would run; Fluree gains an
explain_cypher method mirroring explain_sparql.
- The ledger-scoped /explain endpoint handles Content-Type:
application/cypher (raw text or {cypher, params} envelope), with the
same bearer-scope and Min-T handling as the Cypher query path. The
connection-scoped /explain returns 400 pointing at the ledger-scoped
route, matching the query endpoints.
- fluree query --cypher --explain works locally and remotely (new
RemoteLedgerClient::explain_cypher); the --at rejection now also
covers remote explain.
Lexes =~, parses it at string-predicate precedence (STARTS WITH /
CONTAINS level), and lowers to the engine's REGEX — which already exists
for SPARQL — wrapping the pattern as CONCAT("^(?:", pat, ")$") so
Cypher's whole-string match semantics hold, alternations stay grouped,
and runtime (non-literal) patterns anchor too. Wired through both the
read path and the write-path filter lowering. Inline flags like (?i)
pass through to the regex engine.
cypher_name_from_iri blindly stripped everything before the last # or /, so a relationship typed http://other.example/kb#order came back as "order" — unparseable as an IRI on the client, and inconsistent with db.labels()/db.relationshipTypes(), which are @vocab-aware and keep foreign IRIs whole. Naming now follows the db.labels() rule everywhere: strip the ledger context's @vocab prefix when it applies, otherwise return the full IRI. The vocab rides from Cypher lowering on Query::cypher_vocab into ExecutionContext for scalar evaluation, and the typed formatter derives it from the view's default context, so labels()/type()/keys() and rendered node/relationship names agree. Default-mode (namespace-0 bare names) output is unchanged.
Reproduces the KB Elasticsearch-reindex query shape — UNWIND over a URI list feeding a property-equality match plus one-hop expansion against an n10s-style ledger — with knobs for ledger construction (reindex / bulk import / novelty-only), a mixed string/langString value predicate, and reified (annotated) edges. Timings at the reported 72k-node/190k-edge, 21k-URI scale: - reindex- and import-built roots, plain edges: ~0.4s (fast) - mixed value datatypes: point lookup degrades 0.1ms -> ~7ms (the untyped-string bound-object broad scan; UNWIND lane unaffected) - annotated edges: the query wedges at 100% CPU (reproduces the field report; per-row edge-annotation probe under investigation)
A Cypher relationship variable used only for its value surface
(RETURN p, type(p), ...) lowers to OPTIONAL { EdgeAnnotation } so
reified parallel edges bind per annotation. On a ledger with reified
edges that OPTIONAL evaluated its expanded chain per row, and with no
stats for the system f:reifies* predicates the chain drove from
f:reifiesPredicate — per row it enumerated ~(sidecar / #relationship
types) candidate reifiers and existence-checked each with its own scan.
Measured: 13s for a 520-row UNWIND over 13k reified edges (~25ms/row),
matching a field report of 35s-with-LIMIT-20 / OOM at 21k rows.
- AnnotationValueOptionalBuilder answers the whole required batch with
three set-wise f:reifies* scans (planned scans: overlay-merged,
policy-filtered) plus hash lookups per row; falls back to the generic
per-row path for history/dataset/multi-ledger contexts. 520-row probe:
13.2s -> 0.04s.
- The batched OPTIONAL hash-join lane now admits single-source
DefaultGraphSource chains (the edge-annotation expansion wrapper).
- The forward-arena probe accepts a per-row (variable) predicate, so
untyped -[p]-> also qualifies where the arena exists.
- estimate_triple_row_count gets structural selectivities for bound-
object f:reifies* lookups (stats never cover system predicates).
- cypher_unwind_probe example grows annotated/mixed/import modes and
query variants that isolate the relationship-binding cost.
The batched annotation OPTIONAL probe re-drained the three f:reifies* predicates for every required batch — ~55 sidecar passes on a 54k-row result, ~35s at 190k reified edges. The builder lives for exactly one query run against one snapshot/overlay/to_t, so the maps are built once on first use and reused.
…lementers fluree query --cypher --explain --remote now POSTs application/cypher to the ledger-scoped /explain endpoint (verbatim body, params substituted server-side); drop the stale 'explain not supported for Cypher' claim.
The negotiated /import-upload handshake now accepts raw source data — the same formats fluree create --from ingests locally — and runs the chunked bulk-import pipeline on complete instead of a .flpack restore. Mint takes source_kind: "source" plus filename (its extension drives the pipeline's format sniffing; only the extension is trusted). CSV and Cypher uploads are converted to JSON-LD shards server-side via new shared helpers in fluree-db-api::import_source, which the CLI's local import now also uses (one conversion implementation for both surfaces). - Discovery advertises "source-upload" + import.source_formats; refused (and unadvertised) on Raft-replicated servers, where the pipeline's direct nameservice publishing would bypass the log. - fluree create <ledger> --remote <name> --from data.ttl|dump.cypher|… uploads single files through the same mint→PUT→complete→poll flow and errors with a .flpack fallback hint when the server lacks the mode. - Import job registry carries the source kind; complete branches restore vs pipeline in the same background-task slot. - The import commit paths held tracing EnteredSpan guards across the blob-store awaits, making their futures !Send (invisible until a tokio::spawn needed them); store steps now use .instrument.
A bulk-imported root carries annotation_index: None; queries binding relationship variables then take the scan-fallback path. The CLI's local import already follows up with one reindex so the api attachment provider's bulk-import bootstrap scan seals an authoritative arena — the server's source-upload complete step now does the same when the import reports has_annotations. Verified end-to-end: an uploaded Cypher dump with relationship CREATEs finishes with a sealed arena and the relationship queryable through the arena probe. The probe example gains knobs used while verifying this (PROBE_REINDEX2 / PROBE_LOAD_FIRST / PROBE_CACHE_CFG) and property-bearing annotations.
An annotated insert followed by indexing left annotation_index=None
permanently. Three cooperating defects:
1. The api attachment provider returned None ('delta unknown') when the
ledger wasn't resident in the LedgerManager — the norm for a
write-only ingest, where the background indexer fires before any
read loads the ledger. The build defensively dropped the arena and
stamped had_annotation_arena, blocking the bootstrap scan forever.
The provider now falls back to a *transient* backend load (never
inserted into the cache — cache insertion from a background context
disturbs the running handle's novelty bookkeeping) whose replay of
un-indexed commits yields the complete event history.
2. A full rebuild under Augment coverage discarded a previously-sealed
arena: root assembly had no way to reach the base arena. Fir6Inputs
now carries the previous root's CID, and the Augment arm merges the
prior arena's events with the delta (the incremental Phase 3d
contract) before resealing. Prior-arena blobs follow the normal
old-generation GC lifecycle.
3. Fluree::reindex read the NsRecord before quiescing the background
indexer; a racing background publish left it rebuilding from a stale
record (index_head_id gone), losing the just-sealed arena. The
record is re-fetched after cancel + wait_for_idle.
Pinned by a four-cell seal matrix in it_edge_annotations_indexed,
including the background-race reproduction at probe scale. The
deliberate post-defensive-drop scan-fallback contract is unchanged and
its existing test still passes.
…alStager The SPARQL ;-separated update loop (D-10: N ordered operations, each observing the previous through a virtual state, one atomic commit) moves from an inline loop in stage_transaction_from_txns into a reusable SequentialStager: stage-per-op with last-wins fact folding, simulated sequential graph-id assignment, namespace-delta union, and a finish() that layers the merged fold over the original base. The single-op fast path and the skip-last-apply optimization are unchanged; finish() now also tolerates a zero-op run (empty staged view) for callers whose operation count is data-dependent.
Three small mechanisms the sequential write driver threads rows through: - UnresolvedValue::PreBound(Binding): an already-resolved VALUES cell, passed through lower_values_cell verbatim. Never parser-produced — bindings extracted from a prior query result re-enter a Txn's WHERE losslessly (entities as Sids, typed literals as-is). - CypherLowerOpts::seeded_vars: columns the caller binds by appending a Values row block to the lowered Txn post-hoc. Lowering registers them bound (SET/REMOVE target checks pass, templates may reference them) and stages the statement as an update. InlineRows vars now register as bound too. - The UNWIND $batch last-wins row dedup now applies only to single-write-clause statements: it keys on the first node MERGE, which would drop legitimate rows of a multi-clause chain (two rows sharing src but differing in dst are distinct work). Multi-clause statements own duplicate-key semantics in the driver (first row per key creates, later rows match). Materializer's JoinKeyMode is re-exported for callers that materialize encoded bindings at extraction boundaries.
Lifts the 'MERGE must be the only write clause' restriction. A statement
may now chain write clauses — MERGE chains, CREATE+MERGE mixes,
interleaved SET/REMOVE — with each clause observing the writes before it
and rows piping downstream, publishing as ONE atomic commit:
UNWIND $rows AS row
MERGE (a:Person {id: row.src})
MERGE (b:Person {id: row.dst})
MERGE (a)-[:KNOWS]->(b)
The driver (cypher_seq.rs) executes over a SequentialStager: the read
prefix evaluates as a read query into a row table; each write clause
stages one or two seeded sub-Txns against the accumulated virtual state.
A MERGE clause partitions rows by probing (match probe over a seeded
VALUES join), dedups absent rows by their engine-evaluated identity-key
tuple (first row per key creates, running ON CREATE SET), re-binds every
row against the post-create state (Cypher's multi-match multiplication
included), and applies ON MATCH SET to the non-creating rows — exactly
sequential per-row semantics, vectorized. CREATE-bound variables thread
as reconstructed skolem Sids; computed values in merge keys and SET
positions ({name: p.team}, p.age + 1) hoist into engine-evaluated row
columns before lowering.
The previously-rejected node MERGE under a plain leading MATCH routes
here too — one create per distinct absent key, every row bound.
A trailing RETURN is answered from the final row table with the full
read expression surface (matched AND created bindings, modifiers), not
just created-entity reconstruction.
Wiring: WritePlan::Sequential dispatches on all three write paths —
autocommit (transact_cypher*), Bolt explicit transactions (staging
against the private state, so probes see earlier statements), and the
consensus committers via RefTransactBuilder::cypher_sequential (probes
run under the write lock, preserving the no-TOCTOU contract; the retry
loop re-resolves per attempt). The local committer carries the RETURN
envelope on TransactionReceipt::cypher_return; Raft supports sequential
writes but rejects a RETURN-carrying one at the route (its receipt path
has no channel for the rows). Existing single-clause plans — including
the conditional MergeSet/MergePerRowSet shapes — keep their exact paths;
detection fires only on statements that previously errored, and the
JSON-LD / SPARQL transaction paths are untouched.
This was referenced Jul 21, 2026
aaj3f
changed the title
aaj3f
marked this pull request as ready for review
July 21, 2026 20:27
This was referenced Jul 21, 2026
bplatz
requested changes
Jul 26, 2026
bplatz
left a comment
bplatz
left a comment
Contributor
There was a problem hiding this comment.
The code here is the strongest in the program — my only substantive ask is about where these fixes live, not what they do.
All three headline items verified correct:
- Cache-arm guard (R-CACHE-ARM).
has_delete_manifestsis the OR of both signals (summary_indicates_deletes(snapshot) || delete_manifests > 0) in both planners — I specifically went looking for a summary-only stamp, which would have let an override-cached delete-bearing selection come back with the flagfalse. It isn't there.guard_cached_scan_filesis called from both hit arms (r2rml.rs:1916in-memory,:1950disk) and unit-tested on both sides. /infomor-approximate-tables. Using the zero-I/O summary counters is the right mechanism — I'd flagged on #1520 that this path is structurally metadata-only and can't reach the manifest list, so a guard call wouldn't have worked. This does the reachable thing instead.- ASC top-k (F-AUD-6). The soundness gate is real:
r2rml.rs:2062filters on!tk.ascending || field.required, and a declined directive falls through to the parallel scan rather than proceeding. The worst-firstOrdKeyreorientation is correct for both directions.
The ask: two of the defects this PR fixes are live in #1507, which merges three PRs earlier. Detail inline.
Two worth a look items inline as well.
| // (last-wins), and even EQUAL group attrs would UNDER-COUNT the fan-out. So any | ||
| // duplicate parent key DECLINES the fused plan (caller returns `Ok(None)` → | ||
| // materialize), the conservative posture the whole operator takes when a shape | ||
| // is outside what it can fold soundly. Returns `false` on any duplicate (was |
Contributor
There was a problem hiding this comment.
Definite issue (sequencing, not code) — this fix is correct, and it's four PRs downstream of the bug it fixes.
This is exactly the defect I raised on #1507: the equal-value duplicate was a latent fan-out under-count, and declining on any duplicate parent key is the right resolution. The comment even documents the reasoning better than I did.
The problem is placement. Under the stated merge order (#1520 → #1507 → #1514 → #1528), main carries the under-count for the duration of three merges, and #1514 widens fused-aggregate admission on top of it (C5/E1/E2/W4-2), so the exposure grows before it shrinks. The same is true of the cache-arm guard in this PR, which fixes the MoR override-leak I raised on #1507's persisted scan-files cache.
Both are small, self-contained commits. Please cherry-pick them onto #1507 — the insert_dim_gkeys change is a one-line behavior flip plus the comment, and the cache-arm work is the delete flag + guard_cached_scan_files + the CACHE_FORMAT_VERSION bump. That way no intermediate state of main ships a known silent-wrong-answer, and this PR keeps only the work that's genuinely its own.
If cherry-picking is awkward against the branch topology, the alternative is to land #1528 ahead of #1514 — but forward-porting the two fixes is cleaner and lets #1507 be correct on its own merits.
Contributor
Author
There was a problem hiding this comment.
Done, with one compile-reality adjustment to the exact mechanics. (a) The insert_dim_gkeys decline is forward-ported to #1507 (0b4269a), byte-identical with this branch's version down to the in-body comment, so nothing diverges at merge. (b) The cache-arm work could not ride down as-is: #1507's rung predates the MoR guard entirely (no mor_guard.rs, no override — those arrive with #1520 off newer main), so the cherry-pick does not compile there. #1507 instead received a guard-independent equivalent (0247c79): same PersistedScanFiles schema and version bump byte-identical to this branch's, manifest-list-only detection mirroring its COUNT path, and hit arms that decline flagged entries outright. That closes the override-persistence leak standalone at #1507 — no intermediate main ships it in any order — and this branch's override-aware guard_cached_scan_files remains the final form, superseding the conservative arm at merge. The convergence merge on this branch (8effa91) reconciles the shared regions so that supersession is an auto-resolve rather than a maintainer conflict.
| /// [`crate::scan::ScanPlan::has_delete_manifests`] flag so cached scan-file | ||
| /// selections carry the signal downstream. | ||
| pub fn summary_indicates_deletes(snapshot: &Snapshot) -> bool { | ||
| snapshot.total_delete_files().unwrap_or(0) > 0 |
Contributor
There was a problem hiding this comment.
Worth a look — this fails open on a malformed counter, and /info is now the one consumer with no backstop.
total_delete_files() and friends are .parse().ok(), so a present-but-unparseable counter becomes None → unwrap_or(0) → > 0 is false → "no deletes." On the planners that's fine, because the manifest-list count is OR'd in right after. But fetch_virtual_table_row_counts calls this on its own, and that path is metadata-only by design — there is no manifest list to fall back to.
So the failure mode is a table that silently doesn't appear in mor-approximate-tables and has its over-counted row total presented as exact. That's a quieter version of the same class this whole thread is about: the operator gets a number with no signal that it's an upper bound.
Cheapest fix is at the accessors — treat present-but-unparseable as "deletes present" rather than absent, so the summary arm fails closed on its own and every consumer inherits it. Absent stays 0, which is correct.
Contributor
Author
There was a problem hiding this comment.
Confirmed and fixed at the source PR as suggested: #1520's guard now fails closed on present-but-unparseable (and negative) counters at the guard level (83c13d9 there), and this branch's copy is reconciled to the same semantics in 8effa91 — with the placement nuance that the flip lives in the shared predicate rather than the raw Option<i64> accessors, so /info inherits "deletes-present" for a malformed counter and lists the table as approximate instead of silently presenting an over-count as exact.
| /// aggregate builds, retained lookups) must never be released, or the guard would | ||
| /// under-count live memory. No-op on a disabled handle. | ||
| #[inline] | ||
| pub fn release(&self, bytes: usize) { |
Contributor
There was a problem hiding this comment.
Worth a look — the mechanism is right, but it doesn't reach the case that most needs it.
Saturating so it can't underflow, and the caller invariant is stated clearly. It currently has one caller (r2rml/operator.rs:1507, scan windows), which fixes streaming-window accumulation.
What it deliberately excludes — persistent join/aggregate buffers — is where I think the real over-count is. I've raised this on #1514: GraphOperator builds a fresh inner subplan per parent batch (graph.rs:357, :536), and batched-OPTIONAL rebuilds similarly, so a hash join inside one re-records its whole build on every rebuild while the previous is already dropped. The counter grows linearly in driving batches against a flat true peak.
A rebuilt build table isn't really a "persistent buffer" — its drop point is exactly as provable as a scan window's, it's just at inner-subplan teardown rather than batch handoff. If the rebuild sites released on teardown, this mechanism would cover it without weakening the invariant for genuinely retained buffers. Worth deciding whether that's in scope here or a follow-up, but I don't think it should stay unaddressed — it's a false-abort path on native queries with the guard default-on.
Contributor
Author
There was a problem hiding this comment.
Agreed that it shouldn't stay unaddressed, and it hasn't: fixed at #1514 (5b33104), which is where the counter itself lives — rebuild-boundary releases (snapshot before inner open, release the delta after drain) at both GraphOperator sites and the batched-OPTIONAL driver, using this branch's release primitive ported byte-identically so the two rungs converge. Your framing that a rebuilt build table's drop point is exactly as provable as a scan window's is the argument the site comments now make. Regression test included; the false-abort repro it pins is recorded in the PR #1514 thread.
…lated boundary (D1) + label the budget estimate (W1) D1 (false 507 on rebuilt operators): the query memory counter was add-only, so a correlated GraphOperator / batched-OPTIONAL that rebuilds its inner subplan once per parent row/batch summed each rebuild's build-table charge (hash-join / GROUP BY / fused-dim) into the shared counter while true peak memory stayed at one build. With N driving rows the counter reported ~Nx the true peak and false-aborted a legitimate correlated query with a typed MemoryBudgetExceeded (507) on a small (Lambda-sized) ceiling. - Forward-port the release primitive verbatim from cfd773d: QueryCancellation::release (saturating fetch_update decrement) + its invariant doc, plus the ExecutionContext::release wrapper. Byte-identical to that rung so a later merge-up auto-resolves. (Its scan-window caller in r2rml/operator.rs is NOT ported — that belongs to the other rung.) - Release at the rebuild BOUNDARY, never inside a live build: snapshot mem_used() before each correlated inner opens and release exactly the delta after it drains and closes, at both GraphOperator sites (execute_in_graph per-row, execute_in_graph_batched per-batch) and the batched-OPTIONAL rebuild driver (OptionalOperator::build_batch). The whole inner (build tables included) is provably dropped there, so the delta is precisely its transient charge; execution on one handle is sequential so nothing races the delta. An early ?-exit skips the release (over-counts, the safe direction) on a path that aborts the query anyway. - Regression test hash_join::tests::d1_rebuild_boundary_release_no_false_abort: two independent builds on one handle under a budget admitting one build's peak but not the two-build sum complete only with the boundary release (a negative-control assertion proves the un-released sum would exceed the budget). W1 (estimate honesty): MemoryBudgetExceeded.used_bytes is rows x declared-schema cols x a flat per-binding estimate, not measured bytes. Make that explicit at both surfaces the reviewer reads: append "(estimated)" to the Display and document the field (name kept to avoid public-API churn); rename the GROUP BY telemetry span field to mem_used_bytes_est and declare it so the record is no longer a silent no-op. No span allowlist/registry references the key, so nothing else needs updating.
An equal-value duplicate parent join key was kept as a single GroupKeyResolver map entry, so the fused fold emitted one row where the generic pipeline emits the fan-out of a materialized RefObjectMap join — halving COUNT(*)/SUM when the duplicates agree. Collapse insert_dim_gkeys to decline on ANY duplicate (function body + comment ported verbatim from the #1528 rung so the eventual merges auto-resolve identical), and invert+rename the pinning test to dim_dup_join_key_always_declines. Also correct the outer doc comment's stale "harmless and kept" claim, which is wrong at both rungs.
…ty + stable disk-cache hasher (B2 Option A, W3) B2 (Option A, guard-independent): PersistedScanFiles/CachedScanFiles gain has_delete_manifests and CACHE_FORMAT_VERSION bumps 2->3 (struct + version + conversions + tests ported verbatim from the #1528 rung). The scan-files miss arm stamps the flag from a manifest-list-only delete probe (send_snapshot_has_delete_manifests, mirroring the COUNT(*) path's detection without re-reading data manifests); both cache-hit arms route the fetched entry through servable_scan_files, treating a delete-bearing entry as a MISS (re-plan). No merge-on-read guard exists at this rung, so this is a plain fall-through, never an override re-check — the planner is left untouched to avoid colliding with #1520 when it lands on main first. The version bump alone invalidates every v2 scanfiles entry. W3: replace std DefaultHasher (unstable across toolchains) with a fixed xxh64 stable_key_hash (seed 0) + CACHE_SCOPE filename segment + a golden pin test, so a toolchain bump no longer silently orphans the on-disk cache. Ported verbatim from the #1528 rung; disk_catalog_cache.rs is byte-identical to it.
…(W1) The can_project_distinct_before_sort branch computes k/can_topk identically to its sibling but deliberately offers no scan-side set_topk: DISTINCT sits below the sort, so k scan rows can dedup to fewer than k and the push-down would under-produce. Add a rationale comment mirroring the ASC-declines note so the asymmetry reads as a decision, not an oversight.
…LING note (W2) Extend the COUPLING note to name GraphOperator, which is neither pure-forward nor pure-absorb in the set_row_budget taxonomy: it absorbs w.r.t. self.child but threads the budget into the per-parent-batch correlated inner subplan. Soundness holds because each batch rebuilds the inner tree with a fresh budget whose rows flow to the same LIMIT, but that is the subtler argument a future editor is most likely to miss.
Merge the PR base up to pick up B1 (fused_aggregate declines any duplicate dim join key, 0b4269a — the fix D2 depends on) plus B2+W3 disk-cache delete-manifest safety and f17's absorbed main-merge (DEC-004 consolidation, 7e949f1). Almost everything auto-merged, including the D1 cancellation.rs release primitive (byte-identical port, zero-conflict) and the D1/W1 context.rs / group_aggregate.rs / optional.rs hunks (disjoint regions). Two files conflicted, both in the iceberg disk-cache layer, and both provably reconciled: - fluree-db-api/src/graph_source/disk_catalog_cache.rs: taken from f17 wholesale. Provably lossless: f17's version is byte-identical to family-c dab3b99 (which inherited this branch's own #1500 §4 "stable key hash" rework and layered B2 on top), and diffing this branch's version against dab3b99 shows only the B2 additions (has_delete_manifests, CACHE_FORMAT_VERSION 2->3, F-AUD-1 guard tests) with none of the §4 content removed. Same origin, one direction of supersession. - fluree-db-api/src/graph_source/r2rml.rs: UNION of the test modules — this branch's §1 (#1500) inline-loadTable-metadata tests kept first, f17's servable_scan_files / delete-manifest (F-AUD-1) tests appended. f17's servable_scan_files production hunks auto-merged outside the conflict; verified they gate BOTH scan-file cache-hit arms (in-memory + disk), a delete-bearing entry falling through to a guarded re-plan, and that CACHE_FORMAT_VERSION appears once and equals 3.
…widened-shape regression) The #1514 review (D2) flagged that this PR widens fused-aggregate admission into the JOIN fold (W4-2 mixed fact+dim GROUP BY keys, dim-side E2), which reaches insert_dim_gkeys — whose equal-value-duplicate under-count B1 (0b4269a, merged from the f17 base) has now closed by declining ANY duplicate parent join key. B1's dim_dup_join_key_always_declines guards the primitive. This adds the missing other half: join_fold_declines_on_non_pk_dim_join_key drives resolve_join_at_open (reached only by the widened join shapes) with a real R2RML mapping + terminal-dim table provider, and asserts the CALLER propagates the decline to Ok(None) — falling back to the sound materialize path — when the dim has a non-PK (repeating) join key. A unique-key control run FUSES (Ok(Some)), so a broken fixture fails the control rather than passing vacuously; SF01's unique PKs mean the corpus can never raise this. The test doc records the boundary: E1, C5, and fact-only E2 take the single-scan fold (resolve_at_open, scan_table n=1) and never build a FK->GKey map, so the widened-shape concern does not reach them (route_group_key_sources_mixed_and_declines covers routing). Also refresh the two resolve_join_at_open caller comments that still said "CONFLICTING duplicate": post-B1 the decline fires on ANY duplicate dim join key.
…convergence) Resolve the scan-files cache-hit arms family-c-side: family-c's guard-aware `guard_cached_scan_files` supersedes lambda's `servable_scan_files` decline — same end-state semantics (flagged + no override refuses with a typed error; flagged + override serves with a caveat). r2rml.rs converges byte-identical to family-c's scan-files region; the orphaned `servable_scan_files` helper, its miss-arm probe, and its unit test are dropped — family-c's `cached_scan_files_guard_refuses_delete_bearing_entry` covers the same case with a stronger (typed-refusal) assertion. B1 `insert_dim_gkeys` (doc + body) and the refreshed "ANY duplicate" caller comments auto-resolve to the corrected f17 wording. disk_catalog_cache.rs and cancellation.rs are byte-identical across both heads; lambda's #1500 §1 loadTable tests carry over verbatim. hash_join.rs UNION resolution: lambda's D1 boundary-release test (d1_rebuild_boundary_release_no_false_abort) is adapted to family-c's membudget era — its per-build charge is derived from BINDING_EST_BYTES (size_of::<Binding>() = 88 here) instead of the removed flat-64 assumption, so its two assertions (charge retained until released; no false-abort after the boundary release) are preserved against family-c's constant.
…formed-counter test (#1520 W1/W2) Absorb #1520's W1/W2 fail-closed guard into family-c's re-applied mor_guard.rs. `summary_delete_evidence` reads the raw summary map: a present counter that is non-zero, negative, or unparseable refuses (naming key+value); an absent one proceeds (manifest-list backstop). `ensure_no_summary_deletes` and the reconciled `summary_indicates_deletes` route through it — one fail-closed definition serving the guard, the planner's ScanPlan.has_delete_manifests stamp, the /info row-count path, and override-read caveats. W1 tests (garbage / -1 / absent / classifier) and the W2 `error_message_is_actionable` retarget (planner location + preview "qualified (location)" shapes) ported. /info side: extract `metadata_indicates_mor_approximate_count` (the exact predicate `fetch_virtual_table_row_counts` uses) and pin the 1528-W1 outcome — a malformed or negative delete counter lists the table in mor-approximate-tables (row count reported as an upper bound), never counted as exact; a clean append snapshot is not flagged.
Contributor
Author
|
The placement ask is executed (with the B2 compile-reality adjustment detailed inline — the guard doesn't exist at #1507's rung, so it got a guard-independent equivalent instead of the literal cherry-pick), both worth-a-look items are fixed at their source rungs (#1520 fail-closed counters, #1514 rebuild-boundary releases), and this branch's convergence merge (8effa91) reconciles every shared region — dup-key decline, cache schema, hasher, guard copy, release sites — so the stack's remaining merges auto-resolve. Gates re-run green at the new head. |
bplatz
approved these changes
Jul 28, 2026
bplatz
left a comment
bplatz
left a comment
Contributor
There was a problem hiding this comment.
All three items resolved: the two fixes now live at #1507/#1514 with this branch containing them via the convergence merges (supersession auto-resolves — verified insert_dim_gkeys and summary_delete_evidence hash-identical across rungs), and the fail-closed counter semantics reach every consumer — the fail-open accessors have zero remaining production call sites, and the /info test pins the malformed-counter scenario. mor_guard, /info, iceberg (250+), and query (1323) suites all pass locally.
One non-blocking nit: the new smoke_minio example is missing a [[example]] required-features = ["aws"] stanza in fluree-db-iceberg/Cargo.toml (its two siblings have one), so cargo test -p fluree-db-iceberg at default features fails to compile the example target — and CI can't see it since it runs --all-features. Fine to fold into a later commit.
This was referenced Jul 29, 2026
aaj3f
added a commit
that referenced
this pull request
Jul 30, 2026
Re-ran p1/p2 under clean load (load1 7.7-9.0) on substrate B: DuckDB + fluree-main + fluree-#1528, both cache modes, N=5. Replaces the load-contaminated baseline rows. Cold: DuckDB wins p2 ~1.8x, near-tie p1; Warm: Fluree wins p1 (count-manifest shortcut) and p2 ~2x. main==#1528 on p1/p2 (no regression).
aaj3f
added a commit
that referenced
this pull request
Jul 30, 2026
…e arm64 linux) Full uncontested table (host-identity stamped: c7g.4xlarge/AL2023, duckdb 1.5.5 d8cdaa33fd sha 9882c99a, fluree 5598ffd + c81862d), predictions-vs-outcomes vs R1 §3.7, and named-gaps. Findings: #1528 fusion collapses cq038 (74s->99ms, ~750x) and cq016 (30s->1.0s, ~30x) beyond its scoped p3 case; crt_join_reorder (multi-FACT join) is the largest still-open gap (~130s, ~900x, unaffected by #1528); crt_highcard 14x + 8x RSS; fused/relational pairs 3-6x. Local contended legs demoted to labeled secondary. Correctness gate green all 10.
aaj3f
added a commit
that referenced
this pull request
Jul 31, 2026
Full pair set at 10x scale, same uncontested host. Scale-delta: the fused/scan-heavy family's 3-6x tail widened to 6-52x (fluree per-row materialization scales linearly vs DuckDB sub-linear); the multi-FACT join (crt_join_reorder) went from 917x-completes to DNF (>200s) on both binaries (DuckDB 321ms) — a hard wall at scale; #1528 fusion becomes essential (cq038 ~100ms vs main 72s; cq016 completes ~22s vs main DNF); fluree RSS widened (crt_highcard 4.4GB vs DuckDB 512MB). 153 rows accounted for (scoped legs to bound instance-hours; n/a vs DNF distinguished).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
perf(virtual): big-Iceberg-audit implementation — correctness/capacity guards, coverage widenings, harness re-bless, browse-parity, family-C (consolidates the audit program + #1525)
This is one of five surviving pull requests in the consolidated virtual-dataset review program, which collapses the #1475–#1529 lineage into five reviewable units at the maintainers' request. #1528 carries the big-Iceberg-audit implementation: the memory-budget guard, the Tier-2 coverage widenings, the harness re-bless and switch registry, the browse-parity package, and the family-C filter-join at the tip — together with the retraction-resurrection merge-up and the merge-on-read cache-arm guard that live on its integration base. It is the deepest and widest survivor, and it carries the one native-observable edit in the whole program that no switch can revert; that edit is foregrounded in the native-impact appendix below. Reading order across the program: #1520 (standalone MoR correctness fix) → #1507 (performance program) → #1514 (serverless Lambda-usability + BYO-IAM) → #1528 (this PR) → #1529 (native-twin materialize builder). The outward-facing audit record is on the
docs/virtual-dataset-audit-2026-07branch — start with 00-MASTER-AUDIT.md §6 (the recommendations ladder) and the implementation reviews in pr-reviews-impl.md.Original PR body
Scope expanded by consolidation; the tip PR's own change (the family-C filter-join) is the narrative below. The four audit PRs plus the retraction-resurrection merge-up and the MoR cache-arm guard folded in are rolled up under "Consolidated scope."
Forest map
Two deployed production queries on
enterprise-byo-test-10DNF (>180s) on the virtual (R2RML/Iceberg) dataset. P4 (parity/P4-famc-probe.md) traced BOTH to one line — the join-path blanket filter decline infused_aggregate.rs::resolve_join_at_open— and DEC-003 §1/§7 authorized the fix (AJ, 2026-07-21). This PR lifts that guard and ports the single-tableFilterPlanmachinery to the join path. All the join fold, F1 combine, and E2/W4-2 machinery already exist beneath (baseperf/browse-parity); this is the finite widening P4 scoped.The two production shapes (adapted to the SF0.1 bench star, verbatim from P4):
Mechanism
route_filter_sourceroutes the FILTER to the single chain pattern that owns EVERY referenced variable:Resolved.filter, applied per fact row innext_batch. This is the exact machinery the single-table path already uses, extracted into the sharedbuild_filter_planso a filtered join is byte-parity with the single-table fold. (The fact row-validity already null-drops the filter's member columns before the filter runs.)row_passes_filter_plan), so a fact row probing a filtered-out dim key drops — equivalent to applying the residual post-join because the dim attrs are functionally determined by the dim PK.Q2's COUNT + two AVG folds need no new aggregate machinery —
resolve_agg_foldsalready folds several from the fact scan; only the filter port is new.Semantics — the D-c5 bar (a wrong count is the one unacceptable outcome)
The fused filter matches the MATERIALIZED
FilterOperatorby construction: the materialized operator (filter.rs),passes_filters(eval.rs), and the fusedFilterPlanall evaluate the SAMEPreparedBoolExpressionthrougheval_to_bool_non_strict, which demotes a can-demote expression error tofalse(row excluded).A NULL/absent column value in a comparison is an unbound BGP member ⇒ the row is EXCLUDED (e.g.
?status != "Closed"with NULL status does NOT count as "not Closed";?onHand < ?reorderwith a NULL on either side is excluded). On the fact side this is enforced by the R2RML star row-validity (validity_cols), which fires before the filter — fact filter vars are always fact object vars, so a null member drops the row first (BGP parity, identical to single-table). On the dim siderow_passes_filter_planenforces the same null-drop inline (a null column short-circuits tofalsebefore the boolean eval), because the dim scan does not otherwise null-check a non-group filter column — this also keeps!BOUND/COALESCE-style filters sound. Semantics-divergence check (task item 2): none. The single-table plan already matches the materialized operator (same primitive), so it is reused verbatim; nothing to report.Tested:
family_c_row_passes_filter_plan_null_excludes_and_comparesexercises!=-with-NULL and var-to-var with a NULL on each side with null-bearing hermetic fixtures (SF01 has no null STATUS/onHand rows, so the corpus members cover the non-null path and the unit tests cover the null path).Admission / decline enumeration (each a soundness line, all in code)
ADMIT: a filter whose vars all resolve to scalar columns on ONE pattern that is the FACT or the TERMINAL dim.
DECLINE (
Ok(None)→ materialize): a var-free/constant filter; a var bound as an object on ≥2 patterns (cross-source value-equality) or a filter spanning ≥2 patterns (fact AND dim, or two dims); an INTERIOR-dim owner (v1, symmetric with the interior-dim group-key decline); aRefObjectMapFK / template / constant / multi-valued object var (build_filter_plan→object_map_for_var); EXISTS/NOT-EXISTS/subquery filters and any non-bare-FILTER shape (detection only captures a singlePattern::Filter(expr)).Switch
Rides
FLUREE_FUSED_R2RML_AGG_JOINas a widening (program switch policy: widenings ride their family switch; OFF reverts). With the join sub-switch OFF,resolve_join_at_openis never reached, so a filtered join reverts to the materialize path.Gates (all local; CI does not fire on a non-main base) — record in
docs/audit-impl/family-c-gates.mdfused_aggregate.rs,corpus.rs); the pre-existinghash_join.rsdrift untouched.-p fluree-db-query --all-targets --no-deps: 0 new lints (only the 5 pre-existing doc-list warnings atfused_aggregate.rs:112-116).cargo test -p fluree-db-query: 1319 lib (+4 family-C tests) + all integration bins, 0 failed.cargo test -p fluree-bench-virtual --bins: 36 passed (corpus validates at 87 members; smoke covers all tags).cargo check -p fluree-db-api --no-default-features --features aws,iceberg,shacl→ PASS.69006c4ffd39…(3 rows), q087 hashd7aba410bb32…(10 rows) — byte-identical to P4's measured native.LIVE (virtual-sf01, single-flight, 1 rep, PAT in-memory) —
compare --gate: 4 records, 0 hash mismatches, 0 perf violations!=FILTERBoth new members collapse from seconds to sub-second AND hash-match the native materialized oracle byte-for-byte (q087's AVG decimals included) — the fused fold fired and is exactly correct. The two regression sentinels are unchanged.
Residuals (still decline → materialize)
Interior-dim filters (v1); filters spanning fact AND dim (or two dims); filters over a
RefObjectMapFK / non-scalar object; EXISTS/subquery filters. Each is a sound decline (falls back to the materialize path), not a wrong answer.Consolidated scope
The four audit PRs below stack in order onto an integration base branch that begins with a merge of
main(bringing the retraction-resurrection fix) and then re-applies the merge-on-read cache-arm guard; the family-C filter-join is the tip. Each superseded PR closes as consolidated-at-maintainer-request, its pre-consolidation tip preserved underarchive/pre-refactor-2026-07-21/<branch>. The merge-up and cache-arm guard never had their own PR and are anchored by the tagsarchive/pre-refactor-2026-07-21/perf_audit-tier012andarchive/pre-refactor-2026-07-21/audit_mor-cache-arm.mainmerge-up (#1508 fix)main, bringing the F-AUD-2 fix for retraction-resurrection (a hard/CoW delete resurrected by a re-index in an un-indexed window). Folded in via the base branch, not a separate PR.plan_scan, with a delete-flag on the cache entry and aCACHE_FORMAT_VERSION2→3 bump so pre-guard entries miss and recompute. This completes the #1520 guard on the stacked path. Folded in via the base branch, not a separate PR.releasedecrement primitive, a per-query budget divisor, and scan-window plus fact-as-parent memory accounting, so a wide crawl or point-lookup that previously OOMed now aborts typed./infonative+virtual member routing) — each an admission gap that previously fell to full per-row materialization.{"@id": …}, typed-json honored, booleans as real bools), constant-IRI select-map routing onto the bound-subject machinery, a loud refusal for a dropped VALUES clause, a typederr:r2rml/UnsupportedPatternrefusal envelope, property-var crawl budget forwarding, an unmapped-class short-circuit, the cfg-orphan build fix, and the F1 constrained-COUNT fusion.Verification of record
CI does not fire on a non-main base, so the verification of record is the final live gate plus the six independent adversarial reviews.
The whole-stack live gate at the final integrated head reported 77 records, 0 hash mismatches, 0 perf violations (
vbench compare --run <merged> --gate, from the re-bless in #1523) — correctness passes end-to-end. The native re-bless went 54→77 entries with all 74 oracles reproduced exactly (no native regression), and the harness bless guard now refuses to pin a DNF or an abort as a budget. The stale-baseline analysis and the run JSONLs are in C2-bench-wave1.md and data/.The six adversarial diff reviews are recorded in pr-reviews-impl.md: R-1521 (memory-budget guard) verdict SHIP with every residual bounded, documented, and in the over-count-safe direction; R-CACHE-ARM (the perf-line cache-bypass fix) verdict CLOSED — the guard is now complete on the stacked path as it already was on
main; R-1522 (the eight widenings) verdict SHIP with no blocking finding, every mechanism admission-only and the in-engine FILTER/SortOperator authoritative; R-1523 (the harness/trust layer) verdict SHIP — the gate can no longer pin a timeout or abort as a budget; R-PARITY (#1525 browse-parity) verdict SHIP with the native-untouched rule holding for every cluster; and R-1528 (family-C) verdict SHIP — null semantics sound on all four attack sub-points, no admitted-and-wrong shape. Family-C is live-verified: q086 and q087 collapse from seconds to sub-second and hash-match the native materialized oracle byte-for-byte.The correctness findings these PRs implement are verified independently: the merge-on-read gap in V1-mor-verification.md, the memory-budget blind spot in V2-membudget-verification.md, and the browse-parity serialization and shape chains in parity/P1-serializer-verify.md, parity/P3-shape-matrix-empirical.md, and the two production-DNF shapes in parity/P4-famc-probe.md. The coverage gaps each widening closes were reproduced live in C2b-bench-wave2-probes.md.
Native-impact appendix
Foregrounded first: the one edit in the whole program that a switch cannot revert.
BINDING_EST_BYTES64 →size_of::<Binding>()= 88 incontext.rsis shared/native and NOT switch-revertible. After this change, native fold and join memory accounting counts 88 bytes per binding regardless ofFLUREE_SCAN_MEM_ACCOUNTING. It is defended as conservative — over-count, fail-late-not-wrong — but it is not proven byte-neutral: a native query under memory pressure aborts marginally earlier than before. It is verified only by the compile-time canarybinding_est_bytes_is_at_least_binding_stack_size(the constant is>= size_of::<Binding>()), with no native corpus leg. This is the single native-observable edit in the whole program that no switch can revert, and reviewers should treat it as such.releasedecrement primitive is added tofluree-db-core/src/cancellation.rs(git-confirmed 58 insertions; inert for native — only the R2RML window-release pairing calls it). A budget-share divisor lands inexecute/runner.rs(FLUREE_QUERY_BUDGET_SHARE_DIV, default 1 = full budget per query = today's behavior, gated ondiv>1 && memory_limit().is_none()so it is a no-op for native). Scan-window and parent-build accounting land in the R2RMLoperator.rs(gatedFLUREE_SCAN_MEM_ACCOUNTING, default on, R2RML-scan-only; off is byte-identical except that the unconditional pull-loopcheck_cancelledis upgraded tocheckpoint, whose outcome is byte-identical).set_row_budgetforwarding — a third generic-operator budget extension. It forwards the LIMIT budget on the genericOptionalOperatorto the required/outer side only. SwitchFLUREE_R2RML_BUDGET_OPTIONAL(default on); off swallows the budget and is byte-identical. MINUS, FILTER, SORT, DISTINCT, HAVING, and subquery explicitly decline (documented, observable spans) — MINUS declines because its anti-join output can under-produce under a finite primary budget./infomember-routing is a genuinely native-reachable path (gated).merge_virtual_into_nativeinledger_info.rssits on the native-reachable/infopath; it unions classes and properties by IRI (a graph-source member replaces a native collision, never sums; it keeps the top-level nativestats.flakes/size). SwitchFLUREE_R2RML_INFO_MEMBER_ROUTING(default on); off restores the pre-mechanism behavior byte-identically. Minor caveat: an IRI-form mismatch would double-LIST (not double-count). The MoR rider on this path flags delete-bearing tables into anmor-approximate-tableslist (upper-bound counts) — this is where fix(iceberg): fail closed on merge-on-read delete files (silent stale-row risk) #1520's deferred/infoover-count is closed.format_node_object_bindinghas four callers, all on the graph-source crawl path, so no native or hydration path reaches it. But the shared JSON-LD default-format literal arm (format/mod.rs) is touched:xsd:booleanstrings now coerce to real bools (the D3 fix). That boolean rendering is the sole shared/default-path behavioral delta; every non-boolean literal is untouched (coerce_bool_literalreturnsNone), and it is regression-tested (crawl_boolean_literal_renders_as_bool).build()'s#[cfg(native)]gating and the correct feature-gating onwith_secret_resolverthat the feat(iceberg): BYO-IAM on operator-owned S3 — inline metadata, fail-closed credentials, SecretRef rotation fix, typed storage errors, verify probe (#1500, #1497, #1498) #1505 SecretRef insertion had dropped — build-time feature-gating only, no runtime path, restoring no-native builds. It adds a new permanent gate,cargo check -p fluree-db-api --no-default-features --features aws,iceberg,shacl, which CI should adopt.Pattern::R2rmlarm — native producesPattern::Triple, so it is never reached, and it is gated. Every fold widening (perf(r2rml): coverage widenings — scan-side pruning/budget/IO + fold-side aggregate admission (audit Tier-2) #1522 MIN/MAX and filtered-COUNT, fix(r2rml): browse-parity — native-shape crawl output, bound-root select-map routing, typed refusals, constrained-COUNT fusion #1525 F1, and this PR's family-C filter-join) lives infused_aggregate.rs, which returnsNonethe instant any non-R2rml pattern is present — it cannot fire for native. The review confirmed this PR's own diff is onlyfused_aggregate.rs(+625), reusing the sharedeval_to_bool_non_strict/can_demote_in_expressionwithout modifying them. test(bench): re-bless at the audit head + resilience hermetics + SWITCHES.md regeneration (audit Tier-1/hygiene) #1523 is bench tooling and docs; its native re-bless reproduced all 74 oracles exactly.Confirmed negatives across the span (git-checked, name-only diff clean): no
arrow/datafusion/parquetdependency bump, and no.github/workflowsorrust-toolchainedits.Landing and hand-off notes
Land #1520 first. #1520 and this line both carry the fail-closed merge-on-read guard; the added and removed lines are byte-identical (only the git patch-id differs, because the two were cut against different base trees). Once #1520 merges to
main, this PR's guard portion resolves as an identical-content merge and the cache-arm guard (CACHE_FORMAT_VERSION2→3) rides on top cleanly. A reviewer diffing this PR will see themainmerge-up as a large block on the integration base; it is the retraction-resurrection fix, folded in via the base branch and anchored by the archive tags named in the consolidated-scope roll-up, not lost work.