feat(materialize): fluree materialize — native-twin builder from virtual R2RML/Iceberg datasets - #1529
feat(materialize): fluree materialize — native-twin builder from virtual R2RML/Iceberg datasets#1529aaj3f wants to merge 24 commits into
Conversation
…hunk A) The engine-free core of `fluree materialize`: enumerate every (s,p,o) triple for a compiled R2RML mapping directly off row batches, with no query plan, operator, or fuel tracker. It reuses the exact per-term materialize helpers the query operator uses (materialize_subject/predicate/object_from_batch, get_join_key_from_batch), so term and datatype fidelity match the engine by construction — the only new logic is the whole-graph orchestration. Foreign-key edges are resolved lookup-free and dims-first: parent triples maps are processed before their children (topological order), and a parent key->subject index built during the parent scan resolves each child FK by hash lookup. A dangling or null FK produces no triple (R2RML inner-join semantics). Cyclic and self-referential parents are fully pre-indexed. Emits into a TripleObserver (an N-Triples collector for the parity diff; an ingest-sink adapter lands in chunk B) and collects MaterializeStats incl. per-(child, predicate) FK-edge counts for the parity gate. 9 hermetic golden tests cover multi-class type emission, null-subject and null-object drop, typed/lang/constant/template objects, FK inner-join, dangling and null FK drop, composite keys, self-referential FK pre-indexing, and dims-first ordering determinism.
… chunk B) materialize_graph(provider, gs, observer) streams each logical table through the whole-graph enumerator over the R2rmlProvider/R2rmlTableProvider scan path — the provider-backed counterpart to the in-memory reference driver, with identical dims-first ordering and the same per-batch index/emit calls. The ordering + pre-index logic is now a shared materialize::plan() so both drivers stay in lock step, and ParentIndexSet::needed_parent_columns feeds scan projection. Engine-equivalence checkpoint: materializer_driver_matches_engine_wildcard runs the driver over the same airline mock the engine `?s ?p ?o` wildcard uses and asserts the two decoded triple sets are identical (12 triples) — the deterministic hermetic analog of the live native-sf01 full-triple diff, proving the new enumerator agrees with the proven query engine over an identical scan. Plus materializer_driver_resolves_fk_across_tables: dims-first FK resolution and dangling-drop over streamed per-table scans. Full it_graph_source_r2rml suite green (55 passed).
ImportSinkObserver bridges the whole-graph enumerator to the native bulk ingestion pipeline: each materialized (s,p,o) is interned and encoded through the SAME ImportSink term_iri/term_blank/term_literal + emit_triple the Turtle and JSON-LD parsers use, so term/datatype/language fidelity is preserved by construction (a "9.99"^^xsd:decimal interns to a decimal flake, a "hola"@es to a lang-tagged string). The lang-vs-explicit-datatype-vs-implicit-xsd:string decision — the one spot literal fidelity could silently narrow — is factored into literal_sink_args and unit-tested for all three cases.
…1, chunk B) The materialization side of the end-to-end builder: build_virtual_chunk encodes a buffer of materialized triples into a ParsedChunk by mirroring parse_chunk exactly (WorkerCache + ImportSink::new_cached + SpoolContext + finish), replaying interned triples through ImportSinkObserver instead of parsing Turtle. Zero transact edits — every primitive is public and ParsedChunk fields are pub. drive_virtual_import streams a virtual R2RML source through the enumerator (dims-first, same pre-index pass) into a byte-budgeted ChunkingObserver and hands each completed ParsedChunk to an emit callback in contiguous idx order (t=idx+1). Chunk size is derived from BYTES (the memory budget), never a fixed row count — the machine-safety chunk-size knob. R2rmlBuildProvider is the combined provider trait for the builder, with a compile-time Send+Sync assertion (the ?Send/Send trait-duality guard). Byte-budget chunk cutting is unit-tested.
…03 D1, chunk B COMPLETE) Wires the materializer into the bulk import pipeline as a new ImportSource::Virtual branch that REUSES the battle-tested downstream unchanged. A single materializer thread scans the R2RML source (async, driven via Handle::block_on off a dedicated thread), enumerates triples dims-first, and encodes byte-budgeted ParsedChunks that feed the SAME commit_parsed_chunks_in_order consumer + spool-sort + build_all_indexes + publish_index the text paths use. Provider rides on ImportConfig::virtual_source (Arc<dyn R2rmlBuildProvider>, Send+Sync asserted); the arm is inserted before the remote-parallel arm so every existing Local/Remote/streaming/Files arm body is byte-identical (diff = 138 insertions, 1 deletion — only is_channel_fed gains `|| is_virtual`). Chunk byte-size derives from the memory budget (machine-safety). Entry: CreateBuilder::import_r2rml, doc'd to set explicit --parallelism/--memory-budget-mb. End-to-end test (multi-thread, tight bounds threads=2 + memory_budget_mb=256): builds a real indexed+published twin from a dim+fact+FK mock with lang and xsd:decimal literals, then queries it back — class counts, FK-edge resolution to the parent IRI, xsd:decimal value+datatype round-trip (not narrowed to string), and lang-tag round-trip all confirmed. All 83 bundled import tests + the 56-test R2RML suite green.
…DEC-003 D1) The virtual materializer drives its async scan via Handle::block_on off a dedicated thread, which deadlocks on a single-threaded tokio runtime. Detect the runtime flavor at the arm entry and return a typed ImportError::UnsupportedRuntime instead of hanging; production servers and the CLI run multi-thread. A code comment + this error document the remote-style two-stage producer as the runtime-flavor-agnostic future alternative. Guard test on a current-thread runtime confirms the loud failure.
…DEC-003 D1, chunk C) Adds the per-table build-watermark capture seam: R2rmlProvider::build_watermark returns HashMap<table, TableWatermark> with a DEFAULT of empty, so every existing provider keeps compiling and a real Iceberg-backed provider overrides it to report its catalog session's pins. TableWatermark carries the authoritative metadata_location pin plus best-effort snapshot_id/sequence_number (typed Option, for the Deliverable-3 snapshot diff). The bulk builder will fail loud when this is empty for a non-empty table set, so a provider that forgets to override cannot publish an unstamped twin.
…og session (DEC-003 D1, chunk C2)
Record each table's pinned Iceberg snapshot (metadata_location + snapshot_id +
sequence_number) into a new IcebergCatalogSession.snapshots map at the single
scan_table_inner caller of load_table_context, first-writer-wins. Expose it as
pinned_tables(gs) -> {table -> TableWatermark} and override
FlureeR2rmlProvider::build_watermark to surface it. Harmless for ordinary
queries; the driver reads it only during a materialize build.
…003 D1, chunk C3) Add a metadata-only pin-all pre-pass to drive_virtual_import so every table's snapshot is pinned before emission (cross-table skew bounded to seconds; emit scans reuse the pins, no duplicate loadTable GETs). Assemble the twin's WatermarkStamp — per-table watermark vector + SHA-256 mapping hash + builder version — with a fail-loud guard that errors on an empty watermark for a non-empty table set (§17(a)/(b)). The stamp is logged here; chunk D persists it into the final commit. Mocks override build_watermark with fakes.
…ommit (DEC-003 D1, chunk D) Add ParsedChunk.txn_meta (empty for every text-import chunk → byte-identical commits) and consume it in finalize_parsed_chunk. The materialize driver ns-encodes the WatermarkStamp (builder version + mapping hash + watermark vector as sorted JSON) into ONE chunk's txn_meta via the sink's new intern_meta_predicate (so the stamp predicates' ns codes publish in that commit's namespace_delta). A one-chunk lookahead makes the stamp ride the LAST non-empty commit — completion-marker semantics: a valid twin is one whose head commit carries a parseable stamp. TableWatermark gains serde derives; read_stamp parses it back. Round-trip test: build -> archive_ledger -> restore_ledger -> stamp survives byte-identical and parses.
…e + full diff (DEC-003 D1, chunk E) verify_twin compares the built twin against its virtual source using the Chunk-A enumerator as the oracle and a native wildcard query on the twin, both rendered to identical N-Triples: class counts (always-on), a stratified per-subject sample (Quick, always-on, full node compare of the first N subjects/class), and a full-triple diff (Full). Per-property counts are skipped-with-note (the manifest-sound fast path is a real-Iceberg optimization). Numeric object literals are value-canonicalized on both sides so a value-preserving typed- literal normalization (stored xsd:decimal re-rendering '5.00' as '5') is not misreported — the same treatment the LIVE full-triple diff will need. Gate failure => report FAILED, twin not announced (publish-then-verify-then-drop, the sound ordering; the CLI drops on failure). Hermetic tests: faithful twin passes Quick+Full; a corrupted twin (injected instance) fails on the count + full diff.
…3 D1, chunks F+G) New `fluree materialize <graph-source>` command: build a native twin from a virtual R2RML source, verify it, and write it. Flags: --into (default -twin), --output pack|ledger|s3 (pack default; s3 rejected up front as not-yet-wired in the file-backed CLI), --output-path, --verify quick|full, --max-performance, --home, and (G) --allow-mor-deletes. §14 co-resident posture: conservative fixed defaults (parallelism=2, memory_budget_mb=512) unless the user sets an explicit value or --max-performance (own-the-box auto). Publish-then-verify-then-drop: a failed gate drops the twin so nothing unverified stays announced. One-shot CLI leaks the Fluree for a 'static real provider (single build-scoped session). G: the MoR scan-path guard's refusal is surfaced with the --allow-mor-deletes override (sets FLUREE_ICEBERG_ALLOW_MOR_DELETES). Command self-gates on iceberg (default on); a not(iceberg) build gets a clear error. 5 helper unit tests.
…dget honesty, drop-branch fix (DEC-003 D1) Four review-response items (all nonblocking, one commit): 1. Verify-default honesty: print the Quick gate's shared-oracle epistemics at run time (catches ingest/index corruption, NOT enumerator logic — run --verify full + the native diff before cutover). Make the stratified sample seeded-deterministic (WatermarkStamp.sample_seed, recorded as a sampleSeed Long, derived from the mapping hash) and spread across the class via a seed-mixed hash instead of the lexicographic first-3 — tail-subject bugs are now sampleable, reproducible, and coverage widens as the seed rotates. 2. Quick negative test: a per-subject property corruption (counts unchanged) fails the Quick sample arm. 3. Budget-floor honesty in --help + the gate doc: --memory-budget-mb is advisory sub-2GB (chunk floors at 128MB), peak RAM ~= 2GB + parallelism*128MB*2.5, and the FK parent index sits OUTSIDE the budget (~ largest parent table). 4. Drop-on-fail test (dropped_twin_is_unpublished) — it surfaced a real CLI bug: drop_ledger rejects a :branch suffix (HTTP 400), so the gate-fail drop would have errored and left the unverified twin announced. FIXED: strip the branch (ledger_name_no_branch) and surface a loud manual-drop warning if the drop still errors instead of swallowing it.
…i-table sources (DEC-003 D1) AJ-driven registration-gap fix: the local CLI's Direct mode was single-table, so a multi-table R2RML source over a catalog-less bucket copy (a Snowflake-managed Iceberg database whose table dirs carry random suffixes, e.g. fact_order.UIHGsQex/) could not register. Now, when a Direct table_location is a WAREHOUSE ROOT (auto-detected: its leaf dir does not name the requested table), each rr:tableName resolves to its own dir under the root via ONE session-cached S3 LIST, matching <name>.<suffix>/ or bare <name>/ case-insensitively (namespace-stripped). Ambiguity or a miss fails loud, naming the candidates. - New SendIcebergStorage::list_dir (S3 ListObjectsV2 + delimiter; default-errors for other backends; LazyStorage forces + delegates). - Pure match_warehouse_table_dir in fluree-db-iceberg (5 hermetic tests: suffixed, bare, case, ambiguity, miss). - FlureeR2rmlProvider::resolve_direct_table_location + IcebergCatalogSession warehouse-listing cache (one LIST per build). - Single-table direct behavior byte-unchanged (self-diff: only the info! moved; api it_graph_source_r2rml 62 green, iceberg lib 251 green). - No new flag / no config-schema change (table_identifier already tolerates a root; Direct registration does no probe). Rule documented in iceberg map --help.
aaj3f
changed the title
… feat/materialize-builder # Conflicts: # fluree-db-transact/src/import.rs
Per-phase tracing spans on the materialize produce side: pin_all_tables, per-table table_scan_emit (scan+render+emit wall + triple delta), and an encode_total accumulator across build_virtual_chunk calls. These are the spans the A2 A/B phase attribution was measured with; they ship with the fix slate so the same numbers are reproducible on any build. The two verify-phase spans in this commit are superseded by the O2 verify redesign that follows in this slate.
…zation
verify_twin double-buffered the ENTIRE graph twice in RAM even in Quick mode
(full source re-enumeration into a BTreeSet + full twin wildcard into another;
~26.8GB footprint at 6.1M triples, certain OOM at 35M). Redesign both modes to
be memory-bounded:
- Quick (default): peak RSS = O(sampled subjects). Class counts come from a
single streamed source pass (SampleAndCountObserver — counts per-class type
triples, retains full triple sets ONLY for the sampled subjects, drops the
rest as it streams) plus bounded twin COUNT queries; the per-subject sample
pulls a seeded window of subjects on each side via bound-subject queries.
- Full (--verify full): spools BOTH sides to on-disk N-Triples (source streamed
through the enumerator, twin paged through bounded wildcard queries), external
merge-sorts each under a bounded working set, then streams a k-way diff. Peak
RSS = O(one sort run). Replaces the whole-graph double-buffer.
- Decimal false-reject fix (A2 receipt): canonicalize_numeric_nt is now value-
canonical for double/float — parse to f64 and re-render the shortest round-trip
form so XSD E-notation (Fluree storage, '-1.5521762E1') and the enumerator's
plain f64::to_string() ('-15.521762') compare equal. decimal/integer keep exact
lexical canonicalization (no lossy float round-trip). Regression tests included.
Drop-on-fail already checks the drop result and warns if the drop errored (CLI).
The two verify-phase spans from the instrumentation commit are superseded here.
…-parallelism) The produce side pinned snapshots and pre-indexed FK-parent tables strictly serially. A2 measured ~30s of serial loadTable round-trips (pin_all_tables) and ~217s of strictly-serial cold-S3 fact-table fetch (Pass-1 pre-index) at 35M. - pin_all_tables: the 8-16 serial loadTable GETs now run concurrently via buffer_unordered, bounded by the --parallelism knob. - Pass-1 pre-index: pre-index tables now scan concurrently (bounded by --parallelism), each into its OWN partial ParentIndexSet, merged back once all scans complete (ParentIndexSet::split_empty + merge_from). Parent IRIs are disjoint across pre-index tables, so the merge is a union. Peak extra memory = parallelism x (one batch + one table's partial index); the partials sum to the final index, resident regardless of ordering. drive_virtual_import takes a parallelism argument (the existing knob, previously only sizing the result channel); the virtual arm passes the effective parse threads. Machine-safety: all new concurrency is capped by the knob (default 2, co-resident). O5(c) - eliminating the fact double-read (Pass-1 index scan + Pass-2 emit scan of the same table) - is DEFERRED: it is naturally subsumed by O1's produce-side pool (topological parent-first emit that indexes during the emit scan), and doing it standalone is more invasive than warranted here. Recorded as a follow-up.
…x accounting
Two bounds-correctness holes the audit flagged (O6):
(a) effective_chunk_size_mb underflowed for sub-2GB budgets: (budget-2048)
saturated to 0 and the clamp then floored the chunk at 128MB REGARDLESS of
budget — so a 512MB-budget import used the same 128MB chunk as a 2GB one, and
a 128MB chunk under a 512MB budget blows the budget once the ~2.5x parse/
inflight expansion is counted. Below 2GB the chunk now scales as
budget*0.6 / working-set, clamped to [16,128] MB (continuous with the
large-budget floor at the 2GB boundary). The >=2GB path is unchanged.
(b) the FK ParentIndexSet is fully resident for the whole build and sat OUTSIDE
any budget. It is now charged against a fraction (~50%) of the import memory
budget and the build FAILS LOUD (MaterializeError::ParentIndexBudgetExceeded,
mirroring the QueryError::MemoryBudgetExceeded pattern) if it would overflow —
checked after Pass-1 and after each lazily-indexed dimension. Estimate via
ParentIndexSet::estimated_bytes (key + term string bytes + per-entry
overhead). drive_virtual_import takes a memory_budget_bytes argument; the
virtual arm passes the effective import budget (0 disables the guard).
The materialize --help BUDGET note is rewritten to describe the new model (it
claimed the budget was advisory below 2GB and the FK index was outside it — both
now fixed). Tests: sub-2GB chunk scaling, large-budget unchanged, budget-fraction
math, empty/disabled guard, estimated_bytes nonzero on a populated index.
…t path Byte-identical allocation reductions on the per-row produce path (verified by the exact-triple-set enumerator tests and the template tests — 121 r2rml tests green): - expand_template_from_batch: replace the per-placeholder `result.replace(...)` loop — each call a full re-scan + re-alloc of the whole growing string, O(placeholders x len) per row, and prone to double-expanding a value containing another placeholder's text — with a single-pass build from the original template's segments into one reserved String. One pass, one allocation, exact R2RML semantics. - emit_batch: hoist per-batch what was rebuilt per row: the rdf:type object terms (was RdfTerm::iri(class.clone()) cloning every class IRI on every row) are built once; each POM's constant predicate is borrowed via Cow (the common case) so the hot data-triple path skips the per-row predicate clone; and this TriplesMap's triple total folds into stats.per_tm ONCE per batch (one tm.iri clone per batch, not one per emitted triple). SCOPE: this is the low-risk, self-contained half of O3/O4. The OwnedTriple double- materialization drop (O3a — intern directly from the enumerator into the chunk sink, cutting on true encoded size) and the per-TriplesMap compiled-template cache are DEFERRED to the O1 produce-parallelism PR: they restructure the same emit->chunk pipeline O1 rearchitects, so landing them together (validated by the 35M re-measure) avoids conflicting rework. Recorded as a follow-up.
…rallelism The entire produce side ran on ONE virtual-materializer thread; --parallelism only sized the result channel, so the create-from path's measured 3.9x (1->2 threads) scaling was fully foregone (A2). This activates the knob as a real produce pool. Design (mirrors the proven text-import N-worker + reordering-consumer pattern): - Pass-1 now pre-indexes EVERY FK parent (not just cyclic/self-referential), so the ParentIndexSet is COMPLETE and READ-ONLY before Pass-2 — a child may emit on any worker in any order, so its parent must already be indexed. Acyclic dim parents that were indexed lazily during their own emit scan are therefore scanned twice (index + emit); the double-read on (usually small) dimensions is the cost O1 accepts (eliminating it is the deferred O5(c)). The pre-index scans stay concurrent (O5), each into a partial index merged back after. - Pass-2: `parallelism` sync worker threads render + intern + encode batches, each with its OWN private chunker + encoding sink. This driver scans each table (async) and dispatches its batches over a bounded channel; a blocking send on the dedicated producer thread is natural backpressure. The r2rml scan futures are !Send, but workers do only SYNC render/encode (the async scans stay on the driver), so no Send bound is needed — sidestepping the non-Send trap. - Chunk idx comes from a shared atomic; commit_parsed_chunks_in_order reorders by idx, so out-of-order arrivals are fine. The completion stamp rides ONE dedicated final chunk (highest idx -> committed last -> the twin's head), produced after the workers join — the already-supported empty-stamp-commit shape. Peak produce memory = parallelism x (one chunk buffer + one encoding sink) — the same N x chunk model the text path lives under; bound = --parallelism (default 2, co-resident), no new knob. drive_virtual_import now takes a Send+Clone SyncSender instead of an FnMut sink so the pool can fan chunks in; the virtual arm keeps a sender clone to surface a build error after the driver drops its copy. Correctness is validated hermetically (no 35M re-measure needed — that stays the separate perf phase): parallel_produce_matches_serial_and_stamps_once drives a fixture mapping at p=1 and p=2 and asserts identical MaterializeStats, identical total op_count (no triple lost/duplicated), unique + contiguous chunk indices, and EXACTLY ONE stamped chunk at the highest idx. SpoolConfig gains #[derive(Clone)] (all-Arc fields) so each worker gets its own handle; fluree-db-tabular becomes a direct api dep to name ColumnBatch on the work channel. DEFERRED (recorded here, in this PR — not a second PR): O3a (drop the OwnedTriple intermediate; intern directly from the enumerator into the chunk sink) does not ride naturally on this worker restructure — it rewrites the same per-worker emit->chunk path — so it lands as a follow-up commit on this branch.
…ect to sink The produce path materialized every triple TWICE: the enumerator emitted it, the ChunkingObserver cloned it into an owned (RdfTerm, String, RdfTerm) buffer, then build_virtual_chunk re-interned the whole buffer into a fresh ImportSink — three touches per term (alloc, clone, hash+copy). This removes the intermediate. Each produce worker now interns each batch's triples DIRECTLY into an open per-chunk ImportSink as the enumerator emits them (new InterningObserver), so a term is allocated once by the enumerator and interned once into the sink dictionary — the OwnedTriple buffer and the second intern pass are gone. The worker cuts a chunk on the running byte estimate at BATCH boundaries: it secures a batch, claims a chunk idx (only once a batch is in hand, so no idx is wasted on an empty chunk — a gap would stall the in-order consumer), fills the sink batch by batch until the estimate crosses the budget, then finalizes + ships. Chunk sizing stays byte-budgeted (triple_weight estimate, same threshold as before); the one change is granularity — a chunk may overshoot by at most one batch, which the memory model already tolerates (peak = parallelism × chunk, unchanged). ImportSink borrows its allocator (&mut), so an open sink cannot be stored across recv() calls without a self-reference; the per-chunk sink therefore lives in an inner scope (created fresh per chunk, mirroring build_virtual_chunk's finalize), which sidesteps the lifetime problem cleanly. build_virtual_chunk is retained for the one dedicated stamp chunk (empty triples + completion stamp). ChunkingObserver and its now-obsolete unit test are removed (VecDeque import with them). Re-ran the O1 differential after this rewrite of the per-worker emit→chunk path: parallel_produce_matches_serial_and_stamps_once PASSES — p=1 and p=2 still produce identical MaterializeStats, identical total op_count (no triple lost/duplicated), unique + contiguous chunk indices, and exactly one stamped final chunk at the highest idx. NOTE: "cut on true encoded size" is approximated by the triple_weight byte estimate — ImportSink exposes no mid-stream encoded-size accessor, and adding one is out of scope; the estimate matches the prior chunker's semantics.
…n receipt)
The 35M acceptance run false-rejected a PROVEN-identical twin (independent-oracle
diff: 35,238,778 triples both sides, zero diff both directions) on FACT_WEB_EVENT
— the same class of verify-canonicalization gap as the decimal fix, on the one
datatype it hadn't covered. EVENT_TS is the schema's only xsd:dateTime: storage/
export pads whole-second timestamps to 6-digit microseconds
("2026-06-28T00:16:42.000000Z") while the enumerator oracle's format_timestamp
(r2rml term.rs) DROPS the fraction when it is zero ("2026-06-28T00:16:42Z"). The
numeric canonicalizer never touched dateTime, so the sample compare reported
TripleDiff{missing:1,extra:1} per sampled event.
Fix (mirrors the decimal trailing-zero strip, applied to BOTH sides):
canonicalize_numeric_nt is generalized to canonicalize_value_nt, which now also
normalizes xsd:dateTime / dateTimeStamp / date / time — drop insignificant
trailing fractional-second zeros (and a bare "."), and normalize a zero UTC offset
(+00:00 / -00:00) or Z to canonical Z. So "…:42.000000Z" and "…:42Z" agree, as do
"…:42.500000Z" and "…:42.5Z" and "…Z" vs "…+00:00"; distinct instants and non-zero
offsets stay distinct. Lexical (dependency-free) rather than parse-to-instant — it
directly matches term.rs's formatter shapes; a non-zero offset is left intact (it
denotes a different instant without date arithmetic, and this schema is all-UTC —
a documented residual). Regression tests mirror the decimal ones: padded-vs-
unpadded, Z-vs-+00:00, and distinct-instant / non-zero-offset non-collapse.
OPEN QUESTION for the maintainer (NOT implemented here, per instruction): the
re-measure worktree carries a throwaway `--verify none` CLI flag. Whether a
first-class "skip verify" mode deserves productizing — for operators who run their
own independent-oracle diff and want to skip the in-process gate — is a product
call, flagged not decided.
|
Flipped to ready: the draft-status reason (the unrun independent live gate) is discharged — see the 2026-07-29 addendum at the top of the body for the fix slate, the full-scale acceptance numbers, and the zero-diff oracle verdict at 35,238,778 triples. Reading order still applies: this reviews on top of #1554 (the landing PR for the audit program content). |
…parent double-read The O1 parallel-produce rewrite pre-indexed EVERY foreign-key parent in a separate Pass-1 and then re-scanned each table to emit, so every big parent fact (e.g. FACT_ORDER, parent of FACT_ORDER_LINE) was scanned twice over cold S3 — the dominant cold-build lever the B-REMEASURE O5 verdict named. Replace the global "pre-index every parent" pass with topological WAVE scheduling: tables are layered so a non-preindex FK parent emits in a strictly earlier wave than its children, and its key->subject index is built DURING its own single emit scan (index-during-emit). Each wave freezes the cumulative parent index behind an Arc for a fresh O1 worker pool; after the wave drains, its wave-local index folds into the cumulative for the next wave. A non-preindex parent is therefore scanned exactly once for the build. Only cyclic / self-referential parents keep the pre-index fallback (they cannot be indexed in a single forward pass). r2rml: MaterializationPlan gains `waves` (dependency_waves) + an enumerate_by_waves reference enumerator with tests (star, two-level chain, cyclic fallback, triple-set parity vs enumerate_from_batches). api: drive_virtual_import rewritten to the per-wave loop; spawn_produce_workers helper extracts the O1 worker body; the hermetic p=1-vs-p=2 differential is extended with a two-level chain (a parent that is also a child) and a mutual-cycle fixture. Peak produce memory is unchanged (parallelism x chunk + the resident parent index, one live wave); no new knob. Chunk idx stays contiguous across waves via the shared atomic, with exactly one stamped final chunk. verify_twin's source-oracle re-scan is kept (considered-and-declined removal): the Quick sample subjects are chosen post-publish from the twin, so the pre-publish build cannot retain them, and the re-scan's independent re-execution is the gate's value.
bplatz
left a comment
bplatz
left a comment
There was a problem hiding this comment.
Approving. Please walk the inline notes below and decide which belong in this PR versus a follow-up — a few of them (1, 4) I think are worth folding in here; the rest are your call.
Verified locally
cargo clippy --all --all-features --all-targets -- -D warnings— clean.cargo nextest run -p fluree-db-api -p fluree-db-r2rml -p fluree-db-cli -p fluree-db-transact -p fluree-db-iceberg --all-features --no-fail-fast— 4373 passed, 0 failed.
The one finding that is not inline
No docs. This adds a new top-level fluree materialize command with no docs/cli/materialize.md and no docs/SUMMARY.md entry; every one of the other ~43 CLI commands has both. docs/graph-sources/r2rml.md and docs/graph-sources/iceberg.md are the natural home for the twin / watermark / completion-stamp concept and for the new warehouse-root --table-location resolution rule, which the body says is documented only in iceberg map --help. Per the repo's docs mandate this belongs in the same changeset.
Process note
The base audit-program/land-on-main is 45 commits and ~10k lines ahead of main, so this diff is ~6.4k of the ~16.5k that actually reaches main through it. Worth stating explicitly for whoever sequences the landing.
What is strong
The wave-scheduled index-during-emit design is the right shape, and claiming a chunk idx only AFTER securing the first batch — so an empty chunk can never leave a gap that stalls the in-order consumer forever — is exactly the kind of detail that is easy to get wrong and was gotten right. Publish-then-verify-then-drop is sound: I confirmed create ... import rejects a ledger that already has commits, so the hard drop can only ever hit a ledger this build just created, and the :branch strip closes the one way that guard could have been defeated. The verify-default honesty message printed to the operator is the right instinct. And apart from the chunk-sizing note, the shared-path footprint really is as small as the appendix claims.
| // buffers + index build and divide the rest across the working set; | ||
| // floor at 16MB (still worth a commit), cap at 128MB (continuous with | ||
| // the large-budget floor at the 2GB boundary). | ||
| let working = budget_mb as f64 * 0.6 / denominator; |
There was a problem hiding this comment.
Shared-path behavior change that the native-impact appendix does not list.
The appendix says the only shared-path change is ParsedChunk.txn_meta, "self-diff byte-identical for every text import." This rewrite of effective_chunk_size_mb changes chunk sizing for EVERY import whose budget is under 2048 MB — Turtle/TriG/JSON-LD included, not just materialize.
Concretely, at memory_budget_mb = 512 with max_inflight = 2: before, 512.saturating_sub(2048) = 0 → clamp(128, 768) = 128 MB. After: 512 * 0.6 / 6 = 51 MB. That is ~2.5x the commit count and a different ledger shape for the same input.
The default budget is 80% of RAM, so most hosts land above 2048 and are unaffected. What IS affected: containers under roughly 2.5 GB, and anyone who passes --memory-budget-mb explicitly — which the PR's own machine-safety posture actively encourages, and which every finalizer-touching test in this repo already does (memory_budget_mb = 256).
The fix itself reads correct to me — the old floor genuinely ignored the budget below 2 GB and the comment explains exactly why. The ask is just that the appendix name it as a second shared-path change with its blast radius, rather than the byte-identical claim standing unqualified.
| let mut offset = 0usize; | ||
| loop { | ||
| let q = format!( | ||
| "SELECT ?s ?p ?o WHERE {{ ?s ?p ?o }} ORDER BY ?s ?p ?o LIMIT {PAGE} OFFSET {offset}" |
There was a problem hiding this comment.
OFFSET paging over the whole twin is O(n^2), and moves the memory cost onto the query side.
At 35M triples this is ~176 pages, and each page asks the engine for a full ORDER BY ?s ?p ?o over the entire twin and then skips M rows. Each page also materializes 200k bindings as serde_json::Value via to_sparql_json.
So the boundedness O2 won is real on the verify side (external sort, SORT_RUN_LINES runs) but the twin-read side is neither bounded nor linear. The addendum's 35.2M full-triple diff reads like an external same-renderer N-Triples export rather than a run of verify_twin_full, so I do not think this specific path has been exercised at that scale — worth confirming.
An index-ordered scan or the existing export path would give a single linear pass; spool_twin_ntriples is most of the way to re-implementing one already.
| let mapping = provider.compiled_mapping(graph_source_id, None).await?; | ||
| let classes = mapping_classes(&mapping); | ||
|
|
||
| let dir = std::env::temp_dir().join(format!( |
There was a problem hiding this comment.
std::env::temp_dir() may be tmpfs, which would undo the bounded-memory rewrite.
Full mode writes two complete N-Triples renderings of the graph plus their sorted runs here — tens of GB at the scales this builder targets. On many Linux hosts /tmp is tmpfs and therefore RAM-backed, so the verify that was just rewritten to be memory-bounded would put it all back in memory, on exactly the large datasets where it matters.
There is also no override. Suggest defaulting to the import's spool dir (or somewhere under .fluree) and adding a --tmp-dir. TmpDirGuard cleans up on drop but not on SIGKILL, which is a second reason not to leave GBs in a shared /tmp.
| /// under `materialize:watermark`. The whole stamp must fit the 64 KiB txn_meta | ||
| /// cap; a schema with a very large table count would need a compacter encoding | ||
| /// (documented residual). | ||
| fn encode_stamp( |
There was a problem hiding this comment.
The 64 KiB cap this doc comment asserts is not enforced on this path.
MAX_TXN_META_BYTES is checked in exactly two places — fluree-db-transact/src/parse/txn_meta.rs:407 and parse/trig_meta.rs:1398 — both of which validate txn-meta arriving from user TriG/JSON-LD. encode_stamp builds TxnMetaEntrys directly and hands them to finalize_parsed_chunk, so neither validator runs.
The watermark JSON grows with table count times metadata_location length; S3 metadata URIs plus JSON framing run roughly 350 bytes per table, so ~190 tables. That is not an unreasonable warehouse.
Every other guard on this path is fail-loud with a typed error (EmptyWatermark, ParentIndexBudgetExceeded). This one should be too, rather than a comment describing a cap and a downstream failure mode that is not stated.
| /// `Some` here (all three stamp fields present). Keys on the stable predicate | ||
| /// LOCAL NAMES (the namespace code is per-ledger); the local names are | ||
| /// materialize-specific, so requiring all three together identifies the stamp. | ||
| pub fn read_stamp(txn_meta: &[TxnMetaEntry]) -> Option<WatermarkStamp> { |
There was a problem hiding this comment.
This ignores the namespace that intern_meta_predicate was added to publish.
The reason GraphSink::intern_meta_predicate exists (and the reason encode_stamp routes the stamp predicates through the chunk's own sink) is so their namespace codes land in this commit's namespace_delta and resolve when the commit is read back. Then read_stamp matches on bare local names, so any commit carrying builderVersion + mappingHash + watermark string entries in ANY namespace parses as a valid twin stamp.
Given the stated contract is "a twin is valid iff a head-walk finds the stamp," it seems worth resolving the ns code and requiring it to be MATERIALIZE_NS. The information is already there by construction — the mechanism was built and then not used on the read side.
| /// sink, so the stamp predicates' namespace codes publish in this commit's | ||
| /// namespace_delta. Every other chunk passes `None`, leaving `txn_meta` empty | ||
| /// (byte-identical to a text-import chunk). | ||
| pub fn build_virtual_chunk( |
There was a problem hiding this comment.
Vestigial after O3a — triples is now always empty.
Since the O3a commit removed the OwnedTriple intermediate, the only remaining call is build_virtual_chunk(&[], ctx, t, idx, Some(&stamp)) at line 1006. So the triples parameter is dead on every path, ImportSinkObserver exists solely to serve it (its only use is the loop at 335-337 that never iterates), and the OwnedTriple alias exists only to name the dead parameter.
All three are pub, so no dead-code warning fires and nothing will surface this later — precisely the rot the repo's dead-code policy is aimed at. Suggest build_stamp_chunk(ctx, t, idx, stamp) and deleting ImportSinkObserver + OwnedTriple, or keeping them with the #[expect(dead_code)]-style why/when comment the policy asks for.
|
|
||
| /// Subjects per class fully compared by the stratified sample (deterministic: | ||
| /// the lexicographically first N subjects of each class). | ||
| const SAMPLE_SUBJECTS_PER_CLASS: usize = 3; |
There was a problem hiding this comment.
Worth surfacing in --help: the default gate's sample strength is three subjects per class. Combined with the class-count check that is a reasonable smoke test, but "quick verify" reads stronger than it is, and the CLI already prints an honesty note about the shared oracle — the sample size belongs in the same breath.
Separately, seeded_offset can put the window near the end of a class, so twin_sample_subject_iris issues ORDER BY ?s LIMIT 3 OFFSET N with N close to the class size — a full sort and skip per class. Fine at the sizes tested; the same shape as the full-mode paging note above.
| entry | ||
| .entry(cols.clone()) | ||
| .or_default() | ||
| .insert(key, subject.clone()); |
There was a problem hiding this comment.
Last-wins on a duplicate parent join key. This MATCHES the generic query operator (R2rmlScanOperator's ParentLookup is also a plain HashMap<Vec<String>, RdfTerm>), so the twin agrees with the virtual path and the parity gate is right to pass — no divergence introduced here.
The reason it is still worth a note: the fused aggregate deliberately DECLINES this shape (insert_dim_gkeys returns false on any duplicate, with a comment explaining that a single-entry-per-key map cannot represent the fan-out), and materializing bakes the last-wins result into a permanent native ledger where fixing the operator no longer fixes the data. If the intent is that the twin is the durable artifact, a loud decline here — matching the fused path's posture — may be worth more than silent agreement with the deviation.
| /// The env var read by fluree-db-iceberg's fail-closed merge-on-read guard. A | ||
| /// stable public contract (`fluree_db_iceberg::mor_guard::ALLOW_MOR_DELETES_ENV`); | ||
| /// hard-coded here so the CLI need not take a direct dependency on that crate. | ||
| const ALLOW_MOR_DELETES_ENV: &str = "FLUREE_ICEBERG_ALLOW_MOR_DELETES"; |
There was a problem hiding this comment.
The comment calls this a stable public contract, but a hard-copied literal with no compile-time link is the drift hazard the repo handles elsewhere with // SYNC: comments at both ends (the OTEL init pair in telemetry.rs / main.rs, and the Cypher "5.4.0" version string). A // SYNC: here and on mor_guard::ALLOW_MOR_DELETES_ENV would at least make a rename visible to whoever does it.
Note also that this file is #![cfg(feature = "iceberg")], and under that feature fluree-db-iceberg is already in the tree via fluree-db-api — so importing the constant may be cheaper than the comment suggests.
2 participants
feat(materialize): fluree materialize — native-twin builder from virtual R2RML/Iceberg datasets
This is one of five surviving pull requests in the consolidated virtual-dataset review program, which collapses the #1475–#1529 lineage into five reviewable units at the maintainers' request. #1529 implements
fluree materialize: a bulk builder that turns a virtual R2RML-over-Iceberg graph source into a native Fluree ledger ("twin"), verifies it against the source, and writes it as a pack or a local ledger. It is the last member of the reading order. Since the consolidated body below was written, the previously-open live gate has been RUN AND PASSED (an independent full-triple diff against a natively-imported ledger at full 35.2M scale came back identical — see the acceptance section directly below), and a measured performance fix slate has landed on the branch; the sections below the acceptance report are the original consolidated body, kept intact for review continuity. Reading order across the program: #1520 (standalone MoR correctness fix) → #1507 (performance program) → #1514 (serverless Lambda-usability + BYO-IAM) → #1528 (big-Iceberg-audit implementation) → #1529 (this PR). The outward-facing audit record is on thedocs/virtual-dataset-audit-2026-07branch — the fact base for materializing Iceberg into native ledgers is in C3-materialization-facts.md and the strategy framing (with the snapshot-keyed correctness lemma) is in D1-strategy-options.md.2026-07-29 addendum — performance fix slate + full-scale acceptance (commits
74780fd25…476d08192)A stage-by-stage performance audit of this builder against the repo's own bulk-import path (
fluree create --from) found that bounded-by-design had been conflated with slow-by-implementation, and a measured fix slate landed here as eight commits on this branch (per the maintainers' one-PR consolidation rule). The audit's baseline, at equal bounds (--parallelism 2 --memory-budget-mb 1024, the 35,238,778-triple sf01 dataset over a Snowflake-managed REST catalog): build+publish 348.7s, and a verify phase that double-buffered the whole graph in RAM even in quick mode — 16.5GB RSS at 6.1M-triple dims scale, projected 95–155GB at 35M, i.e. certain OOM on the build host.The slate, in commit order on this branch: a merge-up of the consolidated program base (
74780fd25); produce-phase instrumentation spans (5e429a9fd); O2 — memory-bounded verify in both modes plus decimal value-canonicalization in the parity compare (9649c3e4d); O5 — concurrent table pin + Pass-1 FK pre-index bounded by--parallelism(2caec9194); O6 — budget honesty: sub-2GB budgets now scale the chunk size instead of silently flooring at 128MB, and the FK parent index is charged against the budget with a typedParentIndexBudgetExceededinstead of living outside it (03787651a); O4 — per-row allocation cleanup in the enumerator hot path, byte-identical across the r2rml exact-output suite (0e6cf38bb); O1 — a parallel produce side:--parallelismsync worker threads render/intern/encode with private sinks while the non-SendIceberg scans stay on the async driver, with an in-commit differential test asserting p=1 and p=2 produce identical triple sets, contiguous chunk indices, and exactly one stamped final chunk (e77bebc15); O3a — theOwnedTripleintermediate buffer removed, terms interned directly from the enumerator into the chunk sink (06e00004f); and a dateTime value-canonicalization fix in the verify compare (476d08192, receipt below). A ninth commit closes the recorded O5(c) residual (fa96388d3): tables are scheduled into topological waves so a non-cyclic FK parent is scanned exactly ONCE — its key→subject index is built during its own emit scan and frozen per-wave for the parallel workers — eliminating the separate pre-index pass that made cold fact tables the dominant wall; only cyclic/self-referential parents keep the pre-index fallback, and the quick gate's independent source re-scan is deliberately kept (its samples are chosen post-publish, so reusing build-pass statistics would let the build self-certify).Measured acceptance at full 35.2M scale (record: the A1/A2/B-REMEASURE documents of the materialize-perf audit series): verify now completes bounded where it previously OOM'd — the entire quick-verified 35M run peaked at 1.41GB RSS (dims scale: 16.46GB → 1.24GB, 13.3× less); a clean-load warm-cache build ran 56.0s wall with encode spread across both workers (the cold-source wall remains dominated by fact tables being scanned for Pass-1 indexing, Pass-2 emission, and the verify oracle — the recorded O5(c) follow-up on this branch, not a separate PR); and the independent oracle gate that kept this PR draft is now closed: a full-triple streaming diff (same-renderer N-Triples export of both sides → external sort → set compare, ~1.2GB RSS) between the materialized twin and the blessed natively-imported
enterprise-sf01ledger returned zero differences in either direction at 35,238,778 triples — the enumerator is proven correct against an oracle that is not itself.Two verify-gate false-reject classes were found by that acceptance run and fixed in the parity compare's value canonicalization: decimals (stored E-notation vs the oracle's plain
f64rendering) and dateTimes (storage pads fractional seconds to six digits,…T00:16:42.000000Z, while the oracle's formatter drops the fraction when zero — the gate now normalizes insignificant fractional-second zeros and zero UTC offsets on both sides). Both were fail-CLOSED failures — the gate refused correct twins and dropped them loudly, never the reverse. Known residuals, recorded here rather than deferred to other PRs: a non-zero-UTC-offset dateTime is deliberately not collapsed to its UTC instant (the schema under test is all-UTC); graph-source resolution is CWD-relative (./.fluree) and does not honorFLUREE_HOMEfor sources — a UX wart worth a look; and whether a first-class skip-verify mode for operators running their own independent diff deserves productizing is an open maintainer question.Original PR body
Base re-pointed by consolidation onto the family-C branch beneath it; no other PR's scope folded in — this survivor remains the standalone materialize builder, original text below.
fluree materialize— a native twin from a virtual (R2RML-over-Iceberg) datasetImplements the materialization design's first deliverable: bulk-build a native Fluree ledger ("twin") from a virtual R2RML-over-Iceberg graph source, verify it against the source, and write it as a
.flpackpack or a local ledger. This is the categorical fix identified by the materialization assessment — it ends the per-query cold-parquet storm and the admission-gate tail-chase for datasets that warrant a native copy, with guaranteed 1:1 native query semantics.First-principles per the AJ directive — zero anchoring to PR #1422; no transact-replay. The build reuses the existing bulk
create --fromfinalize → spool-sort → k-way-merge index → publish pipeline verbatim, pointed at the R2RML scan stream instead of a TTL parser.Design + assessment record: C3-materialization-facts.md (machinery + arithmetic) and D1-strategy-options.md (the strategy space).
What a reviewer should walk
The build is one new producer branch feeding an unchanged downstream. Read it in dependency order:
fluree-db-r2rml/src/materialize/graph.rs— the engine-freeTriplesMap → (s,p,o)whole-graph enumerator. Composes the SAME per-term helpers the query operator uses (term/datatype fidelity by construction), dims-first FK resolution.fluree-db-api/src/materialize.rs— scans each table through the enumerator, encodes byte-budgetedParsedChunks viaImportSinkObserver, ships them into the existing finalizer channel (ImportSource::Virtual).ac21c0711— per-table snapshot capture:IcebergCatalogSession.snapshotsrecorded at the singlescan_table_innercaller ofload_table_context(first-writer-wins),pinned_tables(gs),FlureeR2rmlProvider::build_watermarkoverride.fluree-db-api/src/graph_source/{catalog_session.rs,r2rml.rs}.fdcbc11f7— pin-all pre-pass (all snapshots pinned up front → cross-table skew bounded to seconds) +WatermarkStampassembly (watermark vector + SHA-256 mapping hash + builder version) + fail-loud empty-watermark guard.ee7897e1e— persist the completion stamp in the twin's FINAL commit:ParsedChunk.txn_meta(empty → byte-identical for every text import) +finalize_parsed_chunkconsume; the stamp is ns-encoded through the sink and rides the last non-empty commit via a one-chunk lookahead (completion-marker semantics: a valid twin is one whose head commit carries a parseable stamp).fluree-db-transact/src/{import.rs,import_sink.rs},fluree-db-query/src/r2rml/provider.rs.743f5e4ae— the post-build parity gateverify_twin: class counts + stratified per-subject sample (quick, always-on) and a full-triple diff (full), enumerator-as-oracle vs a native wildcard query, both rendered to identical N-Triples.bf27d545a— thefluree materializeCLI + MoR build-time UX.fluree-db-cli/src/commands/materialize.rs.Correctness proofs (all hermetic; no live runs in this PR)
materializer_driver_matches_engine_wildcard: the driver's triple set is byte-identical to the query engine's?s ?p ?owildcard over the same scan.materialize_builds_queryable_native_twin: a real indexed+published twin round-trips class counts, an FK edge to the parent IRI, anxsd:decimal(value+datatype), and a lang tag (value+@en).materialize_stamp_survives_pack_restore:archive_ledger → restore_ledgerpreserves the head-commit stamp byte-for-byte andread_stampparses it identically.verify_twin_passes_on_faithful_twin(pass,quick+full) andverify_twin_fails_on_corrupted_twin(an injected instance fails on both the class-count check and the full diff).Parity-gate value-semantics (ratified)
A parity gate asks "same graph in value space."
"5.00"^^xsd:decimaland"5"^^xsd:decimalare one value; comparing pre-intern lexical (enumerator) against post-intern canonical (twin) is a diff artifact, not a data defect. The gate strips trailing fractional zeros for xsd numeric datatypes on both sides. Where lexical renderings genuinely differ for the same value, the twin's native rendering is contract-correct by the parity direction (native is the fixed contract) — so any client-observable decimal-lexical drift on cutover is the virtual path shedding a deviation, not the twin introducing one. The lead-run live native-sf01 full-triple diff must carry the same value-level canonicalization.Gate record
fmtclean; scopedclippy0 new warnings;cargo check -p fluree-db-api --no-default-features --features aws,iceberg,shacl(no-native featureset) compiles; CLI compiles with and withouticeberg. Tests: fluree-db-transact lib 303, fluree-db-query lib 116, fluree-db-api materialize lib 6,it_graph_source_r2rml60 (2 ignored ext-infra), fluree-db-cli materialize helpers 5.Machine-safety posture
The CLI default is co-resident-tolerant (fixed
parallelism=2,memory_budget_mb=512, never own-the-box auto-sizing);--memory-budget-mb/--parallelismraise it explicitly,--max-performanceopts into host auto-sizing on a cleared machine. Every finalizer-touching test runs withthreads=2+memory_budget_mb=256. Verification uses publish-then-verify-then-drop: the gate needs a queryable twin, so a failed gate drops the twin — nothing unverified stays announced. (Deliverable-2 rider: solo's alias repoint must sequence strictly AFTER gate-pass.)Residuals (deferred, tracked)
key → parent-subject-term(general); the exists-set + apply-parent-template form saves memory when parent subject-template columns ⊆ the join key. Guarded byMaterializeStats.ref_edgesper-(child,predicate) counts. Not trivial → left as a guarded-threshold note.block_onoff a dedicated thread); a runtime-flavor-agnostic async-task → bridge-thread producer is deferred until a caller needs it.--output s3— not yet wired in the file-backed CLI (rejected up front). The direct-S3 CAS publish path exists (designed and verified CAS-safe); it needs an S3-backed home the CLI does not construct.read_task_sample-bounded sample is the scale follow-up.Package-end gate (lead-run, not in this PR): the live native-sf01 full-triple diff and any live build against Snowflake. No live runs were performed here.
Warehouse-root direct resolution (follow-on, AJ live-gate registration-gap fix)
bf27d545a→6fcf7fc64. The local CLI's Direct mode was single-table, blocking registration of a multi-table R2RML source over a catalog-less bucket copy (a Snowflake-managed Iceberg database whose table dirs carry random suffixes,fact_order.UIHGsQex/). Now a Direct--table-locationmay be a warehouse ROOT: auto-detected when its leaf dir doesn't name the requested table, eachrr:tableNameresolves to its own dir under the root via one session-cached S3 LIST (SendIcebergStorage::list_dir), matched by the purematch_warehouse_table_dir(<name>.<suffix>/or bare<name>/, case-insensitive, namespace-stripped). Ambiguity/miss fails loud naming the candidates. Single-table direct is byte-unchanged (self-diff), no new flag / config change (rule documented iniceberg map --help). Registration shape:fluree iceberg map <name> --mode direct --table-location s3://bucket/warehouse/dw --s3-region <r> --r2rml <ttl>— pure ambient IAM, zero catalog/OAuth. Hermetic: 5 match tests; iceberg lib 251, api it_graph_source_r2rml 62 green. The live S3 LIST is part of the lead-run gate.Consolidated scope
#1529 folds in no other pull requests. It implements the native-twin builder set out in the materialization design record and was built first-principles, reusing the existing bulk-ingestion finalize → spool-sort → k-way-merge → publish pipeline pointed at the R2RML scan stream. Its pre-consolidation tip is preserved at tag
archive/pre-refactor-2026-07-21/feat_materialize-builder; nothing was rewritten. Its base is re-pointed onto the family-C branch beneath it (perf/family-c-filter-join), which is a sibling of the fork point, so the diff renders builder-only while GitHub notes the base is some commits ahead.Verification of record
Every proof in this PR is hermetic; the live gate is deferred and is the basis for the draft status.
The hermetic proofs, all green:
materializer_driver_matches_engine_wildcard(the driver's triple set is byte-identical to the query engine's?s ?p ?owildcard over the same scan);materialize_builds_queryable_native_twin(a real indexed and published twin round-trips class counts, an FK edge, anxsd:decimal, and a lang tag);materialize_stamp_survives_pack_restore(archive→restore preserves the head-commit stamp byte-for-byte); and the parity-gate pairverify_twin_passes_on_faithful_twinandverify_twin_fails_on_corrupted_twin(an injected instance fails on both the class-count check and the full diff). The independent adversarial review is recorded under R-1529 in pr-reviews-impl.md: verdict SHIP (draft) with no blocking finding — the shared text-import path is byte-untouched, the completion stamp rides the structurally-final commit so partial builds are detectable, the CLI'sBox::leakis shell-only and one-per-process, and every fail-loud guard fires. The review's load-bearing observation, which a reader must internalize: the parity gate is an ingestion-integrity gate, not an enumerator-correctness gate — its oracle is the enumerator, so a fault in the enumerator itself is only caught by the deferred live native-sf01 diff.The build/serve/sync arithmetic and the pack substrate anchors are in C3-materialization-facts.md; the native-twin-promotion strategy and its snapshot-keyed correctness lemma are in D1-strategy-options.md.
Native-impact appendix
ParsedChunk.txn_metaplus afinalize_parsed_chunkconsume land on the shared bulk import / transact path (fluree-db-transact/src/import.rsandimport_sink.rs). Every text-importParsedChunksite setstxn_meta = Vec::new()(parse_chunk, the ndjson constructor, andfinalize), andis_virtual = config.virtual_source.is_some()is false for all file and remote imports; the newImportSource::Virtualarm is inserted before the remote-parallel path with no existing body touched. Theimport.rsdiff is 138 insertions / 1 deletion (the only change to an existing line isis_channel_fedgaining|| is_virtual). Thebuild_watermarktrait default is empty, so non-materialize providers are unaffected, andrecord_snapshotis fire-and-forget first-writer-wins with no extra I/O. Verified: transact lib 303 + import 10 green, andmaterialize_stamp_survives_pack_restore.fluree-db-r2rml/src/materialize/graph.rs,fluree-db-api/src/materialize.rs, the CLI, and the warehouse-rootlist_dir) compose the shared per-term helpers the query operator uses without modifying them.Confirmed negatives (git-checked): no
arrow/datafusion/parquetdependency bump, and no.github/workflowsorrust-toolchainedits.Draft status and open gates
This PR is deliberately a draft. Two gates are open and both must be run before any live cutover relies on the twin:
Until both are exercised, the builder invites hermetic review as a draft with the live parity gate stated as pending; it should not be flipped ready on the hermetics alone.