fix(format): CURIE-align virtual graph-source IRIs in sparql_json (F9) by aaj3f · Pull Request #1499 · fluree/db

aaj3f · 2026-07-14T17:13:38Z

What

Virtual (R2RML / Iceberg graph-source) queries rendered predicate and @type IRIs as full IRIs in SPARQL-results JSON, while native compacted them to CURIEs from the query's PREFIX declarations. This closes that divergence (finding F9) so virtual output matches native.

q002 / q042 flip — now byte-identical to the native oracle:

query	virtual before (raw)	virtual after (this PR)	native oracle	rows
q002	`8cd3964a…`	`d65ea385…`	`d65ea385…`	8
q042	`0615a095…`	`33b91b54…`	`33b91b54…`	24

Head row-diff verified: every changed cell is exactly full-IRI → CURIE — <edw:openDate>, <edw:name>, <edw:channel>, <edw:storeId>, <edw:storeType>, <edw:geography>, <edw:regionManager>, and the rdf:type object <edw:Store>. The rdf:type predicate stays a full IRI (no rdf: prefix declared), the data.fluree.dev/edw/ reference objects stay full (that namespace isn't declared), and every literal is unchanged — all matching the oracle. No baseline re-bless: the expected/ oracles were blessed from native and already carry the CURIE form, so virtual now simply matches them.

Why (mechanism)

SPARQL-results-JSON compaction is @context/PREFIX-driven: ContextCompactor::compact_id keys only on the query's declared prefixes and never consults the namespace-code map. Native predicates arrive as Binding::Sid and run through compact_id_sid → compact_id_iri (compacts); virtual predicates arrive as Binding::Iri (a raw graph-source string) and were written verbatim at format/sparql.rs (streaming write_term and DOM format_binding), bypassing the compactor. The fix routes Binding::Iri through the same compact_id_iri native already uses — so parity holds by using the identical compaction call.

(This corrects an earlier "namespace-map gap" hypothesis: native renders rdf:type and data.fluree.dev/edw/ references full despite both being in the ledger namespace map, precisely because compaction tracks declared prefixes, not the map.)

Scope

Change is gated three ways: from_graph_source && FLUREE_R2RML_CURIE_ALIGN && format == SparqlJson. Native results (Binding::Sid) are untouched; only virtual/graph-source Binding::Iri in sparql_json is affected.
QueryResult.from_graph_source is set in exactly one place — query_view_with_r2rml_options, from the resolved view's graph_source_id. Native query_with_options and mixed-dataset paths leave it false (raw, unchanged); the native-ledger-reached-with-.with_r2rml()-but-no-mapping edge is documented at the set site.
Other output formatters (JSON-LD, SPARQL XML, typed, delimited) share the compactor but receive the flag as false and keep raw graph-source IRIs — that cross-format consistency work is tracked as register F16.
Kill switch FLUREE_R2RML_CURIE_ALIGN (default on) forces the pre-F9 raw rendering — powers the revert differential and gives production a revert lever.

Design alternatives were refuted at review gates: encoding at the R2RML operator (hot-path namespace work) and emitting Binding::IriMatch (unsound — it carries a load-bearing originating-ledger primary_sid that eq_for_join compares; virtual IRIs have none). The broader question of whether native should also compact its constructed/federated/GRAPH IRIs (a provenance-dependent-rendering finding) is deferred to #1496.

Blast radius — exactly q002 + q042

Among full-hash-gated queries, the switch moves only q002 and q042 — rerun-proven. The raw ON-vs-OFF differential listed more, which decompose cleanly: hash_gate=rows_only queries (unordered LIMIT) vary run-to-run by subset nondeterminism, not the switch (they project only literals — the switch is a definitional no-op); every other full-gated query was re-run with the switch off on a healthy network and is byte-identical to its switch-on hash. A transient Snowflake network reset during the original OFF phase errored a block of queries (confirmed transport, not the switch); all completed cleanly on rerun.

Gates

q002/q042 flip green vs the native oracle (heads verified full-IRI→CURIE only).
Full virtual corpus ON compare: 54 records, 0 hash mismatches, 0 perf violations (proper per-query gates).
Native 54/54: 0 mismatches. W3C SPARQL suite: green. fluree-db-api lib: 664/664 including three new tests — a context-driven-compaction mechanism proof (namespace map irrelevant; rdf:type stays full), a flag-off-raw / flag-on-CURIE scoping guard on the same Binding::Iri row, and a DOM↔streaming parity assertion with a Binding::Iri row so both arms must flip together (closing the diagnosed partial-fix no-op loophole).

Stacked on perf/r2rml-pr5-scan-topk.

Rider-1 mechanism investigation: SPARQL-JSON compaction is @context/PREFIX-driven via ContextCompactor::compact_id, which never reads the namespace-code map. The virtual gap is a formatter dispatch (Binding::Iri written raw at sparql.rs:333/487) vs native's Binding::Sid -> compact_id_sid -> compact_id_iri. Approved Option B (seed the snapshot namespace map) is a no-op for the hash; corrected fix is to route Binding::Iri through compact_id_iri. Pending lead re-review.

…ites + measured option-C exposure Pre-code proof obligation (lead rider): enumerated all Binding::Iri construction sites workspace-wide. Native-reachable sites exist (eval/value.rs:522 BIND(IRI)/UUID, sparql_results.rs SERVICE, graph.rs GRAPH ?g, bm25/vector search), so the unconditional shared-formatter fix is native-visible — blocks it pending scope decision. Measured exposure: option C regresses zero test gates (W3C runner uses empty ParsedContext => compact_id_iri no-op, plus CURIE-expansion+isomorphic compare; native 54/54 byte-identical, no corpus GRAPH/SERVICE/BIND(IRI)). Options C (two-line, needs AJ sign-off on narrow native consistency change) vs A (provenance-scoped enum variant). Awaiting lead C-or-A + AJ.

… design D (emit IriMatch from r2rml/operator.rs) is UNSOUND: IriMatch carries a real originating-ledger primary_sid that eq_for_join (join.rs:1045) compares for same-ledger bindings (binding.rs:778); virtual IRIs have no SID, so fabricating one breaks virtual joins. A real SID = option B (ruled out). Recommend option A via the existing GraphDb.graph_source_id (None for native, Some for virtual) -> QueryResult.from_graph_source -> IriCompactor flag -> the two sparql.rs Iri arms. Structural native untouchability, no enum change. Awaiting lead nod on the marker design.

…s-link #1496 F9 SHARPENED namespace-map claim refuted via dated addendum (preserved, not rewritten): compaction is @context/PREFIX-driven (compact.rs:60), never the namespace map; real gap is the raw Binding::Iri formatter arm. New F16: other formatters (jsonld/sparql_xml/typed/delimited) render virtual graph-source IRIs raw — consistency follow-up needing own gates. Native provenance-rendering question filed as #1496; cross-linked both ways. Summary table rows updated.

Virtual (R2RML) predicate/type IRIs rendered as full IRIs in SPARQL-results JSON while native compacted them to CURIEs from the query PREFIX, because the formatter's Binding::Iri arm was written raw whereas native's Binding::Sid arm runs through compact_id_iri. compact_id is @context/PREFIX-driven and never consults the namespace map, so the fix is a formatter dispatch change, scoped to graph-source results and sparql_json only. - QueryResult.from_graph_source (set in query_view_with_r2rml_options from the resolved view's graph_source_id); native/mixed-dataset paths stay false. - IriCompactor.compact_graph_source_iris flag, built into the compactor only in the sparql_json entry points; every other formatter keeps raw IRIs (F16). - The two sparql.rs Binding::Iri arms compact via compact_id_iri when set; DOM/streaming parity preserved. - Kill switch FLUREE_R2RML_CURIE_ALIGN (default on) for the revert differential and production safety. Gate (DW_SF01): q002 8cd3964a->d65ea385, q042 0615a095->33b91b54 now MATCH the native oracle byte-for-byte (heads verified: every changed cell full-IRI->CURIE; rdf:type, data.fluree.dev/edw/ references, and literals unchanged). No re-bless: the expected/ oracles were blessed from native and already carry the CURIE form, so F9 was 'virtual diverges from oracle' and the fix closes it with zero baseline churn. Blast radius exactly q002+q042 among full-hash-gated queries (rerun-proven: every other full-gated query is switch-invariant, OFF==ON). Full virtual corpus ON compare 0 mismatches; native 54/54 clean; W3C green; fluree-db-api lib 664/664 incl. 3 new tests (context-driven compaction proof, flag-scoping guard, DOM/stream parity with a Binding::Iri row).

…ate) q029 (rows_only, unordered LIMIT) DNF'd at the 120s deadline in the F9 gate; standalone healthy-network rerun completed ok/rows=100/~150s. Correct but slow; F9-neutral (formatter-only, DNF'd identically on/off); upstream scan variance. Revisit for its own F-number if still ~150s post-PAT-swap.

…e q029 note q029 DNF in the F9 gate was a harness deadline mismatch, not a regression: manifest timeout_s=180, completes ~142-150s (scan-bound/re-scan-amplified, 09-stacked-rebaseline §3), but the gate applied a blanket 120s. F17 records the mechanism (UNION absorbs the LIMIT budget -> both FACT_WEB_EVENT branch scans re-driven ~253x -> 1.94M reads for 100 rows) and remedy (budget propagation through UNION, mechanism-class D; mirror PR-5 wrapper-forwarding; likely subsumed by F14/PR-4d). pf9_gate.sh now honors per-query timeout_s.

Clean 3-rep hot baseline (supersedes contaminated pf9_virt_on): 42/50 ok-queries already <=3s; 8-query tail with mechanism+remedy. Headline: q031 72s is a REGRESSION - PR-8b query-scoped parent-memo present + unit tests pass but q031's plan no longer reaches it (1448 un-memoized dim re-scans); A/B FLUREE_R2RML_LIMIT_PUSHDOWN=0 -> scan_table 1448->8; PR-4c /PR-7/topk refuted; leading culprit PR-5 GraphOperator forwarding refactor (51aafc7), confirm via bisect @2a07bbbc4. Plus q029 (F17), q016 (F14/4d), q038 un-fused COUNT (48s/1row, wall in generic eval), q041 ref-OR not pushed, q053/q017 anti-joins. Slate to AJ first.

…nesty pass Bisect + PR-8b gate forensics exonerate ALL PRs: the 188ms 'sentinel' was a fully-warm reps=3 artifact (q029 also 156ms in that phase, vs 125s cold); the gate's recorded q031 was 65s/scan_table 1448 = same as now. So q031 is cache-sensitive (188ms warm <-> 65-72s thrashed), pre-existing since PR-8b - NOT a regression; retract the PR-5 indictment. File F18 (parent-memo x LIMIT-pushdown interaction). Corrected the burndown end-state (low-single-digit s cache-thrashed, NOT the 0.2s warm-only figure) + slate. Gate-protocol lesson: the 188ms sentinel hid the 65s cold cost in the same gate (~350x optimism) -> sentinels must be cache-thrashed; full-corpus baseline per stacked-PR head.

… source The horizon-demo 'enterprise' source carried a stale/revoked stored PAT (client_secret) -> OAuth unauthorized_client 401 (not a grant/scope gap; ICEBERG_READER already has full ENTERPRISE_DEMO.DW). Re-registered a harness-owned source in ~/vbench/.fluree via 'fluree iceberg map enterprise-sf20-v' (16 DW.* TriplesMaps) with a fresh client_secret; retarget fluree_home -> ~/vbench/.fluree, alias -> enterprise-sf20-v. Verified: exec-one virtual-sf20 q001 = ok, 2236 rows, 6.2s (401->ok).

…spelling The new write_term Binding::Iri arm (block form) tripped clippy:: semicolon_if_nothing_returned (a workspace-denied lint) at sparql.rs:337/339 — add the two semicolons. curie_align_enabled accepted only "0" as falsy while the switch family accepts 0/false/off/no, so FLUREE_R2RML_CURIE_ALIGN=false silently stayed on — widen to the family set. Verified q002/q042 carry no stale F9 markers (hash_gate=full, no expected_status).

aaj3f · 2026-07-21T20:28:58Z

Consolidating at maintainer request. #1499 (F9 — CURIE-alignment of virtual graph-source IRIs in sparql_json) is carried forward in #1507 (the R2RML/Iceberg virtual-dataset performance program), which brings the #1475–#1507 lineage forward as a single reviewable unit. Its verification of record is the program's corpus benchmark, published as C2-bench-wave1.md (native == virtual, 0 hash mismatches). The pre-consolidation branch tip is preserved at tag archive/pre-refactor-2026-07-21/fix_f9-virtual-curie — no history was rewritten. Closing in favor of #1507; discussion continues there.

aaj3f force-pushed the perf/r2rml-pr5-scan-topk branch from a8de70e to 83347b7 Compare July 15, 2026 17:06

aaj3f added 11 commits July 15, 2026 13:08

aaj3f force-pushed the fix/f9-virtual-curie branch from e1ac131 to f815669 Compare July 15, 2026 17:11

This was referenced Jul 15, 2026

docs(audit): F18→F20 investigation record + PR-4d/loadTable slate reservations #1509

Closed

perf(virtual): R2RML/Iceberg virtual-dataset performance program — corpus, pruning, fused aggregation, budget forwarding (consolidates #1475–#1507) #1507

Merged

aaj3f closed this Jul 21, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(format): CURIE-align virtual graph-source IRIs in sparql_json (F9) - #1499

fix(format): CURIE-align virtual graph-source IRIs in sparql_json (F9)#1499
aaj3f wants to merge 11 commits into
perf/r2rml-pr5-scan-topkfrom
fix/f9-virtual-curie

aaj3f commented Jul 14, 2026

Uh oh!

aaj3f commented Jul 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

aaj3f commented Jul 14, 2026

What

Why (mechanism)

Scope

Blast radius — exactly q002 + q042

Gates

Uh oh!

aaj3f commented Jul 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant