feat(rdf): RDF toolkit core — quad IR, IR-level writers, W3C conformance harness, fluree rdf check/count/convert - #1555
Open
aaj3f wants to merge 43 commits into
Open
feat(rdf): RDF toolkit core — quad IR, IR-level writers, W3C conformance harness, fluree rdf check/count/convert#1555aaj3f wants to merge 43 commits into
aaj3f wants to merge 43 commits into
Conversation
The IR is triple-only today, so a faithful `.nq`/`.trig` conversion has
nowhere to put a graph name. Add the two types that give it one:
- `Quad` = a whole `Triple` plus `Option<Term>` graph. Wrapping the triple
(rather than flattening s/p/o beside a fourth term) keeps every existing
triple consumer — ordering, list indices, formatters — working unchanged.
`None` is the default graph, a real graph and not "no graph": the same
statement in the default graph and in a named one are two distinct quads.
Ordering is graph-major so sorting groups each graph's statements, which
is the shape both TriG and readable N-Quads want.
- `Dataset` = a default `Graph` plus named graphs in a `BTreeMap<Term, Graph>`.
The default graph is an ordinary `Graph` with unchanged semantics, which is
what makes triple-only input and quad input agree on it. Named graphs are
key-ordered for the same determinism reason `Graph::prefixes` is a BTreeMap.
`@base`/`@prefix` are document-scoped, so they live on the default graph
and are reached through dataset-level accessors — one authoritative home,
not one per graph.
An empty named graph is representable (TriG's `<g> {}`) and distinguishable
from an absent one: `is_empty()` counts statements, `named_graph_count()`
counts graphs. Dedupe is per-graph — the same triple in two graphs stays two
quads.
The quad-capable analog of GraphCollectorSink: `supports_quads()` is true,
`emit_quad` appends to the named graph (created on first use), and
`emit_triple`/`emit_list_item` go to the default graph — so the same
triple-only event stream builds a default graph indistinguishable from the
`Graph` the collector would build, asserted directly.
Term interning, blank-node identity, and statement-scoped literal recycling
are now one `TermTable` behind both sinks rather than two copies that can
drift on term lifetime — the invariant recycling's soundness rests on.
GraphCollectorSink's behavior is unchanged by the move: same id allocation
order, same `-b{N}` disjoint mint namespace, same slot reuse, and its
adversarial recycling differential stays green.
Rollback spans graphs. A statement can write to the default graph and to
several named ones before failing, so `abort_statement` rewinds each graph
it touched, and REMOVES a named graph that only the failed statement brought
into existence — a statement that fails must contribute nothing, including
no empty graph. Marks are keyed by TermId (no allocation on the hot path);
because two ids can denote the same graph within a statement, rewinding runs
in reverse so the earliest mark wins.
A literal graph name is refused: the dataset keys graphs by that term and no
dataset syntax can serialize a literal there, so accepting one would build a
dataset that cannot be written back out.
…s to them Parsers here report byte offsets because that is what a u32 token span costs nothing to carry. Humans need line and column, and `--continue-on-error` needs a *value* it can collect, count and filter — not a Result that ends the parse. Three additions close that gap: - `LineIndex`: line starts over an input str, built on the FIRST lookup through a OnceCell. Constructing one is free, so a driver can hold one for a whole parse and a clean document still never scans for newlines; the thousandth diagnostic reuses the table the first one built. Columns are 1-based CHARACTER counts (what makes a caret land under the glyph it blames), \r belongs to the terminator, offsets past the end clamp, and an offset inside a multi-byte character resolves instead of panicking. Pinned against the lexer's previous algorithm as an oracle over every offset. - `Diagnostic` + `Severity`: severity, stable `&'static str` code, 1-based line/col derived from the span start (so the two cannot disagree), byte span, one-line message. Line 0 marks "no position" — some conditions are facts about a document, not about an offset, and inventing offset 0 for them would point the caret at a lie. Serialize but not Deserialize: diagnostics are produced and reported, never read back. - `TurtleError::to_diagnostic`: an additive adapter. TurtleError's shape is untouched — every parser caller depends on it. The lexer's caret rendering is now the shared `LineIndex::caret_block` rather than four inline format strings, so a raw lexer error and a rendered Diagnostic describe a position identically — checked by comparing the two blocks, not asserted. A byte-for-byte pin on the lexer's messages was written and shown green BEFORE that refactor, so it proves the output is unchanged rather than documenting whatever it became.
…list-item gap
Three things the quad path needs pinned rather than assumed:
- A TriG document's actual event shape — directives, statements outside any
block, a `<g> { … }` block, then statements back outside it — fed by hand,
since no TriG parser exists in the light crates yet. Re-entering the
default graph after a block must append to it, not start a second one, and
directives must stay document-scoped instead of leaking into the block.
- The capability probe is what routes a producer between the two sinks, so
the split is asserted from both sides. Now that they share a term table,
this is the guard against the triple-only collector quietly inheriting quad
support it has nowhere to honor.
- `emit_list_item` has no quad form in the protocol, so an indexed collection
inside a named graph cannot be expressed and lands in the default graph.
Documented on the method as a protocol gap rather than a routing choice,
and pinned by test — if the trait grows a quad form, that test is what
should fail. Narrow in practice: under the spine collection style a
collection is ordinary triples and goes through emit_quad.
…n the caret gutter
Review found the comments wrong, not the behavior — except the gutter, which
was wrong in both.
F2a. term_table.rs claimed the minted `-b{N}` label "still serializes"
because '-' is legal medially. It is not: the '-' here LEADS, and `_:-b1` is
rejected by this repo's own lexer (`_:b-1` is fine — that is the distinction
the comment blurred). The mint is deliberately NON-serializable: disjointness
is bought with illegality. So a writer must relabel every anonymous mint
rather than pass it through, now stated as the writers' contract and proved
by a lexer test instead of asserted in prose.
F2b. LineIndex claimed an offset inside a multi-byte character resolves to
"that character's column". It resolves to the column AFTER it — the count of
characters starting before the offset — which is what the pre-shared lexer
algorithm did. Prose and test name now say that.
F4. line_text strips a trailing \r unconditionally; str::lines strips it only
from an \n-terminated line, so a final unterminated line ending in \r
differs. Kept deliberately — a raw CR inside a caret block returns the cursor
over the gutter and shreds the diagnostic — and documented as the one
deviation. The oracle now encodes that carve-out and covers "\r", "a\r",
"a\r\r\n" and "a\rb\n" at every offset, instead of not asking.
F5. The caret gutter was a hardcoded two spaces against an unpadded
`{line} | ` prefix, so from line 10 on the bar and the caret sat one cell
left of the source text, and further with each digit. Now padded to the line
number's width. THIS DELIBERATELY CHANGES OUTPUT for lines >= 10 — the one
sanctioned break in the byte-identity pin, which gains a 12-line fixture
covering it. Single-digit lines, every previously pinned case, are unchanged.
Fixing it here rather than in each writer is the point of caret_block being
the shared definition.
…view's probes F1. The list-item test did not trip: a reviewer added a hypothetical default-bodied `emit_quad_list_item` and the suite stayed green, because the test pins where today's event ROUTES and nothing more. Two changes. `PROTOCOL_QUAD_EVENTS` now counts the trait's quad-shaped events, sited beside the methods it counts so the diff that adds a sibling touches it, with a test whose failure message names the follow-up work. Being honest about its limits: Rust cannot enumerate trait items, and a default-bodied addition compiles against every impl — which is exactly how such an event would reach a dataset sink and be dropped into the default graph. So this is a forced, greppable decision point, not a detector, and the test comment now says that instead of claiming the routing test would catch it. F6. Adopt the review's two probe files verbatim, credited to it. quad_rollback_adversarial.rs (10 cases) attacks what the implementation's own tests sat closest to: one graph marked through three ids, a recycled literal slot offered as a graph name, what a refused quad leaves behind, and which layer actually enforces the graph-name invariant. dataset_parser_parity.rs (3 cases, 7 abort shapes x 2 collection modes) is the load-bearing version of the default-graph equivalence claim — it drives the REAL parser, which is what decides when end_statement and abort_statement fire, where the in-crate test only hand-feeds a tidy stream. A9 turned up a contract detail worth naming: abort_statement rolls back statements, not directives. A prefix accepted before the failure stays accepted — on_base/on_prefix are infallible and a producer cannot un-resolve IRIs it already expanded — same as GraphCollectorSink. Documented on the method, since --continue-on-error inherits it.
No golden tests over export existed anywhere, so the coming lift of its escaping/prefix primitives into fluree-graph-format had nothing holding it to "behavior changes: none". Export is not line-deterministic — the untranslated-overlay tail drains from a HashMap — so the fixtures compare SORTED lines. That is a fingerprint of what was written, not a document: a sorted Turtle golden is deliberately not valid Turtle. Escaping, lexical forms, prefix declarations and row-skipping all move it; statement ordering alone is invisible to it. The fixture pins the cases the formatting layer has to get right: N-Triples string escaping incl. a control character, IRIREF percent-escaping, the canonical xsd:double form, a local name with no PN_LOCAL spelling, longest- prefix-first compaction, language tags, refs, and the rdf:type -> a abbreviation. Every node has an explicit @id because a skolemized _:fdb-<ulid> would differ per run.
…luree-graph-format
The IR-level streaming writers need exactly the pure half of export.rs:
N-Triples string escaping, IRIREF percent-escaping, typed-literal writing,
and the Turtle prefix-compaction path. Duplicating it would have given the
two writers of Fluree's RDF two spellings of one grammar.
Moved verbatim into fluree-graph-format (escape.rs, prefix.rs). The
store-coupled half stays in db-api: write_turtle_object resolves datatypes
and subject ids through BinaryIndexStore inside the formatter, so it does
not move.
export.rs re-exports the lifted names, keeping
fluree_db_api::export::{PrefixMap, write_escaped_iri, ...} resolving for
export_builder.rs and any external caller; its tests for them stay put and
now pin that re-export path.
Two additions, both additive: PrefixMap::insert (a streaming writer declares
prefixes as on_prefix events arrive, and a redeclaration must shadow rather
than append) and PrefixMap::new/Default.
The sorted goldens from the previous commit are byte-identical across the
move, which is the whole reason they were built first.
Five GraphSink implementations in fluree-graph-format, so a parser can emit
straight into an output syntax with nothing materialized in between. This is
the bridge the fluree rdf effort was scoped around: everything upstream
produces GraphSink events, and until now nothing consumed them into bytes.
NTriplesWriter / NQuadsWriter — one statement per line, O(1) memory.
TurtleWriter / TrigWriter — the blocks tier: consecutive same-subject runs
fold with ';', same-subject-and-predicate with ',', document order otherwise.
No cross-document regrouping, no [ ] reconstruction, no ( ) re-collapse; all
three need the whole document in memory, which is what a streaming writer
exists not to need. A buffered --pretty tier is the follow-up, and riot draws
the same line.
JsonLdWriter — document mode over the existing format_jsonld, O(document) and
said so in the type docs rather than implied by silence.
TriG writes quads where they arrive, reopening a GRAPH block when the input
returns to a graph it left. That is valid TriG — a graph is the union of its
blocks — and it keeps memory O(1) against interleaved input, where buffering
per graph would be O(dataset). Input already grouped by graph still gets one
block per graph, so the tidy case costs nothing.
Blank-node policy (plan H-6): relabel by default to a fresh b{N} per input
label, with _:fdb- passing through verbatim for #1432's addressability
contract. Disjointness is by construction, not by check: every output label is
either a b{N} mint or begins fdb-, and no mint does. --preserve-bnode-labels
emits labels verbatim and mints anonymous nodes into a reserved fdbw-
namespace; there disjointness cannot be established by construction, so an
input label inside that namespace is refused rather than allowed to merge, as
is one that cannot lex as a BLANK_NODE_LABEL.
Two refusals worth calling out. Named-graph events against a triple-only
writer are refused via the capability probe rather than flattened. Indexed
list items are refused by the text writers — one triple where RDF means three
is data loss — but kept by JsonLdWriter, because JSON-LD has @list to say it
with.
abort_statement is honest about what it can do: unbuffered, bytes already
handed to the writer stay written; WriterConfig::buffer_statements makes it a
true rollback at O(widest statement), which is what --continue-on-error will
need. Turtle's run state rolls back with it, so a rejected statement leaves no
half-open subject.
Round-trip tests parse with the real Turtle parser on both ends and compare
under a blank-node bijection; blank-node identity counts are checked
separately, since isomorphism cannot see a merge. Lexical fidelity is pinned
for +1, 01, 1.0, 1e0 and a bignum, which is why literals are always written
fully quoted and typed rather than in Turtle's numeric shorthand.
…ilures Two holes in the deferred-failure channel, both found reading back over the writers rather than by a failing test. A refusal stashed by term_blank was *taken* by the next emission, so a producer that ignored one Err could emit again and have the label the writer had just refused written out. The protocol permits a failed sink to misbehave; writing the thing it refused is not a reading worth keeping. Failures now latch: every later emission refuses too. In buffered mode nothing reaches the underlying writer until end_statement, which has no error channel — so a broken pipe was swallowed there and only surfaced at finish(). Early termination is the reason the emit methods are fallible at all, so the commit failure is stashed and the next emission raises it, exactly as an unbuffered writer would. Both now go through one Deferred type shared by all five writers instead of an Option<SinkError> per writer.
One Arc bump per quad, not two. No behavior change.
…g them
Review point from the quad workstream, and its framing is the right one:
preserve applies to labels a *user* wrote. A label that cannot lex as a
BLANK_NODE_LABEL was necessarily never written by one — no parser would have
accepted the document — so it is an internal mint, and the IR's own anonymous
nodes are exactly that shape (-b{N}, deliberately unlexable so they cannot
collide in memory).
Refusing such a label, as this did, fails a whole document over a name its
author never chose. Now it is minted afresh into the reserved fdbw- namespace,
same as an anonymous node, which is the only outcome that is both legal and
lossless. The refusal stays for the one case where it is warranted: a label
that DOES lex and sits inside the reserved namespace, where preserving risks a
merge and relabelling would silently break the promise the mode makes.
So the guarantee is now total: every emitted label is legal and disjoint, and
every label a user could have written is either verbatim or a loud error.
Term's Display writes lexical forms with no escaping at all. Nothing in a writer output path uses it — everything goes through the lifted escaping primitives — and these two tests are what keeps that true. The second one is the interesting half. Writing it turned up that the danger is worse than "produces something the parser rejects": Display of the three characters a \ b yields "a\b", which the parser accepts happily as the two characters a + U+0008, because \b is a valid Turtle ECHAR. The backslash became an escape and ate the character after it. Silent corruption, no error anywhere — so a guard that only asks "did the output parse?" would not catch it, and the assertion is that Display's output does not round-trip faithfully, not merely that it fails.
This is a behavior change to shipped output — `fluree export` routes every IRI through this function — and it is a bug fix, not a preference. Percent-encoding is not injective. Given the IRI `http://ex/a b`, the old code emitted `http://ex/a%20b`, which is a DIFFERENT IRI, and which is exactly what a caller separately holding the IRI whose text really is `http://ex/a%20b` would also emit. Two distinct resources came out as one, silently, with no error anywhere. Export has been doing this to every IRI containing a forbidden character. The IRIREF grammar provides UCHAR for precisely this: `<http://ex/a\u0020b>` reads back as `http://ex/a b` — the IRI we were handed, unchanged — while `http://ex/a%20b` stays itself. Escaping is now injective, which is the property that makes it a spelling rather than a transformation. Also narrows the forbidden set to what the grammar actually forbids: #x00-#x20 plus nine punctuation characters. U+007F-U+009F were being escaped and are perfectly legal in an IRIREF, so a raw DEL or NBSP now passes through as itself. The export goldens move by exactly four lines, all IRIs, all in this direction: has%20space -> has\u0020space, odd%7Ciri -> odd\u007Ciri. The JSON-LD golden is untouched, since that path escapes for JSON, not IRIREF. This is what the goldens were built to make visible. Reviewer's j1/j2/j3 and zz_merge probes adopted as tests. Also removes the unreachable eight-hex UCHAR branch (char::is_control stops at U+009F, so it could never fire) and the per-byte format! allocation in escape_iri_into.
Two ways the prefix map could emit a name no parser reads. H-3: '-' is in PN_CHARS but not PN_CHARS_U, so it cannot START a local name. The position-blind check allowed it anywhere, so an IRI ending '/-dash' compacted to 'ex:-dash' — invalid output, silently, wherever such an IRI appears. Local names are now checked positionally and fall back to a full IRIREF when they cannot be spelled. L-3: prefix NAMES were never validated at all. A JSON-LD @context is the live source here — its keys are arbitrary strings, so '1st', 'has space' and '-x' all reach PrefixMap::from_context, and any IRI matching one came out as '1st:thing'. Prefixes that cannot be written as a PNAME_NS are now dropped at insert; the empty prefix stays, since '@Prefix : <…>' is legal. Both directions are conservative: the real grammar allows more than either check accepts, and declining to compact costs only verbosity while compacting wrongly costs validity. The new surface test walks every local-name shape through compaction and parses whatever comes out, so a future widening of either predicate has to stay honest.
H-1. finish() could return Ok after part of the document was lost. Three ways in, all of them ending with no emission left to raise the stashed failure: a broken pipe hit while committing the LAST buffered statement, a refusal as the last event, and a write failure inside on_prefix (which returns nothing). In each case a driver was told the conversion succeeded. finish() now consults the deferred channel — flushing first, so the stashed error, which carries the context, wins over a secondary error from writing to an already-broken pipe. M-2. emit_list_item and the literal-position refusals returned an error without latching, so a producer that ignored one Err got a clean finish(). Both now go through the same channel: counted, latched, and visible at finish. L-2. A literal is refused in subject and predicate position too, not just as a graph name. The protocol documents the invariant on Triple without enforcing it, so a hand-fed producer can present one and the writer would emit a document no parser reads. M-1. A directive arriving while a TriG graph block is open now closes the block first. TriG's wrappedGraph holds statements and nothing else, so a @Prefix between the braces produced a document no TriG parser accepts; the arrival-order design reopens the block on the next quad for free. L-1. close_statement consults the WORKING run state, not the committed one, so a directive arriving mid-statement terminates the statement already written instead of leaving it dangling. Reachable from the public API: a producer may call on_prefix at any point. M-3. BlankLabeler no longer keeps its own input-to-output map. WriterTerms already keys input label to TermId — and that entry holds the rewritten term — so the second map was written on every first sighting and read never, doubling per-label memory for nothing. WriterStats grows "aborted" and "refused" so --continue-on-error can report without keeping its own tally. Latched re-refusals are deliberately NOT counted: they are not separate events. The three tests that asserted Ok-after-loss are flipped, and each of the three finish paths above has its own test — the on_prefix one through a writer whose pipe the test closes at a chosen point, so it exercises the stash rather than just failing again on the next write. TriG directive shape adopted from the reviewer's c1 probe.
…mation M-4. The "every emitted label is legal" guarantee was narrower than the production it claimed. unlexable_first_char checked the FIRST character and only for ASCII, so preserve mode emitted verbatim: any non-ASCII character outside PN_CHARS_BASE (× ÷ · combining marks U+FFFE, and every gap between the ranges), and anything illegal LATER in the label — spaces, quotes, #, comma, semicolon, the empty label, and a trailing '.'. The trailing '.' is the one that matters. `_:ab.` does not fail to parse; it lexes as `_:ab` followed by the statement terminator, so the node is silently RENAMED and the document still loads. Same class as the `a\b` literal corruption, and combined with the finish() fix it would have been an unparseable-or-wrong file reported as success. Turtle-parser-sourced labels cannot reach this, because the parser would have rejected them on the way in. Externally-sourced ones can — R2RML, JSON-LD and skolem identifiers — and that is preserve mode's entire population. The fix is the reviewer's unification: H-3 and M-4 were one bug in two places, both of them a hand-rolled approximation of a grammar this workspace already transcribes. fluree-graph-turtle/src/lex/chars.rs moves into fluree-graph-format (the same lift as escape.rs and prefix.rs) and grows the whole-token productions — is_blank_node_label, is_pn_local, is_pn_prefix — built from the character classes already there. The lexer keeps using them through a re-export at lex::chars, so the reader and the writers now share one transcription and cannot drift. escape.rs's IRIREF predicate is the lexer's own is_iri_char inverted, for the same reason. Two consequences worth stating. Local-name compaction now follows PN_LOCAL, which ADMITS ':' — so `ex:has:colon` compacts where it used to expand. That is a widening of shipped export output; the goldens do not move (no fixture IRI has a colon or a non-ASCII local part) and the surface test parses everything compaction produces and checks it denotes the IRI it came from. And prefix names follow PN_PREFIX rather than an ASCII subset. The Display-guard's re-read now parses in conformant mode, so the hostile set can grow a numeric without the test quietly stopping testing what it means to. w1/w2 probes adopted as an end-to-end test that compares blank labels VERBATIM rather than up to isomorphism — an isomorphism check cannot see a rename, which is the whole failure being looked for.
fluree-graph-turtle gained a dependency on fluree-graph-format for the shared character predicates, and testsuite-shacl and testsuite-sparql are separate workspaces with their own lockfiles that reach it transitively.
Placement correction. Putting them in fluree-graph-format made the Turtle PARSER depend on the FORMATTERS, which builds fine and reads backwards. fluree-graph-ir is the crate both already depend on, and format-agnostic RDF grammar predicates are what it is for. The lexer keeps reaching them at lex::chars, now re-exported from ir, and fluree-graph-turtle drops the dependency it briefly had on fluree-graph-format. Also re-narrows local-name compaction. PN_LOCAL admits ':', so the faithful predicate accepted `ex:has:colon` where export previously wrote the full IRI — a change to shipped output with no correctness behind it. The other narrowings in prefix.rs are there because the wider behavior emitted INVALID output; this one only emits DIFFERENT output, which is a product decision and should be made as one rather than arrive as a side effect of the unification. The carve-out is one line in the writer, not in the grammar predicate: is_pn_local stays faithful to PN_LOCAL and is tested as such, and is_emittable_local is where the writer says it will emit less than the grammar allows. Dropping the ':' clause there is the whole change if we later decide to widen. Goldens are byte-identical to the H-2 commit, confirming the re-narrow put export's output back where it was.
…aces The dependency it recorded is gone: the shared character predicates moved to fluree-graph-ir, which fluree-graph-turtle already depended on.
The last path that emitted an input label with no legality check, and it was
in the DEFAULT Relabel mode rather than an opt-in one: `_:fdb-…` passed
through verbatim on the strength of its prefix alone. Six of seven hostile
fdb- labels produced output no reader accepts, with finish() reporting Ok.
Falls through to the mint, on the same reasoning that settled preserve mode: a
label that is not a BLANK_NODE_LABEL was provably never written by a user, and
it is provably not a Fluree skolem either, because those are fdb-<ulid> and
are always legal. So there is no addressability to preserve and nothing is
lost by minting — the bijection keeps the node's identity, only its name
changes. Verbatim passthrough stays for legal fdb- labels, which is every real
one.
Disjointness is unaffected: mints are still b{N}, passthroughs still begin
fdb-, and fewer labels pass through than before.
y3 adopted inverted to expect the fix; y1/y2 adopted as the identity
properties that make minting a rename rather than a merge — one repeated
illegal label stays one node, and distinct labels (illegal, legal and
anonymous, all sharing one counter) stay distinct.
Adds `testsuite-rdf/`, an excluded workspace that gates the readers (fluree-graph-turtle and the graph IR it emits into) against the W3C rdf-tests syntax suites — the counterpart to testsuite-sparql, which gates the query engine. Without it these suites run in no CI at all. Scope: RDF 1.1 Turtle and N-Triples gate in conformant parser mode (CollectionStyle::Spine + NumericStyle::PreserveLexical); the ingest default mode and the RDF 1.2 suites run in the same invocation but only report. Registered failures stay in the denominator and out of the numerator, so registering a known failure lowers the published rate. Baseline at rdf-tests efccbc6b8: Turtle 297/313 (94.9%), N-Triples 55/70 (78.6%), with all 31 remaining failures registered and attributed in tests/registers/mod.rs. Notes on the design: - The rdf-tests submodule is SHARED, read through ../testsuite-sparql/rdf-tests rather than registered a second time, so "conformance at submodule SHA X" keeps one answer. The harness fails with the exact `git submodule update --init` command when it is missing, rather than reporting a vacuously green empty suite. - Manifests are parsed with our own Turtle parser in conformant mode, walking mf:entries as a real rdf:first/rdf:rest spine. A harness that read the suite's index with a third-party parser could report green while the parser under test could not read the suite at all. - The isomorphism checker is lifted from testsuite-sparql's result_comparison.rs and retargeted to the shared IR term, with set semantics (a graph is a set), a ground/blank split so the common all-ground case is a sort-and-compare, and O(1) injectivity. - The RDF 1.2 suites run their rdf12-only sub-manifests, not the published root: that root mf:includes ../../rdf11/..., which would fold the RDF 1.1 results into the RDF 1.2 number. - The register is policed in both directions, and that enforcement is itself tested — a stale entry, an unregistered failure, a dangling entry and an empty manifest each have a test proving they fail the suite. Main workspace untouched apart from the root Cargo.toml exclude and the new ci.yml job.
…t of `failed` Two ways the harness could report a number that did not mean what it said. F1 — a pass RATE cannot notice its own denominator shrinking. Drop half a manifest and the survivors still score 100%. The only guard was `total == 0`, which catches total loss; the realistic failure is PARTIAL — a spine walk that stops early, a manifest restructure, a half-initialized submodule. The N-Triples suite was the worst exposure: its manifest has no `mf:include` chain, so a truncated walk could silently drop up to 23 tests and still look green. Each suite now declares `expected_total` (Turtle 313, N-Triples 70, rdf12 sub-manifests 96 and 70) and a mismatch fails loudly. This doubles as the submodule-drift tripwire: when rdf-tests legitimately updates, this assert is what fails, and its message says to re-count and update the number in the SAME commit that moves the submodule. F7 — a register entry naming no discovered test was pushed onto the same list as test failures, so it landed in `failed` and broke the report's total = passed + failed + ignored identity, precisely in the runs that were already failing. Register-integrity problems are not test outcomes: nothing ran. They are tracked separately, reported in their own labelled section, and still fail the suite. A `debug_assert` now pins the identity. Both are tested: a deliberately-wrong `expected_total` must fail even though every discovered test passes, and a dangling register entry must be reported without inflating `failed` or breaking the identity.
The freshness contract in `riot-analog-bench-strategy.md` §6b lets a Tier-2 matrix publish only when a conformance run exists at the same `git_sha`. That check was defeatable by accident: a run from a dirty working tree recorded the bare HEAD SHA, so it compared equal to a clean run at the same commit while measuring code that was never committed. `git_sha()` now appends `-dirty` when `git status --porcelain` reports anything, and `Conformance` carries a `dirty: bool` alongside. Both, on purpose: the flag serves structured consumers, and the suffix means a comparator that knows nothing about the flag still refuses the run rather than accepting it. The binary also warns on stderr. An unreadable git state counts as dirty — the point is to refuse to certify a number we cannot tie to a commit, and "git did not answer" is that case.
…efutations F4 — RDF 1.1 §3.3 makes a literal's language tag case-insensitive for term equality: `"chat"@FR` and `"chat"@fr` are the SAME term. The IR compares tags case-sensitively, so a gold `.nt` file spelling a tag in a different case than the `.ttl` it checks would have failed for a difference RDF says does not exist. Folded in the CHECKER ONLY, during normalization — the IR is untouched, because the parser's job is to preserve what the document said and only the comparison needs the equivalence. Normalizing (rather than special-casing `terms_match`) is what makes it apply to the ground-triple set comparison too, which uses plain `Term` equality. Tests both directions: tags fold, while different tags, different subtags, and the literal VALUE's case all stay distinguishing. F3 — locks two properties the implementation already had but nothing pinned: a 2-cycle between two blank nodes is NOT two self-loops (same triple count, same blank count, same per-node degree — a checker matching on counts alone would pass it), and disconnected blank components are matched across components rather than only within the one the search starts in, including the case where each component is individually well-shaped but paired with the wrong partner.
The burn-down was described as six causes, which is what you get by counting the Turtle causes and forgetting E — the N-Triples class, where the suites run through the Turtle parser and a negative test whose document is valid Turtle cannot fail. E reads as a footnote and then turns out to be the largest single group in the register (14 of 31 entries). Enumerate all seven in the module docs so the count is not reconstructed by eye each time.
…items
`predicateObjectList ::= verb objectList (';' (verb objectList)?)*` — the
tail item is optional, so a run of semicolons is legal Turtle that denotes
nothing. The parser consumed one `;`, checked for a statement terminator,
and then demanded a predicate, so `;;` and `; ;` were rejected anywhere
they appeared. Generators emit trailing and doubled semicolons routinely.
Consume the whole run before deciding whether a predicate follows. A
leading `;` stays an error: the production starts with a verb.
W3C rdf11 Turtle 297/313 -> 301/313 (94.9% -> 96.2%); the four register
entries are removed here, as the register's both-way policing requires.
Mode-independent, so the informational ingest-default run moves with it
(272 -> 276) and the conformant >= ingest-default direction assert holds.
PN_LOCAL ::= (PN_CHARS_U | ':' | [0-9] | PLX)
((PN_CHARS | '.' | ':' | PLX)* (PN_CHARS | ':' | PLX))?
Dots are ordinary interior characters and only the FINAL character is
constrained. The scan looked one char past a dot and stopped if that char
was itself a dot, so `:s..2` lexed as `:s` plus a statement terminator —
`:s.1` worked, `:s..2` did not.
One char of lookahead cannot decide this, since a dot after a dot may
still be interior. Scan greedily over PN_CHARS | ':' | '.' and rewind to
the last legal end, the same shape `parse_blank_node_name` already uses
for BLANK_NODE_LABEL. The end is TRACKED rather than trimmed off the span
afterwards, because PLX is a legal final element and a trailing PLX may
itself be an escaped dot (`:a\.`) that trimming would eat.
W3C rdf11 Turtle 301/313 -> 302/313 (96.2% -> 96.5%). Mode-independent
(276 -> 277 ingest-default). PN_PREFIX keeps its single-dot lookahead:
no W3C test exercises a dot run there, so it is left for the reader that
needs it rather than changed unverified.
RFC 3986 §5.1.1: a base declared relative resolves against the base already in scope. `@base <foo/>` under `http://example.org/ns/` denotes `http://example.org/ns/foo/`. The parser installed the reference verbatim, so the base became the string "foo/" and every IRI after it resolved against a scheme-less base. Nothing failed while this happened. A converter would have written a document full of wrong IRIs and reported success, which makes this the most consequential item in the burn-down even though it is one test. The corruption's visible shape was a leading colon — `<a3>` came out as `<:foo/a3>` — and it is worth being precise about why, because the obvious reading (something prepends a colon) is wrong. With a scheme-less base, `parse_iri_components` finds no `:`, so it reports an empty scheme and treats the whole string as the path; RFC 3986 §5.3 recomposition then unconditionally emits `scheme ":"`, which prints as a bare `:` when the scheme is empty. Verified directly: `resolve_iri("foo/", "a3")` returns `:foo/a3`. The shared resolver is behaving as specified for an input that cannot occur once the base is guaranteed absolute. Route the declared IRI through the existing `resolve_iri`, which already serves every other IRI position and delegates to fluree-vocab's shared RFC 3986 resolver. A relative base with nothing in scope now fails loudly instead of installing a broken base. Relative prefix namespaces needed no change — they already resolved — but they inherited the corruption through the base, so they are covered by a test too. W3C rdf11 Turtle 302/313 -> 303/313 (96.5% -> 96.8%), plus units beyond the W3C case for compounding redeclarations and the no-base error.
Turtle has two directive spellings with different case rules, and the lexer had each one backwards. `@prefix` / `@base` are literal terminals in the Turtle grammar, so they are case-SENSITIVE — the lexer lowercased the word after `@` and so accepted `@BASE`. The SPARQL-style `PREFIX` / `BASE` (no `@`) are defined by reference to the SPARQL grammar, whose keywords are case-INSENSITIVE — the lexer matched them exactly and so rejected `base <...>` and `PreFIX : <...>`. Match `@`-directives exactly; compare the bare forms with `eq_ignore_ascii_case`. `a`, `true` and `false` are literal terminals too and stay case-sensitive, with a test pinning that the widening did not reach them. `@BASE` now lexes as a language tag, which the parser already rejects in directive position. W3C rdf11 Turtle 303/313 -> 306/313 (96.8% -> 97.8%), clearing all three directive-case tests at once (two over-rejections, one over-acceptance). GRAPH is left exact: it is TriG's keyword, no test here exercises its case, and widening it unverified would be a guess.
`true` and `"true"^^xsd:boolean` are one RDF term written two ways, but they produced two different IR terms: the keyword became `LiteralValue::Boolean`, the longhand `LiteralValue::String`. `Term`'s equality and hash are variant-sensitive, so the two spellings compared unequal and hashed apart while `Display` rendered both identically. A graph stating a fact both ways held two triples where RDF says one — dedup and set semantics were wrong, not just the W3C comparison. `NumericStyle::PreserveLexical` already fixed exactly this for integers and doubles; the conformant preset simply never reached the `KwTrue`/`KwFalse` arms. Route them through the same typed-string lane, so under the conformant preset both spellings yield `"true"^^xsd:boolean`. Term equality and hash are untouched. The ingest default is unchanged — booleans stay native `LiteralValue` values on the storage path — with a test pinning that, alongside a dedupe test proving the two spellings collapse under the preset. `NumericStyle`'s docs now state the widened scope: the switch governs every token Turtle spells as a bare value (integer, double, boolean). They stay ONE knob deliberately — splitting it would let a caller select a combination where one spelling of one datatype silently fails to deduplicate. W3C rdf11 Turtle 306/313 -> 309/313 (97.8% -> 98.7%). Conformant-only, so the mode delta widens: the preset is now worth 28 tests (89.8% -> 98.7%) and the direction assert holds. This closes the burn-down for A1/A2/A3/B/D; the 4 remaining registered failures are all cause C (IRI/language-tag validation), which belongs to the H-8 workstream.
G1. `parse_with_prefixes_base[_options]` installed its `base` argument
verbatim, so the corruption the `@base` directive fix just closed was still
reachable through the API. Reproduced before fixing:
parse_with_prefixes_base("<a> <b> <c> .", &mut sink, &[], Some("foo/"))
=> <:foo/a> <:foo/b> <:foo/c> .
Identical shape to the directive bug, and identical mechanism: a
scheme-less base makes `parse_iri_components` report an empty scheme, and
RFC 3986 §5.3 recomposition then emits `scheme ":"` as a bare colon. This
one is on the immediate M1 path — the CLI's `--base` flag feeds exactly
this entry point.
A relative parameter is REFUSED, not resolved. The directive has a document
base in scope to resolve against; a caller-supplied base has no context at
all, so resolving would mean inventing one and guessing at the caller's
intent. Refusal matches what a relative `@base` already gets when nothing is
in scope, and the error names the offending value so a bad `--base` says
which input was wrong. Callers holding a possibly-relative base must resolve
it themselves; both public entry points now document that contract.
G2. `TurtlePrelude::base` was documented as "base IRI (as declared)". Since
the directive fix it is as-RESOLVED — resolution happens upstream, at the
directive, before `on_base` fires. That is load-bearing rather than
cosmetic: the prelude is fed straight back into `parse_with_prefixes_base`
for every chunk of a parallel import, which now refuses a non-absolute base.
A relative value there would fail an import loudly rather than mis-resolve
it, but the reason it cannot arise is upstream resolution — so the doc has
to say which.
G3. The two isomorphism refutations guard DIFFERENT properties, and the
comments now say so: the 2-cycle-vs-self-loops case locks blank-MATCHING
(a candidate pairing that cannot extend consistently is rejected), while
the collapse case locks INJECTIVITY (the reverse map). Recorded so nobody
carries one test's expectation onto the other.
Conformance unchanged: Turtle 309/313 (98.7%), N-Triples 55/70.
…file Stands up the `fluree rdf` verb group: RDF syntax tooling that reads files rather than ledgers. No `.fluree/` lookup, no ledger, no connection — the verbs open a file (or stdin), parse it, and report. `check` parses and reports diagnostics; `count` reports statement and term tallies. `convert` is registered as a named stub — the flag surface is settled, the writers are not, and it refuses with a message naming what is missing rather than "not implemented". Three pieces the rest of the effort builds on: - **One syntax table.** `RdfSyntax` names all eight target syntaxes, not just the readable two, so unsupported input is refused by name with what it waits on instead of being sniffed into Turtle and failing thirty lines in. Resolution is explicit flag > extension > content sniff, with the compression suffix stripped first and resolved independently. `detect.rs` is untouched: it still serves insert/upsert/validate and the server, and unifying them is a follow-up. - **PhaseTimings + TimingSink in fluree-graph-ir.** A lane-switching stopwatch (one clock read per phase transition, not per unit of work) and a decorator that instruments any `GraphSink` from the outside. Events are counted on every call and *timed* on a 1-in-127 statement sample, then scaled: bracketing every call would cost two clock reads per event and, on a counting sink, would report mostly its own overhead. - **`--profile`.** A human box table sharing the bench lane's renderer, and `--profile=json` (schema `fluree.rdf.profile.v1`) carrying tool version, host and thread count, corpus fingerprint, per-phase shares, counts, rates, and the profiler's disclosure of its own overhead — marked untrusted above 2% of wall. Both go to stderr so piped output stays clean. Exit codes split "the document is bad" (1) from "the invocation is bad" (2), which is what makes these verbs wrappable. Transparent `.gz`/`.zst` on both files and pipes: gzip is decoded multi-member so `pigz`/`bgzip` output does not silently truncate, and a pipe (or an unlabelled file) is detected from its magic bytes. The stream is opened exactly once — a pipe cannot be reopened at byte 0. flate2/zstd move from dev-dependencies to dependencies; input is capped at 4 GiB, the parser's `u32` token-span limit, with the chunked reader deferred to the parallel pipeline.
Review found the `--profile` sink phase was ~98% instrument artifact and
stamped "trusted". The reviewer's probe (adopted here as
`fluree-graph-turtle/examples/sink_bias_probe.rs` and as unit fixtures in
`fluree-graph-ir::timing`) measured a bracketed `DiscardSink` call at 20.3 ns
against a ~19 ns clock pair: the estimator was scaling its own overhead by the
sample ratio and printing the result as a phase.
Three corrections to `TimingSink`:
- **Subtract before scaling.** `sampled_calls × clock_pair_cost` comes off the
sample first, so the artifact is not multiplied along with the work. What
the extrapolation would have carried is published as
`SinkTiming::artifact`, and folded into `overhead_pct` — the `>2%` untrusted
marker previously could not fire on a `count` at all, because only the wall
cost of the reads actually taken was counted.
- **A measurement floor.** Below 3× the artifact an estimate is inside the
error of the calibration subtracted from it, so `SinkTiming::body` is `None`
and both outputs say "below the measurement floor". That is not zero, and
the JSON says `null` rather than `0` so it cannot be read as "free". The
same rule applies to `finish`, whose no-op case measured tens of ns.
- **Jittered sampling.** A fixed 127-stride reaches only `P/gcd(127,P)` of a
periodic corpus's shapes; on the probe's one-fat-statement-per-127 corpus
that was a 425× over-report when the fat shape was sampled and a total miss
when it was not. The gap is now drawn from `1..2×SAMPLE_STRIDE`, seeded from
the corpus, so there is no residue class to be confined to for any P — and
a given input still samples the same statements across runs.
`finish()` also moved out of the sampled body into its own exactly-measured
field: routed through the scaling, a 50 ms writer flush was reported as 6.4 s.
Probe verification, before → after: cheap sink 70 ms → below floor; periodic
corpus 425× → declines; 50 ms finish 6366 ms → 60 ms exact.
Also in this pass:
- Diagnostics are one line. The lexer pre-renders a block with its own
numbered excerpt and caret, which printed beside this module's snippet and
caret as two errors at two positions.
- `check` writes human diagnostics to stderr, matching riot and `count`;
`--format json` stays on stdout.
- `--profile json` (space) explains itself instead of hunting for a file
called `json`.
- Empty stdin exits 0 like an empty file; a BOM no longer defeats syntax
detection.
- `--profile` and `--time` fire on the failure path too.
- One vocabulary: `triples` for RDF edges, `grammar_statements` for Turtle's
`statement` production. They differ by directives and predicate-object
lists and were both being called "statements".
- profile.v1 gains `host_class` (`FLUREE_BENCH_HOST_CLASS`, `{os}-{arch}`
default), a runtime-discovered `git_sha` (no build script), and
`peak_rss_bytes` normalized across the Darwin-bytes/Linux-kilobytes split.
- Corrected two false doc claims: `wall_ns` now states exactly what its
window excludes (startup, fingerprinting), and the worked example is real
output rather than invented numbers. Peak RSS is documented as measured —
2.4× to 6.2× of input depending on how many distinct IRIs a corpus has,
because the excess is the parser's IRI cache.
Two artifacts in the profile output contradicted the report beside them. A no-op `finish()` measures 100-200ns of dispatch and clock variance, which clears three clock pairs, so it earned a `sink (est) 0.00ms` phase row sitting directly under the line saying the sink was below the measurement floor. The finish floor is now the larger of three clock pairs and one microsecond: a flush worth naming is not a sub-microsecond one, and a real writer's is microseconds to milliseconds. `human_duration` integer-divided sub-microsecond values into microseconds and printed "0μs" — a measured value rendered as nothing. Below a microsecond it now prints nanoseconds.
N1 (blocker). `git_sha` shelled to the *working directory*, so it answered "which commit is the shell sitting on". A binary built from 9869167d8 and run from the main checkout reported ecc059f — worse than having no field, since a baseline would be attributed to code that never produced it. It now runs `git -C <dir of current_exe()>`, which reports the commit a `cargo build` artifact came from and honestly says "unknown" for a binary installed outside any checkout. Test builds a throwaway git repo, runs the binary with that repo as cwd, and asserts the reported SHA is not that repo's HEAD. N2. The below-floor sentence quoted the aggregate floor while describing it as per-event: "under 82ms" across 720,004 calls is seven orders of magnitude from what a reader takes it to mean. `SinkTiming::floor_per_call` divides through, and both the human line and `sink.floor_ns_per_call` now state ~96ns/call. N3. `SinkTiming::floor()` returned the artifact, not the threshold, so a report printing it named a bar three times lower than the one being applied. It returns `FLOOR_MULTIPLE × artifact`; the artifact keeps its own field. N4. No sink line, and no floor arithmetic, when nothing was forwarded — a floor computed from zero calls is a statement about nothing. Trusted split. One `trusted` flag went false on essentially every `count` run via the sink artifact, which made it useless for gating the phases that were fine. Now `phases_trusted` (driven by the clock reads actually taken — negligible by construction, and what a Tier-1 gate should key on) and `sink_trusted` (driven by the extrapolated artifact). The human output prints whichever tripped, naming it. Sampling honesty. The claim that jitter leaves "no residue class to be confined to" was true and incomplete: the schedule is a deterministic function of the corpus — that is what makes two profiles comparable — and both `corpus_seed` and the generator are public, so anyone choosing the input can replay the schedule and park the expensive work in the gaps. A reviewer did exactly that and hid 315ms. Reproducible and unpredictable are incompatible and reproducible is the one a baseline needs, so the docs now scope the guarantee to accidental structure and say plainly that the estimate describes a cooperating corpus, never an untrusted one. Also: `SinkTiming::relative_std_error` (Welford over per-statement sampled costs) bounds ordinary sampling error and surfaces as `sink.relative_std_error_pct`; `normalize_max_rss` takes the platform as a parameter so both the Darwin-bytes and Linux-kilobytes branches are reachable from a test on either host, rather than the branch only one CI ever runs.
The worked `--profile` example was captured from a debug build, which calibrates a reader roughly 20x low. Regenerated from a release build and labelled as one. The peak-RSS figures were also debug-build, and a reviewer measuring them independently got different numbers — reasonably, since the ratio depends on IRI length and corpus shape, which nothing in the doc pinned down. Both fixtures now ship as scripts/rdf-rss-fixture.py, the table quotes release measurements against them (6.7x with every subject distinct, 1.9x with a hundred reused), and the text says why the two differ: the excess is the parser's IRI cache, which grows with distinct IRIs rather than bytes. Also documents the new profile.v1 fields in a table — including which of `phases_trusted` / `sink_trusted` a regression gate should key on — and the limits of the sink estimate against a corpus that was built to hide from it.
The verb the effort exists for. Input (file or stdin, gz/zst) → Turtle parser
in conformant mode → a writer chosen by output syntax → file or stdout.
Streaming throughout: the parser emits into the writer and bytes leave as they
are produced, so `convert big.ttl | head -5` costs five statements and memory
stays flat over a large dump.
Format pairs: {turtle, ntriples} in → {turtle, ntriples, nquads, trig, jsonld}
out. Quad *input* waits on the M2 parsers; JSON-LD input still goes through
`fluree insert`. A syntax with no writer is refused by name with what it is
waiting on.
Output syntax resolves `--to` > `-o` extension > N-Quads, matching riot's
default. `--bnode-policy relabel|preserve` and `--prefixes <JSON|PATH>` reach
the writers, the latter accepting a JSON-LD `@context` document unchanged and
meaning one thing across syntaxes — Turtle and TriG compact with it, JSON-LD
takes it as its context. Prefixes the input declares are always carried.
Three seams worth naming:
- **`parse_into`** generalizes `parse_document` over the sink, so all three
verbs run the same parser under the same options with the same sampling
seed. It hands the sink *back*: a `BufWriter` left to flush on drop
discards the error that says the output is truncated, so convert flushes
through a handle that can still return one.
- **`TimedWriter`** is what makes a `write` phase measurable without the
per-emit clock the review ruled out. Layered writer → BufWriter →
TimedWriter → destination, the writer's small writes cost no clock and the
destination sees 64 KiB chunks, so `write` is real I/O time measured at a
few clock reads per document. `serialize` is then the sink estimate minus
`write` — and, like the estimate it derives from, absent entirely when that
estimate does not clear its measurement floor. A decomposition supersedes
the `sink` row rather than sitting beside it.
- **Conformant parser options are mandated, not preferred.** Under the ingest
default a collection arrives as indexed list items, which every writer
refuses because an indexed list item is a Fluree storage shape with no RDF
serialization. A test pins the observable consequence: the rdf:first/rdf:rest
spine, and rdf:nil for `()`.
Error semantics: a closed downstream pipe exits 0 in silence at whichever
layer reports EPIPE (riot's behaviour); a parse failure exits 1 and says the
output is a prefix of the conversion, because the written bytes cannot be
recalled; everything about reaching the destination exits 2. `--pretty` and
`-o out.nt.gz` both refuse rather than quietly doing something else — the
first would give blocks-tier output under a flag promising more, the second a
`.gz` full of plain text.
`--continue-on-error` is not here: it needs statement-scoped output buffering
wired to a flag, which the writers support and the driver does not yet.
…he latch
H2 (blocker). `--prefixes '{"ok":"not an iri"}'` emitted
`@prefix ok: <not an iri> .` and exited 0 — a document this tool's own
reader rejects, written by this tool, reported as a success. Every namespace is
now checked with `fluree_vocab::iri::is_absolute_iri` before a writer can see
it, and refused with the offending prefix, the offending value, and why a
namespace needs a scheme. A converter that emits what it cannot read back has
failed at the only thing it is for.
M5. Three separate problems around refusals:
- The writers latch, deliberately — a sink that failed once keeps failing
rather than pretending. But the latched message ("this writer already
refused an event") carries no cause, and convert reported `finish()` before
the parse error, so the placeholder is what users saw. The first cause is in
the error the parse loop got; that is now what is reported, with a remedy in
this CLI's vocabulary when the cause is a reserved-namespace collision.
- `File::create` truncates. A refusal that needs no input at all — an output
syntax with no writer — happened *after* the destination was opened, so a
run that was never going to produce anything emptied the file it was pointed
at. Those checks moved ahead of the open. A failure only the parse can
discover still leaves a partial file, which the error already says.
- The library's refusal named `--preserve-bnode-labels`, a flag that does not
exist in the shipped CLI. Corrected to describe the remedy rather than name
a flag, since the crate cannot know its callers' surface.
M4. `is_broken_pipe` matched the *text* "Broken pipe". glibc translates
`strerror` under `LC_MESSAGES`, so that check passes for whoever wrote it and
fails everywhere else — and the failure mode is a spurious error at the end of
an ordinary `| head`. `ParseRun::finished` now carries `SinkError` and the
flush its `io::Result`, so all three sites check `ErrorKind`.
M1. The completion summary is suppressed under `--profile=json`: `2> run.json`
is the bench lane's idiom, and one ✓ line ahead of the document made the file
unparseable — but only with `-o`, which is exactly when a harness uses it.
L1. Refusals blame the output syntax the way it was actually chosen. "--to
nquads has no pretty form" reads as a lie to someone who never passed --to.
L2. A `--prefixes` argument that is valid JSON of the wrong shape reports the
shape, instead of "No such file or directory" sending the reader to look for a
file called `[1,2]`.
…doubled artifact subtraction M2. The suite claimed byte-identical N-Triples across a Turtle round trip as a general invariant, "deliberately stronger than isomorphism". It is not general. Blank-node labels survive only when the parser mints them in the same order the writer emits them, which holds for flat shapes and fails the moment they nest: the parser mints outermost-first, the writer emits deepest-first because the inner node must exist before the triple referencing it. The fixture that carried the claim happened to be flat. Split accordingly. `FLAT_SHAPES` keeps byte equality with the reason it holds written down; `NESTED_SHAPES` — the review's collection-of-collections, triple nesting, and nested anonymous nodes — asserts isomorphism, and *also* asserts that relabelling really does occur, so nobody quietly reinstates the stronger claim without the test objecting. `nt_isomorphic` is lifted from the review's rdflib oracle: brute force over the blank-node bijection, with an N-Triples term tokenizer, because whitespace splitting breaks on the first literal containing a space. It has its own test proving it can tell graphs apart — a checker that returned true for everything would make every round-trip assertion above it vacuous. M3. The above-floor estimator test bounded its result within 3× of truth, which is far too loose to notice the correction being applied twice. That matters now that a real writer clears the floor: a doubled subtraction removes an extra `calls × clock_pair`, which at N-Triples scale deletes a serialize row worth 30% of wall with every other assertion still green. Now asserts exactly one artifact is removed. Verified by mutation — doubling the subtraction fails it with "off by 2.00×". Plus integration coverage for the fixes in the previous commit: the bad-IRI refusal in both inline and file form, the wrong-JSON-shape message, first-cause reporting without the latch placeholder, the output file surviving a refusal that needed no input, `--profile=json` parsing cleanly with `-o`, and each way a refusal can blame the output syntax.
H1. The page said conversion is streaming with JSON-LD as "the one exception", which undersold it to the point of being misleading. JSON-LD's writer is document-at-once by construction — no streaming form exists short of NDJSON, which is deferred — so the whole graph is held and nothing reaches the output until the input is exhausted. Two consequences now stated with measurements: - Peak RSS 871 MiB against N-Triples' 81 MiB on the same 9.6 MiB corpus: ~96x the input against ~9x. On a large dump that is a conversion versus an OOM. - No early pipe exit. `--to jsonld | head -5` parses everything first, so the trick that makes `--to nt | head -5` cheap does not apply. The format table gains a streaming column rather than leaving the reader to infer it from prose, and the text points at the writer module's own notes. Framed as a documented property, not a defect — riot makes the same call, and RDF/XML will when it lands. Also documents the `_:fdbw-` reserved namespace and the collision refusal it can produce under `--bnode-policy preserve`, and the absolute-IRI requirement on `--prefixes`, both with the real error text.
…eproducible Two corrections the review asked for before the fold. The page described a partial `-o` as what happens when a document fails partway through, which left the *other* mid-stream case — a writer refusing an event — undescribed, and left the narrow promise that does hold unstated. Now split explicitly: a refusal that needs no input (unwritable syntax, bad `--prefixes` namespace, `--pretty`) is raised before the destination is opened and leaves `-o` untouched; anything discovered mid-stream has already created and written to it. A streaming converter cannot honestly promise otherwise — writing to a temporary and renaming would trade that for double the disk and no streaming to the final path — so the promise is stated at exactly the width it holds, and the blank-node collision case is named as the second instance. The JSON-LD ~96× RSS figure was a number with no way to check it, which is the same gap the peak-RSS table already had once. It now carries a recipe over the corpus `scripts/rdf-rss-fixture.py` generates, with the output to expect and a note that the ratio reproduces while the absolutes move with allocator, platform and build profile. Verified: that fixture gives 9x for N-Triples and 95x for JSON-LD on a release build.
…ed load `a_sink_far_above_the_floor_is_reported_as_a_number` asserted the estimate within 3× of truth. It passes in isolation and failed once in a 969-test parallel run, which is the honest verdict on the assertion rather than bad luck: the estimator extrapolates a mean over ~16 sampled statements, a mean is not robust to outliers, and the outlier it meets on a loaded machine is the scheduler — one sampled statement preempted mid-spin reads 10 ms instead of 40 µs and drags the extrapolation orders high. Preemption can only inflate a wall-clock sample, never deflate it, so an upper bound here was a bound on machine load. Replaced with a lower bound, which gates the direction load cannot fake: the estimator must not lose a real sink cost. Accuracy in the other direction belongs to the differential probe on a quiet machine, and the comment says so. The exact-one-artifact assertion is untouched and remains the real gate — it is algebra over a single measured `clock_pair` rather than a second timing, so it holds under any load. Re-verified by mutation: doubling the subtraction still fails with "off by 2.00×". The 969-test group now passes 3/3.
aaj3f
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
RDF toolkit core: quad IR, IR-level writers, W3C conformance,
fluree rdfStacks on #1552 (GraphSink protocol). Base:
feat/graphsink-protocol@869703009.Five workstreams of the
fluree rdfeffort, integrated into one branch. They ship together because they are only useful together —#1552 defined a
GraphSinkprotocol that nothing consumed into bytes, the IRcould not represent a quad, no harness could say whether the parser was
correct, and there was no command to point at a file.
After this branch:
and
fluree-graph-formatexposes five streamingGraphSinkwriters(N-Triples, N-Quads, Turtle, TriG, JSON-LD), so a parser can emit straight
into an output syntax with nothing materialized in between — which
convertnow does end to end.
W3C RDF 1.1 Turtle conformance: 309/313 — 98.7%, gated in CI, with the
failure register policed in both directions so it can only shrink.
Three bugs in shipped behavior, fixed here
Each was found by a workstream below, and each was live before this branch.
A relative
@basecorrupted every IRI in the document. RFC 3986 §5.1.1says a base declared relative resolves against the base in scope; the parser
installed the reference verbatim, so
@base <foo/>made the document base thestring
"foo/"and every subsequent IRI resolved against a scheme-less baseinto
<:foo/a>. Nothing failed while this happened — an import or conversionwrote a document full of wrong IRIs and reported success. This is on the
shared ingest parser, not just the new verbs.
Exported IRIs were percent-encoded, which is not injective. Given the IRI
http://ex/a b,fluree exportemittedhttp://ex/a%20b— a differentIRI, and exactly what a caller separately holding the IRI whose text really is
http://ex/a%20bwould also emit. Two distinct resources came out as one,silently. The
IRIREFgrammar providesUCHARfor precisely this, so we nowwrite
<http://ex/a b>, which reads back as the IRI we were handed. Thismoves the export goldens by four lines, all IRIs; the JSON-LD golden is
untouched because that path escapes for JSON, not
IRIREF.Prefixed names could be emitted that no parser accepts.
-is inPN_CHARSbut notPN_CHARS_U, so it cannot start a local name, and prefixnames were never validated at all — a JSON-LD
@contextkey like1storhas spacebecame1st:thingon every IRI it matched.A fourth defect was IR-level rather than export-level:
truebecameLiteralValue::Booleanwhile"true"^^xsd:boolean— the same RDF termwritten longhand — became
LiteralValue::String.Term's equality and hashare variant-sensitive, so a graph stating a fact both ways held two
triples where RDF says one. Dedup and set semantics were wrong, entirely
independent of any test suite.
1. Quad-capable IR and diagnostics (6 commits)
Quad/Dataset(fluree-graph-ir/src/quad.rs,dataset.rs). #1552added
supports_quads()and anemit_quadthat refuses by default —deliberately, because silently folding named graphs into the default graph is
data loss. Nothing could answer
trueto that probe until now.Quadwraps awhole
TripleplusOption<Term>, so every existing triple consumer keepsworking and
Nonemeans the default graph — a real graph, not "no graph".Ordering is graph-major, which is the grouping TriG and readable N-Quads both
want. An empty named graph (TriG's
<g> {}) is representable anddistinguishable from an absent one. A literal graph name is refused rather
than stored.
DatasetCollectorSink(dataset_sink.rs). Term interning, blank-nodeidentity, and statement-scoped literal recycling moved into one
TermTableshared by both sinks rather than two copies that can drift on term lifetime —
the invariant recycling's soundness rests on.
GraphCollectorSink's behavioris unchanged by that move (same id order, same
-b{N}disjoint mintnamespace, same slot reuse). Rollback spans graphs: a statement can write to
the default graph and several named ones before failing, so
abort_statementrewinds each and removes a named graph only the failed statement created.
Diagnostic+Severity+LineIndex(diagnostic.rs,line_index.rs). Parse errors carried byte offsets, not positions — not avalue a
--continue-on-errorrun can collect, count, or filter.LineIndexbuilds its line-start table on the first lookup through a
OnceCell, soconstructing one is free and a clean parse never scans for newlines. Columns
are 1-based character counts (what makes a caret land under the glyph it
blames). Line 0 marks "no position", because some conditions are facts about a
document rather than about an offset, and inventing offset 0 for them points
the caret at a lie.
Two behaviors worth knowing before building on this:
-b{N}buys disjointness from user-written labels through illegality:
_:-b1isrejected by this repo's own lexer. A writer must relabel every anonymous
mint — it cannot pass one through the way it passes a user-written label
through.
abort_statementrolls back statements, not directives. A@base/@prefixaccepted before a statement fails stays accepted;on_base/on_prefixare infallible and a producer cannot un-resolve IRIsit already expanded.
Known protocol gap, deferred to M2.
emit_list_itemhas no quad form, soa Fluree-style indexed collection inside a named graph cannot be expressed
and lands in the default graph. That is a gap in #1552's protocol, not in this
sink, and nothing reaches it until the TriG/N-Quads parsers land.
PROTOCOL_QUAD_EVENTScounts the trait's quad-shaped events with a test whosefailure names the follow-up work — a forced decision point, not a detector,
since Rust cannot enumerate trait items and a default-bodied addition compiles
against every impl.
2. IR-level streaming RDF writers (15 commits)
Five
GraphSinkimplementations influree-graph-format.NTriplesWriterNQuadsWriterTurtleWriterTrigWriterJsonLdWriterThe two export bugs above were found by lifting
export.rs's formattingprimitives into a place where they could be tested properly.
On
<http://ex/a b>looking alarmingThe obvious review question: does an escaped space inside
<…>mean we areemitting something a tool will reject? No.
IRIREFis'<' ([^#x00-#x20<>"{}|^] | UCHAR)* '>', andis aUCHAR` — theproduction exists precisely so a forbidden character can be spelled. Our own
parser reads it back as the IRI we were given, which the round-trip tests
assert.
riot may still warn on such an IRI, but the warning is about the IRI value
— a space is not permitted in an RFC 3987 IRI — and that is a property of the
data we were handed, not of how we spelled it. The old percent-encoding made
that warning disappear by changing the IRI to a different one, which is the
bug: it silenced a complaint about the input by corrupting it. Emitting the
IRI faithfully and letting a validator object to it is the correct division of
labor;
fluree rdf checkis where that objection belongs.Deliberately NOT changed:
:in local namesPN_LOCALadmits:in every position, soex:has:colonis a legal prefixedname that this writer declines to emit. Accepting it would be correct by the
grammar and would change what
fluree exportproduces for every IRI with acolon in its local part — a change with no correctness behind it.
The two fixes above changed shipped output because the previous behavior was
wrong. This one would only make it different. It is left as its own
decision rather than arriving as a side effect:
is_emittable_localinfluree-graph-format/src/prefix.rsis where it lives, and dropping its:clause is the whole change. Open question — decide separately.
Rollback semantics, and the one case that cannot roll back
abort_statementis honest about what a streaming writer can do. Unbuffered —the default — bytes already handed to the underlying writer stay written, and
the docs say so rather than implying otherwise.
WriterConfig::buffer_statementsholds each statement back until the producer commits it, making rollback real
at O(widest statement); that is the switch
--continue-on-errorturns on. TheTurtle run state (
;/,folding) rolls back with it, so a rejected statementleaves no half-open subject.
One residual case, inherent rather than an oversight: a directive arriving
mid-statement force-commits that statement, so a later
abort_statementcannot retract it even in buffered mode. A
@prefixis a top-level thing andmust flush whatever precedes it — it cannot be written inside an unterminated
statement, or inside a TriG
GRAPHblock — so the statement is terminated andreleased before the directive goes out. Producers do not do this (Turtle
directives sit between statements), but the protocol permits it and the public
API allows it, so it is stated. Output is valid in both modes; the only thing
lost is the ability to un-emit that one statement.
Fidelity tiers and blank-node policy
Turtle and TriG implement the blocks tier: consecutive same-subject runs
fold with
;, same-subject-and-predicate with,, document order otherwise.No cross-document regrouping, no
[ ]reconstruction, no( )re-collapse —all three need the whole document in memory, which is what a streaming writer
exists not to need. riot draws the same line. A buffered
--prettytier isthe follow-up. TriG writes quads where they arrive, reopening a
GRAPHblockwhen the input returns to a graph it left — valid TriG, and O(1) against
interleaved input where buffering per graph would be O(dataset).
Blank nodes relabel by default: a fresh
b{N}per input label, with_:fdb-…passing through verbatim for #1432's addressability contract.Disjointness is by construction rather than by check.
--preserve-bnode-labelsemits user-written labels verbatim; a label thatis not a
BLANK_NODE_LABELwas never written by a user, so it is an internalmint or an externally-sourced identifier and is relabelled into a reserved
fdbw-namespace instead. The label check is the whole production, not anapproximation:
_:ab.does not fail to parse — it lexes as_:abfollowed bythe statement terminator, so the node would be silently renamed.
3. W3C conformance harness and parser burn-down (11 commits)
A new excluded workspace,
testsuite-rdf, plus a CI job. It reads theexisting
testsuite-sparql/rdf-testssubmodule through a relative path, soone checkout serves both harnesses at one commit.
The RDF 1.1 Turtle and N-Triples suites gate: every test must pass or be
listed in
tests/registers/mod.rs, policed in both directions — anunexpected failure and a stale register entry each fail the suite — so the
register can only shrink. The RDF 1.2 suites and the ingest-default mode run
in the same invocation and are reported as informational, never gated.
The harness landed at 297/313 (94.9%) with 16 Turtle failures under six
causes. This branch clears five:
;as empty predicateObjectList items@baseresolutionRDF 1.1 Turtle 297/313 → 309/313 (94.9% → 98.7%). N-Triples is unchanged
at 55/70 — every one of its remaining failures needs a strict N-Triples
reader, which is M1 scope, not a parser bug. The 4 Turtle tests still
registered are all cause C (IRI and language-tag validation), which is the
H-8 "IRI validation ships in the light crates" workstream and was deliberately
left alone. Register entries were removed in the same commit as each fix,
which the both-way policing requires.
Mode delta, since the harness asserts conformant mode never scores below the
ingest default: ingest-default Turtle 272 → 281 (86.9% → 89.8%),
conformant Turtle 297 → 309, and
ParserOptions::conformantis now worth28 tests, up from 25. No test regressed in either mode — every delta
is a test moving from fail to pass.
The two that are more than a test count
A3 was silent data corruption, described in the lede. One detail worth
recording: the observed
<:foo/a3>has a leading colon that nothingprepends. With a scheme-less base,
fluree_vocab::iri::parse_iri_componentsfinds no
:, reports an EMPTY scheme, and treats the whole string as thepath; RFC 3986 §5.3 recomposition then unconditionally emits
scheme ":",which prints as a bare
:when the scheme is empty. Verified directly ratherthan reasoned about — the shared resolver is behaving as specified for an
input that cannot occur once the base is guaranteed absolute, so there is no
second bug and the resolver needed no change.
The same corruption was also reachable through the API without any syntax at
all:
parse_with_prefixes_base[_options]installed itsbaseargumentverbatim, and the CLI's
--baseflag feeds that exact entry point. A relativeparameter is now refused, not resolved — the directive has a document base
in scope to resolve against, but a caller-supplied base has no context, so
resolving would mean inventing one.
D was an IR defect, not a comparison artifact — the boolean duality in the
lede. Fixed exactly the way integers and doubles already were
(
NumericStyle::PreserveLexicalroutes the keywords through the typed-stringlane), so
Term's equality and hash are untouched. The ingest default isunchanged: booleans stay native
LiteralValuevalues on the storage path,pinned by a test.
Harness review fixes (the four leading commits)
shrinking: drop half a manifest and the survivors still score 100%. Each
suite now declares
expected_total(313 / 70 / 96 / 70) and a mismatchfails with a message naming the re-count step, which doubles as the
submodule-drift tripwire.
landing in
failedand breakingtotal = passed + failed + ignoredprecisely in runs that were already failing.
git_sha()appends-dirtyandConformancecarriesdirty: bool— the flag servesstructured consumers, the suffix means a comparator that knows nothing about
the flag still refuses the run.
tag case-insensitive for term equality. Folded during normalization in the
CHECKER ONLY — the IR is untouched, since the parser's job is to preserve
what the document said.
Judgment calls a reviewer should check
NumericStylecovers booleans without being renamed. Booleans are notnumerics, so the name is now slightly narrow for what it governs. Renaming
is a breaking change to a type on the feat(graph-ir): fallible GraphSink protocol, quad capability, statement lifecycle + Turtle ParserOptions conformance #1552 base that other workstreams
already build against; the documentation was widened instead.
GRAPHwas left case-sensitive. It is TriG's keyword, no test hereexercises its case, and widening it unverified would be a guess.
which is what the grammar production and the failing test are about. No W3C
test exercises a dot run in the prefix, so it was left rather than changed
unverified — flagged because the two productions share a rule.
4.
fluree rdf— check, count, profile (5 commits)RDF syntax tooling that reads files rather than ledgers: no
.fluree/lookup,no ledger, no connection.
checkparses and reports diagnostics;countreports statement and term tallies;
convertis registered as a named stubthat refuses with a message naming what is missing rather than "not
implemented".
One syntax table.
RdfSyntaxnames all eight target syntaxes, not justthe readable two, so unsupported input is refused by name with what it waits
on, instead of being sniffed into Turtle and failing thirty lines in.
Resolution is explicit flag > extension > content sniff, with the compression
suffix stripped first and resolved independently.
detect.rsis untouched —it still serves insert/upsert/validate and the server; unifying them is a
follow-up.
Exit codes split "the document is bad" (1) from "the invocation is bad"
(2), which is what makes these verbs wrappable. Transparent
.gz/.zstonboth files and pipes; gzip is decoded multi-member so
pigz/bgzipoutputdoes not silently truncate, and the stream is opened exactly once — a pipe
cannot be reopened at byte 0. Input is capped at 4 GiB, the parser's
u32token-span limit, with chunked reading deferred to the parallel pipeline.
The profiler is honest about its own instrument
--profile(human) and--profile=json(schemafluree.rdf.profile.v1) bothgo to stderr so piped output stays clean. This part went through three
correction rounds under review and the result is worth reading before trusting
any number it prints.
The sink usually cannot be measured, and says so. Timing every sink call
costs two clock reads per event, and on these verbs a clock read costs more
than the work it measures — a bracketed discard-sink call is about 20 ns, of
which 19 ns is the clock. Review found the first implementation's sink phase
was ~98% instrument artifact and stamped "trusted". Now: events are counted
exactly and timed on a jittered sample of statements, the clock's own cost is
subtracted from the sample before it is scaled up, and if what remains
does not clear 3× the extrapolated artifact the tool reports "below the
measurement floor" rather than a number — on the documented host, ~96 ns per
call, which is where the clock's own ~32 ns/pair stops being separable from
the work. The JSON says
null, never0, so it cannot be read as "free".Sampling is jittered, because a fixed stride is a residue class. A fixed
127-stride reaches only
P/gcd(127,P)of a periodic corpus's shapes; on aone-fat-statement-per-127 corpus that was a 425× over-report when the fat
shape was sampled and a total miss when it was not. The gap is now drawn from
1..2×SAMPLE_STRIDE, seeded from the corpus, so a given input still samplesthe same statements across runs.
But the estimate assumes a cooperating corpus, and the docs now say so.
The schedule is a deterministic function of the input — that is what makes two
profiles comparable — and both
corpus_seedand the generator are public, soanyone choosing the input can replay the schedule and park the expensive work
in the gaps. A reviewer did exactly that and hid 315 ms. Reproducible and
unpredictable are incompatible, and reproducible is the one a baseline needs,
so the guarantee is scoped to accidental structure and stated as never
covering an untrusted corpus.
Two trust verdicts, not one.
phases_trustedis driven by the clock readsactually taken;
sink_trustedby the extrapolated artifact. They are separatebecause the second is false on essentially every
count— a discard sink'sartifact is a large share of a fast parse — and one combined flag that is
always false is a flag nobody reads. A regression gate should key on
phases_trusted, and the docs say so in the field table.git_shanames the build, not the shell. It shelled to the workingdirectory, so a binary built from one commit and run from another checkout
reported the wrong SHA — worse than having no field, since a baseline would be
attributed to code that never produced it. It now runs
git -C <dir of current_exe()>and honestly says"unknown"for a binary installed outsideany checkout.
Also: a costly
finish()is measured exactly rather than passed through thesample scaling (routed through it, a 50 ms writer flush was reported as
6.4 s); phase shares are of wall clock, not of the phase sum, so unclaimed
time shows as
unattributedinstead of being redistributed; and the workedexample in the docs is real output from a release build, since a debug
binary calibrates a reader roughly 20× low. Peak-RSS figures ship with their
fixtures (
scripts/rdf-rss-fixture.py) because the ratio depends on corpusshape — 6.7× with every subject distinct, 1.9× with a hundred reused — the
excess being the parser's IRI cache, which grows with distinct IRIs rather
than bytes.
fluree rdf convert— the serial conversion pathThe verb the effort exists for. Input (file or stdin,
.gz/.zst) → Turtleparser in conformant mode → a writer chosen by output syntax → file or stdout.
Format pairs:
{turtle, ntriples}in →{turtle, ntriples, nquads, trig, jsonld}out. Quad input waits on the M2 parsers. Output syntax resolves--to>-oextension > N-Quads, matching riot's documented default.--bnode-policy relabel|preserveand--prefixes <JSON|PATH>reach thewriters; the latter accepts a JSON-LD
@contextdocument unchanged and meansone thing across syntaxes — Turtle and TriG compact with it, JSON-LD takes it
as its context.
Conformant parser options are mandated, not preferred. Under the ingest
default a collection arrives as indexed list items, which every writer refuses
because an indexed list item is a Fluree storage shape with no RDF
serialization. A test pins the observable consequence: the
rdf:first/rdf:restspine, andrdf:nilfor().Two properties a user has to know before pointing this at real data
JSON-LD buffers the whole document, and it costs about 10× what the
streaming syntaxes do. Its writer is document-at-once by construction — the
format has no streaming form short of NDJSON, which is deferred — so the whole
graph is held and nothing reaches the output until the input is exhausted.
Measured on the shipped
scripts/rdf-rss-fixture.pycorpus, release build:peak RSS 2100 MiB for JSON-LD against 197 MiB for N-Triples on the same 22 MiB
input, ~95× against ~9×. There is also no early pipe exit:
--to jsonld | head -5parses everything first, so the trick that makes--to nt | head -5cheap does not apply. This is the same call riot makes and the same one
RDF/XML will make; it is documented, with a reproduction recipe, not fixed.
A mid-stream failure truncates
-o. Refusals that need no input — anoutput syntax with no writer, a malformed
--prefixesnamespace,--pretty—are raised before the destination is opened and leave the file untouched.
Anything discovered mid-stream, a parse error or a writer refusing an event,
happens after
-owas created and written to, so it now holds a prefix of theconversion. A streaming converter cannot honestly promise otherwise; writing
to a temporary and renaming would trade that for double the disk and no
streaming to the final path. The error says which case you got.
Telemetry: how
serializeandwritebecame measurableThe review had already ruled out a per-emit clock (at ~96 ns/call the
instrument costs more than the work).
writeis instead measured at thedestination, behind a 64 KiB buffer — writer →
BufWriter→TimedWriter→file — so the writer's small writes cost no clock and the phase is real I/O
time at a few clock reads per document.
serializeis then the sink estimateminus
write, and is absent entirely when that estimate does not clear itsmeasurement floor rather than being invented from a number that is not there.
A decomposition supersedes the
sinkrow instead of sitting beside it, sonobody adds a total to its own parts.
Two disclosures that belong with those numbers:
serializecarries about 17%. Differential truth is ~0.83× of thereported figure — bracketed timing on a superscalar core measures something
slightly different from marginal wall time. It is presented as a
measurement, so the uncertainty travels with it.
TimedWriteris!Send. It shares its clock throughRc<Cell<Duration>>, chosen because the serial driver is single-threadedand an
Arc<Mutex>would put a lock on the write path. The parallel bucketneeds either a per-worker clock aggregated at join or an atomic swap;
recorded here so it is not discovered at thread-spawn time.
Round-trip fidelity, scoped to what is actually true
An earlier version of this branch asserted byte-identical N-Triples across a
Turtle round trip as a general invariant, "deliberately stronger than
isomorphism". It is not general, and review caught it. Blank-node labels
survive only when the parser mints them in the same order the writer emits
them, which holds for flat shapes and fails the moment they nest — the parser
mints outermost-first, the writer emits deepest-first because an inner node
must exist before the triple referencing it. The fixture carrying the claim
happened to be flat.
The suite now splits: flat shapes keep byte equality with the reason written
down, and nested shapes (collection-of-collections, triple nesting, nested
anonymous nodes — the reviewer's fixtures) assert isomorphism and that
relabelling really occurs, so the stronger claim cannot be quietly reinstated.
The isomorphism checker is lifted from the review's rdflib oracle and has its
own test proving it can distinguish graphs; without that, every round-trip
assertion above it would have been vacuous.
Error semantics
0converted, and also a closed downstream pipe — every layer reportsEPIPEon
| headand the right answer to all of them is to stop quietly, which isriot's behaviour.
1the input did not parse.2the invocation or thedestination was wrong. Broken-pipe detection is on
io::ErrorKind, never onthe message: glibc translates
strerrorunderLC_MESSAGES, so a substringcheck passes for whoever wrote it and fails for everyone else.
--continue-on-erroris not here. It needs statement-scoped outputbuffering wired to a flag —
WriterConfig::buffer_statementsalready makesabort_statementa true rollback — plus a decision on the exit code whenstatements were skipped.
Review provenance
Each of the four lines passed multi-round adversarial review with independent
verification before integration, and the fixes are folded in here rather than
listed as future work. Concretely, review changed the shipped result in ways
worth naming: the sink estimator was rebuilt after a reviewer proved it was
measuring the clock; a reviewer's hand-built corpus defeated the sampling and
the docs were rescoped rather than the claim defended; the
@basefix wasfound incomplete through the API and extended; two probe files authored by
review are adopted verbatim and credited in-tree
(
fluree-graph-ir/tests/quad_rollback_adversarial.rs,fluree-graph-turtle/tests/dataset_parser_parity.rs); and the byte-for-bytepin on the lexer's messages was written and shown green against the
unmodified lexer first, so it proves the output is unchanged rather than
documenting whatever it became.
CI does not fire on this PR
The base is
feat/graphsink-protocol(#1552's branch), notmain, and thisrepo's CI triggers only on
main-based PRs. Nothing here has been exercisedby GitHub Actions. The full local battery below stands in for it, and CI will
fire for real when the stack rebases onto
main.nextest— graph-ir, graph-format, graph-turtle, graph-json-ld, fluree-vocab, db-transactnextest -p fluree-db-api(full — the shared-abstraction blast radius)nextest -p fluree-db-cli -p fluree-db-docsrdf::)cargo test --doc(graph-ir, graph-turtle, graph-format)testsuite-rdf(cargo test)cargo check --workspace --all-targetscargo check --all-targets— excludedtestsuite-sparqlcargo check --all-targets— excludedtestsuite-shaclcargo clippy --all-targets --no-deps— graph-ir, graph-turtle, graph-format, db-cli, db-api, bench-supportcargo fmt --all --checkFollow-ups recorded, not silently dropped
--prettybuffered fidelity tier for Turtle/TriG (H-8).convert --prettyrefuses today rather than quietly emitting blocks-tier output under a flag
that promised more.
--continue-on-errorforconvert: the writers already support thestatement buffering it needs; the driver wiring and the skipped-statement
exit code are what is left.
-o out.nt.gz), which is refused today rather thanwriting plain text into a file whose name promises gzip.
Sendwrite clock (above).JsonLdWriteris triple-only, becauseformat_jsonldcannot express adataset — refusing beats reproducing the read-side adapter's silent
@graphdrop.
ECHARtable) belongs with thecanonverb in M5.
emit_list_itemhas no quad form (M2, above).registrations, and belongs to the H-8 light-crate IRI validation workstream.
detect.rsandRdfSyntaxshould become one syntax table.conversion pipeline.