feat(rdf): M1 completion — dataset formats, validation, parallel convert, competitor harness - #1569
Open
aaj3f wants to merge 40 commits into
Open
feat(rdf): M1 completion — dataset formats, validation, parallel convert, competitor harness#1569aaj3f wants to merge 40 commits into
aaj3f wants to merge 40 commits into
Conversation
… event
W-quad left this decision to whoever owns the protocol, and a real TriG
reader is what forces it. Under CollectionStyle::Spine — the conformant
shape — a collection is ordinary triples, so a collection inside a named
graph already routes through emit_quad and nothing was missing. Under the
indexed style, which exists for Fluree ingest, a list element is metadata
on the edge with no triple form: a producer parsing
GRAPH <g> { :s :p ( 1 2 ) }
had no way to say WHICH graph the collection belonged to, and
emit_list_item put it in the DEFAULT graph. Silent cross-graph data
movement, and unreachable until now only because no quad producer existed.
Follows the established pattern exactly:
- `emit_quad_list_item(s, p, o, index, graph)` with a default body that
debug_asserts the capability and then ERRORS, like emit_quad — a sink
that takes quads but cannot place an indexed item in one must say so
rather than relocate it.
- PROTOCOL_QUAD_EVENTS 1 -> 2, which is what forces the routing sweep; the
test that pins it fails until every dataset sink is updated.
- DatasetCollectorSink routes to the named graph, refusing a literal graph
name for the same reason emit_quad does.
- TimingSink forwards it (a decorator that dropped it would break the
chain); AnyWriter dispatches it.
- The N-Quads and TriG writers refuse it through the SAME
`refuse_list_item` helper the triple form uses — the problem is the
indexed item, not the graph. Overridden explicitly rather than left to
the default, because those writers do claim quad support and the refusal
should sit beside the capability. JSON-LD needs nothing: it does not
claim quads at all.
- `check`/`count`'s DiscardSink counts it, like every other event.
DatasetCollectorSink::emit_list_item's doc comment said this gap was "a
gap in the protocol, not in this sink" and left the routing pinned by
test. That reasoning is now retired and the comment says so — which is
exactly the follow-up the PROTOCOL_QUAD_EVENTS failure message asks for.
Four tests: the indexed item lands in its named graph and NOT the default
one, the triple form still means the default graph, a literal graph name
is refused, and the default body's refusal is exercised through a sink
that claims quad support without honoring it.
TriG as a MODE of the Turtle parser, not a fork: one lexer, one set of term
and statement productions, `ParserOptions::dialect` selecting the grammar,
for the same reason CollectionStyle is an option rather than a second
parser. `parse_trig` is the conformant preset with Dialect::TriG.
All three block forms: `GRAPH label { … }`, the bare `label { … }` form,
and a bare `{ … }` default-graph block. Graph labels are IRIs and blank
nodes (including `[]`) — deliberately narrower than `parse_subject`,
because a collection or a blank-node property list cannot name a graph and
accepting one would emit its contents into a graph named by a node the
document is simultaneously describing.
The routing is one field. Every emission already funnelled through
`sink_emit_triple` / `sink_emit_list_item`, so `current_graph: Option<TermId>`
set at the block boundary is what makes ALL the Turtle productions — object
lists, collections, property lists, reification — land in the right graph.
It is saved and restored around each block, including on error, so a failure
inside a block cannot leak graph scope into the rest of the document.
The capability probe fires once per named block, not per triple: a
triple-only sink cannot represent a named graph at all, and discovering
that on the thousandth statement would mean the first 999 were already
misplaced. A TriG document whose graphs are all the default graph still
parses into any sink.
The bare-label form is where TriG needs the parser's one token of
lookahead: `label {` and `label :p :o .` diverge only after the label. The
term is parsed and then reinterpreted, and only for the label-shaped
tokens — a `[ … ]` or `( … )` has already emitted by the time we could
look, and neither can name a graph.
W3C rdf11 TriG SYNTAX: 208/209 (99.5%). Two defects this suite caught in
the new code, both fixed here rather than registered:
- a trailing `;` may be closed by `}`, as it already was by `.` and `]`
(trig-syntax-struct-07);
- inside a block, only a blankNodePropertyList subject may omit the
predicate-object list — a bare subject or a bare collection is a syntax
error, not an empty statement (trig-syntax-bad-struct-12, bad-list-03/04,
and both bad-n3-extras cases).
The one remaining syntax failure is `trig-syntax-bad-lang-01`, the TriG
member of the H-8 language-tag validation gap.
The suite's 147 eval tests are registered pending an N-Quads reader: they
compare against `.nq` gold files, so the TriG parser is not what is missing.
They should clear as a block when that reader lands.
A purpose-built line scanner, NOT the Turtle parser in a narrower mood. The character of these formats is what they REFUSE — directives, prefixed names, relative IRIs, `,`/`;` lists, `'''`/`"""` strings, bare numeric and boolean literals, collections — and every one of those is valid Turtle. A reader built by restricting the Turtle parser therefore cannot reject them, which is exactly why 14 W3C negative-syntax tests were unenforceable and registered as cause E. One scanner serves both grammars, with `LineDialect` deciding whether a fourth term is allowed. Parameterizing is honest here: N-Triples IS N-Quads without the graph label, unlike Turtle-vs-N-Triples which differ nearly everywhere. Statements never span lines, which is also the property the parallel pipeline wants — a chunk boundary is any newline. Results, against the pre-registration: N-Triples 55/70 -> 70/70 (78.6% -> 100%) predicted 70/70 N-Quads new -> 87/87 (100%) predicted 84/87 TriG 208/356 -> 352/356 (58.4% -> 98.9%) predicted 351/356 N-Triples beat its own cause-C entry too: the strict language-tag production rejects `"string"@1`, so `nt-syntax-bad-lang-01` cleared without H-8. Both line-format registers are now EMPTY. TriG's 147 eval tests all cleared with no TriG change, confirming the diagnosis that they were blocked on the gold-file reader rather than on the parser. Eval compares datasets: default graphs isomorphic, named graphs paired by name for IRIs and by content for blank-node names. That last part is a documented approximation — a true dataset isomorphism solves one bijection across names AND contents together, so a blank node used both as a graph name and inside a graph would be forced to one image. Nothing in this suite needs it, and the approximation can only over-accept, never over-reject. Cause E is closed and cause F never outlived its diagnosis, so C (H-8 IRI/language-tag validation) is the only cause left with entries: 4 Turtle and 4 TriG, the same four shapes in each. conformance.json gains the nquads and trig rows; all four rdf11 formats gate.
`read_support` said quad input was "waiting on streaming quad readers". The readers landed, so the gate opens and `parse_into` routes by resolved syntax: Turtle and TriG through the token-level parser (TriG being Turtle plus graph blocks), N-Triples and N-Quads through the strict line scanner. Two readers, four formats — and the split is deliberate, not incidental. Reading the line formats with the Turtle parser would accept documents `fluree rdf check` is being asked to reject, which is the whole reason the strict scanner exists. `--base` is not threaded into the line formats. They have no base and no relative IRIs, so there is nothing for it to apply to; passing one alongside an `.nq` input is inert rather than silently mis-applied. Found by running the binary rather than trusting the unit tests: every crate test passed while `fluree rdf convert` still refused `.trig` at the resolver, one layer above the dispatch I had just wired. The round trip is now an integration test through the actual binary — TriG in, N-Quads out, TriG back, asserting the graph names survive as the fourth term and that a default-graph statement keeps exactly three. Before this both legs could only be hand-fed, so nothing proved the two formats agreed. Two tests that pinned the old world are updated rather than deleted: the readable set is now four formats (with a note on why it is two readers), and the "known but unreadable" refusal now uses RDF/XML, since that distinction — nameable versus readable — is what the test protects.
Review note, and correct: the strict line reader transcribed its own copy of `ECHAR ::= '\' [tbnrf"'\]`, which the Turtle lexer already had. One grammar production, two transcriptions — the exact failure mode the shared character-class predicates in chars.rs exist to prevent. Nobody would have noticed until someone corrected one row and the two readers began disagreeing about what a string contains. `simple_escape` and `unicode_escape_value` move to fluree-graph-ir::chars, beside the predicates, and BOTH readers call them. The payload-carrying `\u`/`\U` forms stay scanned locally — they are scanning, not a lookup — but their hex-to-scalar step is shared too, which is where the surrogate/out-of-range rejection lives. No behavior change; the tables were identical. Tests pin the table row for row, pin that characters outside it are NOT escapes (notably `\a`, which C-family grammars have and RDF does not), and pin that a lone surrogate is refused. Also documents the `.nt` input decision at the dispatch: the strict reader, which is riot parity and a deliberate behavior change — a `.nt` file containing a directive, a prefixed name, a bare number or a long-quoted string used to be accepted, because all of that is valid Turtle, and is now rejected, because none of it is valid N-Triples. And closes the L-8 gap from both sides with a real test. The writers fold only CONSECUTIVE same-subject runs, so a subject recurring non-consecutively is emitted as a second block for the same subject; that output now round-trips through the new readers, asserted through the binary. The test first checks the writer actually DID emit repeated blocks, so it cannot silently stop exercising the thing it names, then checks trig -> nq -> trig -> nq is a fixpoint in content. Not in bytes: the first TriG carries a prefix from its source and the N-Quads intermediate has none, so the second writes full IRIs. Prefixes are presentation, not statements.
M1 (HIGH, crash). `unicode_escape` sliced the 4/8-byte hex window by BYTE
index after checking only that the bytes existed. A multi-byte character
straddling that window is not a char boundary, so the slice panicked:
$ fluree rdf check panic.nq # "\u0éé"
thread 'main' panicked at nquads.rs:307:30:
end byte index 33 is not a char boundary; it is inside 'é'
exit 101
Reachable from a one-line file, in both escape positions (literal and IRI,
which share the scanner), with 2-byte and 4-byte characters alike.
M2 (MED). `u32::from_str_radix` accepts a leading `+`, so `\u+041` decoded
as `A`. To answer the question in the brief: the 795cabc7e sharing did NOT
already fix this. It moved the hex-to-scalar step into
fluree-graph-ir::chars, but the permissiveness was inside that step. The
Turtle lexer was never exposed because it gates on `is_hex_digit` while
scanning and so never hands a signed payload over; the strict reader sliced
first and converted second, and inherited it. That asymmetry is exactly the
"one grammar, two behaviors" hazard sharing was supposed to remove, so the
gate goes in the SHARED function — both readers now refuse it, and neither
can drift back.
One idea closes both: require exactly `width` ASCII hex digits BEFORE
slicing or converting. Hex digits are ASCII, so proving the window is all
hex digits also proves it ends on a char boundary — the crash and the
grammar leak have the same root, which is why one check retires both.
Verified through the BINARY, and reproduced before fixing (the standing
broken-fix rule): the panic fixtures exited 101 beforehand and exit
EXIT_DOCUMENT_INVALID now, `\u+041` decoded to "A" beforehand and is
refused now, and `A\U0001F600` still decodes to "A😀". All 21 of the
reviewer's fixtures run panic-free.
The regression tests assert an EXACT exit code rather than `!success()`,
because a panic also fails `success()` — the weaker assertion would have
passed against the bug it exists to catch.
Also from the review:
- M3: `the_triple_form_of_a_list_item_still_means_the_default_graph` ->
`a_list_item_without_a_graph_term_means_the_default_graph`. The old name
encoded the retired reason ("the protocol has no quad form"); the
protocol has one now, and the surviving reason is the absent graph term.
- M4: the `sink_bias_probe` example's `FullSink` decorator now forwards
`emit_quad_list_item`. Examples get copy-pasted, and a decorator that
silently drops an event is the worst thing to copy.
- M5: the register comment claimed "all 147 eval tests cleared". Wrong
arithmetic — 3 of the 4 entries below it ARE `TestTrigNegativeEval`. The
true statement is all 143 POSITIVE eval tests, and 144 of the 147 eval
family; the 3 that remain are ill-formed-IRI tests, not eval-machinery
tests, which is why the reader did not clear them.
- M7: the conformance artifact's policy string now records that the
informational rdf12 N-Triples row moved 18 -> 19 when the strict reader
landed — an RDF 1.2 test that needed a real N-Triples reader, not an RDF
1.2 capability gain.
The grammar checks a document's SOURCE. Neither grammar constrains what a
`\uXXXX` escape expands to, nor what base resolution produces — and RDF
requires the result to be a term. That gap is the whole of burn-down cause C.
`iri::iri_violation` checks a RESOLVED IRI: the `IRIREF` exclusion set
(`#x00-#x20 <>\"{}|^`\\`) and then absoluteness. Exactly that set and no
more — U+007F is deliberately NOT excluded, because the production does not
exclude it, and over-rejection is the standing hazard for a conformance
parser: negative tests cannot see it and only the positive suites catch it.
`lang::language_tag_violation` checks the `LANGTAG` production,
`[a-zA-Z]+('-'[a-zA-Z0-9]+)*`. That is BCP 47's shape without its length
rules and without a registry, and stopping there is the point. Enforcing
subtag lengths would reject tags the RDF grammars accept; registry validity
is a data question needing a registry this crate has no business carrying.
The name says well-formedness, not validity.
Both return an enum carrying the offending character and its BYTE offset
within the checked string, and both are allocation-free on the passing path.
The offsets are into the checked value, not the source — expansion and
resolution change the length — which the docs say, because a caller reporting
a source position must use the token's own start instead.
Closes burn-down cause C. RDF 1.1 Turtle reaches 313/313 and its register is now EMPTY; N-Triples drops its one genuine miss and keeps only cause E, the 14 tests waiting on a strict N-Triples reader. `ParserOptions::validate` is OFF by default and ON in `conformant()`. Off is the load-bearing half: bulk import runs the default on the 2M-statements/sec path and consumes documents this database wrote, so it must not pay for a scan of every resolved IRI. Pinned as behavior, not as a flag — the ingest test parses each of the five bad documents successfully. Validation runs AFTER escape expansion and base resolution, because that is where the problem is: `<http://ex/ >` is legal source denoting a non-IRI. It reaches every position a term does — subject, predicate, object, datatype, and prefixed names via the expanded name, which is where a bad `@prefix` namespace surfaces. Errors are `TurtleError::Parse`, so they carry a byte offset and render as a located diagnostic like any other. The offset blames the TOKEN, not the character: expansion changes the length, so an index into the resolved value maps back to nothing. The message carries that finer index instead. Language tags are validated in the parser, NOT the lexer. `@BASE` has to keep lexing as a LangTag so the parser can reject it in directive position — Turtle's `@`-directives are case-sensitive, which is cause B, already fixed. A lexer-level check would undo it. A leading U+FEFF is now consumed. Consumed, not stripped: token spans are absolute offsets into the original source, so removing the three bytes would shift every one of them — tested by asserting a diagnostic still lands on the right line and column with a BOM in front. Only at the start; elsewhere U+FEFF is a zero-width no-break space and a real error. One writer test needed correcting rather than the writers themselves. The round-trip harness re-parsed writer output in conformant mode, which now rejects the IRIs those tests exist to pin — terms that came from a STORE, which never promised to hold only grammatical IRIs (export.rs has always been able to emit one). The re-parse is a decoder, not a judge, so it no longer validates; conformance is covered where it belongs.
…path check, count and convert already parsed in `ParserOptions::conformant`, so term validation switched on for all three the moment the preset carried it. This makes the opt-out explicit and labels it. `--nocheck` turns validation off and nothing else — the grammar is still the grammar, pinned by a test that feeds it a syntax error and still expects exit 1. It exists for measurement and for input you already trust, and the help text says so rather than advertising it as a speed knob. The three verbs share one `verb_options()`. They must describe the same parse or their answers stop agreeing with each other, and threading the flag as part of the options is what makes a divergence impossible rather than merely unlikely. `--profile` output carries `validated`. A --nocheck run and a validating run are not comparable measurements, and every other RDF tool worth benchmarking against validates by default — so an unlabelled --nocheck number would be a faster answer to an easier question. Added as a field, not a schema bump, per the schema's own policy.
…ics, validate directives Six review findings on the parser half of H-8. F1. The comment above the prefixed-name check claimed validating on the cache-miss path was SOUND. It is not, on this branch. `prefixed_term_cache` is keyed by span text and nothing here invalidates it when `@prefix e:` is redeclared, so a second `e:x` returns the first one's term and never reaches the check at all. Now stated as a DEPENDENCY on fix/turtle-prefix-redefinition rather than as a property this branch has. The test that pins the two together needs both branches and is noted as wave-3 integration work. F2. `check_iri` and `check_lang` interpolated their subject raw. The offending character is by definition one the grammar forbids, and the two worst damage the report rather than the data: an expanded newline ends the message's first line, truncating it wherever a caller reads a headline, and an expanded NUL makes captured stderr binary — at which point grep answers "no match" for text that is right there. This session hit that trap twice in committed source. Both messages now use `escape_debug`. F5. `check_iri` now runs on the `@base` and `@prefix` directive values themselves, not only on terms built from them. A document that declares a non-IRI namespace and never uses the prefix used to pass `check` clean, and `check`'s answer is "this is valid RDF". A bad base is worse than inert: every relative reference in the document resolves against it. F6. Recorded a known limitation instead of carrying it silently: `"x"@base` and `"x"@prefix` are well-formed LANGTAGs this parser rejects, because the lexer decides those are directive keywords before anything knows a string precedes them. No W3C test covers either (neither is a registered subtag) and fixing it means giving the lexer the parser's context. Deliberate non-fix. F7. Corrected the NotAbsolute reachability note: M2 built a SEPARATE strict N-Triples reader rather than a profile on this parser, so the predicate's justification holds but by a different mechanism than the comment described. Optional perf item, taken. A verbatim `<...>` token with no base in force needs no validation at all: the lexer already scanned that span with `is_iri_char`, and the branch condition established absoluteness. An `IriSource` parameter distinguishes lexed spans from `\uXXXX`-expanded text, so the skip cannot reach the escaped case — which is the whole turtle-eval-bad class. Licensed by the new all-codepoints differential: if `is_iri_char` and the forbidden set ever drift, that test fails and this skip stops being sound. A behavioral guard also pins that validated and unvalidated parses agree on every base-less document with lexed IRIs.
… and de-NUL a fixture Two review items in the writers' crate, which is the only one that depends on every copy of the set. The IRIREF exclusion set is transcribed independently in more than one place, and the copies serve opposite directions: the reader decides what to ACCEPT, the validator what to REJECT, the writer what to ESCAPE. Drift between any two is silent and asymmetric — a character the writer emits raw but the reader refuses produces output this project cannot read back, and one the validator blesses but the writer escapes changes an IRI's identity on the way out. Each copy already had its own hand-picked unit tests, which is exactly how all three could look tested while disagreeing. So the agreement is now asserted at EVERY codepoint, not over a list: the reader's accept-set against the validator's reject-set, the writer's actual escaping behaviour (through the public entry point, not the private predicate behind it) against the same set, and the set itself against the production the grammar spells. This is also what licenses the base-less validation skip on the turtle side — if the two predicates drift, this test fails and that skip stops being sound. Recorded rather than quietly omitted: `fluree-db-sparql` carries a FOURTH copy, byte-identical today, which this cannot reach — a light crate cannot depend on the SPARQL engine. It is the copy most likely to drift, because nothing binds it to the others. Separately, roundtrip.rs carried a literal NUL byte in a Turtle fixture where the neighbouring escapes are written out. Replaced with the `\u0000` escape the document meant: a Turtle string literal may hold a NUL either way, so the parsed value is identical and the test still checks that the writer escapes it — but the source file is text again rather than binary to grep and `file(1)`. The differential is the review's; the fixture fix is impl-writers'. Credited to both.
The whole argument for putting `validated` in the JSON report is that an unlabelled --nocheck number is a faster answer to an easier question, and every other RDF tool worth benchmarking against validates by default. A human reading the table is at least as likely to quote a figure from it, so scoping the label to --profile=json contradicted its own rationale. Shown only when validation is OFF. The default is validating, so a line on every ordinary run would be noise.
…e-placement note Comment-tier addendum from the spot-verify. No behavior changes. N1. The justification for escaping language-tag diagnostics was BACKWARDS, in three places. It said the lexer accepts more than the grammar; the lexer's scan admits only `[a-zA-Z0-9-]`, which is NARROWER than the grammar in characters and wider only in SHAPE — digits first, leading, trailing or doubled `-`. Verified against the parser: `@1`, `@e1`, `@en-`, `@-en` and `@en--gb` reach the predicate, while `@`, `@en_GB`, `@en.GB` and an accented tag are all LEXICAL errors that never do. Two consequences the docs now state rather than imply: from Turtle, only `NonAlphabeticPrimary` and `EmptySubtag` are reachable, and the `Empty` variant's doc example (`"x"@`) was illustrating something the lexer refuses outright. `NonAlphanumericSubtag` is likewise unreachable from this reader. The escape stays. It is free on a short ASCII string, `language_tag_violation` is a shared predicate whose other callers bring their own lexers, and M2's separate N-Triples reader may be looser — so: unreachable through the Turtle lexer today, kept for the shared predicate and future readers. Renamed `a_language_tag_diagnostic_is_escaped_too` to `a_reachable_language_tag_diagnostic_stays_single_line`, which is what it actually pins, and extended it to assert the other half — that the characters which would need escaping die in the lexer. N2. Softened `with_no_base_lexed_iris_behave_identically_validated_or_not` to say what it is: a smoke test. While `is_iri_char` and the forbidden set remain complements, no document exists that it could distinguish, so it cannot fail while the skip is sound. The real guards are named instead — the all-codepoints differential in `fluree-graph-format`, which fails on drift, and the inertness of the branch, which has composed no string for a scan to have an opinion about. Third item, sited where the next person will look (on `check_iri`): validation runs parser-side, BEFORE `sink_term_iri`, so it fires once per resolved occurrence — while `iri_term_cache` dedupes by resolved IRI, meaning a sink-side move would silently change error multiplicity from per-occurrence to per-unique-term. Moot today and verified so: the parser fails fast, and three copies of one bad IRI produce a single error at the first offset. It becomes live only if `--continue-on-error` lands and validation moves behind that cache, where a document repeating one bad IRI would report it once and a user counting diagnostics would under-count.
…nner The pre-scan path found statement boundaries with a byte-level state machine that tracks strings, IRIs and comments. The streaming path — the one bulk import and parallel conversion actually use — approximated it: split at a line ending in `.`, with an ad-hoc toggle for triple-quoted strings. The two disagreed in three ways, and every one of them was silent. - **A line-final comment ending in `.` looked like a terminator.** `ex:a ex:p "v" ; # see RFC 3986 sec. 5.1.1.` ends with a period, so the chunk was cut mid-statement and neither half parsed. Test: chunk 0 came back "expected Dot, found Eof". - **`'''` inside a short string toggled the long-string flag.** An odd number left it stuck on and every later boundary was refused, so the rest of the file became one chunk however large it grew. Test: 1 chunk where 2+ were due. (An even number self-cancels, which is why the fixture spells out the parity.) - **Mid-file `@prefix`/`@base` was not detected at all.** Only chunk 0 carries the directive block, so a redefinition meant every later chunk resolved the prefix against the *header* binding and produced wrong IRIs with no error. `PrefixAfterData` existed but only the pre-scan path looked. Now both do, and the streaming reader surfaces it so a caller can fall back to serial. All three are now tests, written against the old scanner and shown failing first. The per-byte logic — pending-dot confirmation, CRLF, triple-quote resolution that needs the current state to know whether three quotes open or close — is extracted into `BoundaryScanner`, and both paths drive that and nothing else. Two implementations of one question is what allowed them to diverge. A capability falls out: with no line scan, a newline is incidental, so a dump written as one long line now splits. It could not before — the scanner searched for `\n`, found none, and accumulated the whole document. Gates: fluree-graph-turtle 27 splitter tests (was 24), turtle+transact+ir+ format 779, FULL fluree-db-api 2923 — the bulk-import path this sits under.
…s chunks
Workers parse chunks concurrently into per-chunk collector sinks; one thread
replays them into the real writer in document order. `--parallelism` (the
existing global flag) selects it; `--parallelism 1` is the serial path exactly,
so the flag is never a correctness decision.
THE WRITER STAYS SERIAL, and that is a correctness requirement rather than a
simplification. Blank-node identity is document-wide, so output labels must be
assigned by something that sees the whole document in order. Two failure modes
sit on opposite sides of it:
- A *labelled* `_:x` in chunk 1 and chunk 7 is ONE node (Turtle scopes labels
to the document) and must reach one output label. Independent per-chunk
relabellers split it whenever the chunks hold different numbers of earlier
blanks.
- An *anonymous* `[]` in chunk 1 and another in chunk 7 are TWO nodes and must
not share a label. Every collector mints from the same counter, so both are
`-b1` by the time the replayer sees them, and one relabeller merges them.
`ChunkScopedBlanks` renames only the mints, into `-c{chunk}_{n}`: the leading
`-` cannot begin a Turtle BLANK_NODE_LABEL, so no user-written label can
collide with a mint however adversarial, and the chunk index makes two chunks'
mints disjoint. Labelled nodes pass through untouched, because those must
unify.
Differential gates, all mandatory and all present: byte-identical serial vs
parallel at 2/4/8 workers; a labelled node spanning chunks staying one node;
300 anonymous nodes staying 300; and the adversarial fixture whose own labels
imitate every mint pattern in the system (`b1`, `c0_1`, `g0_1`, `u1`,
`fdbw-1`) — 307 distinct nodes, no merges. Verified by mutation: dropping the
per-chunk scoping fails four of them. Worth noting that the plain
serial-vs-parallel differential still PASSED under that mutation, because its
fixture has no anonymous nodes — which is exactly why the dedicated ones are
separate tests.
Parallel runs only for line-based output. Turtle and TriG fold consecutive
same-subject runs, so a chunk boundary inside a run ends the fold early: the
output is still correct and still isomorphic, but not byte-identical, and byte
equality is the gate. Oxigraph draws the same line. The fallback is silent in
effect but not in report — `--profile` now carries `threads_used` and a
`parallel_reason`, because "why is this not using my cores" otherwise has no
answer from outside the process.
Also: two phases, `workers` (parse time summed across threads) and
`reassembly` (ordered replay). Neither joins the sequential total —
`Phase::SEQUENTIAL` now names the three phases that actually run one after
another, and `workers` exceeds the wall clock on any run that scales. That
correction also fixes `unattributed_ns`, which the serialize and write phases
had been silently consuming since convert landed.
Skip the statements that do not parse, keep the rest, report every skip, and exit 1. A converter that quietly drops input is worse than one that refuses, so this is opt-in and the exit code still says the conversion is partial — riot's rule, and the one thing a script must not do is read a partial conversion as a whole one. A clean document under the flag exits 0 silently. Recovery is: parse, and on failure record a diagnostic, find the next statement boundary after the error, and parse again from there. Two things make that sound rather than merely plausible, and both are mutation-verified. **A rejected statement contributes nothing.** The parser emits during descent, so `ex:bad ex:p "one" ; ex:q "two" ; ex:r ??` has two triples in the writer before the failure is known. The flag turns on `WriterConfig::buffer_statements` so `abort_statement` is a true rollback. Turning that off fails `a_rejected_statement_contributes_nothing_even_when_it_emitted_first` and nothing else — the fragments reach the output while every other test stays green, which is exactly the shape of bug the test exists for. **Prefixes survive the restart.** A resumed parse begins after the `@prefix` block and would resolve every prefixed name against nothing, so `PrefixRecorder` captures directives as they pass and re-seeds each restart. Re-seeding is not replaying: the parser takes them as bindings without emitting `on_prefix`, so the writer does not redeclare what it has already written. `splitter::next_statement_boundary` is the resync primitive, built on the `BoundaryScanner` extracted in the chunker commit — plan §1.5 asked for exactly that, and it now has one caller that needed it. It always advances, which is what makes a document of nothing but errors terminate rather than spin; there is a test for that and one for a trailing unterminated statement. Diagnostics are positioned against the whole document rather than the fragment the resumed parse saw, because a user counting lines is counting the file's. Recovery is serial: resync needs the document as one sequence of statements, so `--continue-on-error` overrides `--parallelism` and `--profile` reports why. Documented, including the limit — a directive skipped over as part of a bad statement is lost, and the statements needing it then fail in turn, each reported and none silent.
…tart
Addendum gap, confirmed as a failing test first. `PrefixCheck` only tried to
match `@prefix`/`@base`/`PREFIX`/`BASE` when it believed it was at the start of
a LINE, so a directive written mid-line evaded the guard the previous commit
had just added:
ex:a ex:p "1" . @Prefix ex: <http://second.example/> . ex:b ex:p "2" .
Turtle has no line structure — that is an ordinary document, and the
redefinition is exactly as corrupting to chunk as one at column 0, because only
the first chunk carries it and every later chunk resolves `ex:` against the
header binding.
The flag now means "a token could begin here", set at the start of input and
after any whitespace. Being in `ScanState::Normal` is what already kept a
keyword inside a literal or comment from matching, so the fix does not widen
detection into strings — there is a test pinning that a `@prefix` written
inside a literal is still just text, and it passed before this change and
after it.
The architecture the G1b measurement called for. Each worker runs its own
writer into a byte buffer and the driver concatenates in chunk order; there is
no shared relabeller and therefore no serial replay, which was ~half the wall
clock of the collect-then-replay design this replaces.
What makes that sound is the label scheme, which is a pure function of (label,
chunk) so workers need no coordination:
- a user's `_:L` becomes `u{L}` — prefixing is injective, so distinct labels
stay distinct, and the same label in two chunks maps to one output label,
which is what document scoping requires;
- an anonymous node in chunk `c` becomes `g{c}_{n}` — disjoint across chunks
by `c`, within a chunk by `n`;
- `_:fdb-…` passes through verbatim, keeping #1432's addressability contract.
The three classes are disjoint by first character rather than by check, and a
user label that imitates a mint is no threat: `_:g0_1` becomes `ug0_1`.
Reassembly is bounded, not "join everything then write". Workers deposit into a
channel whose capacity is at least the worker count — so a worker can never
block holding the chunk the writer is waiting for — and out-of-order arrivals
wait in a reorder buffer. Peak memory is roughly (workers + capacity) × chunk
output, not the whole output.
THE DIFFERENTIAL GATE IS NOW THREE LAYERS, replacing cross-mode byte equality,
which this design deliberately gives up (riot makes no cross-mode byte promise
either):
(a) determinism — same input, same K, byte-identical across runs. Thread
scheduling cannot reach the output, because labels depend on chunk index
and chunks concatenate in order.
(b) equivalence — serial and parallel agree on triple count and on the
number of distinct blank nodes, and a labelled node spanning chunks is
still one node. The byte-identical collect-then-replay implementation is
kept as the oracle side.
(c) adversarial — ten bait labels imitating every pattern the scheme itself
produces (`b1`, `g0_1`, `g7_3`, `u1`, `ug0_1`, `ufdb-x`, `fdbw-1`,
`c0_1`), each keeping its own identity.
The parallel path reports no sink estimate rather than a fabricated one: each
worker has its own writer and no single instrument saw them all.
… load average Two things, both following from the ruling. **`WriterConfig::declare_prefixes`** (additive, default on). Turtle and TriG declare the prefixes they compact with, which is what makes their output readable — so the default can never be the suppressed one, and a test pins that. Off exists for exactly one caller: chunk-parallel conversion, where several writers each produce a fragment of one document. Every chunk re-parses the header prelude, so every chunk's writer would emit the same `@prefix` block into the middle of the output. Chunk 0 declares for the whole document and the rest suppress. Compaction is untouched by the knob — the prefix map is still populated, only the directives are withheld — and there is a test for that, because a chunk that declared nothing AND expanded every IRI would defeat the point of chunking Turtle at all. `finish` has its own declaration site for the empty-document case; it honours the knob too, with its own test. So the parallel path now covers every text syntax. A subject whose statements straddle a chunk boundary comes out as two subject blocks rather than one: valid blocks-tier output, since the tier already declines to regroup a subject that recurs later, and documented as a consequence rather than left to be discovered. **Load average in the profile.** Adopted as permanent methodology after a scaling sweep on this repo's own hardware measured 1.46x while the load average was 65 on 16 cores — the serial baseline, running unchanged code, had slowed by 3.1x, so the number was contention rather than architecture, and nothing in the report said so. Every run now records `host.load_average_1m`, and the human output prints a `LOADED` line when the average exceeds the core count: a duration measured on a contended machine is not a measurement, and this is the only way to tell after the fact. Docs: a parallelism section stating what is guaranteed (per-mode byte determinism; serial/parallel equivalence in triples and distinct blank nodes) and what is not (cross-mode byte equality — riot promises none either), plus the prefix-once and split-subject consequences and the memory trade.
…, and whitespace-anchored Three ways a mid-document directive slipped past the chunker, each producing silently wrong IRIs rather than an error, and each now a fixture. **Case (H2).** Turtle §6.5 makes the SPARQL-style `PREFIX`/`BASE` forms case-INsensitive while the `@`-forms stay case-sensitive, and the lexer implements exactly that — its own test proves `prefix` and `Prefix` lex as directives. The chunker compared every keyword case-sensitively, so a lower- or mixed-case redefinition was ordinary data to it: the directive reached the first chunk and every later chunk resolved against the header binding. Now matched per keyword's own rule, with a fixture for all eight spellings. **Single candidate (found while fixing the above).** The matcher committed to one keyword on the first byte and could not backtrack. `@prefix` and `@base` share `@`, so `@base` was unreachable — it mismatched `@prefix` at byte 1 and reset. A mid-document `@base` therefore evaded detection entirely, which the eight-spelling fixture caught immediately. Candidates are now a bitmask of everything still consistent with the bytes seen. **Token boundaries (M3).** `at_token_start` was set only by whitespace, but Turtle requires none around its punctuation: `ex:a ex:p "1".@Prefix ex: <…> .` is legal and evaded the previous widening. The flag now follows any real delimiter — whitespace plus `. ; , ] ) >` and `"`. The complementary property is pinned too: `BASELINE`, `ex:baseline` and `ex:prefixed` are still not directives, because a keyword match needs a delimiter to confirm it.
…k to serial H1, the default-path deadlock. `convert big.ttl | head` hung forever at 0% CPU for any input over the parallel threshold. The receiver lived OUTSIDE the scope closure, so a write failure returned from the closure while workers were still blocked in `send` on a full bounded channel — and `thread::scope` then joined them, forever, with nobody draining. The serial path's early-termination contract has to survive parallelism. Now the writer owns the receiver, and on any write failure it sets a shutdown flag (so workers stop before parsing another chunk rather than converting the rest of a 4 GiB file for nobody) and drops the receiver (so workers already blocked in `send` wake and exit). The failure carries its `io::ErrorKind` out through a dedicated `ParallelFailure` rather than being flattened into a string, because flattening is what lost the information the broken-pipe check needs: a closed downstream is a *successful* end to `| head -5`, and must exit 0 quietly. The missing coverage is added, at parallelism 4 and 8: a reader that never reads, one that takes a single line, and one that stops mid-stream — all exiting 0 — plus a bound on elapsed time, so a regression fails the test instead of hanging until nextest's kill. M1, spec conformance. A mid-file directive now falls back to SERIAL instead of aborting. §1.4 always said fallback, and aborting was the wrong reading of it: the document is legal Turtle and must convert. Only the *chunking* is impossible, because a redefinition would reach the first chunk and nothing after it. Chunking therefore happens before the parallel path is committed to, and its refusal downgrades the plan; the note is printed and the reason reaches `--profile`'s `parallel_reason`. This also shrinks the blast radius of any directive-detection gap that remains. M2. `--continue-on-error` no longer suppresses `--profile`. They were mutually exclusive by accident, and recovery is exactly when profiling matters, since resync re-parses from every error. Under `--profile=json` the per-skip diagnostics would make stderr unparseable, so the count travels inside the document as `skipped_statements` instead.
…hanging
The H1 coverage relied on `wait_with_output`, which is exactly the wrong shape
for a hang: a deadlocked child does not fail the test, it freezes the suite.
nextest prints SLOW and kills after 6 minutes, `cargo test` waits forever, and
neither reads as "the pool is deadlocked".
The wait now happens on a second thread with a 90s ceiling — an order of
magnitude over a healthy debug-build run of this fixture — and expiry kills the
child and panics with the case name. `wait_with_output` still drains stderr, so
a chatty child cannot be mistaken for a hung one.
The matrix widens to --parallelism 1/4/8 × {immediate close, head -1, head -100,
close mid-read}, with 1 as the serial baseline that always honoured the
contract. `-o FILE` at the same three widths is the negative control: it never
sees EPIPE and passed before the fix, so it separates "termination broke" from
"the pool broke", and it checks that ending quietly on a dead pipe did not cost
ending correctly on a live one.
The fixture now asserts it exceeds MIN_PARALLEL_BYTES. An undersized one
converts serially whatever --parallelism says, which is how a green suite missed
a deadlock on the default path.
Verified by mutation: restoring the pre-fix receiver shape fails the matrix at
`--parallelism 4 × immediate close` in 10s with the control still passing.
Keeping that ownership but dropping the shutdown flag does not hang, which
places the flag correctly — it governs how promptly workers stop, not whether.
Two flags asked for opposite things and the parallel path won without saying so. Workers rename every blank node into the coordination-free (label, chunk) scheme — that is what lets them write bytes with no shared relabeller — so a user's `_:named` came out as `_:unamed` at parallelism ≥ 4: byte-identical to relabel, with the flag silently ignored. The writer's refusal to preserve a label inside its own reserved `_:fdbw-` namespace disappeared with it, because the renamed label no longer collided: a run that must exit 2 exited 0, with labels the user never wrote. Preserve now downgrades to serial, following the mid-file-directive precedent rather than refusing: the user asked for label fidelity, serial delivers it, and the cost is speed rather than correctness. The condition is a parameter of `ParallelPlan::decide` rather than a check at its one call site, so a second call site cannot be added that forgets, and it sits ahead of the syntax and size checks because it is about correctness rather than benefit. `--profile` reports it as the `parallel_reason`, since a silent downgrade is the defect that field exists for. Coverage, all three failing without the fix: the policy test extends to parallelism 4 and 8 over an above-threshold fixture and demands the serial preserve bytes exactly; the collision refusal is re-asserted at parallelism 8 (exit 2, naming the label); and the profile carries `threads_used: 1` with a reason naming the flag. Also: `fixture()` now refuses a name differing from an existing one only by case. On APFS and NTFS `base.ttl` and `BASE.ttl` are one file, so a matrix over `@base`/`@BASE` that names fixtures after the spelling would test the last twice and the first never, and nothing would fail — the fixture would simply overwrite its sibling. An exact repeat stays allowed.
The instrument from riot-analog-bench-strategy.md §3 + §6b. Scripts, not criterion; not in CI; run on demand at milestones. Nothing it produces is publishable, and the tooling is built to make an accidental publication hard rather than to trust that nobody will. tools.lock.json pins riot 6.1.0, serdi 0.32.10, oxigraph 0.5.9 and rapper 2.0.16 with sha256s computed by downloading each artifact, not copied off a page. A --version mismatch REMOVES a tool from the matrix rather than producing an unlabelled column: the reason this lane's numbers rot is that nobody records which build made them. Locally that gate already earns its keep — riot 5.4.0 and oxigraph 0.4.6 are refused, serdi and rapper run. corpora.lock.json carries the §6b F2 replacement set with the reasons the original two choices were dropped (RiverBench ships NT only, so a Turtle column would have meant benchmarking every tool on input shaped by our own writer; sp2b's generator is gone and its rationale was void). Nothing is vendored: dbpedia-live is CC-BY-SA and DBLP republishes in place, so the fetcher records a hash on first download and hard-fails if the bytes later change, which keeps one matrix internally consistent without pretending upstream froze for us. Three measurement traps handled rather than assumed: Peak RSS units. Darwin's getrusage(2) returns ru_maxrss in bytes where Linux returns kibibytes, so the obvious portable fix is a macOS divide-by-1024. That is wrong for GNU time's output, which normalizes before printing — applying it makes every macOS figure 1024x too small, and "1.5 MB peak RSS for a parser" is plausible enough that nobody would question it. Established by measurement, not by reasoning: `bench-harness calibrate` allocates a known 200 MiB and checks the reading. (This build had the wrong assumption first; the calibration is what caught it.) Binary translation. On an Apple-silicon Mac under Rosetta, uname -m reports x86_64, so every result would be stamped with hardware that was not involved and the harness's own clock calls would be emulated. Detected via sysctl.proc_translated, stamped as -translated, and disqualified from publication — recorded, not corrected. Checking parity. riot validates Turtle by default and N-Triples not at all, and our parser does no IRI validation, so a default-vs-default matrix would race a validating run against a non-validating one and call the difference performance. riot runs at both --check=false and --check=true, and the lock already carries our `validate` mode so the comparison is never quietly asymmetric while W-h8 is outstanding. The correctness gate runs alongside speed, not after it: normalized cross-tool diff plus independent rdflib isomorphism, with rdflib's absence reported BY NAME rather than skipped silently. Its normalizer has zero fluree dependencies on purpose — routing both sides of a differential through our own parser would let a bug in our reader cancel itself out. The gate is verified in both directions; a deliberately dropped triple is caught at both levels. differential.sh measures writer cost as convert-minus-count with both sides un-instrumented, because --profile's sink lane has a ~99ns/call floor that is the same order as the thing it would measure. It reports the combined MAD next to the delta and says outright when the delta does not exceed it. .gitignore narrowed from a blanket /benchmarks/ to /benchmarks/* plus a negation, since git cannot re-include a path inside an excluded directory and the strategy puts this harness in-repo at exactly that path. Corpora, fetched tools and run outputs stay untracked. Open and recorded in the README rather than implied: conformance gating (F8) is NOT wired and the run manifest says so in a field; cold-start cells are unimplemented; our own cells need `fluree rdf convert`, which lands with the writers.
A single U+0000 committed inside a Rust string fixture compiled, passed its
own test, and survived cargo test, cargo fmt --check, clippy and the full
fluree-db-api suite. NUL is valid UTF-8 inside a raw string, so nothing in the
toolchain had reason to object.
What it broke was search. file(1) classifies such a source as `data`, and grep
and ripgrep then report NO matches for the whole file — not an error, no
diagnostic, just nothing. A reviewer grepping it is told the symbol does not
exist. That is cheap to introduce and expensive to find, which is the shape of
defect worth one walk of the tree.
Walks `git ls-files -z` and rejects every byte below U+0020 except tab, LF and
CR, plus DEL. Byte-oriented rather than char-oriented so a file that is not
valid UTF-8 is still examined instead of skipped. Exemption is by extension
and the list is deliberately short: the repo's goldens are TEXT and are absent
from it on purpose, because an invisible byte in a golden is exactly what
should fail. The per-path exemption list is empty and should stay so — a path
on it is a file nobody can grep.
Sited beside workspace_reconcile, following that repo-structure-test
precedent.
KNOWN RED, by design. This commit carries the DETECTOR only; both offenders it
finds are owned elsewhere, so the two concerns stay in separate hands:
fluree-graph-format/tests/roundtrip.rs:372 -> h8 fix-pass (impl-quad)
fluree-db-api/src/admin.rs:283 -> main-targeted prefix-fix PR
(impl-quad rider commit)
The admin.rs one predates this effort and is live on origin/main, so that file
is ungreppable on main today; it was found by this test's first run rather
than by anyone reading it.
Both fixes were verified locally before being handed off: with them applied
this test passes 4/4, the 664 db-api lib tests are unchanged (the admin.rs one
is inside a comment), and fluree-graph-format stays at 121/121 with the
corrected fixture. The guard goes green at the integration rung once both land.
Review fix-pass. The design held; the measurement path did not. CLOCK (item 1, the one that invalidates everything else). Timestamps were taken with `python3 -c` on each side of the interval, putting TWO interpreter startups INSIDE every measured region — 26-40ms the pair, measured. On a corpus where serdi does ~10ms of real work that is not a perturbation, it is most of the number, and being near-constant it compressed every ratio toward whichever tool was slowest. Replaced with bash's EPOCHREALTIME builtin, which forks nothing. The brief preferred GNU time's own child-only elapsed field, and I did not use it as the primary clock: measured directly, `gtime -f %e` reports "0.00" for serdi on the smoke corpus. It is centisecond-resolution, which is two orders coarser than the thing being timed. It is now recorded per cell as a cross-check for corpora large enough to register, labelled as corroboration rather than measurement. Verified by timing a known 500ms sleep through the harness path: 0.517s vs 0.533s on the old path. ~15ms of constant removed, matching the reported +12.1ms. Same fix in differential.sh, where it mattered more than anywhere: a differential subtracts two medians, so the constant cancelled but its jitter did not — it inflated both MADs and made real deltas fail their own significance test. LOAD (found while verifying the clock fix, not in the brief). The fixed clock still reported serdi at 21ms against a directly-measured ground truth of 10ms. The residual was not the clock: sibling build jobs had the load average at 10 on a 16-core box. A harness that cannot tell "this tool is slow" from "this machine was busy" is not measuring the tool, and median-and-MAD does not rescue it — load inflates every sample in the same direction, so the median moves with it. The runner now records load per core, warns loudly above 0.5, and marks the run non-publishable. Same treatment as Rosetta: recorded, not corrected. NORMALIZER (item 3, and it was inventing equalities). It collapsed whitespace runs everywhere, so `<http://ex/a b>` and `<http://ex/a b>` compared equal — two different resources fused, on precisely the population of IRIs the escaping rules concern. Rewritten as a term tokenizer: whitespace between terms is not content, whitespace inside a term is. The guard test that was supposed to catch this asserted norm(line)==norm(line), which cannot fail; it now asserts the two DIFFERENT IRIs stay different. UCHAR (item 4a). `\uXXXX` and `\UXXXXXXXX` now decode to their code point in literals AND in IRIs, which is the grammar's own equivalence — raptor emits the escaped form for non-ASCII where others emit the character, so without this every accented term in a real corpus is a false mismatch. An escaped backslash followed by text that merely looks like a UCHAR is not one, and is tested. BLANK NODES (item 4b, as ruled). Level 1 exempts statements mentioning a blank node and reports how many it skipped; level 2 (rdflib isomorphism) owns them. No canonicalization in the zero-dep normalizer: deciding whether `_:b0` here is `_:x` there is graph isomorphism, and faking it would give confident wrong answers on the one question a line transform cannot answer. CORRECTNESS GATE (items 2, 6). It passed zero-triple outputs — two empty files trivially agree. Cells are now checked against the corpus manifest's ground truth first, cells whose status is not ok are EXCLUDED from comparison rather than compared, and the single-tool path FAILS instead of reporting "nothing to cross-check" at exit 0. One tool agreeing with itself verifies nothing. Both previously-passing failure modes are now verified to fail. Also: exact version patterns, so 0.32.4 no longer satisfies a 0.32.10 pin (item 5); mismatch reports name the build actually found rather than the pin it failed to match (5); .part+mv atomic downloads (7); the dead duplicate `dest=` assignment removed (8); bash-5 guard with the reason, since EPOCHREALTIME is a builtin and macOS still ships 3.2 as /bin/bash (9). RULINGS applied: reading notes now state that our validate cell becomes fillable at wave-3 integration and that the primary comparison then flips to check_true-vs-validate per H-8; the README voids every pre-clock-fix number outright — re-measure, do not rescale.
Addendum, from W-parallel-2's correction to the translated-shell finding.
The correction matters and I had it wrong in the disqualifier text. A translated
shell does NOT translate its children: an arm64-only child launched from a
Rosetta parent still runs native. So "the shell is emulated" does not license
"these numbers are emulated", and the renderer said so. Reworded to state what
IS true — the harness's own per-sample overhead is emulated — and to point at
the new per-cell ARCH column instead of inviting the inference.
The real hazard is a COMPETITOR shipping x86_64-only, because then one column is
emulated and another is not, and a matrix reports the emulator as a performance
difference. Every result record now carries the architecture class its
executable actually runs as, and the renderer marks a run whose cells disagree
as non-comparable — the same disqualification tier as host translation, proven
by test.
Two of the five classes are admissions rather than answers:
translated-x86_64 single x86_64 slice on an arm64 host: emulated
universal-indeterminate several slices, and which one the kernel chose is
not observable from outside the process — a universal
binary launched from a translated parent does not
reliably pick the native one
riot lands in the second: it is a shell script over a universal JVM
(x86_64,arm64e). That is worth knowing given riot is the named target, and it is
now printed rather than assumed. Measured on this host: serdi, rapper and
oxigraph are all single-slice native-arm64, so the serdi-vs-rapper cells that
have been produced so far were native-vs-native.
Also adopts W-parallel-2's load methodology permanently: load per core is
recorded per CELL at measurement time, not once per run. A build that starts
midway through the matrix contaminates only the cells after it, and a single
run-level figure would average that away.
…bookkeeping Closing items. All four bite the publication run specifically, which is why they land now rather than being left for it. N1, and it is the worst failure mode in the harness so far. Every timing is derived by STRIPPING EPOCHREALTIME's decimal separator, so under a comma-decimal locale the strip is a no-op, awk parses "1785358343,127180" as 1785358343, and every cell reports 0.000000 seconds with status ok. Reproduced before fixing: a 300ms sleep measures as "0,000000 s" under LC_ALL=de_DE.UTF-8. lib/common.sh now exports LC_ALL=C and LC_NUMERIC=C, which also makes sort and diff byte-ordered for the cross-tool comparison. Belt and braces: require_dot_decimal_clock asserts the property the arithmetic actually relies on rather than trusting the export upstream, and is verified to fire when the locale is re-imposed after sourcing. N2. lipo is macOS-only, so on Linux every real binary fell through to `unknown`, which the renderer treats as not-confirmed-native — a permanent disqualification banner on the one host the arch check exists to bless. Added an ELF branch keyed off `file -b`, classifying x86-64 and aarch64 against the host arch, so both the current x86_64 candidates and the arm64 m8gd locus resolve. `unknown` stays as the honest fallthrough for anything neither branch recognizes. N3. The exempt-count capture swallowed normalize's exit status, so an unreadable cell produced an empty normalized file and was reported downstream as a content DIFFER — a wrong diagnosis of a real problem, and the more expensive kind of wrong because it sends the reader looking at the data instead of the disk. Now checked: a non-UTF-8 cell reports "normalize FAILED: stream did not contain valid UTF-8" and is excluded rather than compared. N4a. The bash-5 guard now runs in all seven scripts, not just the two that read the clock. macOS still ships 3.2 as /bin/bash and the failure without the guard is not a clean error. N4b. The ground-truth check recounts the output FILE instead of trusting the runner's own out_statements field, and flags a disagreement between the two as a BOOKKEEPING MISMATCH distinct from a wrong count. A producer's claim about its own output is not evidence about that output; both paths are verified to fire.
Turtle lets a prefix be rebound part-way through a document. The parser
ignored the rebinding for any prefixed name it had already seen, so
@Prefix e: <http://a/> . e:x <http://p/> "1" .
@Prefix e: <http://b/> . e:x <http://p/> "2" .
put BOTH statements on <http://a/x>. No error, no warning — the document
says one thing and the graph holds another.
Cause: `prefixed_term_cache` is keyed by span text (`e:x`), which identifies
an IRI only relative to the bindings in force when the entry was made, and
nothing invalidated it when a binding changed. `iri_term_cache` was never
affected — it keys on the RESOLVED IRI — and neither was `@base`, since a
namespace is resolved once, when its directive is read. A test pins both, so
the scope of this fix is demonstrated rather than asserted.
Fix: on a prefix directive, clear the expansion cache IF the prefix rebinds to
a different namespace. Only if. Redeclaring the same binding is the common
case, not a curiosity — chunked import prepends the file's prefix block to
every chunk over a pre-seeded map, so the parser sees each prefix declared
twice by construction — and every cached expansion stays correct through it.
The per-term path is untouched: the diff adds no instruction to
`resolve_prefixed_term` or `sink_term_iri`. Added work is one hash lookup per
prefix DIRECTIVE, of which a document has a handful, against a cache consulted
once per prefixed NAME, of which it has millions. Clearing is coarse — a
rebind strands only that prefix's entries, but finding them means scanning
every key — and rare enough that O(cache) once beats a per-lookup epoch check
on the hot path.
Ten tests, eight of which fail without the fix; the two that pass either way
are `@base` and same-binding redeclaration, which is what shows the scope is
right. Found while wiring term validation for H-8; nothing about it is
specific to validation and it reproduces on plain `parse`.
Review fold. The fix was right; what it lacked was anything that could tell if half of it were deleted. THE GAP: `bind_prefix` clears the expansion cache only when a prefix actually rebinds, and that "only when" was invisible in parser output. Clearing on a same-binding redeclaration yields identical triples — just colder — because `iri_term_cache` absorbs the repeated work before it can reach the sink. So a mutation dropping the condition passed every gate in the PR. Confirmed by running it: with the clear made unconditional, all eleven triple-level tests still pass. The guard is the review's `parse_reporting_cache` — test-only, returns the prefixed-name cache hit/miss counters — plus two tests reading them: a same-binding redeclaration must leave the cache warm (1 hit, 1 miss), a real rebinding must drop it (0 hits, 2 misses). The same mutation now fails the first one. That is the whole point: the counters are the only place this behaviour is observable. Also documented, from the review: the comparison in `bind_prefix` is on MEANING, not spelling. `parse_prefix_directive` resolves the namespace before storing it, so `prefixes` holds resolved IRIs — and both directions of that matter. Same spelling under two different bases names two namespaces and MUST clear; two spellings under one base name one namespace and must NOT. Comparing raw text would get each backwards. It was load-bearing and undocumented. Two review probe files adopted as permanent regression tests, credited: - `prefix_redefinition_adversarial.rs` (9 tests) attacks what the fix's own tests could not reach, because they never vary the base: the missed-clear and spurious-clear cases above, A→B→A epochs with overlapping and disjoint names, the coarse whole-cache clear with twenty live prefixes, terms nested in `[ ]` and `( )`, escaped local names, a pre-seeded relative binding, and mixed directive spellings. - `prefix_redefinition_blast_radius.rs` (2 tests) executes the PR's blast-radius claim instead of asserting it, and now quantifies it: of the post-redefinition statements, 0 are mis-resolved inside the chunk carrying the redefinition — the fix holds there — and 1438 are mis-resolved in later chunks, which is the §1.4 chunker gap this PR explicitly does not close. It also confirms the guarded pre-scan path WOULD reject the file, which is what makes "import uses the unguarded reader" the operative fact. That 1438 is an assertion of a live defect, kept live on purpose rather than `#[ignore]`d: an ignored test runs nowhere and rots, while this one fails the moment §1.4 lands and tells whoever closed it to come flip the number to 0. Wording fold: "a rebind invalidates only that prefix's entries, but finding them means scanning every key, so we drop them all", used consistently.
…he fix
The blast-radius test's `wrong_in_carrier == 0` proved nothing. Verified
before fixing: with `bind_prefix` deleted outright — the parser reverted to the
merge base, no fix at all — BOTH tests in the file still passed.
The corpus was the problem. Subjects were `e:s{i}`, a distinct span every time,
and predicates were full IRIs, so no prefixed name was ever looked up twice.
The expansion cache was never consulted for a span it already held, which is
the only shape in which a stale entry can be served. The test walked the whole
blast radius and never touched the defect it was named for.
Every line now carries `e:tag` as its object. It is cached under the first
binding and requested again after the rebinding, so a stale cache serves the
old namespace for a span whose meaning has changed. The new assertion counts
post-redefinition triples inside the CARRIER chunk — the one that actually
contains the redefinition — whose object still resolves to `http://a/tag`:
fix removed (parser at merge base): 379 -> assertion FAILS
fix present: 0 -> assertion passes
`wrong_in_carrier` stays 0 in both cases, which is the demonstration that it
was never the discriminating assertion.
Kept unchanged: the subject-index chunk attribution, and the §1.4 tripwire on
later chunks. The tripwire's count moved 1438 -> 1621 with the corpus, which is
why it remains a printed diagnostic and an inequality rather than an equality —
ordinary chunking changes must not make it flap.
Found by an independent adversarial review of #1565. Two doc corrections ride
along: the meaning-vs-spelling cases are pinned in the ADVERSARIAL file (they
need a moving `@base`, which the other file never varies), and the clearing
comment now states the real trade — `retain` scans every key too, so the cost
is the same order; what it would buy is keeping other prefixes warm, and what
it costs is a predicate that has to compare up to the colon, since the obvious
`starts_with(prefix)` evicts `ex:name` when `e` rebinds. Simplicity over
post-rebind hit rate, not cost over cost.
admin.rs carried a literal NUL byte at line 283, inside a comment that meant
to NAME the character: "callers can otherwise sneak a `<NUL>` past". The
neighbouring lines spell the same class correctly (`U+0000..=U+001F`,
`U+007F`), so this is the one that got away.
The cost is not cosmetic, but it is grep-dependent, and the honest version
names which grep. A single NUL makes the file binary to text tools —
`file(1)` reported it as `data` rather than source — and the search behaviour
then splits:
* ugrep 7.5.0 (what `grep` resolves to in this dev environment) exits 1 and
prints NOTHING for a pattern occurring 23 times in the file. Not "binary
file matches", not a warning: a clean, confident "not found".
* BSD grep 2.6.0-FreeBSD at /usr/bin/grep — GNU-compatible, and what CI and
most contributors get — prints "Binary file ... matches" and exits 0. It
withholds the matching lines but does not lie about the match.
So the severe reading applies to anyone whose `grep` is ugrep, and the milder
one to everyone else; both are worse than a file that just works. `file` now
reports UTF-8 text and both greps find all 23. The earlier version of this
message claimed the silent-exit-1 behaviour universally, which was true only
for the shell it was measured in — corrected on review.
Comment-only and byte-precise: one line changes, the NUL replaced by the text
`U+0000`. 670 lib tests unchanged.
Found by the control-byte guard from the RDF-toolkit work, which flagged this
as its second literal NUL — the first was in a Turtle test fixture. Riding
here rather than standing alone because a one-byte PR against main is not
worth its own review slot.
…close `prefix_redefinition_blast_radius.rs` arrived with #1565 from main, where two of its assertions pinned DEFECTS rather than properties, deliberately, so they would fail the moment the chunker workstream landed and send whoever closed the gap here. Both fired on the integration branch. This is that visit. What actually changed is not "the chunker now handles such a file". It REFUSES it: a mid-file directive makes a document unchunkable — only the first chunk would carry the redefinition — so the reader thread raises `PrefixAfterData` and stops. The important part is how that refusal reaches a consumer, because the old test is what made it look alarming. `recv_chunk` reports the closed channel as a benign `Ok(None)`, so a consumer that only drains sees a SHORT READ AND NO ERROR: 1797 of 4000 statements on this corpus. The error lives on the reader thread and only `join()` surfaces it. Every live import call site joins, and `import.rs` carries a comment saying exactly why. The old test drained without joining, which is why it read as truncation. So the replacement asserts the distinguishing fact rather than the symptom: drain, then join, and require `Err(PrefixAfterData)`. That is the assertion that tells "refused" apart from "silently truncated", and it would fail if the join contract were ever dropped. `the_guard_misses_a_mid_line_directive` becomes `the_guard_catches_a_mid_line_directive` — detection now fires at any token start, not just at line start. The parser property the file also carried — a prefixed name reused across a rebinding must follow the rebinding rather than the span cache — was entangled with the chunk-carrier machinery, which no longer exists because there is no carrier chunk. It is now its own test on a document that is never chunked, since it is a property of the parser either way. One latent hazard noted, not fixed: `ImportSource::recv_next` returns `Ok(None)` on a closed channel without joining. It has no callers today; a future one would inherit the silent-short-read shape.
…ams meet The last four entries in any register, and they cleared without a line of TriG code — which is the point worth recording. They were `TestTrigNegativeEval` tests, so they read as eval-machinery failures and were registered under cause F (no N-Quads reader). They were nothing of the kind: three ill-formed IRIs behind `\uXXXX` escapes and one bad language tag — the exact four-test shape the Turtle suite carried under cause C. The strict line reader closed F and left them behind, correctly listed but under the wrong cause; term validation then closed them as C. Validation reaches TriG for free because the TriG parser is the Turtle grammar with a dialect flag, so `ParserOptions::validate` scans graph-block content by construction. That is the argument for one grammar with a dialect knob instead of a second parser, and this is the evidence for it. The register's both-way policing is what forced this commit rather than letting the entries rot: four tests that started passing while still listed failed the suite until the entries came out. All four RDF 1.1 suites are now 100% with every register empty: Turtle 313/313 N-Triples 70/70 N-Quads 87/87 TriG 356/356 `ParserOptions::conformant` is now worth 32 Turtle tests (89.8% -> 100.0%).
Plan Wave-3 MUST-DO. Two readers reached RDF term validity by two different routes: the Turtle parser calls `fluree-vocab`'s `iri_violation` and `language_tag_violation` under `ParserOptions::validate`, while the strict N-Triples/N-Quads reader carried its own absoluteness check and its own hand-rolled `LANGTAG` loop. The m2 review recorded the gap as symmetric — both accepted post-expansion invalid IRIs — so nothing was WRONG in a way a suite could see. Two implementations of one grammar drift, though, and that is what this closes before they do. The IRI half was a real hole, not just duplication. The reader checked only that the result was absolute, and it checked the SOURCE bytes for the IRIREF exclusion set — but `<http://example.org/a b>` is legal source that expands to a space, so the byte scan cannot object and the absoluteness check has no opinion. Validation has to run on the expanded value, which is where the Turtle parser already ran it. Pinned by a test that fails when the check is reverted to the old one. The language-tag half is now scanner-delimits, predicate-decides: take a maximal run of the tag alphabet and let `language_tag_violation` judge it, rather than encoding the production in an acceptance loop. The run is a superset of LANGTAG, so an ill-formed tag reaches the predicate instead of being cut short by the scan and reported as some other error. Validation is unconditional here, deliberately, where the Turtle parser makes it a flag. The flag exists to keep a scan of every resolved IRI off the bulk import hot path, and that path consumes documents this database wrote. Neither applies to a line format: it has no base, no ingest fast path, and an N-Triples document is either a set of RDF terms or it is not one. Also verifies checklist item F7 against the actual reader rather than by argument: the line grammars carry no base machinery, so a relative IRI has nothing to resolve against and `IriViolation::NotAbsolute` is genuinely reachable — in Turtle the same reference resolves and never reaches it. All four RDF 1.1 suites hold at 100% (Turtle 313, NT 70, NQ 87, TriG 356).
The flip the plan predicted: `a_mid_file_directive_falls_back_to_serial_rather _than_refusing` asserted that exactly one term after a mid-file rebinding denotes the new namespace. It is two, and the second one is the fix. The trailing statement is `ex:z ex:p "z" .`, so its subject AND its predicate both sit after the rebinding. Before #1565 only the subject moved: `ex:z` is a span the document had never used, while `ex:p` had been expanded 80,000 times and came back from the span cache under the old binding. Asserting one occurrence was asserting the bug. The count is now two and the last line is pinned whole, so it cannot drift back to one for some unrelated reason. Fold-ins recorded at the wave close, applied here because this is where the branches converged: - The dead-reader comment claimed receiver ownership stops workers and the shutdown flag only makes them prompt. The reviewer's four-way mutation table says otherwise: either mechanism ALONE keeps the run live, and only removing both hangs. The old phrasing reads as though the flag were an optimization, which invites deleting a second liveness guarantee that no single-mutation test can catch. Restated as: either alone suffices, keep both, and the mutation that proves it removes both. - `list_items_land_in_the_default_graph_because_the_protocol_has_no_quad_form` is now `list_items_without_a_graph_land_in_the_default_graph`. The protocol HAS the quad form — `emit_quad_list_item` landed with the M2 work — so the old name asserted a reason that had stopped being true. - `chars::unicode_escape_value` documents the one hazard its signature cannot express: it takes `&str`, so it can never see a partial UTF-8 sequence, but a caller slicing RAW BYTES can panic before reaching it. Both current readers slice a `&str` at their own scanner's positions, so the class is structurally out of reach for them — a third reader over `&[u8]` would have to bound its own slice. The guarantee lives at the call site, not here. - The CLI tests' `EXIT_DOCUMENT_INVALID` was a bare `1`. It now binds to `fluree_db_cli::error::EXIT_ERROR`, so the assertions track the contract instead of a literal that would keep passing if the binary's code changed.
The three-way differential in `fluree-graph-format` sweeps every codepoint to keep the reader's accept-set, the validator's reject-set and the writer's escape-set from drifting — and recorded in its own docs that a FOURTH copy, in `fluree-db-sparql/src/lex/chars.rs`, was deliberately left uncovered because a light crate cannot depend on the SPARQL engine. That copy is the one most likely to drift, precisely because nothing bound it: the SPARQL lexer is maintained on its own, and a change there would have been invisible to the RDF side until a document round-tripped wrong. Bound here instead of lifted. `fluree-db-cli` is the smallest crate that already depends on both sides, so the binding is a DEV-dependency and adds no production edge in either direction — the reviewer's design, and the reason the copy can stay where it is. A lift would have made the SPARQL lexer depend on the RDF light crates to settle a question that a test can settle. The widening it needs is exactly one item: `pub use chars::is_iri_char` in the SPARQL `lex` module, which already re-exports selectively, rather than making the whole `chars` module public. Both directions are asserted — against the reader's accept-set and against the validator's reject-set in the opposite polarity — over all of Unicode rather than a hand-picked list, which is what let four copies disagree while all four looked tested. Verified by mutation: narrowing the SPARQL copy's control range from `\x20` to `\x1F` fails both tests naming `U+0020`. The sibling differential's "deliberately not covered" note now points here.
Reviewers found three separate cases of one flag silently overriding another — `--continue-on-error` x `--profile`, `-o` x `--profile=json`, and `--parallelism` x `--bnode-policy preserve`. Three of one kind is a pattern, and the pattern is that flag interactions were being discovered rather than specified. So `convert.md` now carries a table of every pair that interacts, and this adds a test per row asserting the DOCUMENTED behavior in both the serial and the parallel path. Two of the three defects existed on only one side, which is why every row is exercised in both. The standing rule is in the doc: a new flag adds a row, and if it interacts with nothing it says so, because "nothing" is a claim a reader can check and an absent row is not. Behaviors were established by running the binary rather than read off the source, and two are worth naming: - Under `--continue-on-error` + `--profile=json` the per-skip diagnostics are NOT printed. stderr is one JSON document there, so a diagnostic per skipped statement would make it unparseable; the count travels as `skipped_statements` instead. The test parses stderr rather than grepping it, so a stray line fails it. - `-o` and `--profile=json` never share a stream: the file gets the bytes, stderr gets the JSON, and stdout stays empty. Without `-o` the bytes take stdout and the JSON still does not. One test needed a control to be worth anything. `--continue-on-error` short-circuits the parallel plan BEFORE the input-size check, so asserting "it went serial" would have passed on a one-line fixture. It now asserts the fixture clears the threshold AND that the same corpus without the flag runs parallel, so the serial choice can only be attributed to the flag.
Emptying the last register broke the harness's own negative self-test, which is a good problem and needed a real fix rather than a deletion. `an_unregistered_failure_fails_the_suite` proves the register is enforced in the direction that matters: a test that FAILS while unregistered must fail the suite. It did that by running RDF 1.1 Turtle in conformant mode against an empty register and requiring an error. That worked for exactly as long as the parser had conformance failures. It has none now — 313/313 with an empty register — so the call returned `Ok`, `expect_err` panicked, and the check that guards every other suite was itself red. It now drives `ParseMode::IngestDefault`, whose 32 failures against the conformance suite are a DESIGN property rather than a defect: indexed list items and canonicalized numerics are deliberately lossy as RDF, which is why that mode is reported and never gated. So the self-test is anchored to something nobody is trying to fix, instead of to a parser defect that was always going to be removed out from under it. Caught only because the gate battery runs plain `cargo test` in this workspace. `cargo test --test w3c_rdf` — the invocation used while working on the registers — runs the integration target alone and never compiles the `--lib` tests where this lives.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
M1 completion: dataset formats, validation, parallel convert, competitor harness
Stacks on #1555 (RDF toolkit core). Base:
feat/rdf-toolkit-m0@f6f6857de.Four workstreams plus the parser fix from #1565, integrated into one
branch: 40 commits, 68 files. Together they finish M1.
All four W3C RDF 1.1 suites are now at 100%, and every known-failure register
is empty.
The register is policed in both directions, so those zeros are load-bearing: a
single regression either re-populates a register or fails the suite.
After this branch
fluree rdf convertreads Turtle, TriG, N-Triples andN-Quads, writes all five text syntaxes, validates the terms it reads, recovers
from bad statements, and parses on as many threads as the input allows.
1. Dataset formats — TriG and the strict line reader (6 commits)
TriG is a MODE of the Turtle parser, not a fork. One lexer, one set of term
and statement productions,
ParserOptions::dialectselecting the grammar — thesame argument that made
CollectionStylean option rather than a secondparser. All three block forms are supported (
GRAPH label { … }, barelabel { … }, and a bare default-graph{ … }).The routing is one field. Every emission already funnelled through
sink_emit_triple/sink_emit_list_item, so acurrent_graph: Option<TermId>set at the block boundary is what makes all the Turtle productions — object
lists, collections, property lists, reification — land in the right graph. It
is saved and restored around each block including on error, so a failure inside
a block cannot leak graph scope into the rest of the document.
The capability probe fires once per named block rather than per triple: a
triple-only sink cannot represent a named graph at all, and discovering that on
the thousandth statement would mean the first 999 were already misplaced.
The strict N-Triples/N-Quads reader is deliberately NOT the Turtle parser in
a narrower mood. The character of these formats is what they refuse —
directives, prefixed names, relative IRIs,
,/;lists,'''/"""strings,bare numeric and boolean literals, collections — and every one of those is
valid Turtle. A reader built by restricting the Turtle parser therefore cannot
reject them, which is exactly why 14 W3C negative-syntax tests were
unenforceable and registered as burn-down cause E. One scanner serves both
grammars, with
LineDialectdeciding whether a fourth term is allowed;parameterizing is honest there because N-Triples genuinely is N-Quads without
the graph label.
This is a deliberate behavior change for
.ntinput: a.ntfilecontaining a directive, a prefixed name, a bare number or a long-quoted string
used to be accepted (all of it is valid Turtle) and is now rejected (none of it
is valid N-Triples). That is riot parity, and accepting it would make the verb
agree with no other tool in the field.
--baseis not threaded into the line formats. They have no base and norelative IRIs, so a
--basepassed alongside an.nqinput is inert ratherthan silently mis-applied.
A crash fixed on the way.
unicode_escapesliced the 4/8-byte hex windowby byte index after checking only that the bytes existed, so a multi-byte
character straddling that window panicked (
fluree rdf checkon a one-linefile, exit 101). The same scan also inherited
u32::from_str_radix'sacceptance of a leading
+, so\u+041decoded asA. One idea retires both:require exactly
widthASCII hex digits before slicing or converting — hexdigits are ASCII, so proving the window is all hex digits also proves it ends
on a char boundary. The gate went into the shared function so neither reader
can drift back. The regression tests assert an exact exit code rather than
!success(), because a panic also failssuccess()and the weaker assertionwould have passed against the bug it exists to catch.
Also shared, for the same reason: the
ECHARescape table, which the linereader had transcribed independently from the Turtle lexer.
2. Term validation (H-8, 7 commits)
Both grammars constrain a document's source text. Neither constrains what a
\uXXXXescape expands to, nor what base resolution produces — and RDFrequires the result to be a term. That is precisely the
TestTurtleNegativeEvalclass: the document lexes, and the term it denotes isnot a term. Raw forbidden characters were already rejected by the lexer; only
escaped ones slipped through, which is why this needed a post-expansion check
rather than a tighter grammar.
fluree-vocabgets two pure predicates besideresolve_iri:iri::iri_violationapplies theIRIREFexclusion set and then absoluteness,and
lang::language_tag_violationapplies theLANGTAGproduction. Both areallocation-free on the passing path.
Two deliberate limits, both to avoid over-rejection — the standing hazard for a
conformance parser, invisible to negative tests and caught only by the positive
suites. U+007F is not excluded, because the
IRIREFproduction does notexclude it. No BCP 47 length rules and no registry lookup, because the RDF
grammars impose neither; the function is named for well-formedness, not
validity.
ParserOptions::validateis off by default and on inconformant(), and offis the load-bearing half. Bulk import runs the default on the
2M-statements/second path over documents this database itself wrote, so it must
not pay for a scan of every resolved IRI. It is pinned as behavior — the
ingest test parses all five bad documents successfully — not merely as a flag
assertion.
The error offset blames the token, not the offending character: expansion
changes the string's length, so an index into the resolved value maps back to
nothing in the source. The message carries the finer index instead, and a test
asserts the two genuinely differ.
Language tags are validated in the parser, not the lexer.
@BASEmust keeplexing as a
LangTagso the parser can reject it in directive position —Turtle's
@-directives are case-sensitive, which is burn-down cause B, alreadyfixed. A lexer-level check would silently undo it.
BOM tolerance — an ungated behavior change, disclosed
A leading U+FEFF is consumed, not stripped: token spans are absolute offsets
into the original source, so removing the three bytes would shift every one of
them. Pinned by asserting a diagnostic still lands on the right line and column
with a BOM in front. Only at the start; elsewhere U+FEFF is a zero-width
no-break space and a real lexical error.
This one is NOT behind
ParserOptions, so it changes bulk import too. ABOM-prefixed Turtle file that import previously rejected with "unexpected
character
\u{feff}at line 1, column 1" now loads. That is almost certainly afix — Windows editors emit BOMs routinely, riot and every other production
reader tolerate a leading one, and the old failure told a user nothing useful
about their file. But it is a change in what the shipping import path accepts,
it is not opt-in, and it is not what H-8 was chartered to do, so it is called
out here rather than left for someone to find in a changelog. Gating it would
mean threading
ParserOptionsinto the lexer, which nothing else needs; thealternative shape, if the team prefers strictness, is to reject in import and
accept in the
rdfverbs.--nocheck, and why the profile reports it--nocheckturns off term validation and nothing else, pinned by a test thatfeeds it a syntax error and still expects exit 1. All three verbs share one
verb_options()so they cannot describe different parses.--profilecarriesvalidated, because a--nocheckrun and a validating runare not comparable measurements. Every other RDF tool worth benchmarking
against validates by default, so an unlabelled
--nochecknumber would be afaster answer to an easier question.
3. Parallel convert (10 commits)
Workers produce bytes. Each worker runs its own writer into a byte buffer
and the driver concatenates in chunk order. There is no shared relabeller and
therefore no serial replay, which was roughly half the wall clock of the
collect-then-replay design this replaces.
What makes that sound is a label scheme that is a pure function of (label,
chunk), so workers need no coordination: a user's
_:Lbecomesu{L}(prefixing is injective, so distinct labels stay distinct and the same label in
two chunks maps to one output label, which is what document scoping requires);
an anonymous node in chunk
cbecomesg{c}_{n}; and_:fdb-…passes throughverbatim, keeping #1432's addressability contract.
The chunker now runs the real byte-level scanner. The streaming reader was
line-based with an ad-hoc triple-quote toggle: it split mid-statement on
#inside a line-final comment, mis-toggled on
'''inside short strings, and didnot detect mid-file directives at all. It now drives the same
BoundaryScannerthe pre-scan path uses, and detects a directive at any token start — the
earlier guard was case-sensitive, single-candidate and whitespace-anchored.
A mid-file directive makes a document unchunkable, because only the first chunk
would carry the redefinition. Per plan §1.4 that is a fallback to serial,
not a refusal: the document is legal Turtle and must still convert.
Two bugs found by review, both fixed rather than documented. Parallel
convert deadlocked on a dead destination — the writer stopped draining on
EPIPE while workers stayed blocked on a full bounded channel, and
thread::scopejoined them forever at 0% CPU, so any input over the parallelthreshold piped to
headhung. And--bnode-policy preservesilentlyrelabelled under parallelism; it now forces the serial path and reports why.
--continue-on-erroris serial by construction: resync needs the document asone sequence of statements, and a chunk boundary is not a place a skipped
statement can be reasoned about.
G1b-2 measurement, with its caveats attached
Dev-host numbers at
48444c026, workers-produce-bytes, lower bounds — readthem with every caveat below, not as a claim.
(up from 1.36× under the replaced architecture), reproduced next morning
within 2%.
92.5% of ideal — so plan §1.3 works. The cap is the single-threaded
chunk scan: 2.85 s, K-independent, 19.6 ms/MB dead-linear (~52 MB/s, about
10× slow for a byte DFA). The model
wall(K) ≈ 2.85 + 6.49/widthfits themeasurements.
nothing. The only lever is the pre-pass.
idle floor of ~5.7 is disclosed rather than subtracted; the scan was 43% of
wall and was charged to
unattributed, which is why aPhase::Chunklane isqueued.
Nothing here is a publishable number. Per plan §5/R5, nothing publishes
before the official m8gd run.
4. Tier-2 competitor harness (6 commits)
Out-of-band bench tooling in
benchmarks/conversion/, an excluded workspace soit never lengthens
cargo check --workspacefor anyone else. Its review pass isworth reading as methodology, because every fix bites the publication run
specifically.
The instrument was measuring itself. Timestamps were taken with
python3 -con each side of the interval, putting two interpreter startupsinside every measured region — 26–40 ms for the pair, measured. On a corpus
where serdi does ~10 ms of real work that is not a perturbation, it is most of
the number, and being near-constant it compressed every ratio toward whichever
tool was slowest. Replaced with bash's
EPOCHREALTIMEbuiltin, which forksnothing.
A locale could zero every cell silently. Timings were derived by stripping
EPOCHREALTIME's decimal separator, so under a comma-decimal locale the stripis a no-op, awk parses
1785358343,127180as1785358343, and every cellreports 0.000000 s with status ok. Reproduced before fixing: a 300 ms sleep
measures as
0,000000 sunderLC_ALL=de_DE.UTF-8. The harness now forcesLC_ALL=C, and asserts the dot-decimal property the arithmetic relies onrather than trusting the export.
Architecture is recorded per tool, per cell. The earlier translated-shell
finding was corrected: a translated shell does not translate its children, so
"the shell is emulated" does not license "these numbers are emulated". The real
hazard is a competitor shipping x86_64-only, because then one column is emulated
and another is not and the matrix reports the emulator as a performance
difference. Two of the five classes are admissions rather than answers —
translated-x86_64anduniversal-indeterminate— and riot lands in thesecond, being a shell script over a universal JVM. Measured on this host, serdi,
rapper and oxigraph are all single-slice native arm64.
Ground truth is recounted from the output file rather than trusted from the
runner's own
out_statementsfield, with a disagreement between the two flaggedas a bookkeeping mismatch distinct from a wrong count — a producer's claim about
its own output is not evidence about that output.
Load per core is recorded per cell at measurement time, so a build that
starts midway through a matrix contaminates only the cells after it instead of
being averaged away.
5. #1565's parser fix, and the two tests it flips
A redeclared prefix must change what later names mean. The parser caches
expanded prefixed names by their span text (
ex:name), which identifies an IRIonly relative to the bindings in force — so a stale cache turned a rebinding
into a silent no-op. Fixed on
mainas #1565; the effort branches predate it,and landing it here completes §1.4's serial-fallback story.
It was built to flip two tests, both of which pinned defects rather than
properties so they would fail the moment the chunker work landed and send
whoever closed the gap to them. Both fired, and both are now flipped to the
closed behavior:
the_guard_misses_a_mid_line_directive→..._catches_...: detection nowfires at any token start.
redefinition and delivers it as chunk data. It now refuses it.
a_mid_file_directive_falls_back_to_serial_rather_than_refusingassertedthat exactly one term after the rebinding denotes the new namespace. It is
two, and the second one is the fix: the trailing statement is
ex:z ex:p "z" ., and before fix(turtle): a redeclared prefix must change what later names mean #1565 only the fresh subject moved while thepredicate — expanded 80,000 times already — came back from the cache under
the old binding. Asserting one was asserting the bug.
One thing worth knowing about that refusal, because the old test made it
look alarming. When the reader thread raises
PrefixAfterDatait stops, andrecv_chunkreports the closed channel as a benignOk(None)— so a consumerthat only drains sees a short read and no error (1797 of 4000 statements on the
test corpus). The error lives on the reader thread and only
join()surfacesit. Every live import call site joins, and
import.rscarries a commentsaying why, so there is no data-loss path in the product; the old test simply
never joined. The replacement asserts drain-then-join returns
Err(PrefixAfterData), which is the assertion that tells "refused" apart from"silently truncated".
ImportSource::recv_nextreturnsOk(None)without joining. It has no callerstoday; a future one would inherit the silent-short-read shape. Noted, not
fixed.
#1565's
admin.rsrider is included (a comment-only one-line changereplacing a literal NUL with the text
U+0000). It is not optional here: thebench line ships a control-byte guard over every git-tracked text file, and the
plan records that guard as RED until both NUL fixes land. The other fix is on
the h8 line. Cherry-picking a change that also sits on a branch targeting
mainis harmless — an identical patch is dropped on rebase — and the alternative was
a red gate.
Integration work beyond the merge
The four lines were built in parallel and had never met. These are the commits
that only exist because they did.
The TriG register emptied where two workstreams met. Its last four entries
were
TestTrigNegativeEvaltests, so they read as eval-machinery failures andwere registered under "no N-Quads reader". They were nothing of the kind: three
ill-formed IRIs behind
\uXXXXescapes and one bad language tag — the exactfour-test shape the Turtle suite carried under cause C. The strict line reader
closed the eval cause and left them behind under the wrong one; term validation
then closed them, reaching TriG for free because the TriG parser is the Turtle
grammar with a dialect flag. That is the argument for one grammar with a
dialect knob rather than a second parser, and it is the evidence for it.
Both readers now call the same term predicates (a plan MUST-DO). The Turtle
parser called
fluree-vocab's predicates; the strict reader carried its ownabsoluteness check and its own hand-rolled
LANGTAGloop. The IRI half was areal hole rather than duplication: the reader checked the source bytes for the
exclusion set, but
<http://example.org/a b>is legal source that expandsto a space, so the byte scan cannot object and absoluteness has no opinion. The
language-tag half is now scanner-delimits, predicate-decides. Both are pinned by
tests verified to fail when the wiring is reverted.
The fourth IRIREF transcription is bound. A full-Unicode differential
already tied the reader's accept-set, the validator's reject-set and the
writer's escape-set together, and recorded that a fourth copy in
fluree-db-sparqlwas deliberately uncovered because a light crate cannotdepend on the SPARQL engine. It is now bound from
fluree-db-cli— the smallestcrate already depending on both sides — as a dev-dependency, so it costs no
production edge and the copy can stay where it is. Verified by mutation.
A documented flag-interaction table, with a test per row in both the
serial and the parallel path. Reviewers found three separate cases of one flag
silently overriding another, and three of a kind is a pattern: interactions were
being discovered rather than specified. The standing rule is in
convert.md— anew flag adds a row, and if it interacts with nothing it says so, because
"nothing" is a claim a reader can check and an absent row is not.
The harness's own negative self-test needed re-anchoring. It proved the
register is enforced by running Turtle in conformant mode against an empty
register and requiring an error — which worked only while the parser had
conformance failures. At 313/313 it returned
Okand the check that guardsevery other suite was itself red. It now drives
ParseMode::IngestDefault,whose failures are a design property nobody is trying to remove. It surfaced
only because the gate battery runs plain
cargo test;--test w3c_rdfnevercompiles the
--libtests where it lives.Four smaller fold-ins recorded at the wave closes are applied: the dead-reader
deadlock comment (either mechanism alone suffices — keep both; the mutation that
proves it removes both), a test name that encoded a retired reason, a
byte-slicing caveat on
unicode_escape_value, and binding the CLI tests' exitcode to the real contract instead of a literal.
Review provenance
Each of the four lines went through build → adversarial review → fix → verify
with independent mutation and probe verification before integration, and the
fixes are folded in rather than listed as future work. Review changed the
shipped result repeatedly and concretely: the bench harness's clock was
rebuilt after a reviewer showed it was timing two Python startups; the parallel
path's deadlock and its silent
--bnode-policyoverride were both found byreview, not by the suite; the
\upanic was reproduced from a one-line filebefore being fixed; and the
@basefix from the previous wave was foundincomplete through the API and extended.
CI does not fire on this PR
The base is
feat/rdf-toolkit-m0(#1555's branch), notmain, and this repo'sCI triggers only on
main-based PRs — verified inci.yml, which isbranches: [main]on bothpushandpull_request. Nothing here has beenexercised by GitHub Actions. The local battery below stands in for it.
nextest— graph-ir, turtle, json-ld, format, vocab, transactnextest -p fluree-db-api(full — shared-parser blast radius)nextest -p fluree-db-cli -p fluree-db-docsnextest -p fluree-db-sparqlcargo test --doc(ir, turtle, format, vocab)testsuite-rdf— fullcargo test(lib + suites)conformantis worth 32 testscargo check --workspace --all-targetstestsuite-sparql/testsuite-shacl/ bench harness —check --all-targetstestsuite-rdf—clippy --all-targets -- -D warnings,fmt --checkclippy --all-targets --no-deps× 8 crates (ir, turtle, format, vocab, cli, api, sparql, bench-support)cargo fmt --all --checkOne pre-existing flaky test, disclosed.
fluree-graph-ir::timing::the_sampling_error_bound_is_reported_and_grows_with_dispersioncompares measured dispersion between a uniform and an erratic sink, and under
full-suite parallelism the uniform sink's busy-wait can be preempted enough to
invert the comparison. Measured at 1/12 full
-p fluree-graph-irruns on thebase
f6f6857deand 2/12 at this tip — the same rate within noise, and thisbranch does not touch the sampling code (the only change to
timing.rshereadds two
Phasevariants). It belongs to #1555's line, not this PR, and isrecorded rather than papered over.
Follow-ups recorded, not silently dropped
A 500 MB/s scan puts K=16 at roughly 8.9×, and the scan is resync-able so it
can itself be parallelized at K offsets.
Phase::Chunklane — the scan is 43% of wall currently charged tounattributed. One line.ImportSource::recv_nextjoins nothing (no callers today).--prettybuffered Turtle tier (H-8) is still unimplemented and refuseshonestly.
canon,inferandqueryare M5.