Skip to content

Count the span-density cost instead of timing it - #146

Open
wildthink wants to merge 2 commits into
nodes-app:mainfrom
wildthink:perf/deterministic-span-cost-test
Open

Count the span-density cost instead of timing it#146
wildthink wants to merge 2 commits into
nodes-app:mainfrom
wildthink:perf/deterministic-span-cost-test

Conversation

@wildthink

Copy link
Copy Markdown
Contributor

Follow-up to #140 and to 350b2d3, and an answer to "happy to hear an argument for a different number": I don't think there is a better number, because the thing being measured is wrong. This replaces the timing with a count.

Sorry for the red CI — the bound was mine and the reasoning behind it was wrong in a way your commit message pins exactly.

Why no threshold worked

I justified those bounds with "minimum of several runs; noise only ever adds time, so the floor is stable." That holds for an absolute measurement on an idle machine. It does not hold for the ratio I actually asserted: under sustained contention there is no quiet run to find a floor in, and the smaller measurement inflates proportionally more, so the ratio drifts up with load rather than staying put. 5.3x here, 10.9x on the runner, same parser. Your point that the post-rewrite CI reading sits above the pre-rewrite floor is the end of the argument — any bound green on CI would have passed the parser the test exists to catch.

Counting instead

InlineParser.parse can now report an InlineParseCost: claimed-range probes and containment tests. Both are pure functions of the input, so they read the same on your laptop, mine, and a loaded runner — and the separation is decisive instead of marginal:

40 spans 240 spans growth
current parser 508 3248 6.0x
pre-rewrite pairwise containment 1840 58320 33.9x

Exactly 6.0x for every construct — code, links, emphasis, highlight, mixed, and link labels with nested code spans. I got the 33.9x by actually restoring the old pairwise containment and re-running, not by extrapolating. The bound stays at 8, which now sits inside a gap that doesn't move.

These run on CI. The timed assertions stay exactly as you left them — opt-in, useful for absolute numbers — and I corrected the msPerParse comment, which still asserted the floor reasoning I'd used to justify the ratio.

On the instrumentation

Counters are plain Ints bumped beside comparisons the loops already perform, threaded through rather than parked in a global. The global would have been a smaller diff and wrong: swift-testing runs suites in parallel, so every other suite that parses markdown would land in the counter mid-measurement. ClaimedIndex owns its own probe count and the scans take it inout so the caller reads it back; buildTree takes the cost inout.

That is a real cost in the hot path — two integer increments — which I judged worth it against a global that races or a #if DEBUG counter that leaves release untested. Say the word if you'd rather have it behind a compile flag and I'll move it.

Two things worth being explicit about, since both are limitations rather than features:

  • claimedProbes is zero for a paragraph of links. Nothing is claimed before the link pass, and there are no * or \ characters to ask about, so the query never fires. containmentTests carries the signal there and claimedProbes carries it for code spans, which is why the assertion sums them — and why it also asserts the count is non-zero, so it can't pass by measuring nothing.
  • It won't catch a brand-new scan that bypasses ClaimedIndex and buildTree entirely. It holds the existing structures to linear; it doesn't prove nothing quadratic exists anywhere. The timed version had the opposite trade — measured everything, meant nothing off this machine.

corpusFingerprint is untouched and still b74649ffbbbe237a, which is also the check that the instrumentation changed no parse. 324 tests green.

Thanks for rebasing #140 rather than bouncing it — and for catching this on CI instead of letting it rot as a flake.

wildthink-pub and others added 2 commits August 18, 2026 12:06
The wall-clock ratio these assertions used was not portable, and gating CI
on it turned main red: the same parser reads 5.3x on an idle laptop and
10.9x on a contended runner, which is above the 11.3x pre-rewrite floor,
so no threshold separates the two parsers. 350b2d3 made them opt-in, which
left the rewrite with no CI-visible cost guard.

InlineParser.parse can now report an InlineParseCost — claimed-range probes
and containment tests — and the assertions use that. Both quantities are
pure functions of the input, so they read identically everywhere, and the
separation is decisive rather than marginal: 6.0x for 6x the spans against
33.9x with the pre-rewrite pairwise containment restored, verified by
reintroducing it. The bound stays at 8, now inside a deterministic gap.

The counters are plain Ints beside comparisons the loops already do,
threaded through rather than kept in a global, because swift-testing runs
suites in parallel and a global would be raced by every other suite that
parses. ClaimedIndex owns its own probe count; the scans take it inout so
the caller can read it back.

The timed assertions stay, still opt-in, for absolute numbers. What the
counted form does not catch is a new scan bypassing ClaimedIndex and
buildTree entirely — it holds the existing structures to linear rather
than proving nothing quadratic exists anywhere.

Corpus fingerprint unchanged, so the instrumentation changed no parse.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The rebase onto current main puts this branch on top of the directive seam
(nodes-app#120), which added a claimed-span producer that did not exist when these
tests were written. Two counted cases now cover it: 6.0x work for 6x the
spans, on both the self-contained and container forms.

Each shape pairs the directive with a code span deliberately. A paragraph of
bare `@mk` claims nothing in passes 1-2, so `ClaimedIndex` is built EMPTY and
a pairwise scan over it costs nothing — the assertion then passes no matter
what the cursor does, and only `buildTree` is under test. That is exactly what
the first version of these two tests did, and it looked fine: 6.0x, green.

Verified the other way round, by restoring the pre-rewrite pairwise
containment. With the code span both fail at 37.6x; without it both still pass.

`expectLinearWork` takes a registry now, defaulted to the extensions one, so
the existing call sites are unchanged.

455 tests pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@wildthink
wildthink force-pushed the perf/deterministic-span-cost-test branch from d582cb8 to d9680f2 Compare August 18, 2026 16:10
@wildthink

Copy link
Copy Markdown
Contributor Author

Rebased onto current main — this now sits on top of the directive seam (#120/#155), which landed after this PR was opened. Still CLEAN, CI green.

That rebase is worth more than a refresh here: #120 added the directive hook to InlineParser, the same file this PR rewrites. Textual mergeability doesn't say the two compose, so I added counted coverage for the new path — 6.0x work for 6x the spans, on both the self-contained and container forms.

One thing worth flagging, because the first version of those two tests was wrong in a way that looked right.

I originally wrote the shapes as bare @mk / @bx{w} paragraphs. They measured 6.0x and went green. But a paragraph of bare directives claims nothing in passes 1-2, so ClaimedIndex is built empty — a pairwise scan over it costs nothing, and the assertion holds regardless of what the cursor does. Only buildTree was actually under test.

I caught it by restoring the pre-rewrite pairwise containment and checking the new tests fail. They didn't. Pairing each directive with a code span gives the index something to scan:

shape with the rewrite pre-rewrite containment
bare @mk 6.0x 6.0x — passes, detects nothing
`c0` @mk 6.0x 37.6x — fails

expectLinearWork takes a registry now, defaulted to the extensions one, so the existing call sites are unchanged. 455 tests pass.

No rush on this one — I know the directive pair was the priority. Happy to rebase again whenever it suits you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants