From 94db93a60488b75c38f278e7849024a50d47a501 Mon Sep 17 00:00:00 2001 From: Jason Jobe Date: Tue, 4 Aug 2026 16:54:07 +0000 Subject: [PATCH] Inline parse cost linear in spans per region, not quadratic MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Every pass after the first consulted the claimed ranges by scanning the whole array: once per character in scanEscapes and collectDelimiterRuns, once per candidate in scanLinkFamily. buildTree then decided containment by testing each span against every other one. All fine at ordinary densities, ~n^2 when a single paragraph carries hundreds of spans. Both scans are avoidable for the same reason. The passes walk the string left to right and never look back, and claimed ranges never PARTIALLY overlap, so a cursor over the sorted ranges answers "is this claimed?" in amortised constant time. Containment falls out of the same invariant: sorting spans by start ascending and length descending puts every span immediately after the one containing it, so buildTree becomes a single ordered walk. A paragraph of 240 code spans parses in 0.5ms rather than 33ms. 6x the spans now costs ~6x the parse instead of ~30x. Affects every claimed-span construct — code, escapes, links, images, wiki links, inline LaTeX, emphasis, and extension spans. No parse result changes. InlineSpanDensityTests folds the parsed tree of a 4000-input pseudo-random corpus into one fingerprint, recorded on the pre-rewrite parser at this branch's merge base; the scaling assertions fail on that parser at 11.3x-27.8x against an 8x bound. Rebased onto #118, which relaxed the invariant this rests on: a link may now contain a claimed code span inside its label, so claimed ranges are disjoint OR properly nested rather than strictly disjoint. Three consequences, all handled here. * ClaimedIndex gains `overlapping`, which enumerates every claimed range meeting a candidate instead of answering yes/no. Only the link case needs it; everything else still short-circuits on the first overlap. It peeks forward from the cursor rather than advancing it, so the left-to-right walk is unaffected and the cost property holds. * `contains` needed no change: a nested range sorts after its container, which already covers it. * The corpus fingerprint was re-recorded on the pre-rewrite parser at the new merge base. It reproduces that parser's trees exactly, which is what makes this a pure performance change on top of #118 as well. The scaling bound moved from 12x to 8x. Measured on one machine: worst case after the rewrite is 5.6x (code), best case before it is 11.3x (highlight), so 12x let the two cheapest constructs pass on the OLD parser — the one thing the assertion exists to prevent. 8x is the geometric midpoint of that gap, with ~1.4x of room either side. Closes #109 Co-Authored-By: Claude Opus 5 Co-Authored-By: Claude Opus 5 (1M context) --- ARCHITECTURE.md | 5 +- CHANGELOG.md | 12 ++ .../MarkdownEngine/Parser/InlineParser.swift | 171 ++++++++++++------ .../InlineSpanDensityTests.swift | 141 +++++++++++++++ 4 files changed, 273 insertions(+), 56 deletions(-) create mode 100644 Tests/MarkdownEngineTests/InlineSpanDensityTests.swift diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index d541824e..58900de9 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -45,7 +45,10 @@ regexes are gone, replaced by hand-written scanners and a real syntax tree. escapes → link family (`![[…]]`, `[[…]]`, `![…](…)`, `[…](…)`, `~~…~~`, `$…$`) → emphasis (`*`/`_` delimiter runs) → `buildTree`. Each pass claims spans only in regions not already claimed, so there are never partial - overlaps and the tree is a clean containment tree. + overlaps and the tree is a clean containment tree. That invariant is also + what keeps the pass linear in span count: claimed ranges are consulted + through a cursor rather than rescanned, and `buildTree` derives containment + from a sort instead of comparing spans pairwise. 3. **`MarkdownAST` / `DocumentAST.parse`** combines the two into the semantic document AST — `[BlockNode]`, each inline-bearing block carrying its parsed `[InlineNode]` children in absolute document coordinates. `BlockNode`, diff --git a/CHANGELOG.md b/CHANGELOG.md index 3a178c8a..8b2df629 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 wherever a fill should read as a block. ### Changed +- **Inline parse cost is linear in the spans per region, not quadratic.** Every + pass after the first consulted the claimed ranges by scanning the whole array + — once per character in `scanEscapes` and `collectDelimiterRuns`, once per + candidate in `scanLinkFamily` — and `buildTree` decided containment by testing + each span against every other. The passes walk the string left to right and + claimed ranges never partially overlap, so a cursor over the sorted ranges answers both + questions in amortised constant time, and sorting spans by start ascending / + length descending turns containment into a single ordered walk. A paragraph of + 240 code spans parses in 0.5ms rather than 33ms; 6x the spans now costs 6x the + parse instead of ~30x. Affects every claimed-span construct — code, escapes, + links, images, wiki links, inline LaTeX, emphasis, and extension spans. No + parse result changes. - `==highlight==` fills the line box. AppKit paints `.backgroundColor` over ascent + descent only, so the marker fell short of the line height by the leading plus `paragraph.lineHeightExtraSpacing`, and a highlight that wrapped diff --git a/Sources/MarkdownEngine/Parser/InlineParser.swift b/Sources/MarkdownEngine/Parser/InlineParser.swift index 5399ea47..8db8adbd 100644 --- a/Sources/MarkdownEngine/Parser/InlineParser.swift +++ b/Sources/MarkdownEngine/Parser/InlineParser.swift @@ -29,6 +29,13 @@ // recursively; code/image/wiki/embed/latex/escape are // opaque leaves. // +// Claimed spans are therefore either disjoint or properly NESTED — a link +// label may hold one, nothing else may. That is load-bearing for cost as well +// as correctness: it's what lets `ClaimedIndex` answer "is this claimed?" with +// a cursor and `buildTree` derive containment from a sort. A pass that claimed +// a PARTIALLY overlapping span would break both, so keep claiming whole or not +// at all. +// import Foundation @@ -92,9 +99,9 @@ enum InlineParser { guard len > 0 else { return [] } var claimed = scanCodeSpans(ns, len: len) - claimed += scanEscapes(ns, len: len, claimed: claimed.map(\.fullRange)) - claimed += scanLinkFamily(ns, len: len, claimed: claimed.map(\.fullRange), registry: registry) - let emphasis = resolveEmphasis(ns, len: len, claimedRanges: claimed.map(\.fullRange)) + claimed += scanEscapes(ns, len: len, claimed: ClaimedIndex(claimed)) + claimed += scanLinkFamily(ns, len: len, claimed: ClaimedIndex(claimed), registry: registry) + let emphasis = resolveEmphasis(ns, len: len, claimed: ClaimedIndex(claimed)) return buildTree(region: NSRange(location: 0, length: len), spans: claimed + emphasis, ns: ns, registry: registry) } @@ -125,16 +132,59 @@ enum InlineParser { return r } } - /// Region whose interior can own already-collected child spans. - var containerContent: NSRange? { - switch self { - case .emphasis(_, _, let open, let close): - return NSRange(location: NSMaxRange(open), length: close.location - NSMaxRange(open)) - case .link(_, let textRange, _, _): - return textRange - default: - return nil + } + + /// The already-claimed ranges, in a form the later passes can consult in + /// amortised constant time. + /// + /// Every pass that asks "is this claimed?" walks the string left to right + /// and never looks back, and claimed ranges never PARTIALLY overlap (each + /// pass only claims inside regions no earlier pass took). So a cursor over + /// the sorted ranges answers without rescanning: the answer for index `i` + /// only ever involves the first range that ends after `i`. + /// + /// A nested range (a code span inside a link label) sorts after its + /// container, which already covers it, so `contains` stays correct without + /// looking past the cursor. `overlapping` is the one query that must, and + /// it peeks rather than advances. + /// + /// Sortedness is established here rather than assumed of callers, so no + /// call site carries an ordering obligation. + private struct ClaimedIndex { + private let ranges: [NSRange] + private var cursor = 0 + + init(_ spans: [Span]) { + ranges = spans.map(\.fullRange).sorted { $0.location < $1.location } + } + + /// Discard ranges that end at or before `idx`. `idx` must not move backwards. + private mutating func advance(to idx: Int) { + while cursor < ranges.count, NSMaxRange(ranges[cursor]) <= idx { cursor += 1 } + } + + mutating func contains(_ idx: Int) -> Bool { + advance(to: idx) + return cursor < ranges.count && NSLocationInRange(idx, ranges[cursor]) + } + + mutating func overlaps(_ range: NSRange) -> Bool { + advance(to: range.location) + return cursor < ranges.count && ranges[cursor].location < NSMaxRange(range) + } + + /// Every claimed range overlapping `range`. Peeks forward from the + /// cursor without consuming, so the caller's left-to-right walk is + /// unaffected. + mutating func overlapping(_ range: NSRange) -> [NSRange] { + advance(to: range.location) + var out: [NSRange] = [] + var k = cursor + while k < ranges.count, ranges[k].location < NSMaxRange(range) { + if NSIntersectionRange(ranges[k], range).length > 0 { out.append(ranges[k]) } + k += 1 } + return out } } @@ -186,12 +236,12 @@ enum InlineParser { // MARK: - 2. Backslash escapes (claimed → escaped chars are inert everywhere) - private static func scanEscapes(_ ns: NSString, len: Int, claimed: [NSRange]) -> [Span] { - func inClaimed(_ idx: Int) -> Bool { claimed.contains { NSLocationInRange(idx, $0) } } + private static func scanEscapes(_ ns: NSString, len: Int, claimed: ClaimedIndex) -> [Span] { + var claimed = claimed var spans: [Span] = [] var i = 0 while i < len - 1 { - if ns.character(at: i) == backslash, !inClaimed(i), isAsciiPunctuationChar(ns.character(at: i + 1)) { + if ns.character(at: i) == backslash, !claimed.contains(i), isAsciiPunctuationChar(ns.character(at: i + 1)) { spans.append(.escape( range: NSRange(location: i, length: 2), character: NSRange(location: i + 1, length: 1), @@ -207,23 +257,24 @@ enum InlineParser { // MARK: - 3. Link family / inline LaTeX / extension spans - private static func scanLinkFamily(_ ns: NSString, len: Int, claimed: [NSRange], registry: ExtensionRegistry) -> [Span] { + private static func scanLinkFamily(_ ns: NSString, len: Int, claimed: ClaimedIndex, registry: ExtensionRegistry) -> [Span] { + var claimed = claimed + // A candidate overlapping a claimed span is rejected, except for spans + // wholly nested inside a Markdown link's label (#118). Only that case + // needs the full overlap list; everything else short-circuits on the + // first one. func hasDisallowedClaimedOverlap(_ span: Span) -> Bool { - let overlaps = claimed.filter { - NSIntersectionRange($0, span.fullRange).length > 0 - } - guard overlaps.isEmpty == false else { return false } - guard case .link(_, let textRange, _, _) = span else { return true } - return overlaps.contains { - rangeContains(textRange, $0) == false + guard case .link(_, let textRange, _, _) = span else { + return claimed.overlaps(span.fullRange) } + return claimed.overlapping(span.fullRange).contains { !rangeContains(textRange, $0) } } var spans: [Span] = [] var i = 0 while i < len { - if claimed.contains(where: { NSLocationInRange(i, $0) }) { i += 1; continue } + if claimed.contains(i) { i += 1; continue } if let span = matchClaimedSpan(ns, len, at: i, registry: registry), - hasDisallowedClaimedOverlap(span) == false { + !hasDisallowedClaimedOverlap(span) { spans.append(span) i = NSMaxRange(span.fullRange) } else { @@ -515,8 +566,8 @@ enum InlineParser { var remaining: Int { rightEdge - leftEdge } } - private static func resolveEmphasis(_ ns: NSString, len: Int, claimedRanges: [NSRange]) -> [Span] { - var runs = collectDelimiterRuns(ns, len: len, claimedRanges: claimedRanges) + private static func resolveEmphasis(_ ns: NSString, len: Int, claimed: ClaimedIndex) -> [Span] { + var runs = collectDelimiterRuns(ns, len: len, claimed: claimed) guard !runs.isEmpty else { return [] } var stack: [Int] = [] var spans: [Span] = [] @@ -531,18 +582,15 @@ enum InlineParser { return spans } - private static func collectDelimiterRuns(_ ns: NSString, len: Int, claimedRanges: [NSRange]) -> [DelimRun] { - func inClaimed(_ idx: Int) -> Bool { - for r in claimedRanges where NSLocationInRange(idx, r) { return true } - return false - } + private static func collectDelimiterRuns(_ ns: NSString, len: Int, claimed: ClaimedIndex) -> [DelimRun] { + var claimed = claimed var runs: [DelimRun] = [] var lineIdx = 0 var i = 0 while i < len { let c = ns.character(at: i) if c == newline { lineIdx += 1; i += 1; continue } - guard c == asterisk || c == underscore, !inClaimed(i) else { i += 1; continue } + guard c == asterisk || c == underscore, !claimed.contains(i) else { i += 1; continue } var j = i while j < len, ns.character(at: j) == c { j += 1 } @@ -614,33 +662,46 @@ enum InlineParser { // MARK: - 5. Containment tree private static func buildTree(region: NSRange, spans: [Span], ns: NSString, registry: ExtensionRegistry) -> [InlineNode] { - let inRegion = spans.filter { rangeContains(region, $0.fullRange) } - - func isChild(_ s: Span) -> Bool { - for parent in inRegion { - guard !equalRange(parent.fullRange, s.fullRange), let content = parent.containerContent else { continue } - if rangeContains(content, s.fullRange) { return true } + // Spans are non-overlapping or properly nested (each pass claims only + // inside regions no earlier pass took), so ordering by start ascending + // and length descending puts every span immediately after the one that + // contains it. Containment then falls out of a single ordered walk, + // instead of testing each span against every other span. + let ordered = spans + .filter { rangeContains(region, $0.fullRange) } + .sorted { a, b in + let (x, y) = (a.fullRange, b.fullRange) + return x.location == y.location ? x.length > y.length : x.location < y.location } - return false - } + var cursor = 0 + return buildTree(region: region, ordered: ordered, cursor: &cursor, ns: ns, registry: registry) + } - let top = inRegion.filter { !isChild($0) }.sorted { $0.fullRange.location < $1.fullRange.location } + /// Consumes spans from `cursor` for as long as they fall inside `region`, + /// leaving `cursor` on the first span that doesn't. + private static func buildTree( + region: NSRange, ordered: [Span], cursor: inout Int, ns: NSString, registry: ExtensionRegistry + ) -> [InlineNode] { var result: [InlineNode] = [] - var cursor = region.location + var textStart = region.location - for span in top { + while cursor < ordered.count { + let span = ordered[cursor] let fr = span.fullRange - if fr.location > cursor { - result.append(.text(NSRange(location: cursor, length: fr.location - cursor))) + guard rangeContains(region, fr) else { break } + cursor += 1 + + if fr.location > textStart { + result.append(.text(NSRange(location: textStart, length: fr.location - textStart))) } switch span { case .code(let range, let content): result.append(.code(range: range, content: content)) case .emphasis(let kind, let range, let open, let close): let content = NSRange(location: NSMaxRange(open), length: close.location - NSMaxRange(open)) - let childSpans = inRegion.filter { rangeContains(content, $0.fullRange) && !equalRange($0.fullRange, fr) } result.append(.emphasis(kind, range: range, markers: [open, close], - children: buildTree(region: content, spans: childSpans, ns: ns, registry: registry))) + children: buildTree(region: content, ordered: ordered, + cursor: &cursor, ns: ns, registry: registry))) case .link(let range, let textRange, let url, let markers): result.append(.link(range: range, textRange: textRange, url: url, markers: markers, children: reparse(textRange, ns: ns, registry: registry))) @@ -660,10 +721,14 @@ enum InlineParser { children: parsesContent ? reparse(contentRange, ns: ns, registry: registry) : [] ))) } - cursor = NSMaxRange(fr) + // Every span but emphasis is opaque, so nothing should remain + // inside one. Skipping keeps the walk well-formed if that ever + // changes, rather than emitting a node past the cursor. + while cursor < ordered.count, rangeContains(fr, ordered[cursor].fullRange) { cursor += 1 } + textStart = NSMaxRange(fr) } - if cursor < NSMaxRange(region) { - result.append(.text(NSRange(location: cursor, length: NSMaxRange(region) - cursor))) + if textStart < NSMaxRange(region) { + result.append(.text(NSRange(location: textStart, length: NSMaxRange(region) - textStart))) } return result } @@ -701,10 +766,6 @@ enum InlineParser { inner.location >= outer.location && NSMaxRange(inner) <= NSMaxRange(outer) } - private static func equalRange(_ a: NSRange, _ b: NSRange) -> Bool { - a.location == b.location && a.length == b.length - } - private static func isWhitespaceOrBoundary(_ idx: Int, _ ns: NSString, _ len: Int) -> Bool { guard idx >= 0, idx < len else { return true } let c = ns.character(at: idx) diff --git a/Tests/MarkdownEngineTests/InlineSpanDensityTests.swift b/Tests/MarkdownEngineTests/InlineSpanDensityTests.swift new file mode 100644 index 00000000..bf7ca725 --- /dev/null +++ b/Tests/MarkdownEngineTests/InlineSpanDensityTests.swift @@ -0,0 +1,141 @@ +// +// InlineSpanDensityTests.swift +// MarkdownEngineTests +// +// Inline parse cost against span density within one region (#109). +// +// Two halves, and the first is the one that matters: the containment rewrite +// has to be a pure performance change. `corpusFingerprint` folds the parsed +// tree of 4000 pseudo-random inputs into one value, recorded on the PRE-rewrite +// parser at the merge base (1a2bd74, i.e. with #118) — in the spirit of +// `GoldenCorpusTests`, except the baseline covers shapes nobody would think to +// write by hand. +// +// Re-record it ONLY on a parser that predates the rewrite, otherwise it just +// ratifies whatever the rewrite does. It is also a bare hash: when it fails, +// diff `String(describing:)` per input against the old parser to see what moved. +// +// The second half asserts the cost curve is linear in spans rather than +// quadratic, so the scans can't quietly come back. +// + +import Foundation +import Testing +@testable import MarkdownEngine + +@Suite("Inline parse cost vs. span density") +struct InlineSpanDensityTests { + + // MARK: - Corpus + + /// Deterministic LCG — the corpus must be identical across builds for the + /// fingerprint to mean anything, and `SystemRandomNumberGenerator` isn't. + private struct LCG { + var state: UInt64 = 0x2545F4914F6CDD1D + mutating func next(_ bound: Int) -> Int { + state = state &* 6364136223846793005 &+ 1442695040888963407 + return Int((state >> 33) % UInt64(bound)) + } + } + + /// Fragments chosen to collide: bare and paired delimiters, escapes, and + /// the openers of every claimed-span construct, so the corpus is dense in + /// half-formed and nested spans rather than in valid markdown. + private static let atoms = [ + "a", "bb", " ", " ", "*", "**", "_", "__", "`", "``", "\\", "\\*", "\\`", + "[", "]", "(", ")", "![", "[[", "]]", "|", "$", "==", "~~", "url", "http://e.com/x", + "\n", "word", ".", "!", "*a*", "**b**", "`c`", "[d](e)", "[[f|g]]", "$h$", + ] + + private func corpus(_ count: Int) -> [String] { + var rng = LCG() + return (0.. String { + var fnv: UInt64 = 0xcbf29ce484222325 + func fold(_ s: String) { + for b in s.utf8 { fnv = (fnv ^ UInt64(b)) &* 0x100000001b3 } + } + for s in strings { + fold(s) + fold(String(describing: InlineParser.parse(s, registry: registry))) + } + return String(fnv, radix: 16) + } + + @Test("the containment rewrite changes no tree in a 4000-input corpus") + func corpusFingerprint() { + let registry = MarkdownEditorConfiguration( + extensions: [HighlightExtension(), StrikethroughExtension()] + ).extensionRegistry + + #expect(fingerprint(corpus(4000), registry: registry) == "b74649ffbbbe237a") + } + + // MARK: - Cost curve + + /// Minimum of several runs: scheduler noise only ever adds time, so the + /// floor is the stable statistic. Means would make this a flake. + private func msPerParse(_ text: String, registry: ExtensionRegistry) -> Double { + var best = Double.infinity + for _ in 0..<7 { + let start = DispatchTime.now().uptimeNanoseconds + for _ in 0..<20 { _ = DocumentAST.parse(text, registry: registry) } + let ms = Double(DispatchTime.now().uptimeNanoseconds - start) / 20 / 1_000_000 + best = min(best, ms) + } + return best + } + + private func paragraph(_ n: Int, _ make: (Int) -> String) -> String { + (0.. String) { + let registry = MarkdownEditorConfiguration(extensions: [HighlightExtension()]).extensionRegistry + let small = msPerParse(paragraph(40, make), registry: registry) + let large = msPerParse(paragraph(240, make), registry: registry) + let growth = large / small + + #expect(growth < 8, "\(label): 6x spans cost \(String(format: "%.1f", growth))x parse") + } + + @Test("code spans: parse cost is linear in spans per paragraph") + func codeSpanDensity() { expectLinearInSpans("code") { "`word\($0)`" } } + + @Test("links: parse cost is linear in spans per paragraph") + func linkDensity() { expectLinearInSpans("links") { "[word\($0)](https://e.com/\($0))" } } + + @Test("emphasis: parse cost is linear in spans per paragraph") + func emphasisDensity() { expectLinearInSpans("emphasis") { "*word\($0)*" } } + + @Test("highlights: parse cost is linear in spans per paragraph") + func highlightDensity() { expectLinearInSpans("highlight") { "==word\($0)==" } } + + /// The pathological case the issue was filed from: escapes and code spans + /// together, where every later pass used to rescan every claimed range. + @Test("a paragraph mixing claimed-span kinds stays linear") + func mixedDensity() { + expectLinearInSpans("mixed") { i in + switch i % 4 { + case 0: return "`code\(i)`" + case 1: return "\\*lit\(i)\\*" + case 2: return "*em\(i)*" + default: return "[l\(i)](u\(i))" + } + } + } +}