Directives (1/4): the seam — parsing, registry, and argument coercion

[wildthink]

PR 1 of 4 for the directive seam designed in #108. As you asked, this is the one to sign off the projection — the parsing is thin on purpose, and nothing is styled yet.

Rebased on current main (0.11.0), so cursorFollowsSpanInk, the ordered-list numbering, and the block-background work are all in. 356 tests green, demo builds.

What this is

MarkdownExtension handles delimiter-shaped constructs. It can't express a construct with a name and typed arguments — InlineSyntax is a pair of delimiter strings, so @font(size: 18){…} has nowhere to live. MarkdownDirective is the parallel seam, built to the same isolation contract: syntax and schema in, never ranges.

Two forms, both tree-shaped: self-contained (@pagebreak) and container (@font(size: 18){text}, body re-parsed as markdown). No "applies to everything after me" form — it would make styling depend on document position rather than tree position, and its effect would outlive its own block.

The projection

Directives do not add a node kind. A match becomes an InlineNode.ext whose id carries a reserved directive. prefix:

return .ext(id: match.nodeID, range: match.range, contentRange: match.contentRange,
            markers: match.markers, parsesContent: match.parsesContent)

So InlineNode, buildTree, offsetNodes, InlineASTAdapter, MarkdownToken, and shrinkInlineMarkers are all untouched, and directives inherit marker shrink, caret reveal, token projection, incremental restyle, and rich copy for free rather than reimplementing any of it.

The cost is one namespace convention (DirectiveRegistry.idPrefix, with nodeID(for:) / directiveID(forNodeID:) as the only two places that know about it). An extension whose own id began directive. would collide; the prefix contains a ., which no bundled extension id uses. If you'd rather have a real InlineNode case and take the switch churn, say so — I went this way specifically because your CONTRIBUTING says new constructs shouldn't thread a case through parser, styler, and renderer.

The fingerprint property

DirectiveRegistry is carried by ExtensionRegistry:

self.fingerprint = directives.isEmpty
    ? extensionFingerprint
    : extensionFingerprint + "~" + directives.fingerprint

Two consequences worth checking me on:

• No second cache key exists. Every parse cache already keys on the grammar fingerprint, so registering a directive at runtime invalidates them with nothing new threaded through the pipeline.
• A directive-free registry is byte-identical to before. The empty half contributes nothing, so no existing document re-parses and no existing cache entry is invalidated by this PR landing. There's a test asserting the fingerprint contains no ~ when no directives are registered.

Only fields that change the PARSE participate (name, marker, form, parsesBody) — presentation-only edits must not invalidate parse caches. Free-text fields are length-prefixed so the concatenation is injective.

Safety over an existing corpus

Two rules, both tested:

Registered names only. @home in prose stays literal unless home is registered — same as unregistered extension syntax.

Left boundary as a deny list. Only letters and digits reject. I wrote it as an allow list of "opening punctuation" first and it silently dropped every directive abutting markup — *@font(size: 18){x}*, **…**, - @pagebreak — because the preceding character is a delimiter I hadn't listed. Deny-list is all the email rule (name@example.com) ever needed and can't fail that way.

Rejection is always total: unregistered name, malformed call, wrong form, unbalanced delimiters, or a run crossing a line break all leave the candidate literal. Nothing here produces a partial construct.

One interaction found while testing: a directive containing a backslash escape (@font(size: 18){a \} b}) stays literal, because the escape pass claims before the link-family pass and a candidate overlapping a claimed span is rejected. That's existing engine behaviour — [a \* b](url) and ==a \* b== are rejected identically — so directives inherit it rather than special-casing. The scanner still measures the body correctly in isolation, so if escapes ever stop pre-claiming, directives need no change. Both facts are asserted.

Footprint

File	Lines	What
InlineParser.swift	10	scanner hook in matchClaimedSpan, after every built-in
MarkdownExtension.swift	33	ExtensionRegistry carries the directive registry + folds its fingerprint
MarkdownEditorConfiguration.swift	12	directives + directiveSettings

46 lines across 3 existing files. Everything else is new files under Sources/MarkdownEngine/Directives/.

Deliberately not here

• No directive ships. FontDirective / ColorDirective are pure presentation, so they arrive with styling in PR2. The parser tests declare their own shapes instead, which keeps them testing the seam rather than a bundled implementation.
• Nothing is styled. A registered directive currently renders as literal text. DirectiveStyle, presentation, and glyphs are PR2/PR3.
• No autocomplete. valueCompletions and the completion types are PR4.
• Arguments coerce at styling time, not parse time, so a directive-free document pays nothing for the schema machinery.

Testing

swift build / swift test green — 356 tests, including upstream's. New coverage: boundary and rejection cases, form enforcement, balanced/nested/escaped delimiters, single-line enforcement, precedence against code spans and inline LaTeX, body re-parsing, token projection, alternate and coexisting markers, multi-scalar marker rejection, and the fingerprint properties above.

[luca-chen198]

Design approved — this is what I asked for in #108, and the description does exactly the job I wanted it to: the namespace choice and the fingerprint property are both spelled out, and the parsing stayed thin. Projecting onto .ext under a reserved namespace instead of threading a new node kind through parser, styler and renderer is the right call, and the byte-identical fingerprint for a directive-free registry is the property that makes the seam safe to add at all.

On merging. I'd like to land PR1 and PR2 together rather than merge this on its own. It's a fair amount of new public API for a library that's semver-bound, and standalone a registered directive parses and then renders as plain text — which is a worse state than not registering it. Since the whole thing already exists on feat/directives, holding PR1 until PR2 is open costs you nothing and means the API arrives together with the behaviour it exists for. Nothing to redo — open PR2 whenever suits you and I'll take them as a pair. The fixes below can land here in the meantime.

Two fixes I'd like in this PR, both confined to DirectiveArguments.swift, so neither touches anything #140 is rewriting:

1. defaultValue on a positional parameter is never applied. applyingDefaults only fills labelled ones, so a positional parameter carrying a default silently yields nothing. Implement it or drop the field — it's public API, and once it's released I'm stuck with a parameter that does nothing.
2. Escape handling is split across the feature. balanced() is escape-aware, splitArguments / splitLabel / coerce are not. @font(size: "a \" b"){x} matches, but the value arrives as a \" b, and an escaped quote flips inQuote in the comma split, so a multi-argument call splits wrong.

One known limitation to document rather than fix here. A code span or $…$ in the body rejects the whole directive: @font(size: 18){a `b` c} produces no directive node at all, while emphasis, links and nesting are all fine. That's the same shape #118 just granted an exemption for in link labels, so it deserves a doc line now and its own issue. Please don't fix it in this PR — it lives in scanLinkFamily's overlap rule, which is exactly what #140 rewrites, and #140's rebase has to build the "enumerate overlapping claims" machinery for #118 anyway. Cleanest as a small follow-up once that lands.

Later, not now:

• MarkdownHTMLRenderer.html(from:extensions:) has no directives: parameter, so the clean-copy path can't see them and MarkdownDirective.html(arguments:bodyHTML:) is currently unreachable. Harmless while nothing is styled, but it has to land with PR2 or copied text diverges from what's on screen.
• Worth a doc line that the hook sits before the extension loop, so directives beat extensions. Only "after every built-in" is written down.

On the bundled directives: agreed, Font and Color belong with PR2 — no point shipping presentation types before there's presentation. And I still want the directive-heavy restyle perf scenario from #108; PR2 is the right place for it.

[wildthink]

Thanks — and agreed on landing as a pair. PR2 is open as #155, based on this branch rather than on main, so it reads as the diff on top of this one. Rebased both onto current main too.

Fix 1 — positional defaultValue: implemented

You were right that it did nothing: applyingDefaults guarded on parameter.label, so positional parameters skipped the fill entirely. Implemented rather than dropped.

One semantic I had to pin down: positional defaults fill by position, which makes them a tail-only affair. Given (a, b = 2, c), @x(1) yields 1, 2 and still reports #2 missing — there is no syntax for "default here, but supply the next one", so the first positional without a default ends the fill. The missing-positional diagnostic also names its own index now instead of reporting once at the count. Four tests, all of which fail without the change.

Fix 2 — I think this one is unreachable, and the real question is different

I tried to reproduce your example and it doesn't behave as described:

@font(family: "a \" b"){x}   → no directive at all, not a directive with a wrong value

The \" is claimed by the escape pass (pass 2), the directive candidate overlaps a claimed span, and scanLinkFamily rejects it outright. So splitArguments never receives a backslash-escape — any backslash before ASCII punctuation kills the whole directive before argument parsing runs. Making the splitters escape-aware to match balanced() would be hardening a path nothing reaches.

The reachable case is a backslash before a non-punctuation character ("a \z b"), which isn't an escape at all and passes through raw — arguably correct, and certainly not what balanced()'s escape-awareness would change.

So I've left the splitters alone. The actual question is what an escape should mean inside a directive, and that's the same overlap rule as the limitation below — which is why I put both in one issue rather than patching half of it here. Say the word if you'd rather I harden the splitters defensively anyway; I just didn't want to add code whose only test would have to construct a state the parser can't produce.

The limitation — documented, and narrower than we both thought

Not fixed here, as you asked. But measuring it moved the boundary:

body	result
@font(size: 18){a `b` c}	rejected
@font(size: 18){a \* c}	rejected
@font(size: 18){a $x^2$ c}	directive
@font(size: 18){a [l](u) c}	directive
@font(size: 18){a *b* c}	directive

$…$ does not reject — it's claimed in pass 3 alongside directives, so it never overlaps a prior claim. Only genuinely pre-claimed spans bite: code spans (pass 1) and escapes (pass 2). That narrows the eventual fix to exempting two passes rather than "everything else in a body".

Documented in the DirectiveScanner header, at the InlineParser hook, and in the changelog — and pinned by tests asserting both halves, so whoever lifts it flips them deliberately instead of discovering a surprise. Filed as #154 with the analysis and the ClaimedIndex.overlapping-based fix sketch; happy to take it once this pair lands, since it touches the same rule.

Your other two

• html(from:extensions:directives:) — done, it's in Directives (2/4): styling — font composition, colour, and rich copy #155, so MarkdownDirective.html is reachable and rich copy matches the screen.
• Hook ordering — the comment now says directives match before the extension loop, not just after the built-ins, and why that ordering is safe.
• The Design: directive seam — named inline commands with typed arguments (@font(size: 18){…}) #108 perf scenario is in Directives (2/4): styling — font composition, colour, and rich copy #155, with its timed assertions opt-in behind MDE_PERF and a structural test carrying the load on CI. After 350b2d3 I wasn't going to ship three more wall-clock ratios.

407 tests green here, 446 on #155, demo builds.