feat: -mode encode-marked, and optional MySQL executable-comment lexing

[blotus]

Two changes that make injection structure visible in the token stream, where it is currently dropped. Nothing existing changes — both are opt-in and every default is byte-identical to today.

1. -mode encode-marked, which keeps quote positions (cmd/sqlprocessor).
2. WithExecutableComments, which lexes MySQL executable comment bodies as SQL (lexer).

---

1. -mode encode-marked

Why

encode deletes quotes before lexing. That is the right call and has to stay: a single dangling quote makes the lexer swallow the rest of the input as one INCOMPLETE_STRING, and the whole sample loses its structure.

But deleting them also erases the shape of the commonest injection there is. After the strip:

"anything' OR 'x'='x"   ->  IDENT SPACE KEYWORD SPACE IDENT OPERATOR IDENT
"anything or x=x"       ->  IDENT SPACE KEYWORD SPACE IDENT OPERATOR IDENT

An attack and an ordinary English phrase produce the identical sequence. Anything consuming this encoding is structurally unable to separate them — it is not a matter of a better consumer, the information is gone before it arrives.

What

encode-marked splits the line at each quote, lexes each segment separately, and emits a QUOTE token between them:

"anything' OR 'x'='x"   ->  IDENT QUOTE SPACE KEYWORD SPACE QUOTE IDENT QUOTE OPERATOR QUOTE IDENT
"O'Brien"               ->  IDENT QUOTE IDENT
"'-'"                   ->  QUOTE OPERATOR QUOTE
"'; DROP TABLE users; --"  ->  QUOTE PUNCTUATION SPACE COMMAND SPACE KEYWORD SPACE IDENT PUNCTUATION SPACE COMMENT

A quote still never reaches the lexer, so the swallowing problem stays fixed. QUOTE is inserted by the encoding, not produced by the lexer; it is upper-case and space-free like every other type name, so it survives a whitespace split unchanged.

readLine now takes a stripQuotes flag instead of always stripping, since this mode needs the quotes the other deletes. Every other caller passes true.

Two behaviours worth reviewing

• This is a different encoding, not a refinement. A quote becomes a lexical boundary, so 12'34 is NUMBER QUOTE NUMBER here and a single NUMBER under the strip. Anything fitted to one encoding cannot consume the other — they are not mix-and-match.
• Quote-only lines stop being skipped. readLine strips before processReader's blank check, so "'" currently arrives empty and is dropped. Under encode-marked it is not empty and encodes as QUOTE. That is intended, but it means output line numbers differ between the two modes for such inputs.

Testing

Adds the first tests for cmd/sqlprocessor: both encodings, the stripQuotes flag, and the property that no quote ever reaches the lexer (no STRING/INCOMPLETE_STRING/QUOTED_IDENT in marked output). The existing suite is unchanged and passes.

Separately verified against an independent Go reimplementation of this encoding on 154 captured vectors — byte-identical output.

Context

This came out of investigating why a consumer of encode missed quoted tautologies that a rule-based detector caught. Retraining on encode-marked closed most of that gap and improved detection at every false-positive budget we measured. Happy to share the numbers internally; keeping them out of a public PR body.

---

2. WithExecutableComments

MySQL executes the body of /*! ... */ and /*!NNNNN ... */ when the server is at least version NNNNN. So this is not a comment:

SELECT * FROM t WHERE id=1/*!50000union select pw from users*/

On any MySQL >= 5.0 that runs the union. The lexer emits the whole construct as a single MULTILINE_COMMENT, so the payload arrives as:

NUMBER MULTILINE_COMMENT

The statement is still there, inside the comment token's value, but anything reading token types cannot see it. Worse, the construct that hides a full union select also shrinks the sample to two tokens, so a consumer that treats short fragments as uninteresting will wave it through.

With the option enabled the opening delimiter and version gate are consumed and the body is lexed as ordinary SQL:

NUMBER KEYWORD SPACE COMMAND SPACE IDENT SPACE KEYWORD SPACE IDENT

which is what the server executes.

Details worth reviewing

• Off by default, deliberately. Normalization wants a MySQL optimizer hint (/*!40101 SET NAMES utf8 */) to stay a comment; enabling this would move obfuscated output for existing callers. sqlprocessor gains -executable-comments, also off, because the encodings it emits are what downstream models were fitted to — enable and retrain together.
• A version gate is exactly five digits. 50000 is 5.0.0. Shorter runs are not gates, so /*!500or 1=1*/ keeps 500 in the body, where it lexes as a NUMBER.
• A */ inside a string does not end the body. The body is scanned as SQL rather than searched for a delimiter, so a quoted literal containing */ is consumed as a string — matching MySQL. This incidentally fixes 1/*!50000select '*/' */, which the current path mis-lexes into an INCOMPLETE_STRING.
• No recursion. The delimiters emit no token, so scanning has to continue past them. Scan does that in a loop; recursing once per delimiter would grow the stack in proportion to the input, which matters for a lexer reading attacker-controlled bytes. There is a test for 200k empty comments.
• The disabled path is unchanged in cost. A benchmark A/B is inside the noise floor — measured by comparing the base branch against itself, which swings −4.9% to +5.3% on this machine, wider than any delta the change shows.
• Interaction with encode-marked: a quote ends a segment and the lexer scanning it, so an executable comment opened before a quote does not stay open across it. Quote positions are the point of that encoding, so they win. Pinned in a test.

Testing

Lexer table tests for both settings: the default single-token behaviour, body lexing with and without a version gate, the sub-five-digit case, plain comments left untouched, the */-in-a-string case, an empty comment, an unterminated one, and the 200k-comment stack test. Plus a sqlprocessor test covering the flag and the quote-boundary interaction. Existing suite unchanged and passing.

Context

Found while checking which mutations evade a token-type encoding. This one is not a near-miss: the payload is dropped by a short-fragment rule because the evasion shrinks it, so it never reaches scoring at all.

🤖 Generated with Claude Code