feat(like): support LIKE over Binary arrays with byte semantics - #9497
feat(like): support LIKE over Binary arrays with byte semantics#9497moshap-firebolt wants to merge 1 commit into
Conversation
LIKE was restricted to Utf8 inputs: `Like::return_dtype` rejected Binary haystacks. But SQL LIKE over binary/bytea is well defined as a byte-wise match, and the matcher already runs over the haystack bytes. - `Like::return_dtype` accepts Binary input for case-sensitive LIKE. ILIKE over Binary is still rejected, since case folding is only well defined over text. - The regex fallback in `LikePattern` compiles with `.unicode(false)` when the haystack is Binary, so `_`/`.` match a single byte and `%`/`.*` span arbitrary bytes (including invalid UTF-8). Utf8 haystacks keep Unicode codepoint semantics, unchanged. The literal fast paths (Eq/StartsWith/EndsWith/Contains) were already byte-based. Utf8 LIKE behavior is unchanged. Added tests covering the byte-vs-codepoint distinction on multi-byte and invalid-UTF-8 data, and the ILIKE-over-Binary rejection. On `aЖb` (Ж = 2 bytes), `a_b` matches under Utf8 (one codepoint) but not under Binary (one byte); `a__b` matches under Binary. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Merging this PR will improve performance by 12.87%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ⚡ | WallTime | words_gather_scalar[65536] |
9.4 µs | 8.2 µs | +14.04% |
| ⚡ | Simulation | compress_fsst[(500, 64, 8)] |
585.9 µs | 524.5 µs | +11.71% |
Tip
Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.
Comparing like-on-binary (721b719) with develop (b81420e)
Footnotes
-
517 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
connortsui20
left a comment
There was a problem hiding this comment.
Thanks for the PR!
This causes an issue with how we rewrite LIKE, which assumes that its input is only UTF8.
vortex/vortex-array/src/stats/rewrite/builtins.rs
Lines 331 to 383 in b81420e
This approximately translates the LIKE to min > UTF8_LITERAL OR max < UTF8_LITERAL, where min and max are stats with the same dtype as the array/column. So once we allow binary columns as input to LIKE, we will try to compare a binary scalar (min/max) with a Utf8 literal scalar.
I think that there are a few possible solutions here, and to be honest neither of them seem the most ideal:
- We could just not use stats rewriting / pruning for binary
LIKE, and this is probably the simplest thing - We could coerce the
Utf8pattern intoBinarywhen we see this pattern, and then the comparison comes for free (we are trying to move away from implicit coercions in Vortex) - I also thought about requiring exact-typed patterns (we could say that you must prefix a
LIKEon a binary with ab, socol_a LIKE b"prefix%", but that is a larger change.
Interested in what you are thinking?
| /// Unicode mode, matching identically to a `&str` regex on valid UTF-8 input. For Binary haystacks | ||
| /// (`byte_mode`) Unicode is disabled, so `.`/`.*` operate on single bytes and match arbitrary byte | ||
| /// sequences (including invalid UTF-8), giving SQL byte semantics for `_`/`%`. | ||
| fn regex_like(pattern: &str, case_insensitive: bool, byte_mode: bool) -> VortexResult<Regex> { |
There was a problem hiding this comment.
can this take a struct of options
There was a problem hiding this comment.
is byte_mode ascii mode?
Summary
LIKEwas restricted toUtf8inputs —Like::return_dtyperejectedBinaryhaystacks. But SQLLIKEover binary/byteais well defined as a byte-wise match, and the matcher already runs over the haystack bytes. This addsLIKEsupport forBinaryarrays with byte semantics.Changes
Like::return_dtypeacceptsBinaryinput for case-sensitiveLIKE.ILIKEoverBinaryis still rejected, since case folding is only well defined over text.LikePatternregex fallback compiles with.unicode(false)when the haystack isBinary, so_/.match a single byte and%/.*span arbitrary bytes (including invalid UTF-8).Utf8haystacks keep Unicode codepoint semantics — unchanged. The literal fast paths (Eq/StartsWith/EndsWith/Contains) were already byte-based.Semantics
For a multi-byte haystack
aЖb(Ж= 2 bytes):Utf8(codepoint)Binary(byte)a_ba__b%and the literal patterns match byte-for-byte onBinary, including across invalid-UTF-8 byte sequences.Tests
test_like_binary_byte_semantics— covers the byte-vs-codepoint distinction on multi-byte and invalid-UTF-8 data.test_ilike_over_binary_is_rejected— assertsILIKEoverBinaryerrors.Utf8LIKE/ILIKEtests continue to pass (no behavior change forUtf8).