feat: Implement map-to-string casting#4647
Conversation
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
andygrove
changed the title
| query expect_fallback(Cast from MapType) | ||
| -- Empty map: still falls back because planning sees `map()` as `Map<NullType, NullType>`, | ||
| -- which reaches the existing NullType -> StringType cast fallback. | ||
| query expect_fallback(Cast from NullType to StringType is not supported) |
There was a problem hiding this comment.
not sure if this is the intended behavior. How should we handle the empty map(Map<NullType, NullType) case?
There was a problem hiding this comment.
technically we can cast it, we have everything for it.
Perhaps it would be enough to add NullType to list of supported types?
There was a problem hiding this comment.
@comphead Thanks for review. I updated the code to support empty maps. Please refer to the PR description for details.
Allow map nulltype key value to execute natively for cast map to string by enabling nulltype support in cast and codegen input output paths. Update SQL expectation for empty map from fallback to native query. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
| @@ -672,6 +673,8 @@ private[codegen] object CometBatchKernelCodegenInput { | |||
| if (elementNullable) " if (isNullAt(i)) return null;\n" | |||
There was a problem hiding this comment.
This is intentional. NullType has no typed getter to override (there is no getNull API), so we emit no scalar getter here. Null semantics are handled via isNullAt (always true for NullVector) and generic get(..., NullType) dispatch, which returns null.
2 participants
Which issue does this PR close?
Closes #4631.
Rationale for this change
This PR implements native map-to-string casting in Comet to match Spark formatting (
{k -> v, ...}).As part of completing this feature, it also covers the empty-map case (
cast(map() as string)) where Catalyst infersMap<NullType, NullType>.What changes are included in this PR?
native/spark-expr.MapType -> StringTypeinCometCast.NullType -> StringTypein cast support checks.Map<NullType, NullType>:NullTypeNullVector, null-only write/getter handling)cast(map() as string)runs natively instead of fallback.How are these changes tested?
PROFILES='-Pspark-4.0' make format./mvnw -q test -DwildcardSuites=CometSqlFileTestSuite -Dsuites='org.apache.comet.CometSqlFileTestSuite cast_complex_types_to_string' -Dtest=none