Apply nullable struct validity during child evaluation - #9502
Conversation
2e2b2dc to
99687e3
Compare
|
Two questions:
|
|
You're right on both points. I reworked the PR around that.
The writer and serialized
I no longer discard all child zone maps. For a strict predicate, parent nulls can only remove rows from the logical field. The raw child rows are a superset of the visible rows, so if the existing child statistics prove the predicate false for every raw row, that proof remains sound after applying parent validity. Those predicates delegate to the child reader's normal pruning path. For non-strict predicates such as I also added regression coverage for projection, The fresh Actions are still |
caa73d4 to
20b1fbf
Compare
Signed-off-by: Liam Abourousse <75449798+Bardakor@users.noreply.github.com>
20b1fbf to
daada40
Compare
0518f4d to
daada40
Compare
Rationale for this change
Children of nullable structs are logically null whenever the parent struct row is null. Existing Vortex files store struct validity separately from the child layouts, so evaluating a child predicate without the parent validity can return incorrect results. The reproducer in #9205 shows
is_null(s.a)returning no rows when one row should match.What changes are included in this PR?
vortex.structserialized representation and declared child dtypes unchanged.is_null, where parent validity can make the predicate true even when child statistics say there are no nulls.is_null,is_not_null, and a strict comparison whose raw child value matches under a null parent.What APIs are changed? Are there any user-facing changes?
No public API or file-format changes. Existing files remain readable and newly written files keep the same struct layout representation. Scans now preserve nullable-struct semantics while retaining zone-map pruning for strict child predicates.
AI assistance
I used ChatGPT to inspect the struct layout reader and pruning paths, reason about inherited validity, backward compatibility, and strict versus non-strict predicates, implement the change, and prepare the regression tests. I reviewed the resulting diff and repository contribution requirements before updating this PR.