Skip to content

Apply nullable struct validity during child evaluation - #9502

Draft
Bardakor wants to merge 1 commit into
vortex-data:developfrom
Bardakor:fix/nullable-struct-zone-maps
Draft

Apply nullable struct validity during child evaluation#9502
Bardakor wants to merge 1 commit into
vortex-data:developfrom
Bardakor:fix/nullable-struct-zone-maps

Conversation

@Bardakor

@Bardakor Bardakor commented Aug 20, 2026

Copy link
Copy Markdown

Rationale for this change

Children of nullable structs are logically null whenever the parent struct row is null. Existing Vortex files store struct validity separately from the child layouts, so evaluating a child predicate without the parent validity can return incorrect results. The reproducer in #9205 shows is_null(s.a) returning no rows when one row should match.

What changes are included in this PR?

  • Keep the existing vortex.struct serialized representation and declared child dtypes unchanged.
  • Use a nullable struct reader that applies parent validity when evaluating child projections and filters.
  • Preserve child pruning for strict predicates. Parent nulls only remove logical rows, so a falsity proof over the raw child rows remains sound.
  • Skip child-only pruning for non-strict predicates such as is_null, where parent validity can make the predicate true even when child statistics say there are no nulls.
  • Preserve the field lookup and partition-expression caches used by the existing struct reader.
  • Add an end-to-end regression test covering projection, is_null, is_not_null, and a strict comparison whose raw child value matches under a null parent.

What APIs are changed? Are there any user-facing changes?

No public API or file-format changes. Existing files remain readable and newly written files keep the same struct layout representation. Scans now preserve nullable-struct semantics while retaining zone-map pruning for strict child predicates.

AI assistance

I used ChatGPT to inspect the struct layout reader and pruning paths, reason about inherited validity, backward compatibility, and strict versus non-strict predicates, implement the change, and prepare the regression tests. I reviewed the resulting diff and repository contribution requirements before updating this PR.

@Bardakor
Bardakor force-pushed the fix/nullable-struct-zone-maps branch from 2e2b2dc to 99687e3 Compare August 20, 2026 09:07
@myrrc

myrrc commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Two questions:

  1. How do you handle backward compatibility?
  2. What about performance implications? Discarding statistics from old zone maps means we can't reasonably do zone-map pruning.

Bardakor commented Aug 20, 2026

Copy link
Copy Markdown
Author

You're right on both points. I reworked the PR around that.

  1. Backward compatibility

The writer and serialized vortex.struct representation are now unchanged. Child layouts keep their declared dtypes, and the nullable struct reader combines parent validity at evaluation time. Existing files with physical child values underneath null parent rows therefore recover the correct logical semantics in new readers without requiring a file format change. Newly written files keep the same representation, so this PR does not introduce a new wire contract.

  1. Performance and pruning

I no longer discard all child zone maps. For a strict predicate, parent nulls can only remove rows from the logical field. The raw child rows are a superset of the visible rows, so if the existing child statistics prove the predicate false for every raw row, that proof remains sound after applying parent validity. Those predicates delegate to the child reader's normal pruning path.

For non-strict predicates such as is_null, child-only null_count or all_non_null cannot safely falsify the predicate because parent validity may introduce logical nulls. Only that unsafe pruning path is skipped, and parent validity is applied before evaluation.

I also added regression coverage for projection, is_null, is_not_null, and a strict comparison where the raw child value matches but the parent struct row is null.

The fresh Actions are still action_required with no jobs started and are waiting for repository approval. I am leaving the PR in draft until the actual checks run.

@Bardakor
Bardakor force-pushed the fix/nullable-struct-zone-maps branch from caa73d4 to 20b1fbf Compare August 20, 2026 16:56
Signed-off-by: Liam Abourousse <75449798+Bardakor@users.noreply.github.com>
@Bardakor
Bardakor force-pushed the fix/nullable-struct-zone-maps branch from 20b1fbf to daada40 Compare August 20, 2026 17:00
@Bardakor Bardakor changed the title Propagate nullable struct validity to child layouts Apply nullable struct validity during child evaluation Aug 20, 2026
@Bardakor
Bardakor force-pushed the fix/nullable-struct-zone-maps branch from 0518f4d to daada40 Compare August 20, 2026 17:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Children of nullable Struct may have null values propagated to zone maps

2 participants