perf(table): index positional deletes by path and partition

[fallintoplace]

Summary

Position delete matching currently sorts all delete entries by sequence, then builds a file_path metrics evaluator and scans the remaining suffix for every data file. This makes scan planning grow with the product of data and delete file counts.

This change builds a small index before creating scan tasks:

• single-file deletes are grouped by referenced data path, using referenced_data_file or equal file_path bounds
• the remaining deletes are grouped by partition spec and normalized partition value
• each bucket is sequence-sorted so data-file lookups can binary-search the first applicable delete

Partitioned matching follows Java DeleteFileIndex: unresolved position deletes apply within the same spec and partition, while path-scoped deletes match directly. The common path with only single-file deletes avoids partition-key construction. Existing deletion vector indexing and suppression behavior remain unchanged.

Equality delete indexing is kept separate in #1752. The two changes are related but independent.

Benchmarks

The sparse path benchmark includes index construction and matching. Each delete file targets one data file.

Data files	Delete files	Before	After	Speedup
2,000	500	162.32 ms	0.14 ms	1,159x
20,000	5,000	15.31 s	1.34 ms	11,425x

The full 20,000 by 5,000 baseline was run once because the suffix scan allocated about 12.4 GB. The smaller comparison remains in the benchmark for a practical repeatable before-and-after measurement.

The committed suite also covers partition-scoped and mixed workloads. Representative medians from three runs with three iterations each on an Apple M1 Pro:

Workload	Before	Indexed	Speedup	Memory before	Memory indexed
100 selective partitions	116.72 ms	2.07 ms	56.5x	41.8 MB	1.96 MB
Mixed path and partition deletes	154.08 ms	72.45 ms	2.1x	124.8 MB	57.85 MB

Commands:

go test ./table -run '^$' -bench '^BenchmarkPositionalDelete(IndexSparsePaths20KBy5K|PlanningSparsePaths)$' -benchmem -benchtime=3x -count=3
go test ./table -run '^$' -bench 'BenchmarkPositionalDeletePlanning(SelectivePartition|Mixed)$' -benchmem -benchtime=3x -count=3

Testing

• full repository test suite
• focused positional-delete tests under the race detector
• golangci-lint
• direct reference-equivalence matrix covering 78 mixed data/delete comparisons across path classification, spec and partition identity, sequence boundaries, and path metrics

[zeroshade]

Faithful to Java's DeleteFileIndex in the ways that are easy to get wrong: >= sequence matching for position deletes (vs the strict > in #1752's equality index — the asymmetry is the spec, and having both documented side by side is valuable), no global bucket since only equality deletes can be global, path-scoped fast path with conservative fallback to (spec, partition) when the reference is ambiguous, and metrics-evaluator pruning retained inside the partition bucket so multi-file deletes still match by bounds. The sparse-path benchmark speaks for itself, and keeping the DV suppression path untouched was the right scoping call.

Same mechanical note as #1752: today's merges touched table/scanner.go, so please rebase both siblings (in whichever order you prefer — they'll also overlap each other) and let CI re-run before merge.

---

This review was drafted with an AI-assisted tool and may contain mistakes; an Apache Iceberg Go maintainer has reviewed and confirmed the submission. See the contributing docs for what the project considers a maintainer review.

[zeroshade]

This is approved but currently conflicts with main after today's batch of merges (several of them touched the scan/delete paths). Could you rebase? I'll merge once CI is green on the updated branch.