Skip to content

perf(table): group positional deletes by file path in one pass - #1724

Open
fallintoplace wants to merge 2 commits into
apache:mainfrom
fallintoplace:perf/group-pos-deletes-by-file-path
Open

perf(table): group positional deletes by file path in one pass#1724
fallintoplace wants to merge 2 commits into
apache:mainfrom
fallintoplace:perf/group-pos-deletes-by-file-path

Conversation

@fallintoplace

@fallintoplace fallintoplace commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

What

Group positional deletes by file path in one pass.

The old implementation found the unique paths and then ran an equality expression and filter over the full columns once per path. This version walks the file_path and pos values once, appends each position to its path's builder, and keeps the existing per-file map semantics.

It also:

  • checks cancellation before and during large grouping passes
  • keeps independent cursors for differently chunked columns
  • releases builders if grouping fails
  • benchmarks both single-chunk and deliberately mismatched multi-chunk inputs

Benchmark

1M rows, identical generated input on base 65005959 and this branch, Apple M1 Pro, darwin/arm64, -benchtime=100ms -count=3. Values are the median of three runs.

paths base ns/op head ns/op speedup base B/op head B/op base allocs/op head allocs/op
1 72,511,542 16,228,399 4.5x 60,418,596 17,147,496 2,000,164 39
10 163,576,708 21,973,542 7.4x 62,849,376 21,887,440 2,000,950 317
100 962,813,292 24,432,517 39.4x 87,080,904 29,639,840 2,008,794 2,513
1,000 8,186,696,000 28,130,344 291.1x 327,849,536 19,723,416 2,087,157 17,026

The benchmark also runs 64 file_path chunks against 65 position chunks. In a 500ms, three-run check, this stayed within run-to-run noise of the single-chunk case, with B/op effectively flat and at most one additional allocation per operation.

Checks

  • go test ./table -count=1
  • go test -race ./table -run '^TestGroupPosDeletesByFilePath' -count=1
  • go vet ./table
  • golangci-lint run ./table --timeout=10m
  • go test $(go list ./... | rg -v '/io/gocloud$') -count=1

@fallintoplace fallintoplace reopened this Aug 14, 2026
@fallintoplace
fallintoplace marked this pull request as ready for review August 14, 2026 19:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant