Skip to content

perf(rest): cache partition decode plans - #1803

Draft
fallintoplace wants to merge 10 commits into
apache:mainfrom
fallintoplace:feat/rest-scan-delegation
Draft

perf(rest): cache partition decode plans#1803
fallintoplace wants to merge 10 commits into
apache:mainfrom
fallintoplace:feat/rest-scan-delegation

Conversation

@fallintoplace

@fallintoplace fallintoplace commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Cache the resolved partition decode plan by spec ID while decoding REST scan tasks.
  • Reuse the plan across inline and fetched task envelopes in one remote scan.
  • Keep decoded partition values and maps independent for every file.
  • Add focused reuse tests and a decoder benchmark.

Performance

Compared with the parent commit on an Apple M1 Pro, using GOMAXPROCS=1 and 5 benchmark samples:

  • 64 tasks: 365 us -> 241 us, about 1.5x faster
  • 1024 tasks: 4.23 ms -> 3.82 ms, about 1.1x faster
  • Allocations: 2,049 -> 1,862 per operation, about 9% fewer

The cache is shared across inline and fetched task envelopes, so the setup cost is paid once per spec for a remote scan.

Testing

  • go test ./... -count=1
  • go test -race ./catalog/rest -count=1
  • go vet ./catalog/rest
  • go test ./catalog/rest -run "^$" -bench "^BenchmarkDecodeScanTasks$" -benchmem -benchtime=300ms -count=3

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant