feat(layout): add a paged layout encoding to bound flatbuffer tables

[wsulais]

Refs #9447. Draft, for design feedback — the unresolved questions in that issue could change the shape of this.

Problem

A layout is a single recursive flatbuffer, verified on open against VerifierOptions::max_tables (1,000,000 by default). Each chunk layout costs one table, so past the limit the writer produces a file the default reader cannot open, and the error appears at open rather than at write.

Change

A vortex.paged layout encoding. It reports no inline children and holds one segment containing its subtree as a nested Layout flatbuffer, resolved on descent. Each page is verified separately, so max_tables and max_depth apply per page.

Opt-in, off by default:

WriteStrategyBuilder::default().with_page_size(1024)
ChunkedLayoutStrategy::new(flat).with_page_size(1024)

No flatbuffer schema change. Three existing properties make that possible: layout encodings are an open registry with ids resolved at read time via Footer.layout_specs; Layout.metadata is opaque per encoding; and LayoutReader's evaluation methods already return futures.

LayoutChildren stays synchronous. child() cannot await a segment fetch, but a page reports no children, so the descent happens in the reader instead.

Results

2,714,867,981 rows, 22 leaf columns, 49,152-row blocks. Same data, block size and encodings in both arms; only the layout differs. The paged file was verified row-identical to the inline one through Arrow at three ranges, including the last 60,000 rows.

	inline	paged @ 1024
layout nodes	1,215,246	1,264 (1,188 pages)
open at default limit	Too many tables.	opens, 54 ms
footer bytes	53,473,504	19,558,192
segments	1,215,194	1,216,382
file bytes	15,801,529,440	15,801,714,832

Page size sweep, 65,536 rows × 13 i32 columns at 64-row blocks, 13,312 chunks. root tables and max/page are the smallest table budget each message verifies under, by bisection. Each arm is asserted to read every row back and to select the same rows under a filter.

page_size	root tables	max/page	layout kB	file kB	segments	batches
inline	13,352	–	366.4	2246.0	13,325	1024
8	1,704	9	125.9	2499.6	14,989	1024
32	456	33	33.3	2309.4	13,741	1024
128	144	129	10.1	2261.9	13,429	1024
1024	53	1025	3.4	2248.1	13,338	1024

max/page is page_size + 1 at every setting, so root ≈ chunks/page_size + 3×columns. Below page_size 32 the per-page encoding dictionary dominates and the file grows measurably.

Split boundaries

batches is 1024 in every arm, including inline.

register_splits is synchronous, so a page cannot fetch its subtree to answer it. The writer promotes the subtree's chunk boundaries to the page instead. One integer per chunk would dominate the footer — 26 kB of a 35 kB layout at page_size 128 — so the uniform case is symbolic:

pub enum ChunkBoundaries {
    Uniform { len: u64, count: usize, row_count: u64 },
    Explicit(Arc<[u64]>),
}

That took page_size 128 from 35.4 kB to 10.1 kB. Non-uniform subtrees use explicit offsets and stay exact.

A page answers register_splits without touching its segment, giving a whole subtree the benefit is_indivisible gives a single child. A test asserts zero segment requests during planning.

One encoding dictionary for the file

FooterSerializer builds the file's dictionary while serializing the root layout — after the page segments are written — so a page initially carried a dictionary of its own. The second commit threads one LayoutContext through the write instead: LayoutWriterContext carries it, FooterSerializer already accepted one, and the file writer hands the same context to both. LayoutBuildContext and LayoutDeserializeArgs gained a layout_read_ctx so the read side resolves a page against the footer's dictionary.

Worth ~32 bytes per page. For 13,312 chunks the serialized layout drops from 125.9 kB to 73.6 kB at page_size 8, 10.1 kB to 6.7 kB at 128, and 3.4 kB to 2.8 kB at 1024. The page-size table above reflects this; paging's file-size overhead at page_size 1024 is now 0.08%.

Tests

12 new, 393 passing across vortex-layout and vortex-file.

• round trip through the layout crate and through a file
• root layout verifies under a budget the inline layout exceeds; so does every page
• row range spanning page boundaries
• splits identical to inline, uniform and non-uniform chunks
• planning issues zero segment requests
• uniform pages record boundaries in constant space
• filters push down through a page into the zone maps below it
• a page carries no encoding dictionary of its own
• a pruning future that is never awaited issues no segment request

Not included

• Paging any layout but chunked. write_page is generic; only ChunkedLayoutStrategy calls it.
• Automatic page sizing, or paging past a threshold rather than always.
• Sharing one encoding dictionary across pages.
• Zone-map summaries on the page boundary. I had assumed this was the largest piece of remaining value, but in the layout the file writer produces the zoned layer sits above the chunked layer and therefore above the pages, so the statistics a pruning decision needs are already outside the page. Promoting them would add metadata nothing consults. It becomes worthwhile only alongside paging layouts other than chunked — a page holding a whole column's zoned layout would need it. What the assumption was actually pointing at is fixed by the third commit: a page no longer requests its segment for a pruning future that is dropped unawaited.

Limitations

• A reader without the encoding can traverse a paged layout tree but cannot scan it (ForeignLayout::dyn_new_reader errors). Hence opt-in and off by default.
• display_tree and depth_first_traversal stop at a page.
• ChunkedLayoutStrategy gained a public field, breaking struct-literal construction.
• Paging does not reduce write-time memory: ChunkedLayoutStrategy collects every chunk layout before paging any. Measured cost of paging is +5% peak RSS on an identical 100 M-row write, 1,565 MB against 1,642 MB.
• Selective filters run about 2x faster on paged files in the sweep, but that is not a benefit of paging — it is paging supplying laziness the inline path lacks, and the same win is available without it. Attributed by counting constructions: for a filter returning 499 of 65,536 rows, the inline layout builds 13,313 FlatReaders and 13,321 array futures, one per chunk in the file, where a paged layout builds 105. ZonedReader::pruning_evaluation constructs its data child's pruning future eagerly, so ChunkedReader walks every chunk in the range building a reader — and filter_evaluation an array future, each issuing a segment_source.request() — before the zone map has pruned anything. Pruning stops those futures being polled, not being constructed; a page collapses them behind one deferred future. SplitBy::RowCount gives identical counts, so this is the evaluation path, not register_splits. Deferring construction in ChunkedReader would give the inline path the same win; filed separately as Selective filter cost scales with total chunk count, not with rows selected #9449.
• Read timings are in-memory via open_buffer. The read-amplification argument for larger pages is untested against object storage.

Notes

Built on develop. cargo fmt --check and cargo clippy --all-targets are clean for both crates, though the workspace rustfmt config wants nightly for several options and I ran stable.

Can split into "the encoding" and "wire it into the file writer" if that reviews better.

AI assistance disclosure

Developed with agentic AI assistance — Claude Opus 5 (claude-opus-5) via Claude Code 2.1.232, recorded as an Assisted-by: trailer on the commit. Written test-first. Every number above comes from a reproducible run rather than an estimate; claims I could not attribute are marked as such.

[robert3005]

Thanks for the contribution!

While I think this is an ok solution I think a better solution is fixing flatbuffer verifier. There's no reason that the verifier has to run at all in principle. However, the code generated by flatbuffers uses unchecked access which means that without upfront validation you can access invalid memory address.

Having option in flatbuffer generator or having an alternative flatbuffer binding generator would a lot better long term solution

[wsulais]

Thanks for clarifying! max_tables is a budget on the verifier walk, not on layouts.
Eager total verification against lazy partial reads is the actual mismatch: opening the 2.7B-row file verifies 1,215,246 tables for a pruned query that reads a handful of them.

I think there are three separate ceilings here:

1. The verifier budget. Your fix removes it, retroactively, for every file already written and with no opt-in. Paging doesn't do either of those.
2. One flatbuffer per layout. root_type Layout in its own Postscript segment, 32-bit offsets, so ~2 GiB — which is what max_apparent_size: 1 << 31 is expressing. Not a policy knob over a verification strategy; lazy checked access doesn't move it. Splitting into multiple buffers is the only thing that does.
3. One flatbuffer per segment map. [SegmentSpec] is a struct vector, so it never consumes tables and never trips the error — but it's 16 bytes per segment in one buffer, and paging leaves it 1:1. On the paged 2.7B-row file the footer is 19.6 MB of which 19.5 MB is segment map. That's the floor this PR can't get under.

(2) is the case where paging is necessary rather than preferable.
Extrapolating from that file at 27.9 bytes/node and 22 leaf columns:

	49,152 rows/block	8,192	2,048
2.7B rows	1.2M nodes, 34 MB	7.3M, 203 MB	29.2M, 814 MB
6.1B rows	2.7M, 76 MB	16.4M, 457 MB	65.6M, 1.83 GB

Not a contrived geometry: block size comes from a byte target with row_block_size: 8192 as the multiple, so 4-byte columns landed at 49,152 while 8-byte columns stay near the floor, and finer blocks are the direction spatial pruning pushes.
(Node/segment sizes extrapolated from the one measured file, assuming leaves dominate.)

So on the 2.7B-row artifact specifically, your fix opens it as-is and paging is an improvement rather than a requirement.
It stops being optional an order of magnitude out, and (3) is waiting behind it either way.

Aside from that, what paging buys that a verifier fix doesn't: the layout's own footer contribution goes from ~34 MB to approximately nothing, and subtrees are fetched on descent instead of at open.
Over object storage the footer still comes down whole however it's validated — lazy access only avoids that where the bytes can be mapped.

On the path, as I read it there are three:

1. a checked-accessor option in flatc's Rust codegen
2. a different binding generator. e.g. planus's read_as_root returns Result with no whole-buffer pass and fallible accessors
3. lazy verification against the current crate: verify the root Layout table and bound its children vector, defer each child to descent

(3) needs no codegen change and no format change, and is the same verify-on-descent idea as this PR applied to verification instead of to the file.
It also means hand-written verification next to unchecked reads.

I suggest that we keep this design for those use cases (e.g. LiDAR scans) and pursue one of the paths in a different PR.

[codspeed-hq]

Merging this PR will degrade performance by 11.8%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

❌ 2 regressed benchmarks
✅ 1998 untouched benchmarks
⏩ 89 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	WallTime	words_gather_scalar[65536]	8.2 µs	9.4 µs	-12.47%
❌	WallTime	words_gather_dispatch[1024]	8 ns	9 ns	-11.11%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing wsulais:feat/hierarchical-layout-pages (fc9979b) with develop (b825c4f)}

Footnotes

1. 89 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩