ci: run the multi-GiB slow tests in one job, not four - #9495
ci: run the multi-GiB slow tests in one job, not four#9495joseph-isaacs wants to merge 6 commits into
Conversation
The three multi-GiB regression tests (`build_views_offsets_overflow_i32`, `compress_large_int`, `fsst_compress_offsets_overflow_i32`) were gated on the ambient `CI` env var, so they ran in every workspace test job: the instrumented coverage run, Windows, linux-arm64 and musl. The coverage job is the worst place for them, since `-Cinstrument-coverage` on an unoptimized build makes the multi-GiB allocation and fill loops far slower, and the coverage they add is negligible. Gate them with plain `#[ignore]` instead, so every test job skips them, and pass `--run-ignored all` in "Rust tests (linux-arm64)" so that one job still runs the whole suite. A `slow-multi-gib` nextest test group holds them to one at a time, since each keeps several GiB live. The `test-with` gate was evaluated at compile time, so whether a job ran the slow tests depended on the environment at build time rather than at run time; `#[ignore]` plus `--run-ignored` is decided at run time. `test-with` is now unused and is dropped, along with the 19 transitive crates it pulled in. Also fix the nextest override for these tests: the old filterset `test(a | b)` matched nothing (`|` is not regex syntax inside `test()`) and the override carried no settings, leaving the tests on the default 150s kill budget that `fsst_compress_offsets_overflow_i32` (~110s locally) was close to. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
0c71282 to
ab76cc8
Compare
`--run-ignored all` on the workspace run also un-ignored the CUDA tests, which `vortex-cuda`'s test macro marks `#[ignore]` when the runner has no GPU: 568 of them ran and failed on linux-arm64. Run the slow tests from their own step instead, selected by name, with the same cargo arguments as the step above so nothing is rebuilt. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Merging this PR will degrade performance by 0.84%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | WallTime | words_gather_scalar[65536] |
8.2 µs | 9.4 µs | -12.26% |
| ⚡ | Simulation | compact[(2048, 90)] |
1.8 µs | 1.6 µs | +12.07% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing claude/slow-ci-tests-coverage-d3dq68 (5fa8c2a) with develop (4080251)
Footnotes
-
442 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
The i32-vs-i64 decision already has a cheap unit test on `fsst_output_fits_in_i32_offsets`, but the i64 branch itself was only reachable through `fsst_compress_offsets_overflow_i32`, which must build ~3.2 GiB to push FSST output past `i32::MAX`. That size is not padding: escape coding is exactly 2x, so crossing the boundary needs ~1.07 GiB of input and the test already sits within 1% of that minimum. `compress_views` is generic over the offset type, so drive it as `i64` directly on three short strings: the branch is covered in 10ms, and the multi-GiB test is left as an end-to-end check of the real boundary rather than the only coverage of the code path. Also build `fsst-rs` optimized under the `ci` profile. Its compression kernel dominates that test's runtime unoptimized: 106s -> 64s locally. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
`compress_large_int` asserts nothing beyond "does not error", so what it guards was not recorded anywhere. Printing the encoding tree at several sizes shows it: at 50M rows the sampled ALP exponents no longer fit every value, and the result carries `patch_indices` / `patch_values` / `patch_chunk_offsets`. Below ~5M rows the sample fits the whole array and no patches are produced at all, so the structure appears only by scale. Reach it deliberately instead: keep whole numbers as the bulk and sprinkle values needing more decimal digits than the sampled exponents can represent. 200K rows reproduce the same patched-ALP tree, and varying the patch density pins the `patch_chunk_offsets` width across u8, u16 and u32. The three cases run in 1.3s against 40s for `compress_large_int`, and unlike it they assert the shape and round-trip the values. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Keeps `#[test_with::no_env(VORTEX_SKIP_SLOW_TESTS)]` alongside the `#[ignore]` gate, so setting the variable at build time still drops the multi-GiB tests from the binary, and restores it in the two sanitizer jobs so they never compile those test bodies under asan/tsan. `test-with` comes back as a dev-dependency of the three crates. The two gates do not overlap: `#[ignore]` decides at run time and is what keeps the tests out of every job's default run, while `no_env` decides at compile time and removes them from the binary entirely. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
`Rust tests (linux-arm64)` does not run everywhere: it is gated on `github.repository == 'vortex-data/vortex'`, so fork pull requests skip it, and it is skipped on pushes to develop along with the windows and coverage jobs. Hanging the multi-GiB regression tests off that job therefore left them running on upstream pull requests only — before this branch they still ran on develop pushes and on forks through the musl job. `Rust tests (linux-musl)` runs in all three cases: it has no repository gate, falls back to `ubuntu-latest` off the upstream org, and runs on pushes to develop. Move the step there, keeping the explicit test names and the cargo arguments that match the step above it so nothing is rebuilt. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Rationale for this change
The three multi-GiB regression tests —
build_views_offsets_overflow_i32,compress_large_intandfsst_compress_offsets_overflow_i32— were gated on#[test_with::env(CI)], and GitHub Actions setsCIin every job, so they ran four times per CI run:Rust tests (coverage),(windows-x64),(linux-arm64)and(linux-musl).The coverage job is the worst host for them:
-Cinstrument-coverageon an unoptimized build makes the multi-GiB allocation and fill loops far slower, and the coverage they contribute is negligible.Two latent problems turned up along the way:
test_with::envis evaluated at compile time, and the variable is not tracked by cargo or sccache — setting it without touching the source leaves the test ignored. Whether a job ran the slow tests therefore depended on the build environment rather than the run environment.filter = 'test(compress_large_int | fsst_compress_offsets_overflow_i32)'matches nothing (|is not regex syntax insidetest()), and the override carried no settings. They were running on the default 150s kill budget, whichfsst_compress_offsets_overflow_i32(~80s on arm64) was uncomfortably close to.What changes are included in this PR?
Run them once, everywhere they used to run
#[ignore], so every job skips them by default — a run-time decision rather than a compile-time one.#[test_with::no_env(VORTEX_SKIP_SLOW_TESTS)]stays alongside it, so setting that variable at build time still drops them from the binary, which is how the two sanitizer jobs avoid compiling those bodies under asan/tsan.Rust tests (linux-musl)with--run-ignored only, naming the three explicitly. That job is the one that covers every event:Rust tests (linux-arm64)is gated ongithub.repository == 'vortex-data/vortex'and is skipped on pushes to develop along with the windows and coverage jobs, so gating them there would leave fork PRs and develop untested.--run-ignored allun-ignores the CUDA tests thatvortex-cuda's test macro marks#[ignore]when the runner has no GPU (568 failures, seen on this branch).slow-multi-gibtest group (max-threads = 1) so they never overlap — each keeps several GiB live — and a 20-minute budget instead of 150s.Make the remaining run cheaper
fsst-rsoptimized under theciprofile. Its compression kernel dominatedfsst_compress_offsets_overflow_i32unoptimized: 79.7s → 7.9s on arm64, no longer even flaggedSLOW.Stop depending on scale for coverage
codes_offsets_i64_path_roundtripsdrives the genericcompress_viewswith the i64 offset type directly, on three short strings (10ms), so that branch is covered without the ~3.2 GiB test. That size is not padding — FSST escape coding is exactly 2×, so crossingi32::MAXneeds ~1.07 GiB of input and the existing test sits within 1% of the minimum.alp_patches_are_chunk_indexedreplaces whatcompress_large_intwas reaching by accident. Dumping the encoding tree by size shows the 50M-row array is the point where the sampled ALP exponents stop fitting and the result gainspatch_indices/patch_values/patch_chunk_offsets; below ~5M rows there are no patches at all. Sprinkling values the sampled exponents can't represent reproduces that structure at 200K rows, and varying the density pins thepatch_chunk_offsetswidth across u8, u16 and u32 — 1.3s for three cases that assert the shape and round-trip the values, against 40s for one that asserted neither.Local repro for a slow test:
Checks
yamllint --strict -c .yamllint.yamlon both changed workflows — clean.cargo clippy -p vortex-array -p vortex-btrblocks -p vortex-fsst --all-targets --all-features— clean.3 tests run: 3 passedin 26.2s with no rebuild.cargo +nightly fmt(no nightly toolchain in this environment); stablecargo fmt --checkreports no diff in any touched file.What APIs are changed? Are there any user-facing changes?
None. CI configuration and test gating only.