Skip to content

ci: run the multi-GiB slow tests in one job, not four - #9495

Draft
joseph-isaacs wants to merge 6 commits into
developfrom
claude/slow-ci-tests-coverage-d3dq68
Draft

ci: run the multi-GiB slow tests in one job, not four#9495
joseph-isaacs wants to merge 6 commits into
developfrom
claude/slow-ci-tests-coverage-d3dq68

Conversation

@joseph-isaacs

@joseph-isaacs joseph-isaacs commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Rationale for this change

The three multi-GiB regression tests — build_views_offsets_overflow_i32, compress_large_int and fsst_compress_offsets_overflow_i32 — were gated on #[test_with::env(CI)], and GitHub Actions sets CI in every job, so they ran four times per CI run: Rust tests (coverage), (windows-x64), (linux-arm64) and (linux-musl).

The coverage job is the worst host for them: -Cinstrument-coverage on an unoptimized build makes the multi-GiB allocation and fill loops far slower, and the coverage they contribute is negligible.

Two latent problems turned up along the way:

  • test_with::env is evaluated at compile time, and the variable is not tracked by cargo or sccache — setting it without touching the source leaves the test ignored. Whether a job ran the slow tests therefore depended on the build environment rather than the run environment.
  • The nextest override meant to protect them was a no-op: filter = 'test(compress_large_int | fsst_compress_offsets_overflow_i32)' matches nothing (| is not regex syntax inside test()), and the override carried no settings. They were running on the default 150s kill budget, which fsst_compress_offsets_overflow_i32 (~80s on arm64) was uncomfortably close to.

What changes are included in this PR?

Run them once, everywhere they used to run

  • Gate the three with plain #[ignore], so every job skips them by default — a run-time decision rather than a compile-time one. #[test_with::no_env(VORTEX_SKIP_SLOW_TESTS)] stays alongside it, so setting that variable at build time still drops them from the binary, which is how the two sanitizer jobs avoid compiling those bodies under asan/tsan.
  • Run them from a second step in Rust tests (linux-musl) with --run-ignored only, naming the three explicitly. That job is the one that covers every event: Rust tests (linux-arm64) is gated on github.repository == 'vortex-data/vortex' and is skipped on pushes to develop along with the windows and coverage jobs, so gating them there would leave fork PRs and develop untested.
  • The step's cargo arguments match the step above it, so nothing is rebuilt. Naming the tests also matters: a bare --run-ignored all un-ignores the CUDA tests that vortex-cuda's test macro marks #[ignore] when the runner has no GPU (568 failures, seen on this branch).
  • Fix the nextest override: a correct filterset, a slow-multi-gib test group (max-threads = 1) so they never overlap — each keeps several GiB live — and a 20-minute budget instead of 150s.

Make the remaining run cheaper

  • Build fsst-rs optimized under the ci profile. Its compression kernel dominated fsst_compress_offsets_overflow_i32 unoptimized: 79.7s → 7.9s on arm64, no longer even flagged SLOW.

Stop depending on scale for coverage

  • codes_offsets_i64_path_roundtrips drives the generic compress_views with the i64 offset type directly, on three short strings (10ms), so that branch is covered without the ~3.2 GiB test. That size is not padding — FSST escape coding is exactly 2×, so crossing i32::MAX needs ~1.07 GiB of input and the existing test sits within 1% of the minimum.
  • alp_patches_are_chunk_indexed replaces what compress_large_int was reaching by accident. Dumping the encoding tree by size shows the 50M-row array is the point where the sampled ALP exponents stop fitting and the result gains patch_indices / patch_values / patch_chunk_offsets; below ~5M rows there are no patches at all. Sprinkling values the sampled exponents can't represent reproduces that structure at 200K rows, and varying the density pins the patch_chunk_offsets width across u8, u16 and u32 — 1.3s for three cases that assert the shape and round-trip the values, against 40s for one that asserted neither.

Local repro for a slow test:

cargo test --release -p vortex-fsst fsst_compress_offsets -- --ignored

Checks

  • yamllint --strict -c .yamllint.yaml on both changed workflows — clean.
  • cargo clippy -p vortex-array -p vortex-btrblocks -p vortex-fsst --all-targets --all-features — clean.
  • CI on this head: all test jobs green; the musl job's slow step reports 3 tests run: 3 passed in 26.2s with no rebuild.
  • Could not run cargo +nightly fmt (no nightly toolchain in this environment); stable cargo fmt --check reports no diff in any touched file.

What APIs are changed? Are there any user-facing changes?

None. CI configuration and test gating only.

The three multi-GiB regression tests (`build_views_offsets_overflow_i32`,
`compress_large_int`, `fsst_compress_offsets_overflow_i32`) were gated on the
ambient `CI` env var, so they ran in every workspace test job: the instrumented
coverage run, Windows, linux-arm64 and musl. The coverage job is the worst place
for them, since `-Cinstrument-coverage` on an unoptimized build makes the
multi-GiB allocation and fill loops far slower, and the coverage they add is
negligible.

Gate them with plain `#[ignore]` instead, so every test job skips them, and pass
`--run-ignored all` in "Rust tests (linux-arm64)" so that one job still runs the
whole suite. A `slow-multi-gib` nextest test group holds them to one at a time,
since each keeps several GiB live.

The `test-with` gate was evaluated at compile time, so whether a job ran the
slow tests depended on the environment at build time rather than at run time;
`#[ignore]` plus `--run-ignored` is decided at run time. `test-with` is now
unused and is dropped, along with the 19 transitive crates it pulled in.

Also fix the nextest override for these tests: the old filterset
`test(a | b)` matched nothing (`|` is not regex syntax inside `test()`) and the
override carried no settings, leaving the tests on the default 150s kill budget
that `fsst_compress_offsets_overflow_i32` (~110s locally) was close to.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
@joseph-isaacs
joseph-isaacs force-pushed the claude/slow-ci-tests-coverage-d3dq68 branch from 0c71282 to ab76cc8 Compare August 19, 2026 17:30
`--run-ignored all` on the workspace run also un-ignored the CUDA tests, which
`vortex-cuda`'s test macro marks `#[ignore]` when the runner has no GPU: 568 of
them ran and failed on linux-arm64. Run the slow tests from their own step
instead, selected by name, with the same cargo arguments as the step above so
nothing is rebuilt.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
@codspeed-hq

codspeed-hq Bot commented Aug 19, 2026

Copy link
Copy Markdown

Merging this PR will degrade performance by 0.84%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚡ 1 improved benchmark
❌ 1 regressed benchmark
✅ 1954 untouched benchmarks
⏩ 442 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
WallTime words_gather_scalar[65536] 8.2 µs 9.4 µs -12.26%
Simulation compact[(2048, 90)] 1.8 µs 1.6 µs +12.07%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing claude/slow-ci-tests-coverage-d3dq68 (5fa8c2a) with develop (4080251)

Open in CodSpeed

Footnotes

  1. 442 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

The i32-vs-i64 decision already has a cheap unit test on
`fsst_output_fits_in_i32_offsets`, but the i64 branch itself was only reachable
through `fsst_compress_offsets_overflow_i32`, which must build ~3.2 GiB to push
FSST output past `i32::MAX`. That size is not padding: escape coding is exactly
2x, so crossing the boundary needs ~1.07 GiB of input and the test already sits
within 1% of that minimum.

`compress_views` is generic over the offset type, so drive it as `i64` directly
on three short strings: the branch is covered in 10ms, and the multi-GiB test is
left as an end-to-end check of the real boundary rather than the only coverage
of the code path.

Also build `fsst-rs` optimized under the `ci` profile. Its compression kernel
dominates that test's runtime unoptimized: 106s -> 64s locally.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
`compress_large_int` asserts nothing beyond "does not error", so what it guards
was not recorded anywhere. Printing the encoding tree at several sizes shows it:
at 50M rows the sampled ALP exponents no longer fit every value, and the result
carries `patch_indices` / `patch_values` / `patch_chunk_offsets`. Below ~5M rows
the sample fits the whole array and no patches are produced at all, so the
structure appears only by scale.

Reach it deliberately instead: keep whole numbers as the bulk and sprinkle values
needing more decimal digits than the sampled exponents can represent. 200K rows
reproduce the same patched-ALP tree, and varying the patch density pins the
`patch_chunk_offsets` width across u8, u16 and u32. The three cases run in 1.3s
against 40s for `compress_large_int`, and unlike it they assert the shape and
round-trip the values.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Keeps `#[test_with::no_env(VORTEX_SKIP_SLOW_TESTS)]` alongside the `#[ignore]`
gate, so setting the variable at build time still drops the multi-GiB tests from
the binary, and restores it in the two sanitizer jobs so they never compile those
test bodies under asan/tsan. `test-with` comes back as a dev-dependency of the
three crates.

The two gates do not overlap: `#[ignore]` decides at run time and is what keeps
the tests out of every job's default run, while `no_env` decides at compile time
and removes them from the binary entirely.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
`Rust tests (linux-arm64)` does not run everywhere: it is gated on
`github.repository == 'vortex-data/vortex'`, so fork pull requests skip it, and
it is skipped on pushes to develop along with the windows and coverage jobs.
Hanging the multi-GiB regression tests off that job therefore left them running
on upstream pull requests only — before this branch they still ran on develop
pushes and on forks through the musl job.

`Rust tests (linux-musl)` runs in all three cases: it has no repository gate,
falls back to `ubuntu-latest` off the upstream org, and runs on pushes to
develop. Move the step there, keeping the explicit test names and the cargo
arguments that match the step above it so nothing is rebuilt.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant