Skip to content

Shard the long-running CORE runs of contracts-dfcc and jbmc-strings#9086

Open
tautschnig wants to merge 1 commit into
diffblue:developfrom
tautschnig:shard-macos-core-suites
Open

Shard the long-running CORE runs of contracts-dfcc and jbmc-strings#9086
tautschnig wants to merge 1 commit into
diffblue:developfrom
tautschnig:shard-macos-core-suites

Conversation

@tautschnig

Copy link
Copy Markdown
Collaborator

These two suites are the slowest non-sharded CORE ctest entries and, at a single 30-minute-capped test each, intermittently time out on the slower and highly variable macOS-14 runners (their per-run time there swings roughly 2-3x). Their timeouts (1800s) are already above the 1200s default, so the right remedy is sharding rather than a further timeout bump.

Split each suite's CORE profile into 4 parallel shards via test.pl's --shard option, reusing the add_sharded_test_pl_profile helper already used for the jbmc-strings symex-driven-lazy-loading profile. The smaller pieces run in parallel as separate ctest entries instead of one indivisible test, so they no longer dominate the job wall-clock or approach the per-test timeout. Each shard keeps the suite's previous 30-minute cap, which is now ample headroom.

The contracts-dfcc smt2_solver PATH environment is applied to each shard, and the THOROUGH/FUTURE/KNOWNBUG levels remain single tests.

  • Each commit message has a non-empty body, explaining why the change was made.
  • n/a Methods or procedures I have added are documented, following the guidelines provided in CODING_STANDARD.md.
  • n/a The feature or user visible behaviour I have added or modified has been documented in the User Guide in doc/cprover-manual/
  • Regression or unit tests are included, or existing tests cover the modified code (in this case I have detailed which ones those are in the commit message).
  • My commit message includes data points confirming performance improvements (if claimed).
  • My PR is restricted to a single feature or bugfix.
  • n/a White-space or formatting changes outside the feature-related changed lines are in commits of their own.

@tautschnig tautschnig self-assigned this Jun 25, 2026
Copilot AI review requested due to automatic review settings June 25, 2026 09:23
@tautschnig tautschnig requested review from a team, kroening and peterschrammel as code owners June 25, 2026 09:23

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR reduces wall-clock time and intermittent macOS runner timeouts by sharding two long-running regression suites’ CORE ctest entries into multiple parallel shards, using the existing add_sharded_test_pl_profile helper.

Changes:

  • Shard contracts-dfcc CORE into 4 parallel ctest tests (each retaining the previous 1800s timeout), and apply the smt2_solver PATH environment to each shard.
  • Shard jbmc-strings CORE into 4 parallel ctest tests (each retaining the previous 1800s timeout), mirroring the already-sharded symex-driven profile.
  • Broaden JBMC regression log ignore pattern to cover shard log filenames (tests-*.log).

Reviewed changes

Copilot reviewed 2 out of 3 changed files in this pull request and generated no comments.

File Description
regression/contracts-dfcc/CMakeLists.txt Replaces a single long CORE test with 4 sharded CORE tests (plus explicit THOROUGH/FUTURE/KNOWNBUG), and applies the smt2_solver PATH env to each shard.
jbmc/regression/jbmc-strings/CMakeLists.txt Converts the suite’s CORE run into 4 shards using add_sharded_test_pl_profile, while keeping other levels as single tests.
.gitignore Ignores JBMC shard log files via jbmc/regression/**/tests-*.log.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@codecov

codecov Bot commented Jun 25, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 80.69%. Comparing base (02c2769) to head (4cd5fce).

Additional details and impacted files
@@           Coverage Diff            @@
##           develop    #9086   +/-   ##
========================================
  Coverage    80.69%   80.69%           
========================================
  Files         1714     1714           
  Lines       189597   189597           
  Branches        73       73           
========================================
+ Hits        152987   152988    +1     
+ Misses       36610    36609    -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

These two suites are the slowest non-sharded CORE ctest entries and, at a
single 30-minute-capped test each, intermittently time out on the slower and
highly variable macOS-14 runners (their per-run time there swings roughly
2-3x). Their timeouts (1800s) are already above the 1200s default, so the
right remedy is sharding rather than a further timeout bump.

Split each suite's CORE profile into 4 parallel shards via test.pl's --shard
option. Add an add_sharded_test_pl_tests macro (next to add_test_pl_tests)
that shards the CORE profile and registers THOROUGH/FUTURE/KNOWNBUG as single
tests, so both suites share a single definition of the pattern rather than
hand-expanding it; it builds on the existing add_sharded_test_pl_profile
helper. The smaller pieces run in parallel as separate ctest entries instead
of one indivisible test, so they no longer dominate the job wall-clock or
approach the per-test timeout. Each shard keeps the suite's previous 30-minute
cap, which is now ample headroom.

The contracts-dfcc smt2_solver PATH environment is applied to each shard, and
the THOROUGH/FUTURE/KNOWNBUG levels remain single tests.

Co-authored-by: Kiro <kiro-agent@users.noreply.github.com>
@tautschnig tautschnig force-pushed the shard-macos-core-suites branch from 3b2b07a to 4cd5fce Compare June 25, 2026 22:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants