Report Test Cycles in PyTest and the CI Job Summary

[runwangdl]

The cycle count is already parsed out of every simulation into TestResult.runtime_cycles, and
run_and_assert_test then drops it. This surfaces it in the PyTest output and, on GitHub Actions,
collects it into a table on the job summary page.

Added

• A Performance Summary block at the end of every PyTest run, listing each test's runtime in cycles.
• The same table appended to $GITHUB_STEP_SUMMARY when running under GitHub Actions, so the numbers
  are on the check page without opening the log.

Changed

• run_and_assert_test prints [deeploy-perf] runtime_cycles=N for each simulation, before the
  assertions so a failing test still reports its cycles.

Notes on the implementation

Collection happens in pytest_runtest_logreport, which fires on the xdist master for every worker's
report. All CI jobs run pytest -n 4, so a module-level list filled inside the tests would stay in
the worker process and never reach the summary; scraping the captured stdout on the master is what
makes the four workers' numbers land in one place.

pytest_terminal_summary does run in the workers as well, each holding only the tests it ran, so
writing the job summary is guarded to the master. Without that guard the shared summary file gets one
partial table per worker plus the complete one — five tables for a four-way run.

Tests on platforms that report no cycles (Generic, XDNA2) contribute no rows, and a run with no cycle
data at all prints nothing.

Verification

Real GAP9 tests through gvsoc, -n 2:

============================= Performance Summary ==============================
  [PASS] test_platforms.py::test_gap9_kernels[Kernels/FP32/Add/Regular]            972 cycles
  [PASS] test_platforms.py::test_gap9_kernels[Kernels/Integer/Add/Regular]       6,257 cycles
  [PASS] test_platforms.py::test_gap9_kernels[Kernels/Integer/MatMul/Regular] 4,066,151 cycles
  [PASS] test_platforms.py::test_gap9_kernels[Kernels/Integer/Pad/Regular_1D]     7,263 cycles

Test	Status	Runtime (cycles)
test_platforms.py::test_gap9_kernels[Kernels/FP32/Add/Regular]	✅	972
test_platforms.py::test_gap9_kernels[Kernels/Integer/Add/Regular]	✅	6,257
test_platforms.py::test_gap9_kernels[Kernels/Integer/MatMul/Regular]	✅	4,066,151
test_platforms.py::test_gap9_kernels[Kernels/Integer/Pad/Regular_1D]	✅	7,263

Also checked: serial (no -n) gives the same table; a failing test still contributes its row; Generic
tests contribute none.

No workflow changes — GitHub sets GITHUB_STEP_SUMMARY for every step, container jobs included.

PR Merge Checklist

1. The PR is rebased on the latest devel commit and pointing to devel.
2. Your PR reviewed and approved.
3. All checks are passing.
4. The CHANGELOG.md file has been updated.
5. If the docker was modified, change back its link after review. (not modified)

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 992dd367-1b76-473a-a120-afcfe603b5d2

📥 Commits

Reviewing files that changed from the base of the PR and between 8c41b9b and bdefd58.

📒 Files selected for processing (3)

• CHANGELOG.md
• DeeployTest/conftest.py
• DeeployTest/testUtils/pytestRunner.py

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.

---

📝 Walkthrough

Summary by CodeRabbit

• New Features

  • Added per-test performance cycle reporting to PyTest output.
  • Added a sorted performance summary showing passed and failed performance checks.
  • Added a Markdown performance table to GitHub Actions job summaries when available.

• Documentation

  • Updated the Unreleased changelog with the new performance reporting capabilities.

Walkthrough

The test utilities now emit runtime-cycle markers. Pytest collects these markers from passed and failed tests, prints sorted results, and can append a performance table to the GitHub Actions job summary. The changelog documents the feature.

Changes

Performance reporting

Layer / File(s)	Summary
Runtime cycle marker emission DeeployTest/testUtils/pytestRunner.py	Adds the exported PERF_MARKER constant. run_and_assert_test prints runtime cycle data before assertions when available.
Performance result collection and reporting DeeployTest/conftest.py, CHANGELOG.md	Adds Pytest hooks that collect and sort runtime cycles, print a summary, and optionally write a Markdown table to GITHUB_STEP_SUMMARY. The changelog documents the output.
Estimated code review effort: 3 (Moderate)	~20 minutes

Merge Risk: ⚪ Minimal · up to bdefd

This PR adds runtime-cycle reporting to test output and GitHub Actions summaries without changing product behavior; no actionable merge-blocking risk remains after normal checks and review.

Sequence Diagram(s)

sequenceDiagram
  participant run_and_assert_test
  participant pytest_runtest_logreport
  participant pytest_terminal_summary
  participant GITHUB_STEP_SUMMARY
  run_and_assert_test->>pytest_runtest_logreport: emit and collect runtime cycle marker
  pytest_runtest_logreport->>pytest_terminal_summary: aggregate performance results
  pytest_terminal_summary->>GITHUB_STEP_SUMMARY: append Markdown performance table

Loading

Suggested reviewers: victor-jung

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description check	✅ Passed	The description clearly explains the per-test cycle reporting, PyTest summary, GitHub Actions integration, implementation, and verification.
Title check	✅ Passed	The title clearly and concisely describes reporting test cycles in PyTest and the CI job summary.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}