perf(gc): minor collections don't walk/rebuild over old-gen (#6181)

[proggeramlug]

Summary

Minor GCs did O(total-heap) work every cycle by rebuilding the old→young
remembered set from a whole-heap walk, so minor pauses grew with heap size /
process lifetime (#6181; 2026-07-09 GC audit §7). This PR ships Fix 2 of
that item and defers Fix 1 with a specific, documented blocker (below).

Fix 2 (shipped): minor collections no longer run the from-scratch
OldToYoungRememberedRebuildState walk (a full ArenaObjectCursor over every
arena object + the entire malloc registry) in
AtomicFinalize → RememberedSetRebuild. Full cycles are unchanged.

The invariant each removed old-gen touch maintained, and how it's preserved

The whole-heap rebuild produced live_old_to_young_sticky, restored in
Reclaim after remembered_set_clear, to re-establish every surviving
old→young edge so the next minor's remembered-set root scan marks the young
children. For a minor that job is already done by three mechanisms that run
regardless:

1. Write barriers record every old→young edge at mutation time (the RS is
   maintained incrementally during the mutator phase).
2. evacuation_sticky (rebuild_evacuated_old_to_young_remembered_set
   over this cycle's moved headers, built in atomic_finalize_minor_prelude
   and restored in Reclaim) re-adds the old→young edges that this cycle's
   evacuation created — evac copies land in OLD_ARENA after the RS scan.
3. restore_surviving_dirty_coverage (CI: gc_write_barrier_stress red on main (missing old→young remembered-set edges → segfault) #5029) rescans the pre-clear dirty
   snapshot with the same slot predicate the evacuation verifier uses and
   re-remembers any slot still pointing into the nursery.

So for minors we skip the rebuild and leave live_old_to_young_sticky = None;
Reclaim restores only evacuation_sticky + the dirty-snapshot repair. The
require_marked=false full walk was not only redundant here but slightly worse:
it re-remembered dead-but-unswept old parents (over-retention).

RS-edge safety argument (no under-remembering / UAF)

Under-remembering is a use-after-free, so this is the load-bearing claim.
Every old→young edge present at reclaim time is one of:

• (a) mutator-created — the barrier dirtied the parent's page when the
  store happened, so the page is in the pre-clear dirty snapshot;
  restore_surviving_dirty_coverage re-remembers the slot iff the child still
  needs tracking (remembered_child_needs_tracking: nursery — including
  TENURED-in-nursery — or a live malloc child). If the child was promoted to
  OLD_ARENA this cycle, the slot was rewritten to the old copy and correctly
  dropped. This is self-maintaining across cycles by induction: each cycle's
  restore keeps the parent's page dirty for the next.
• (b) evacuation-created — covered by evacuation_sticky.

There is no third source in the state-machine minor: it never moves an object
into OLD_ARENA except via tracked evacuation. The age-bump only flips
GC_FLAG_TENURED in place (the object stays physically nursery, hence
classify_heap_generation == Nursery), so a "silent promotion" that would
create an untracked old→young edge cannot happen. The copying fast path is a
separate code path (gc/copying.rs) that this change does not touch and that
already re-remembers its promoted headers via its own
rebuild_evacuated_old_to_young_remembered_set(collector.moved_headers).

restore_surviving_dirty_coverage derives its kept set from the same walk the
PERRY_GC_VERIFY_EVACUATION verifier performs, which per #5029 is
complete-by-construction relative to that verifier — it is the authoritative
mechanism, and historically more complete than the from-scratch rebuild
(#5029 measured the rebuild covering ~10 of ~130 needed pages before the
snapshot repair was added).

Trace-based proof of bounded old work

GcCycleTrace gains remembered_set.rebuild_objects_scanned — the object-visit
count of the whole-heap rebuild walk.

• test_minor_skips_whole_heap_old_to_young_rebuild: a minor over a large
  pinned old-gen (64 old objects) reports rebuild_objects_scanned == 0
  (also asserted in the trace JSON); a full cycle over the same heap reports
  >= 64, proving the counter is wired and the minor's 0 is a genuine skip.
• test_minor_preserves_old_to_young_edge_across_minors: an old→young edge
  recorded (via the write barrier) before a minor survives 4 successive
  minors — after each, RS root marking still reaches and marks the young
  child that is reachable only through the old parent. The child's mark is
  cleared between cycles so each minor must re-derive coverage from the
  remembered set alone. This is the under-remembering guard.

Test results

• PERRY_NO_AUTO_OPTIMIZE=1 cargo test -p perry-runtime --lib -- --test-threads=1
  → 1205 passed; 0 failed.
• PERRY_GC_VERIFY_EVACUATION=1 … gc:: → no evacuation-verify panics
  (no missing_edges / stale-forwarded / old-young-edge-verifier failed).
  Two gc::tests::copying::*_rewrites_exact_*_pointer_*_only tests fail under
  that env var because they hard-assert exact layout_scans.masked_pointer_slots_read
  counts that the VERIFY-gated verify_old_to_young_edges_covered() pass in
  gc/copying.rs perturbs — verified to fail identically on the origin/main
  base (c78903c) without this change (0 passed / 2 failed), i.e.
  pre-existing env-var sensitivity in the copying fast path, which this PR does
  not touch.
• cargo fmt --all -- --check clean. No file > 2000 lines.

Fix 1 (deferred): restrict the minor SWEEP's old-gen object walk — blocker

Fix 1 (make the minor sweep cursor skip the old-gen region) has a concrete
mark-invariant blocker I could not cleanly preserve within a correctness-first
scope, so per the task guidance I did not guess:

• Empirically (probes on this base): a minor does mark root-reachable and
  young→old-reachable old objects (mark_mutable_root_slots_step uses
  try_mark_value, which marks any heap object), and the minor sweep
  finalizes unmarked old objects (freed/dead-accounted). The sweep's old-gen
  walk is therefore load-bearing: it clears the MARKED bit those minor
  marks set on old objects.
• The full GC has no mark-clear pass (clear_marks is #[cfg(test)]
  only) — it assumes marks are clean at entry, which today holds only because
  the minor sweep clears the old marks it set. If a minor skipped the old-gen
  walk, root/young-reachable old objects would retain stale MARKED across to
  the next full GC, which would then treat them as already-marked, skip tracing
  their fields, and free their exclusive old children — a UAF.

A safe Fix 1 needs one of: (i) suppress old-object marking in minors (a
minor-specific gate in the hot, shared try_mark_value path), or (ii) a
full-GC mark-clear pass plus track the (bounded) set of old objects marked
this minor for cheap clearing plus delta-maintain the old_page_*
live/dead/pinned/dirty accounting and preserve old-page evacuation's reliance
on accurate per-cycle marks. That is a larger, riskier change than fits here;
Fix 2 alone removes the bigger cost anyway (an O(all objects) walk, a
superset of the O(old-gen) sweep walk).

Reference: #6181; 2026-07-09 GC audit.

Summary by CodeRabbit

• Bug Fixes

  • Minor garbage collections now skip the heavier old-to-young rebuild step, improving GC behavior and avoiding unnecessary work.
  • Full garbage collections now report how many heap objects were scanned during rebuild, making GC diagnostics more informative.
  • Old-to-young references are preserved more reliably across repeated minor collections.

• Tests

  • Added coverage for minor vs. full collection behavior and remembered-set stability across multiple cycles.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 902faf0c-cc95-4438-aa96-382934fbde63

📥 Commits

Reviewing files that changed from the base of the PR and between 982e49c and 07899c9.

📒 Files selected for processing (4)

• crates/perry-runtime/src/gc/cycle.rs
• crates/perry-runtime/src/gc/telemetry.rs
• crates/perry-runtime/src/gc/tests/oldgen.rs
• crates/perry-runtime/src/gc/verify.rs

---

📝 Walkthrough

Walkthrough

Minor GC cycles now skip the whole-heap old→young remembered-set rebuild, transitioning directly to Sweep, while full cycles record a new objects-scanned count. OldToYoungRememberedRebuildState tracks visited objects, telemetry exposes this via rebuild_objects_scanned, and new tests validate minor-cycle behavior.

Changes

Minor cycle rebuild skip and scanned-object tracking

Layer / File(s)	Summary
Objects-scanned counter in rebuild state crates/perry-runtime/src/gc/verify.rs	OldToYoungRememberedRebuildState adds an objects_scanned field, an accessor method, and increments the counter during arena and malloc-object traversal in step().
Cycle phase logic skips rebuild for minors crates/perry-runtime/src/gc/cycle.rs	AtomicFinalizeSubphase::RememberedSetRebuild transitions minor cycles directly to Sweep without building/stepping the rebuild state; full cycles record scanned objects into trace, set live_old_to_young_sticky, and advance to WeakProcessing.
Telemetry field for rebuild scanned objects crates/perry-runtime/src/gc/telemetry.rs	GcCycleTrace gains old_to_young_rebuild_objects_scanned, initialized to 0 and exported as rebuild_objects_scanned in JSON output.
Tests for minor rebuild skip and edge preservation crates/perry-runtime/src/gc/tests/oldgen.rs	New tests confirm minor cycles skip the whole-heap rebuild while full cycles scan pinned objects, and verify old→young edges persist across repeated minor cycles via remembered-set marking.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Cycle as GcCycle
  participant Rebuild as OldToYoungRememberedRebuildState
  participant Trace as GcCycleTrace

  Cycle->>Cycle: RememberedSetRebuild subphase
  alt minor cycle
    Cycle->>Cycle: skip rebuild, set phase Sweep
  else full cycle
    Cycle->>Rebuild: step(budget)
    Rebuild-->>Cycle: finish() -> live_old_to_young_sticky
    Cycle->>Trace: old_to_young_rebuild_objects_scanned
    Cycle->>Cycle: set subphase WeakProcessing
  end

Loading

Possibly related issues

• [perf] Minor GC does O(old-gen) work: full-region sweep walk, whole-heap RS rebuild, per-cycle page-meta iterations #6181: Targets the same old→young remembered-set rebuild path, making minor GC skip the whole-heap rebuild.

Possibly related PRs

• PerryTS/perry#6178: Both PRs modify gc/verify.rs old→young remembered-set logic, one adjusting edge trackability and the other adding scanned-object counting and minor-cycle skip behavior.

Suggested labels: run-extended-tests

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly matches the main change: skipping minor GC old-gen rebuild work for performance.
Description check	✅ Passed	The description is mostly complete, covering summary, changes, issue reference, and test results, though the template sections are not strictly followed.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch perf/gc-minor-young-regions

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}