ci: start independent checks in parallel

[kyleve]

Posted by an AI agent on kve's behalf.

Summary

• start SwiftFormat, architecture, iOS, snapshot, and macOS checks independently instead of gating every runner behind SwiftFormat
• keep SwiftFormat as its own required check, preserving formatting enforcement while removing the startup dependency
• track deletion of the temporary kve-stuff benchmark organization after its GitHub Team downgrade

Why

The format job was adding a serial startup stage to every expensive CI path even though those checks do not consume its output. The temporary benchmark infrastructure also needs an explicit cleanup task once its paid plan ends.

Testing

• git diff --check
• ruby -e 'require "yaml"; YAML.load_file(".github/workflows/ci.yml", aliases: true)'
• Test suite not run (GitHub Actions scheduling and Markdown-only changes).

[kyleve]

Posted by an AI agent on kve's behalf.

Full CI-runner investigation and outcome

This records the path from “CI is taking too long” to the two concrete changes in this PR. The original option analysis is in PR #219's runner comment, and the disposable experiment is preserved in kve-stuff/Stuff-CI-Benchmark.

The problem and initial candidate set

The expensive path was the macOS-hosted iOS/snapshot work. At the time of the investigation, the representative snapshot job median was about 26.5 minutes. The snapshot suite is deliberately serial within a runner: UIKit and simulator-global state make concurrent captures corrupt one another, and the previous parallel-testing experiment produced 24+ false mismatches while running 1.2–3× slower. Faster hardware can improve compilation, linking, rendering, PNG work, and setup, but cannot remove deliberate settle windows or safely make those captures concurrent in one host process.

The initial comparison was:

Candidate	Advertised shape	Economics considered	Initial concern
Existing GitHub standard	3-core virtual M1, 7 GiB RAM	Free for this public repository	Slowest hardware, but zero administration and the known-good renderer
GitHub XLarge	5-core M2, 14 GiB RAM	$0.102/min; about $2.70 at 26.5 minutes, $1.90 if 30% faster, or $1.62 if 40% faster	Larger runners require an organization on Team/Enterprise Cloud and included/public-repo minutes do not apply
Depot	8-core M2/M4, 24 GiB RAM, 400 GB disk	$0.08/min; about $2.12 unchanged, $1.27 if 40% faster, or $1.06 if 50% faster	Another vendor/permission boundary, finite macOS capacity, and its image had to match the exact renderer
Cirrus Runners	4 M4 Pro vCPUs, 16 GiB RAM	$150/month per concurrent runner; break-even was roughly 1,470 occupied minutes vs GitHub XLarge or 1,875 vs Depot	One slot would serialize the unit and snapshot paths; two slots would be $300/month
Self-hosted Mac mini	Whatever hardware we bought	Hardware plus power, maintenance, and time	Rejected: upgrades, simulator/disk cleanup, runner reliability, state leakage, credentials, and the risk of arbitrary public-PR code on a persistent personal machine
MacStadium/Orka	Managed bare-metal/ephemeral Apple Silicon fleet	Quoted/node-based	Kubernetes/image/fleet administration and a multi-node product shape were excessive for one public repository

The working preference was to benchmark Depot M4 first on price/performance, then GitHub XLarge as the lowest-friction controlled experiment. Cirrus was potentially interesting only at sustained usage, self-hosting was explicitly out, and MacStadium was a capable but enterprise-shaped last resort rather than the only theoretical option.

Account and safety setup

GitHub and Depot's useful macOS paths both pushed us toward an organization, so we created the temporary kve-stuff organization and authenticated the CLI through GitHub's device flow. Enabling GitHub's larger-runner eligibility required the organization plan/billing and trade-screening flow; that UI was also why the payment form was not initially visible.

The organization was deliberately disposable. We made the public Stuff-CI-Benchmark fork so none of the experiment could alter kyleve/Stuff, its branch protection, its snapshot references, or its required checks. The benchmark had:

• manual workflow_dispatch only—nothing ran on pushes or pull requests;
• read-only repository permissions and no secrets;
• a $75 total experiment ceiling;
• metadata-only probes before any paid snapshot workload;
• an exact gate for Xcode 27 beta 4 build 27A5228h plus the iOS 27 simulator;
• a renderer-compatibility requirement that all 381 checked-in references pass without re-recording;
• metadata, 38-image smoke, full, shard-1, and shard-2 workloads, with result bundles and timing reports for real test runs.

The harness landed in benchmark PR #1. During setup, current provider availability invalidated part of the original research: Cirrus CI shut down on June 1, 2026, and Cirrus Runners was no longer accepting new customers. Benchmark PR #2 removed the dead Cirrus configuration while retaining GitHub standard, GitHub XLarge, and Depot targets.

What the probes actually measured

The complete report is CI_BENCHMARK.md, added by benchmark PR #3.

Known-good GitHub control. The xcode-27 metadata run saw an Apple M1 virtual machine with 3 cores and 7 GiB RAM, macOS 26.5.2, Xcode 27 beta 4 (27A5228h), and the iOS 27 simulator. It queued for 7 seconds and completed in 44 seconds.

The 38-image smoke run queued for 8 seconds and passed all 3 selected suites / 38 images without changing references. It took 15m58s end to end. The snapshot step itself took 15m07s, approximately:

• 6m40s of test-harness pre-build/setup work;
• 3m46s building;
• 4m35s executing the 3 suites / 38 images.

About 69% of the snapshot step passed before the selected tests even began. The important finding was therefore that observed queueing—7–8 seconds—was not the source of the delay.

GitHub XLarge. The macos-26-xlarge probe started after only 3 seconds, but image macos-26-arm64/20260728.0273 exposed Xcode 26.6 (17F113), not required build 27A5228h. The exact-build gate stopped it before snapshots, as intended. The 22-second probe cost $0.10 after discounts. The machine may be faster, but running the image suite on another renderer would answer the wrong question and could invalidate all references.

Depot. Depot's current macOS 26 image advertised Xcode 26.x only, so it failed the same compatibility requirement before installation. We did not install the Depot GitHub app and did not create an account or start a trial.

Cirrus. It was not benchmarkable: Cirrus CI had shut down and Cirrus Runners was closed to new customers. No account or charge was created.

MacStadium/Orka and self-hosting. We did not provision either. MacStadium remained technically capable of giving us controlled Apple Silicon images, but doing so would buy and operate a fleet platform to solve one repository's compatibility window. A personal/self-hosted Mac remained contrary to the explicit maintenance and security constraint, especially for a public repository receiving fork PRs.

No paid candidate was renderer-compatible, so there was deliberately no full 381-image paid comparison and no evidence supporting a runner migration. The recommendation was to keep xcode-27, and only rerun the cheap metadata gates when GitHub XLarge or Depot advertises 27A5228h—or when the canonical snapshot references intentionally move to a newer Xcode renderer.

The canonical workflow also already splits the image suite into two duration-balanced matrix shards on independent standard runners. Moving those existing shards to separate runners was therefore not an unclaimed optimization. Adding more tests will still grow execution time; future extra shards can trade more runner/setup overhead for less test wall time, but the large duplicated cold setup makes improving that path valuable regardless of suite growth.

Billing and cleanup state

The experiment's recorded direct costs were:

• GitHub Team: $4 for one month, with downgrade to Free scheduled for September 9, 2026;
• GitHub Actions: $0.10 billable for the XLarge metadata probe;
• Depot and Cirrus: $0.

After the probe, the organization Actions budget was restored to $0 with “stop usage” enabled. No Depot or Cirrus integration/trial was installed. The temporary organization currently exists to retain the public benchmark fork and its evidence until the paid-plan downgrade is complete.

This PR puts the cleanup into the durable root backlog: after September 9, preserve the final report in the canonical repository if it is still useful, verify the organization is on Free with no billable usage or installed integrations, then delete kve-stuff rather than letting the experiment become permanent account clutter.

What we changed immediately

The cold-path recommendation initially sounded unsatisfying because a lot of that time is simply waiting on builds, and the test corpus will continue to grow. While filing the cleanup TODO, though, we found one avoidable serialization point in the canonical workflow: architecture, iOS tests, both snapshot shards, and macOS tests all declared needs: format. None consumes an artifact or output from SwiftFormat, so every expensive runner was waiting for the lint job solely because of the workflow graph.

Commit f422a76a removes those four needs: format edges. It does not weaken enforcement: SwiftFormat Lint remains its own check and branch protection can continue requiring it. It only allows independent checks to request runners immediately.

The first post-change CI run demonstrated the new graph: SwiftFormat, iOS tests, snapshot shard 1, and macOS tests all started at 04:32:28Z, rather than the test jobs waiting for SwiftFormat to finish at 04:33:00Z. Bumper started at 04:33:06Z and snapshot shard 2 at 04:34:34Z, reflecting runner scheduling/capacity rather than declared workflow dependencies. At the time this record was written, SwiftFormat, Bumper, and macOS had passed; iOS and both snapshot jobs were still running.

So the end state is:

1. keep the renderer-compatible GitHub standard runners for now;
2. do not pay for or operate MacStadium/self-hosting without evidence that the repository's scale justifies it;
3. rerun metadata-only paid-runner probes when their Xcode image becomes compatible;
4. attack duplicated cold setup/build cost and reconsider shard count as the suite grows;
5. start today's independent CI jobs in parallel instead of behind formatting; and
6. delete the temporary organization after its scheduled downgrade and final safety checks.