CI: bound apt calls with timeouts and install ccache from staged bundle - #11219
Open
julek-wolfssl wants to merge 2 commits into
Open
CI: bound apt calls with timeouts and install ccache from staged bundle#11219julek-wolfssl wants to merge 2 commits into
julek-wolfssl wants to merge 2 commits into
Conversation
ccache-setup ran a raw apt-get update, so jobs whose ghcr .deb bundle had already installed cleanly still reached the mirror. On 2026-08-19 that step stalled on archive.ubuntu.com for 9-43 min and took out 11 jobs. The .deb is already in /var/cache/apt/archives - install-apt-deps stages the whole bundle - so install it with --no-download first and only fall back to the mirror. ccache was missing from the 22.04-minimal and linuxkm bundles; add it. Neither the fallback in install-apt-deps nor any of the raw apt sites had a timeout, so a wedged mirror hung instead of failing and the retry loops never fired. ci-deps-image already solved this for itself (Acquire timeouts plus `timeout`); apply the same to the consumers, and to the linuxkm producer job that was still missing it. Also recover from a dpkg interrupted by a kill, and bound the bundle pull.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR hardens CI against stalled Ubuntu apt mirrors by adding explicit Acquire timeouts plus timeout wrappers and bounded retry loops, and reduces mirror dependence by installing ccache offline from staged .deb bundles when available.
Changes:
- Add bounded apt update/install retry logic with timeouts across multiple GitHub Actions workflows and scripts.
- Install
ccachepreferentially viaapt-get --no-downloadfrom staged bundles, falling back to mirrored installs only if needed. - Include
ccachein the Ubuntu 22.04 minimal package list and the linuxkm.debbundle closure.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| .github/workflows/sbom.yml | Wrap apt operations with timeouts + retry helper to avoid mirror hangs in SBOM job. |
| .github/workflows/falcon-interop.yml | Add apt retry helper with timeouts for falcon interop/backends CI legs. |
| .github/workflows/cross-library.yml | Add bounded apt retry loops with timeouts for container-based cross-library builds. |
| .github/workflows/ci-deps-image.yml | Add ccache to linuxkm bundle package set and bound apt calls with timeouts. |
| .github/scripts/zephyr-4.x/zephyr-test.sh | Add apt timeouts to best-effort host package install to prevent silent stalls. |
| .github/ci-deps/packages-ubuntu-22.04-minimal.txt | Add ccache to the 22.04 minimal bundle package list. |
| .github/actions/install-apt-deps/action.yml | Bound docker pull and add apt timeouts + dpkg recovery in apt fallback path. |
| .github/actions/ccache-setup/action.yml | Prefer offline ccache install from staged .deb bundle; add timed retry fallback. |
Suppressed comments (2)
.github/workflows/falcon-interop.yml:188
- This job has
timeout-minutes: 20, but the apt retry function allows up to ~50 minutes (3 attempts × (120s update + 900s install) plus backoff). Reduce the retry budget so it can complete within the job timeout (e.g., 2 attempts with a smaller install timeout).
APT_OPTS=(-o Acquire::Retries=3 -o Acquire::http::Timeout=30
-o Acquire::https::Timeout=30)
apt_retry() {
local i
for i in 1 2 3; do
.github/workflows/cross-library.yml:152
- Same retry-budget concern in this job: 3 attempts × a 900s install timeout can exceed
timeout-minutes: 25, so retries may never complete. Tighten the attempt count and/or per-attempt install timeout so the loop can finish within the job budget.
APT_OPTS=(-o Acquire::Retries=3 -o Acquire::http::Timeout=30
-o Acquire::https::Timeout=30)
for i in 1 2 3; do
if timeout -k 10 120 apt-get "${APT_OPTS[@]}" update -q && \
timeout -k 10 900 apt-get "${APT_OPTS[@]}" install -y \
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+53
to
+54
| if sudo apt-get install -y --no-install-recommends \ | ||
| --no-download ccache; then |
Comment on lines
+97
to
+111
| APT_OPTS=(-o Acquire::Retries=3 -o Acquire::http::Timeout=30 | ||
| -o Acquire::https::Timeout=30) | ||
| apt_retry() { | ||
| local i | ||
| for i in 1 2 3; do | ||
| if sudo timeout -k 10 120 apt-get "${APT_OPTS[@]}" update -q && \ | ||
| sudo timeout -k 10 900 apt-get "${APT_OPTS[@]}" install -y "$@"; then | ||
| return 0 | ||
| fi | ||
| echo "::warning::apt-get failed (attempt $i/3)" | ||
| sleep $((5 * i)) | ||
| done | ||
| echo "::error::apt-get failed after 3 attempts" | ||
| return 1 | ||
| } |
Comment on lines
+70
to
+83
| APT_OPTS=(-o Acquire::Retries=3 -o Acquire::http::Timeout=30 | ||
| -o Acquire::https::Timeout=30) | ||
| for i in 1 2 3; do | ||
| if timeout -k 10 120 apt-get "${APT_OPTS[@]}" update -q && \ | ||
| timeout -k 10 900 apt-get "${APT_OPTS[@]}" install -y \ | ||
| --no-install-recommends \ | ||
| build-essential autoconf automake libtool pkg-config \ | ||
| git ca-certificates ${{ inputs.apt_packages }}; then | ||
| break | ||
| fi | ||
| test "$i" -lt 3 || { echo "::error::apt-get failed after 3 attempts"; exit 1; } | ||
| echo "::warning::apt-get failed (attempt $i/3)" | ||
| sleep $((5 * i)) | ||
| done |
| sudo timeout -k 10 120 apt-get "${APT_OPTS[@]}" update -qq >/dev/null 2>&1 \ | ||
| && sudo timeout -k 10 300 apt-get "${APT_OPTS[@]}" install -y -qq \ | ||
| python3-venv libnewlib-dev >/dev/null 2>&1 \ | ||
| || echo "==> [container] host package install skipped (apt unavailable)" |
Comment on lines
+828
to
+842
| APT_OPTS=(-o Acquire::Retries=3 -o Acquire::http::Timeout=30 | ||
| -o Acquire::https::Timeout=30) | ||
| apt_retry() { | ||
| local i | ||
| for i in 1 2 3; do | ||
| if sudo timeout -k 10 120 apt-get "${APT_OPTS[@]}" update -q && \ | ||
| sudo timeout -k 10 900 apt-get "${APT_OPTS[@]}" install -y "$@"; then | ||
| return 0 | ||
| fi | ||
| echo "::warning::apt-get failed (attempt $i/3)" | ||
| sleep $((5 * i)) | ||
| done | ||
| echo "::error::apt-get failed after 3 attempts" | ||
| return 1 | ||
| } |
Comment on lines
+195
to
+197
| APT_OPTS=(-o Acquire::Retries=3 -o Acquire::http::Timeout=30 -o Acquire::https::Timeout=30) | ||
| retry() { local i; for i in 1 2 3 4 5; do "$@" && return 0; sleep $((2**i)); done; "$@"; } | ||
| retry sudo apt-get update -q | ||
| retry sudo timeout -k 10 120 apt-get "${APT_OPTS[@]}" update -q |
cross-library's containers are bare images with no bash, so Actions runs the step under sh; the bash array APT_OPTS=(...) was a syntax error and took out all six cross-library builds. Use a plain word-splitting variable. The retry budget also has to fit the caller's timeout-minutes - install-apt-deps is used by jobs with 4 minutes - or the last attempt is cut off before it can report. Two attempts at 60s/300s instead of three at 120s/900s. The Acquire timeouts are what actually detect a wedge, so the outer bound only backstops apt wedging outside its own I/O loop, and apt resumes from archives/partial/. Also run dpkg --configure -a before the offline ccache probe, not only in the fallback loop: an interrupted dpkg would otherwise push it to the mirror.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ccache-setupran a rawapt-get updateeven when the ghcr.debbundlehad already installed cleanly, so jobs still hit the mirror. On 2026-08-19
this stalled on archive.ubuntu.com for 9-43 min and took out 11 jobs.
--no-downloadfrom the staged bundle in/var/cache/apt/archivesfirst (already staged byinstall-apt-deps), andonly fall back to the mirror if that fails.
timeoutto every raw apt call ininstall-apt-deps's fallback and the linuxkm producer job, matching whatci-deps-imagealready does, so a wedged mirror fails fast instead ofhanging with no retry.