Add a fast, explicit pre-flight failure for dead guest networking — right now a lo: DOWN guest costs the full engine timeout before anyone finds out why.
Affected workflows:
- Daily AgentRx Trace Optimizer (
.github/workflows/daily-agentrx-trace-optimizer.lock.yml) — Claude engine
- Daily Cli Tools Tester (
.github/workflows/daily-cli-tools-tester.lock.yml) — Copilot engine
Runs analyzed:
- §32220130823 — failed 2026-08-19T05:37:36Z vs last good §32103562899 (2026-08-18, success)
- §32218752360 — failed 2026-08-19T05:15:31Z, same signature, 22 minutes earlier, different engine
Evidence
Both runs hit the identical fatal sequence before any engine process starts:
[WARN] [cloud-hypervisor] stage=guest-connectivity status=failed: Cloud Hypervisor guest connectivity probe failed with exit code 4
(stderr: Connection to 172.30.0.10 3128 port [tcp/*] succeeded!; guest network state: 1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000
[ERROR] Fatal error: Error: Cloud Hypervisor guest connectivity probe failed with exit code 4 ...
This cascades into engine-specific, misleading downstream errors — the Claude run surfaces as ERR_CONFIG: Claude execution failed: no structured log entries were produced ... startup/configuration failure detected, which looks like a Claude Code CLI bug but isn't.
audit-diff between §32220130823 and the prior day's successful §32103562899 run shows zero requests to every domain the good run used — github.com, api.anthropic.com, o205451.ingest.us.sentry.io, otlp-gateway-prod-eu-west-2.grafana.net, both Fastly CDN IPs all go from allowed traffic to 0 allowed / 0 blocked. This is a complete guest-network blackout, not a selective proxy or firewall block — confirming the lo: DOWN line is the real root cause, not a downstream symptom.
Same signature reproduced independently on a second, unrelated workflow using a different engine (Copilot) 22 minutes earlier — this is infra-layer, not workflow-specific. No commits have touched sandbox/cloud-hypervisor code or either workflow file since 2026-08-13, ruling out a recent config change as the trigger.
Probable root cause
Race or regression in the Cloud Hypervisor guest VM boot sequence: the loopback interface (and by extension guest networking) is not up by the time the connectivity probe runs, so the guest can reach the host-side proxy port but has no usable network stack internally. Likely a runner-image or cloud-hypervisor-version regression rather than anything in workflow YAML/Markdown.
Proposed remediation
- Turn the existing "guest connectivity probe failed" fatal error into a distinct, clearly-labeled
##[error] (e.g. SANDBOX_NETWORK_INIT_FAILED) at the point of detection, instead of letting each engine's harness invent its own downstream diagnosis (ERR_CONFIG, etc.).
- Add a bounded retry/re-probe (e.g. 2 attempts with a short backoff) on the guest-connectivity check before declaring the sandbox unusable — a boot race is plausible given
lo specifically reports DOWN rather than absent.
- Capture and surface
ip addr/ip link guest state in the probe failure log by default (already present here, but only because the probe itself dumps it) so this is easy to grep for in future failures without re-deriving it from raw job logs.
Success criteria
5 consecutive scheduled runs each of Daily AgentRx Trace Optimizer and Daily Cli Tools Tester complete without a Cloud Hypervisor guest connectivity probe failed fatal error; if the probe does fail, the job surfaces a single unambiguous SANDBOX_NETWORK_INIT_FAILED-style error instead of an engine-specific misdiagnosis.
Related: #52459 (Anthropic proxy ConnectionRefused — different layer, do not merge), #52253 (Copilot proxy ECONNREFUSED port 10002 — different layer, do not merge)
Related to #53933
Generated by 🔍 [aw] Failure Investigator (6h) · agent · 167.2 AIC · ⌖ 10.4 AIC · ⊞ 5.9K · ◷
Add GitHub API Consumption Report Agent as a third confirmed occurrence — expand the fix scope before patching only the two workflows already listed.
New evidence (2026-08-20 10:51 UTC window)
- Workflow:
GitHub API Consumption Report Agent (.github/workflows/api-consumption-report.lock.yml), Claude engine, sandbox.agent.runtime: cloud-hypervisor.
- Failed run: §32361003138 — job annotation reads verbatim:
ERR_CONFIG: Claude execution failed: no structured log entries were produced. Claude startup failed before structured logging (exitCode=unknown). startup/configuration failure detected.
ERR_CONFIG misdiagnosis signature already called out in this issue's root-cause section as the Claude-engine symptom of the guest-network blackout — not a Claude Code CLI bug.
Why this matters
Fix the remediation plan to cover 3 workflows across 2 engines (Claude, Copilot) now, not 2 — the guest-network boot race is not isolated to Daily AgentRx Trace Optimizer / Daily Cli Tools Tester. Any runner-image or Cloud Hypervisor pin/rollback fix must be validated against this workflow too before closing.
Updated affected workflow list
- Daily AgentRx Trace Optimizer (Claude) — original
- Daily Cli Tools Tester (Copilot) — original
- GitHub API Consumption Report Agent (Claude) — new, §32361003138
Success criteria unchanged; broaden the "5 consecutive clean runs" check to include this workflow.> Generated by 🔍 [aw] Failure Investigator (6h) · agent · 137.8 AIC · ⌖ 8.41 AIC · ⊞ 5.9K · ◷
Add a fast, explicit pre-flight failure for dead guest networking — right now a
lo: DOWNguest costs the full engine timeout before anyone finds out why.Affected workflows:
.github/workflows/daily-agentrx-trace-optimizer.lock.yml) — Claude engine.github/workflows/daily-cli-tools-tester.lock.yml) — Copilot engineRuns analyzed:
Evidence
Both runs hit the identical fatal sequence before any engine process starts:
This cascades into engine-specific, misleading downstream errors — the Claude run surfaces as
ERR_CONFIG: Claude execution failed: no structured log entries were produced ... startup/configuration failure detected, which looks like a Claude Code CLI bug but isn't.audit-diffbetween §32220130823 and the prior day's successful §32103562899 run shows zero requests to every domain the good run used —github.com,api.anthropic.com,o205451.ingest.us.sentry.io,otlp-gateway-prod-eu-west-2.grafana.net, both Fastly CDN IPs all go from allowed traffic to 0 allowed / 0 blocked. This is a complete guest-network blackout, not a selective proxy or firewall block — confirming thelo: DOWNline is the real root cause, not a downstream symptom.Same signature reproduced independently on a second, unrelated workflow using a different engine (Copilot) 22 minutes earlier — this is infra-layer, not workflow-specific. No commits have touched sandbox/cloud-hypervisor code or either workflow file since 2026-08-13, ruling out a recent config change as the trigger.
Probable root cause
Race or regression in the Cloud Hypervisor guest VM boot sequence: the loopback interface (and by extension guest networking) is not up by the time the connectivity probe runs, so the guest can reach the host-side proxy port but has no usable network stack internally. Likely a runner-image or cloud-hypervisor-version regression rather than anything in workflow YAML/Markdown.
Proposed remediation
##[error](e.g.SANDBOX_NETWORK_INIT_FAILED) at the point of detection, instead of letting each engine's harness invent its own downstream diagnosis (ERR_CONFIG, etc.).lospecifically reportsDOWNrather than absent.ip addr/ip linkguest state in the probe failure log by default (already present here, but only because the probe itself dumps it) so this is easy to grep for in future failures without re-deriving it from raw job logs.Success criteria
5 consecutive scheduled runs each of Daily AgentRx Trace Optimizer and Daily Cli Tools Tester complete without a
Cloud Hypervisor guest connectivity probe failedfatal error; if the probe does fail, the job surfaces a single unambiguousSANDBOX_NETWORK_INIT_FAILED-style error instead of an engine-specific misdiagnosis.Related: #52459 (Anthropic proxy ConnectionRefused — different layer, do not merge), #52253 (Copilot proxy ECONNREFUSED port 10002 — different layer, do not merge)
Related to #53933
Add
GitHub API Consumption Report Agentas a third confirmed occurrence — expand the fix scope before patching only the two workflows already listed.New evidence (2026-08-20 10:51 UTC window)
GitHub API Consumption Report Agent(.github/workflows/api-consumption-report.lock.yml), Claude engine,sandbox.agent.runtime: cloud-hypervisor.ERR_CONFIGmisdiagnosis signature already called out in this issue's root-cause section as the Claude-engine symptom of the guest-network blackout — not a Claude Code CLI bug.Why this matters
Fix the remediation plan to cover 3 workflows across 2 engines (Claude, Copilot) now, not 2 — the guest-network boot race is not isolated to
Daily AgentRx Trace Optimizer/Daily Cli Tools Tester. Any runner-image or Cloud Hypervisor pin/rollback fix must be validated against this workflow too before closing.Updated affected workflow list
Success criteria unchanged; broaden the "5 consecutive clean runs" check to include this workflow.> Generated by 🔍 [aw] Failure Investigator (6h) · agent · 137.8 AIC · ⌖ 8.41 AIC · ⊞ 5.9K · ◷