Skip to content

Bug: Gerrit Required Verify fails on ~22% of dispatches #4

Description

Summary

The Gerrit Required Verify workflow in this repository fails routinely, and
each failure emails the change author and the releng/collab-it lists. This is
not a recent regression — it has been steady for at least the last 35 days.

Sample: the last 200 failed runs (2026-06-30 → 2026-08-04), against 200
consecutive runs over 2026-07-27 → 2026-08-04 for the rate.

Measure Value
Failure rate (last 200 runs) 44 failed, 6 cancelled, 150 succeeded (22%)
Failure notifications ~5.7 per day
Attributable to cause A below 182 of 200 failures (91%)
Attributable to cause B below 16 of 200 failures (8%)
Genuine INFO.yaml validation failures 1 of 200 (working as intended)

Neither dominant cause is a fault in this repository's own content. Both have
upstream fixes, linked below.

Cause A — verify is dispatched for private Gerrit changes (91%)

lfit/releng-gerrit_to_platform dispatches on patchset-created regardless of
whether the change is private. The Gerrit hook runs server-side with full
visibility; the fdio.required.gha account that has to clear and set the vote
does not have it. Both the prepare and vote jobs then die with:

error: fatal: Not found: 46421
fatal: one or more reviews failed; review output above

Worked example — change 46421:

Gerrit event Time (UTC) Run Result
Uploaded patch set 1 (private) 07:36:49 30888530294 Not found: 46421
Uploaded patch set 2 (private) 07:52:40 30889582764 Not found: 46421
Unset private 07:55:34 30889787320 ✅ SUCCESS

Uploading private and publishing later is a normal VPP developer workflow — 9 of
the last 60 vpp master changes did it — and every patchset pushed while
private produces a failure email.

Granting the service account visibility of private changes would not fix this:
private change refs are not replicated to GitHub, so the verify job could not
fetch the change anyway.

Importantly, unsetting private fires a fresh dispatch on its own, so suppressing
the earlier dispatches costs no verification coverage.

Upstream fix: lfit/releng-gerrit_to_platform#116

Cause B — shallow checkout when the change has already merged (8%)

gerrit-compose-required-info-yaml-verify.yaml checks isolation with
git diff --name-only HEAD~1 over a fetch-depth: 1 checkout. For CSIT
oper-* cherry-picks, which are submitted pre-approved and merge within seconds
of the dispatch, the branch tip is the change commit, so HEAD~1 lies beyond
the shallow boundary:

fatal: ambiguous argument 'HEAD~1': unknown revision or path not in the working tree.
##[error]Process completed with exit code 128.

Worked example: run 30809172607
for change 46399, merged at 11:21:02,
dispatched at 11:21:06.

Upstream fixes: lfreleng-actions/checkout-gerrit-change-action#129 and
lfit/releng-reusable-workflows#843

Contributing factors in this repository

Independent of the two causes above, both Gerrit workflows here reference
retired sources:

  • lfit/gerrit-review-action — the action now lives under lfreleng-actions
  • technote-space/workflow-conclusion-action — archived November 2023, targets
    Node.js 20, and is the source of the deprecation warnings that appear as
    annotations on every run
  • lfit/releng-reusable-workflows/...@main — unpinned; behaviour can change
    without a corresponding change here

Proposed actions

  1. Short term, here. Add a visibility gate to
    gerrit-required-verify-non-voting.yaml: query Gerrit as the voting account
    before doing anything, and skip the remaining jobs when the change is not
    readable. This removes ~91% of the failure emails immediately and is safe to
    revert once the dispatcher is fixed. A change implementing this is ready and
    can be pushed for review on request.
  2. Short term, here. Refresh the action references above and pin the
    reusable workflow call to a release SHA.
  3. Upstream. Land Bug: dispatch fires for changes the platform cannot read lfit/releng-gerrit_to_platform#116 so no dispatch happens
    while a change is unreadable — this is the correct fix and benefits every LF
    project, not only FD.io.
  4. Upstream. Land the checkout/isolation fixes for cause B.
  5. Longer term. Move the Gerrit required-verify reusable workflows to an
    actively maintained home: Feat: host the Gerrit required-verify reusable workflows lfreleng-actions/generic-workflows#23

Happy to supply the full run classification or raw logs behind any of the
figures above.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions