GitDagBundle: promoting a SHA-pinned tracking_ref can fail with 'reference is not a tree' if local storage isn't cleared

[coleheflin]

Description

GitDagBundle supports pinning tracking_ref to a full commit SHA (see #69735). Rolling back to an older SHA works reliably after a config change + Dag processor restart, because the objects are already present in local storage. Promoting to a new SHA that was created/pushed after the bundle's local storage was first populated can fail.

Root cause

In GitDagBundle._initialize:

• The working clone lives at a stable path (self.repo_path, e.g. .../tracking_repo when no separate version is set), not a fresh path per tracking_ref value.
• _clone_repo_if_required() only clones if that path doesn't already exist. On a restart where the bundle's storage directory has survived (the default — dag_bundle_storage_path defaults to /tmp/airflow/dag_bundles), the existing working clone is reopened, not re-fetched.
• Only the bare mirror is fetched inside _clone_bare_repo_if_required → _fetch_bare_repo(). The working clone (tracking_repo) is only fetched inside refresh(), which runs after the self.repo.git.checkout(self.tracking_ref) call at line 215 for the no-version code path.
• If the new SHA (or a newly created tag) isn't yet an object in the stale working clone, checkout raises GitCommandError: fatal: reference is not a tree: <sha> (or pathspec '<tag>' did not match for a new tag).

Repro

1. Configure a GitDagBundle with tracking_ref pinned to commit A. Let the Dag processor initialize the bundle (populates tracking_repo).
2. Push a new commit B to the source repo, then update the bundle config's tracking_ref to commit B's SHA.
3. Restart the Dag processor without clearing the bundle's local storage path.
4. initialize() raises GitCommandError: fatal: reference is not a tree: <B's sha>.

Rolling back to commit A (already present locally) after this failure succeeds normally. A fresh storage path (new pod, or manually deleted bundle directory) also succeeds for promotion, since _clone_repo_if_required() then performs a real clone against the already-updated bare mirror.

Suggested fix

In _initialize, fetch the working clone (or check that the target ref is present and fetch if not) before calling self.repo.git.checkout(self.tracking_ref), mirroring what refresh() already does for the branch/tag tracking case.

Related

• Uncovered during review of Document commit SHA support for GitDagBundle tracking_ref #69735, which documents tracking_ref's SHA support and scopes its restart-behavior docs/tests to this limitation until it's fixed here.