Skip to content

Add owner refs from guest options to config options - #1789

Draft
faisalabujabal wants to merge 5 commits into
vmware-tanzu:mainfrom
faisalabujabal:vmop-3932-guestoptions-ownerrefs
Draft

Add owner refs from guest options to config options#1789
faisalabujabal wants to merge 5 commits into
vmware-tanzu:mainfrom
faisalabujabal:vmop-3932-guestoptions-ownerrefs

Conversation

@faisalabujabal

@faisalabujabal faisalabujabal commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Draft — stacked on #1782, do not review until that merges.

What does this PR do, and why is it needed?

Follow-up to #1782 (vmop-3932). hpannem asked there why
VirtualMachineGuestOptions doesn't carry owner references back to the
VirtualMachineConfigOptions objects that report it, given
controllers/configtarget already uses that pattern one layer up
(VirtualMachineConfigOptions <- ConfigTarget). The status-based GC
landed in #1782 is correct on its own, but it depends on every
reconcile running to completion. Native Kubernetes garbage collection
gives the same end state for free if a VirtualMachineConfigOptions is
force-deleted or otherwise removed outside a normal reconcile.

Each contributing VirtualMachineConfigOptions now adds a
(non-controller) owner reference to every VirtualMachineGuestOptions
it fans out to, mirroring configtarget's reconcileConfigOptions.
status.hardwareVersions[] is unchanged in shape and still carries all
the substantive per-hardware-version data (CPU/memory limits,
recommended controllers, etc.) -- an owner reference has no payload, so
it can't replace that field, only add a second, GC-visible signal
alongside it.

What did change from #1782: the deletion trigger in
removeHardwareVersionAndDeleteIfOrphaned moves from "no
status.hardwareVersions entries remain" to "no owner references
remain". In the normal path these two counts always move together (every
write path adds/removes both in the same pass), so this is not an
observable behavior change on its own -- the reason for the switch is
that gating on the owner-reference count makes a reconcile that fails
between the status patch and the owner-reference patch resume correctly
on retry (next pass finds no status entry, a no-op there, but still
finds the dangling owner reference and finishes removing it), instead of
silently leaking a reference forever.

Reviewing this diff before #1782 merges: because GitHub requires a
PR's base branch to exist in the base repository, and
vmop-3932-gc-unused-guestoptions only exists on my fork, this PR is
opened against main and its diff currently includes #1782's full
changeset. In particular, in .sdd/specs/001-class-policy-resize/tasks.md,
T093a and T093b were completed in #1782, not here -- only T093c
and T093d
are new to this PR. Once #1782 merges this branch will be
rebased onto main and the diff will shrink to just the
owner-reference work (commit 36a972d5).

Which issue(s) is/are addressed by this PR?:

Tracked under vmop-3932 (same ticket as #1782).

Are there any special notes for your reviewer:

Hold review until #1782 merges. To review only the net-new work now,
look at commit 36a972d5 directly rather than the full PR diff.

Please add a release note if necessary:

None

The VirtualMachineConfigOptions controller fans out a
VirtualMachineGuestOptions per guest OS descriptor and upserts a
status.hardwareVersions entry per hardware version, but never
removed that entry once a hardware version stopped reporting the
guest OS -- unlike the analogous ConfigTarget -> VirtualMachineConfigOptions
garbage collection.

Add garbageCollectGuestOptions, invoked at the end of every
VirtualMachineConfigOptions reconcile: it removes the current
hardware version's status.hardwareVersions entry from any
VirtualMachineGuestOptions no longer reporting that guest OS, and
deletes the object outright once no hardware-version entries
remain. The live set is keyed by the sanitized object name (not the
raw guest OS ID) so two IDs that collide onto the same truncated,
DNS-safe name cannot cause a spurious deletion.

Update spec.md, research.md, and tasks.md for 001-class-policy-resize
to reflect that this follow-up (previously called out as deferred)
is now implemented.

vmop-3932
QueryConfigOptionEx returning nil with no error is vSphere's
positive answer that a hardware version has no config option in this
environment -- not a transient signal, which surfaces as a non-nil
error instead. ReconcileNormal treated that nil result as a dead end
and returned before fanOutGuestOptions ran, so a hardware version
that stopped being supported left its previously fanned-out
VirtualMachineGuestOptions entries orphaned forever, even though the
VirtualMachineConfigOptions object itself kept reconciling.

Route the nil-config-option path through fanOutGuestOptions with an
empty descriptor list instead, so it is pruned exactly like the
already-covered non-nil-but-empty case.

vmop-3932
removeHardwareVersionAndDeleteIfOrphaned fetches obj via the
caller's List, then deletes it unconditionally once its last
hardwareVersions entry is being removed. Between that List and the
Delete, another hardware version's reconcile could concurrently
upsert a new entry into this shared object; the unconditional delete
would silently discard that entry along with the object.

Pass client.Preconditions{ResourceVersion: &obj.ResourceVersion} to
Delete so a racing writer causes a conflict here instead, which
surfaces as a normal reconcile error and gets retried with fresh
data on the next attempt -- the same optimistic-locking approach
already used for this file's status patches.

Per automated review on vmop-3932.

vmop-3932
ReconcileDelete only removed the finalizer, so when a
VirtualMachineConfigOptions was deleted -- e.g. by ConfigTarget's own
GC dropping a hardware version -- any VirtualMachineGuestOptions it
had fanned out kept their status.hardwareVersions entry forever, even
though the owning object was gone.

Run the same garbageCollectGuestOptions used by ReconcileNormal
before removing the finalizer, so deletion prunes exactly like a live
reconcile that observes zero descriptors. The finalizer is removed
only after GC succeeds -- it exists to hold the object until cleanup
completes, so removing it unconditionally would let a GC error vanish
along with the object, and the leak would never retry.

Per review feedback on vmop-3932 (why doesn't deleting the CR clean up
its guest options?).

vmop-3932
hpannem asked on PR vmware-tanzu#1782 why VirtualMachineGuestOptions doesn't use
owner references back to the VirtualMachineConfigOptions objects that
report it, given controllers/configtarget already uses that pattern
for VirtualMachineConfigOptions <- ConfigTarget. The status-based GC
added there is correct on its own, but it depends on every reconcile
running to completion; native Kubernetes garbage collection gives the
same outcome for free if a VirtualMachineConfigOptions is force-deleted
or otherwise removed outside a normal reconcile.

Each contributing VirtualMachineConfigOptions now adds a
(non-controller) owner reference to every VirtualMachineGuestOptions it
fans out to, mirroring configtarget's reconcileConfigOptions. Deletion
is gated on the owner-reference count rather than the
status.hardwareVersions entry count, so a reconcile that fails between
the status patch and the owner-reference patch resumes correctly on
retry instead of leaking a reference.

Tracked under vmop-3932 as a follow-up to the GC work in vmware-tanzu#1782.
@faisalabujabal
faisalabujabal requested a review from a team as a code owner July 31, 2026 16:04
@github-actions github-actions Bot added the size/XL Denotes a PR that changes 500-999 lines. label Jul 31, 2026
@github-actions

Copy link
Copy Markdown

Code Coverage

Package Line Rate Health
github.com/vmware-tanzu/vm-operator/controllers/configtarget 74%
github.com/vmware-tanzu/vm-operator/controllers/contentlibrary/clustercontentlibraryitem 67%
github.com/vmware-tanzu/vm-operator/controllers/contentlibrary/contentlibraryitem 67%
github.com/vmware-tanzu/vm-operator/controllers/contentlibrary/utils 85%
github.com/vmware-tanzu/vm-operator/controllers/infra/capability/configmap 92%
github.com/vmware-tanzu/vm-operator/controllers/infra/capability/crd 100%
github.com/vmware-tanzu/vm-operator/controllers/infra/configmap 75%
github.com/vmware-tanzu/vm-operator/controllers/infra/node 77%
github.com/vmware-tanzu/vm-operator/controllers/infra/secret 76%
github.com/vmware-tanzu/vm-operator/controllers/infra/validatingwebhookconfiguration 87%
github.com/vmware-tanzu/vm-operator/controllers/infra/workloadnetworkconfig 86%
github.com/vmware-tanzu/vm-operator/controllers/infra/zone 80%
github.com/vmware-tanzu/vm-operator/controllers/storage/storageclass 93%
github.com/vmware-tanzu/vm-operator/controllers/storage/storagepolicy 96%
github.com/vmware-tanzu/vm-operator/controllers/storage/storagepolicyquota 91%
github.com/vmware-tanzu/vm-operator/controllers/storage/volumeattributesclass 93%
github.com/vmware-tanzu/vm-operator/controllers/util/encoding 73%
github.com/vmware-tanzu/vm-operator/controllers/virtualmachine/storagepolicyusage 96%
github.com/vmware-tanzu/vm-operator/controllers/virtualmachine/virtualmachine 65%
github.com/vmware-tanzu/vm-operator/controllers/virtualmachine/volume 86%
github.com/vmware-tanzu/vm-operator/controllers/virtualmachine/volumebatch 89%
github.com/vmware-tanzu/vm-operator/controllers/virtualmachineclass 73%
github.com/vmware-tanzu/vm-operator/controllers/virtualmachineconfigoptions 85%
github.com/vmware-tanzu/vm-operator/controllers/virtualmachinegroup 90%
github.com/vmware-tanzu/vm-operator/controllers/virtualmachinegrouppublishrequest 88%
github.com/vmware-tanzu/vm-operator/controllers/virtualmachineimagecache 89%
github.com/vmware-tanzu/vm-operator/controllers/virtualmachinepublishrequest 84%
github.com/vmware-tanzu/vm-operator/controllers/virtualmachinereplicaset 68%
github.com/vmware-tanzu/vm-operator/controllers/virtualmachineservice 91%
github.com/vmware-tanzu/vm-operator/controllers/virtualmachineservice/providers 93%
github.com/vmware-tanzu/vm-operator/controllers/virtualmachinesetresourcepolicy 81%
github.com/vmware-tanzu/vm-operator/controllers/virtualmachinesnapshot 91%
github.com/vmware-tanzu/vm-operator/controllers/virtualmachinewebconsolerequest 72%
github.com/vmware-tanzu/vm-operator/controllers/virtualmachinewebconsolerequest/v1alpha1 72%
github.com/vmware-tanzu/vm-operator/controllers/virtualmachinewebconsolerequest/v1alpha1/conditions 88%
github.com/vmware-tanzu/vm-operator/controllers/virtualmachinewebconsolerequest/v1alpha1/patch 78%
github.com/vmware-tanzu/vm-operator/controllers/vspherepolicy/policyevaluation 85%
github.com/vmware-tanzu/vm-operator/pkg/bitmask 100%
github.com/vmware-tanzu/vm-operator/pkg/builder 89%
github.com/vmware-tanzu/vm-operator/pkg/conditions 90%
github.com/vmware-tanzu/vm-operator/pkg/config 100%
github.com/vmware-tanzu/vm-operator/pkg/config/capabilities 97%
github.com/vmware-tanzu/vm-operator/pkg/config/env 100%
github.com/vmware-tanzu/vm-operator/pkg/context 37%
github.com/vmware-tanzu/vm-operator/pkg/context/generic 100%
github.com/vmware-tanzu/vm-operator/pkg/context/operation 100%
github.com/vmware-tanzu/vm-operator/pkg/crd 77%
github.com/vmware-tanzu/vm-operator/pkg/errors 74%
github.com/vmware-tanzu/vm-operator/pkg/exit 100%
github.com/vmware-tanzu/vm-operator/pkg/log 100%
github.com/vmware-tanzu/vm-operator/pkg/mem 100%
github.com/vmware-tanzu/vm-operator/pkg/patch 78%
github.com/vmware-tanzu/vm-operator/pkg/prober 89%
github.com/vmware-tanzu/vm-operator/pkg/prober/probe 90%
github.com/vmware-tanzu/vm-operator/pkg/prober/worker 77%
github.com/vmware-tanzu/vm-operator/pkg/providers/vsphere 74%
github.com/vmware-tanzu/vm-operator/pkg/providers/vsphere/clustermodules 73%
github.com/vmware-tanzu/vm-operator/pkg/providers/vsphere/config 88%
github.com/vmware-tanzu/vm-operator/pkg/providers/vsphere/contentlibrary 75%
github.com/vmware-tanzu/vm-operator/pkg/providers/vsphere/credentials 100%
github.com/vmware-tanzu/vm-operator/pkg/providers/vsphere/network 83%
github.com/vmware-tanzu/vm-operator/pkg/providers/vsphere/placement 70%
github.com/vmware-tanzu/vm-operator/pkg/providers/vsphere/session 60%
github.com/vmware-tanzu/vm-operator/pkg/providers/vsphere/storage 44%
github.com/vmware-tanzu/vm-operator/pkg/providers/vsphere/upgrade/virtualmachine 96%
github.com/vmware-tanzu/vm-operator/pkg/providers/vsphere/upgrade/virtualmachine/backfill 96%
github.com/vmware-tanzu/vm-operator/pkg/providers/vsphere/vcenter 86%
github.com/vmware-tanzu/vm-operator/pkg/providers/vsphere/virtualmachine 84%
github.com/vmware-tanzu/vm-operator/pkg/providers/vsphere/virtualmachine/extraconfig 87%
github.com/vmware-tanzu/vm-operator/pkg/providers/vsphere/vmlifecycle 75%
github.com/vmware-tanzu/vm-operator/pkg/record 84%
github.com/vmware-tanzu/vm-operator/pkg/topology 91%
github.com/vmware-tanzu/vm-operator/pkg/util 79%
github.com/vmware-tanzu/vm-operator/pkg/util/cloudinit 89%
github.com/vmware-tanzu/vm-operator/pkg/util/cloudinit/validate 91%
github.com/vmware-tanzu/vm-operator/pkg/util/image 100%
github.com/vmware-tanzu/vm-operator/pkg/util/kube 91%
github.com/vmware-tanzu/vm-operator/pkg/util/kube/cource 100%
github.com/vmware-tanzu/vm-operator/pkg/util/kube/internal 100%
github.com/vmware-tanzu/vm-operator/pkg/util/kube/networksettings 100%
github.com/vmware-tanzu/vm-operator/pkg/util/kube/proxyaddr 73%
github.com/vmware-tanzu/vm-operator/pkg/util/kube/spq 99%
github.com/vmware-tanzu/vm-operator/pkg/util/linuxprep 97%
github.com/vmware-tanzu/vm-operator/pkg/util/netplan 100%
github.com/vmware-tanzu/vm-operator/pkg/util/nil 100%
github.com/vmware-tanzu/vm-operator/pkg/util/ovfcache 75%
github.com/vmware-tanzu/vm-operator/pkg/util/ovfcache/internal 100%
github.com/vmware-tanzu/vm-operator/pkg/util/paused 100%
github.com/vmware-tanzu/vm-operator/pkg/util/ptr 100%
github.com/vmware-tanzu/vm-operator/pkg/util/resize 98%
github.com/vmware-tanzu/vm-operator/pkg/util/sysprep 98%
github.com/vmware-tanzu/vm-operator/pkg/util/vmopv1 88%
github.com/vmware-tanzu/vm-operator/pkg/util/volumes 100%
github.com/vmware-tanzu/vm-operator/pkg/util/vsphere/client 66%
github.com/vmware-tanzu/vm-operator/pkg/util/vsphere/datastore 100%
github.com/vmware-tanzu/vm-operator/pkg/util/vsphere/fault 100%
github.com/vmware-tanzu/vm-operator/pkg/util/vsphere/library 95%
github.com/vmware-tanzu/vm-operator/pkg/util/vsphere/storage 82%
github.com/vmware-tanzu/vm-operator/pkg/util/vsphere/task 100%
github.com/vmware-tanzu/vm-operator/pkg/util/vsphere/vm 78%
github.com/vmware-tanzu/vm-operator/pkg/util/vsphere/watcher 85%
github.com/vmware-tanzu/vm-operator/pkg/vmconfig 95%
github.com/vmware-tanzu/vm-operator/pkg/vmconfig/anno2extraconfig 100%
github.com/vmware-tanzu/vm-operator/pkg/vmconfig/bootoptions 88%
github.com/vmware-tanzu/vm-operator/pkg/vmconfig/cdrom 88%
github.com/vmware-tanzu/vm-operator/pkg/vmconfig/crypto 92%
github.com/vmware-tanzu/vm-operator/pkg/vmconfig/diskpromo 100%
github.com/vmware-tanzu/vm-operator/pkg/vmconfig/extraconfig 100%
github.com/vmware-tanzu/vm-operator/pkg/vmconfig/networkextraconfig 71%
github.com/vmware-tanzu/vm-operator/pkg/vmconfig/policy 97%
github.com/vmware-tanzu/vm-operator/pkg/vmconfig/virtualcontroller 93%
github.com/vmware-tanzu/vm-operator/pkg/vmconfig/volumes/unmanaged/backfill 98%
github.com/vmware-tanzu/vm-operator/pkg/vmconfig/volumes/unmanaged/register 92%
github.com/vmware-tanzu/vm-operator/pkg/webconsolevalidation 100%
github.com/vmware-tanzu/vm-operator/services/vm-watcher 85%
github.com/vmware-tanzu/vm-operator/webhooks/common 98%
github.com/vmware-tanzu/vm-operator/webhooks/configtarget/validation 87%
github.com/vmware-tanzu/vm-operator/webhooks/persistentvolumeclaim/validation 95%
github.com/vmware-tanzu/vm-operator/webhooks/unifiedstoragequota/validation 89%
github.com/vmware-tanzu/vm-operator/webhooks/virtualmachine/mutation 86%
github.com/vmware-tanzu/vm-operator/webhooks/virtualmachine/validation 96%
github.com/vmware-tanzu/vm-operator/webhooks/virtualmachineclass/mutation 62%
github.com/vmware-tanzu/vm-operator/webhooks/virtualmachineclass/validation 89%
github.com/vmware-tanzu/vm-operator/webhooks/virtualmachineconfigoptions/validation 88%
github.com/vmware-tanzu/vm-operator/webhooks/virtualmachinegroup/mutation 87%
github.com/vmware-tanzu/vm-operator/webhooks/virtualmachinegroup/validation 93%
github.com/vmware-tanzu/vm-operator/webhooks/virtualmachinegrouppublishrequest/mutation 86%
github.com/vmware-tanzu/vm-operator/webhooks/virtualmachinegrouppublishrequest/validation 88%
github.com/vmware-tanzu/vm-operator/webhooks/virtualmachinepublishrequest/validation 90%
github.com/vmware-tanzu/vm-operator/webhooks/virtualmachinereplicaset/validation 90%
github.com/vmware-tanzu/vm-operator/webhooks/virtualmachineservice/mutation 67%
github.com/vmware-tanzu/vm-operator/webhooks/virtualmachineservice/validation 92%
github.com/vmware-tanzu/vm-operator/webhooks/virtualmachinesetresourcepolicy/validation 89%
github.com/vmware-tanzu/vm-operator/webhooks/virtualmachinesnapshot/mutation 86%
github.com/vmware-tanzu/vm-operator/webhooks/virtualmachinesnapshot/validation 91%
github.com/vmware-tanzu/vm-operator/webhooks/virtualmachinewebconsolerequest/v1alpha1/validation 92%
github.com/vmware-tanzu/vm-operator/webhooks/virtualmachinewebconsolerequest/validation 92%
Summary 84% (21490 / 25566)

Minimum allowed line rate is 79%

@faisalabujabal
faisalabujabal marked this pull request as draft July 31, 2026 16:45
@faisalabujabal faisalabujabal added the pro-ai-review-requested Mark the PR as ready for the AI Review bot using a Pro model label Jul 31, 2026
@faisalabujabal

Copy link
Copy Markdown
Contributor Author

Claude Code review — head 36a972d5, net-new commit 36a972d5 only (per the PR notes, #1782's changeset excluded)

Reviewed scope: controllers/virtualmachineconfigoptions/vmconfigoptions_controller.go and its _test.go. Both CRDs are cluster-scoped (+kubebuilder:resource:scope=Cluster on VirtualMachineGuestOptions and VirtualMachineConfigOptions), so a cluster→cluster non-controller owner reference is valid and won't trip the GC's "cluster-scoped dependent with namespaced owner" path. The fan-in write path is exactly what .sdd/memory/operator-best-practices.md prescribes: MergeFromWithOptimisticLock, apiequality.Semantic.DeepEqual for the skip-if-unchanged guard, status patched separately, client.Preconditions on the delete. All 20 checks green.

One real finding.

🟠 Switching the delete gate from status entries to owner references is not upgrade-safe

removeHardwareVersionAndDeleteIfOrphaned now deletes only when hasOwnerRef && len(obj.OwnerReferences) == 1. Every VirtualMachineGuestOptions created before this change has zero owner references, and there are two ways that bites during the upgrade window:

  1. Permanent leak. A pre-existing object that is already stale at upgrade time (no longer reported by any hardware version) never gets its owner reference backfilled, because backfill happens in reconcileGuestOptions, which only runs for live descriptors. GC takes the idx != -1 branch, removes the status entry, and skips the delete because hasOwnerRef is false. On every subsequent pass idx == -1 && !hasOwnerRef returns early — the object is leaked forever. Pre-change code (len(obj.Status.HardwareVersions) == 1) would have deleted it. These are precisely the objects Prune unused VirtualMachineGuestOptions entries #1782's GC exists to clean up.

  2. Premature delete of a still-live object. Mirror image: obj has vmco's freshly-added owner reference (so len(OwnerReferences) == 1) but also a legacy status entry from a second VirtualMachineConfigOptions whose reconcile hasn't backfilled its own owner reference yet. The new gate deletes the shared object even though another hardware version still reports that guest OS. It's recreated on the other VMCO's next reconcile, but the delete is observable to consumers and the surviving version's status entry is lost until then. The old status-count gate (2 entries) would not have deleted it.

The retry-resumption argument in the new doc comment is sound — it just doesn't need to be exclusive. Gating on "no contribution of either kind remains" keeps that property and stays correct across the upgrade:

lastOwnerRef := hasOwnerRef && len(obj.OwnerReferences) == 1
lastStatusEntry := idx != -1 && len(obj.Status.HardwareVersions) == 1
if lastOwnerRef && (lastStatusEntry || len(obj.Status.HardwareVersions) == 0) {
    // ... delete
}
if !hasOwnerRef && lastStatusEntry {
    // legacy object, never backfilled: fall back to the pre-#1789 gate
}

— or, more simply, keep len(obj.Status.HardwareVersions) == 1 || (hasOwnerRef && len(obj.OwnerReferences) == 1) as the delete condition and let the two signals agree in the steady state, which the PR description already argues they do.

🔵 Neither legacy branch is covered

The new unit specs construct every VirtualMachineGuestOptions through the new code, so both cases above are untested. A spec that seeds a VirtualMachineGuestOptions with a status.hardwareVersions entry and no owner references, then reconciles a VMCO that no longer reports it, would pin finding #1.

🔵 E2E: owner references are cluster-observable

Per .sdd/memory/e2e-sync-with-changes.md, the existing GuestOptions fan-out spec in test/e2e/vmservice/vmservice/configpolicy/configpolicy.go is the natural home for a one-line assertion that each fanned-out VirtualMachineGuestOptions carries an owner reference per contributing hardware version. Today that file only sets an owner ref on the synthetic stale fixture (line 359); it never asserts the controller sets them.

🔵 Description will go stale on rebase

The "Reviewing this diff before #1782 merges" section and the T093a/T093b-vs-T093c/T093d note are correct today but become misleading once this rebases onto main. Worth trimming when the draft flips to ready.

@faisalabujabal

Copy link
Copy Markdown
Contributor Author

needs rebase after #1782

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pro-ai-review-requested Mark the PR as ready for the AI Review bot using a Pro model size/XL Denotes a PR that changes 500-999 lines.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant