Skip to content

Fix potential race in interface/volume status update#843

Open
mandre wants to merge 1 commit into
k-orc:mainfrom
shiftstack:fix-attachment-race
Open

Fix potential race in interface/volume status update#843
mandre wants to merge 1 commit into
k-orc:mainfrom
shiftstack:fix-attachment-race

Conversation

@mandre

@mandre mandre commented Jul 8, 2026

Copy link
Copy Markdown
Collaborator

After a port is attached to a server, its Neutron status transitions
asynchronously from DOWN to ACTIVE. Similarly, a volume transitions
from available to in-use after attachment. The cross-controller
signaling via serverTo{Port,Volume}MapFunc is susceptible to a race
with the controller's own status write.

Add defense-in-depth polling: when the port has a device_id but
status is still DOWN, or when a volume has attachments but status
is not in-use, schedule a re-fetch via WaitingOnOpenStack. This
ensures the controller picks up the status transition regardless
of whether the cross-controller signal is lost.

Fixes #842

@github-actions github-actions Bot added the semver:patch No API change label Jul 8, 2026
@mandre mandre added the backport-v2.0 This PR will be backported to v2.0 label Jul 8, 2026
@mandre

mandre commented Jul 8, 2026

Copy link
Copy Markdown
Collaborator Author

Aha, this doesn't work because now we're clearing the other fields. We need to use a different field owner. I'll just keep the second commit.

@mandre
mandre force-pushed the fix-attachment-race branch from c16f12b to de22e4c Compare July 8, 2026 08:39
After a port is attached to a server, its Neutron status transitions
asynchronously from DOWN to ACTIVE. Similarly, a volume transitions
from available to in-use after attachment. The cross-controller
signaling via serverTo{Port,Volume}MapFunc is susceptible to a race
with the controller's own status write.

Add defense-in-depth polling: when the port has a device_id but
status is still DOWN, or when a volume has attachments but status
is not in-use, schedule a re-fetch via WaitingOnOpenStack. This
ensures the controller picks up the status transition regardless
of whether the cross-controller signal is lost.

Fixes k-orc#842
@mandre
mandre force-pushed the fix-attachment-race branch from de22e4c to d9ce2e3 Compare July 8, 2026 08:55
// overwrite Progressing=True with Progressing=False before the
// port becomes ACTIVE. Poll here so we keep Progressing=True
// until the transition completes.
if osResource.Status == PortStatusDown {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The controller can exit from the previous for loop without finding a match server which the port is attached to, so this will not catch the case where port is attached and is DOWN.

Maybe adding a found = true to ensure that we don't make unnecessary API calls to OpenStack?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport-v2.0 This PR will be backported to v2.0 semver:patch No API change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Port/volume status not refreshed after server attachment due to SSA field owner race

2 participants