fix(pipelines): retry terraform apply/destroy on transient Azure control-plane conflicts

[Liyu Ma (liyu-ma)]

Problem

Azure serializes control-plane operations on a managed cluster. When another operation is still in flight, ARM rejects terraform apply / terraform destroy with a transient conflict:

RESPONSE 409: 409 Conflict
ERROR CODE: EtagMismatch
"message": "Operation is not allowed: Another operation is in progress. ..."
"subcode": "PutManagedClusterAndComponents_FailedPrecondition_HCPServerError"

or

ERROR: (OperationNotAllowed) Operation is not allowed because there's an
in-progress PutExtensionAddonHandler.PUT operation (operation ID: ...) on the
managed cluster started on UTC ... Please wait for it to finish before starting
a new operation.

The only protection today is retryCountOnTaskFailure, driven by retry_attempt_count. Several pipelines deliberately set that to 0 to avoid re-running expensive provisioning — e.g. pipelines/perf-eval/Hyperscale Cluster Benchmark/ccp-provisioning-H2.yml — so those pipelines fail immediately on a conflict that clears itself within a minute or two.

Observed failures:

Pipeline	Run	Failure
Hyperscale H2	20260803.1	azure_H2 died after 36s in Run Terraform destroy Command — 409 EtagMismatch
100 Nodes 10k Pods	20260802.1	azure_uksouth* — OperationNotAllowed (in-progress PutExtensionAddonHandler.PUT), cascading into failed destroy + failed Destroy Resource Group

Change

Bounded in-script retry with exponential backoff around the terraform apply/destroy invocation in steps/terraform/run-command.yml:

• Narrowly scoped — only retries when cloud == azure and the terraform log matches a known concurrent-operation conflict (EtagMismatch, OperationNotAllowed, AnotherOperationInProgress, Another operation is in progress, in-progress … operation, operation preempted). Genuine failures (quota, bad config, missing permissions) still fail fast and fall through to the existing cleanup logic unchanged.
• Defaults to 3 retries with 60s / 120s / 240s backoff, tunable via the new transient_conflict_retry_count and transient_conflict_retry_delay_seconds parameters.
• Independent of retry_attempt_count, so it also protects pipelines that opt out of task-level retries.
• Emits an ADO warning on each retry, so the conflict stays visible in the run summary rather than being silently swallowed.

Behaviour is unchanged for AWS/GCP and for every non-conflict failure path.

Testing

• yamllint -c .yamllint steps/terraform/run-command.yml --no-warnings — clean.
• The generated script body passes bash -n.
• Retry loop exercised standalone against four scenarios:

Scenario	Expected	Result
Transient conflict, succeeds on 3rd attempt	retry twice, exit 0	✅ 3 calls, exit 0
Non-transient error (quota exceeded)	no retry, fail fast	✅ 1 call, exit 1
cloud == aws with conflict text	no retry	✅ 1 call, exit 1
OperationNotAllowed / in-progress PUT	retry, recover	✅ 2 calls, exit 0
Conflict never clears	1 + 3 attempts, backoff 60/120/240	✅ exit 1 after 4 attempts

[github-actions]

For reviewers only: reply /run-tf-integration to trigger the terraform integration pipeline before approving the PR.

[Copilot]

Pull request overview

This PR improves Azure pipeline reliability by adding a bounded, in-script retry with exponential backoff around terraform apply / terraform destroy when Azure ARM rejects the operation due to transient “another operation in progress” control-plane conflicts. This makes Azure transient conflict handling independent of task-level retries (retryCountOnTaskFailure), which some pipelines disable.

Changes:

• Added new template parameters to control transient conflict retry count and base delay.
• Wrapped the terraform apply/destroy invocation in a retry loop that only triggers for Azure and only when known transient conflict signatures are detected in the terraform log.
• Emitted Azure DevOps warning log issues on each retry to keep conflicts visible in run summaries.