Acknowledge RP-applied Machine labels and taints

docs/design.md

@@ -152,7 +152,7 @@ See [AKS RP And Flex Node Agent Interaction](design/agent-and-aks.md) for the de
 
 ## State And Idempotency
 
-The agent persists local daemon state so it can recover after restart, reboot, or partial failure. Persisted state includes the applied Kubernetes/settings version and active nspawn machine side.
+The agent persists local daemon state so it can recover after restart, reboot, or partial failure. Persisted state includes the current and previous applied Machine goals and the active nspawn machine side.
 
 The current state model separates desired state, applied state, and runtime discovery:
 

docs/design/agent-and-aks.md

@@ -90,7 +90,7 @@ ARM machine creation must be idempotent. If AKS RP retries creation and the ARM
 
 The Flex Node agent reconciles two external signals.
 
-The ARM machine resource provides desired settings and a version for those settings. The current minimal settings are desired Kubernetes version and settings version. The agent compares the settings version from ARM with its locally applied settings version to detect drift. Future schema extensions can add more settings, but the agent should treat the ARM machine resource as the source of truth for host/nspawn reconciliation.
+The ARM machine resource provides desired Kubernetes version, max pods, custom labels and taints, kubelet image-GC settings, and a version for those settings. The agent compares the settings version from ARM with its locally applied settings version to detect drift. Future schema extensions can add more settings, but the agent should treat the ARM machine resource as the source of truth for host/nspawn reconciliation.
 
 The ARM machine resource does not own the nspawn side. Selecting `kube1` or `kube2` is an internal host implementation detail used by the agent to apply settings atomically.
 
@@ -130,14 +130,14 @@ flowchart TD
 
 ## Current Repave Implementation
 
-AKS Flex Node no longer runs a standalone local drift detector. Desired node settings come from an AKS machine resource. The agent compares the desired machine goal with locally persisted daemon state and repaves the nspawn-backed worker when Kubernetes `Node` deletion indicates AKS has approved replacement.
+AKS Flex Node no longer runs a standalone local drift detector. Desired node settings come from an AKS machine resource. The agent compares the desired machine goal with locally persisted daemon state. It acknowledges label- and taint-only updates that AKS RP already reconciled onto the Kubernetes `Node`; other changes repave the nspawn-backed worker after `Node` deletion indicates AKS has approved replacement.
 
 The current machine goal comes from the ARM machine model:
 
 - `properties.kubernetes` contains the desired Kubernetes version and node settings.
 - `properties.eTag` is exposed internally as the settings version.
 
-The ETag is the drift key. If it differs from the locally applied ETag, the agent waits for the Kubernetes `Node` object to disappear before mutating host state. Status-only updates must not change the ETag.
+The ETag is the drift key. If it differs from the locally applied ETag, the agent first checks whether only labels or taints changed and whether the existing Kubernetes `Node` reflects that complete delta. A matching Node lets the agent persist the new goal without host mutation. Otherwise, it waits for the `Node` object to disappear before mutating host state. Status-only updates must not change the ETag.
 
 The daemon uses two inputs:
 
@@ -170,7 +170,7 @@ This keeps scheduling and disruption decisions outside the agent. AKS RP, an ope
 9. Start node-problem-detector inside the new side.
 10. Clean up the old side's nspawn artifacts.
 
-After successful repave, the daemon patches machine status and persists the applied goal locally.
+After successful repave or in-place acknowledgement, the daemon persists the applied goal locally and reports status through the selected machine client. Direct ARM Machine status is currently read-only, so that client skips the status mutation while retaining local convergence.
 
 AKS Flex Node uses two local nspawn machine names:
 

docs/design/in-cluster-machine.md

@@ -15,7 +15,7 @@ The controller serves the `armcontainerservice.Machine` JSON shape from the `kub
       "orchestratorVersion": "1.34.0",
       "maxPods": 110,
       "nodeLabels": {
-        "kubernetes.azure.com/managed": "false"
+        "workload": "edge"
       }
     },
     "provisioningState": "Succeeded"
@@ -27,18 +27,18 @@ Status updates use a separate patch model because the agent operation status is
 
 ## Bootstrap flow
 
-The local bootstrap configuration is authoritative while `aks-flex-node start` is running:
+The local bootstrap configuration seeds a Machine when one does not already exist. Once the endpoint returns a Machine, its complete goal is authoritative for bootstrap:
 
 1. `NewMachineClient` selects the in-cluster backend without a supplied Kubernetes REST config.
 2. The client builds a REST config from the bootstrap token or configured exec credential.
 3. `EnsureMachine` reads the machine through the Kubernetes service proxy.
 4. If the machine is absent, the client sends a PUT using the local bootstrap goal.
-5. If its Kubernetes version differs, the client sends a PUT that overwrites the remote goal with the local version.
-6. If its Kubernetes version already matches, local bootstrap settings remain authoritative; remote settings other than the ETag do not replace them.
-7. The returned ETag becomes the reconciliation baseline for the locally applied goal.
-8. The daemon state is seeded from that ETag before host or nspawn state is mutated. A later ETag change is treated as a new remote goal.
+5. Whether read or created, the returned Machine is validated and its goal replaces the local bootstrap goal. This includes Kubernetes version, max pods, custom labels, taints, kubelet image-GC thresholds, and the ETag-backed settings version. Scalar defaults omitted by the API retain their validated local bootstrap values.
+6. The daemon resolves nspawn settings and seeds its state from that same effective goal before mutating the host. A later ETag change is treated as a new remote goal.
 
-The ConfigMap-backed controller is read-only: it accepts mutation requests but returns the pre-created machine. Its fixture must therefore already match the local bootstrap version. When machine registration is required, a mismatch fails bootstrap before host mutation.
+When `orchestratorVersion` is a `major.minor` alias, the returned `currentOrchestratorVersion` supplies the exact patch used for artifact resolution.
+
+The ConfigMap-backed controller is read-only: it accepts mutation requests but returns the pre-created Machine. The agent adopts that returned goal even when it differs from local bootstrap configuration. When registration is required, a read, create, or validation failure stops bootstrap before host mutation. When registration is optional, bootstrap continues with the local goal.
 
 ## Daemon flow
 
@@ -48,8 +48,11 @@ After bootstrap, the remote machine is authoritative:
 2. `NewMachineClient` receives that REST config and selects the in-cluster backend.
 3. The client periodically reads the ARM-compatible machine through the service-proxy endpoint.
 4. The daemon compares `properties.eTag` with its locally applied settings version.
-5. A changed ETag represents a new goal. The daemon waits for the Kubernetes `Node` deletion signal before applying it.
-6. Reconciliation status is sent to the endpoint's `/status` subresource without changing the ETag.
+5. A changed ETag represents a new goal. If only labels or taints changed and AKS RP already reconciled them onto the existing Kubernetes `Node`, the daemon acknowledges the observed goal without mutating or repaving the node.
+6. Other goal changes wait for the Kubernetes `Node` deletion signal before the daemon applies them through blue-green repave.
+7. Reconciliation status is sent to the endpoint's `/status` subresource without changing the ETag.
+
+Direct ARM Machine status is currently read-only. In that mode, acknowledgement still advances the local applied goal and ETag, while the status mutation is skipped by the ARM client.
 
 ## Request path
 

hack/demo/aks-flex-node-upgrade.sh

@@ -131,7 +131,7 @@ update_machine_goal() {
       .properties.eTag = $settings |
       .properties.kubernetes = (.properties.kubernetes // {}) |
       .properties.kubernetes.orchestratorVersion = $version |
-      .properties.kubernetes.nodeLabels = (.properties.kubernetes.nodeLabels // {"kubernetes.azure.com/managed":"false"})
+      .properties.kubernetes.nodeLabels = (.properties.kubernetes.nodeLabels // {})
     ' <<<"${current_json}" > "${tmp}"
 
   if [[ -z "${cm_json}" ]]; then

hack/e2e/lib/controller.sh

@@ -332,7 +332,7 @@ ensure_flex_controller() {
 }
 
 _render_machine_json() {
-  local node_name="$1" kubernetes_version="$2" settings_version="$3"
+  local node_name="$1" kubernetes_version="$2" settings_version="$3" max_pods="$4"
   local cluster_id machine_id
   cluster_id="$(state_get cluster_id)"
   machine_id="${cluster_id}/agentPools/${E2E_TARGET_AGENT_POOL_NAME}/machines/${node_name}"
@@ -342,6 +342,7 @@ _render_machine_json() {
     --arg name "${node_name}" \
     --arg kubernetesVersion "${kubernetes_version}" \
     --arg eTag "${settings_version}" \
+    --argjson maxPods "${max_pods}" \
     '{
       id: $id,
       name: $name,
@@ -351,7 +352,7 @@ _render_machine_json() {
         provisioningState: "Succeeded",
         kubernetes: {
           orchestratorVersion: $kubernetesVersion,
-          maxPods: 110,
+          maxPods: $maxPods,
           nodeLabels: {},
           nodeTaints: [],
           kubeletConfig: {
@@ -364,11 +365,11 @@ _render_machine_json() {
 }
 
 _machine_configmap_upsert_unlocked() {
-  local node_name="$1" kubernetes_version="$2" settings_version="$3"
+  local node_name="$1" kubernetes_version="$2" settings_version="$3" max_pods="$4"
   local machine_file patch
   machine_file="${E2E_WORK_DIR}/machine-${node_name}.json"
 
-  _render_machine_json "${node_name}" "${kubernetes_version}" "${settings_version}" > "${machine_file}"
+  _render_machine_json "${node_name}" "${kubernetes_version}" "${settings_version}" "${max_pods}" > "${machine_file}"
   if ! kubectl -n "${E2E_CONTROLLER_NAMESPACE}" get configmap "${E2E_MACHINE_CONFIGMAP}" >/dev/null 2>&1; then
     kubectl -n "${E2E_CONTROLLER_NAMESPACE}" create configmap "${E2E_MACHINE_CONFIGMAP}" >/dev/null
   fi
@@ -379,8 +380,8 @@ _machine_configmap_upsert_unlocked() {
 }
 
 machine_configmap_upsert() {
-  local node_name="$1" kubernetes_version="${2:-${E2E_KUBERNETES_VERSION}}" settings_version="${3:-${kubernetes_version}}"
-  with_cluster_lock _machine_configmap_upsert_unlocked "${node_name}" "${kubernetes_version}" "${settings_version}"
+  local node_name="$1" kubernetes_version="${2:-${E2E_KUBERNETES_VERSION}}" settings_version="${3:-${kubernetes_version}}" max_pods="${4:-110}"
+  with_cluster_lock _machine_configmap_upsert_unlocked "${node_name}" "${kubernetes_version}" "${settings_version}" "${max_pods}"
 }
 
 _machine_configmap_delete_unlocked() {

hack/e2e/lib/node-join-token.sh

@@ -90,7 +90,7 @@ node_join_token() {
   mv "${config_file}.tmp" "${config_file}"
 
   # Step 3: Publish the AKS Machine goal and deploy the agent.
-  machine_configmap_upsert "$(state_get token_vm_name)" "${E2E_KUBERNETES_VERSION}" "${E2E_KUBERNETES_VERSION}"
+  machine_configmap_upsert "$(state_get token_vm_name)" "${E2E_KUBERNETES_VERSION}" "${E2E_KUBERNETES_VERSION}" "${E2E_KUBELET_MAX_PODS}"
   _deploy_and_start_agent "${vm_ip}" "${config_file}" "aks-flex-node-token"
 
   log_success "Token node joined in $(timer_elapsed "${start}")s"

hack/e2e/lib/upgrade-drift.sh

@@ -102,12 +102,16 @@ _ensure_mode_joined() {
 
 _trigger_mode_repave() {
   local mode="$1" desired_version="$2" settings_version="$3"
-  local vm_ip vm_name
+  local vm_ip vm_name max_pods
   vm_ip="$(_mode_vm_ip "${mode}")"
   vm_name="$(_mode_vm_name "${mode}")"
+  max_pods="110"
+  if [[ "${mode}" == "token" ]]; then
+    max_pods="${E2E_KUBELET_MAX_PODS}"
+  fi
 
   log_info "Updating controller machine goal for ${mode} node to Kubernetes ${desired_version} (${settings_version})"
-  machine_configmap_upsert "${vm_name}" "${desired_version}" "${settings_version}"
+  machine_configmap_upsert "${vm_name}" "${desired_version}" "${settings_version}" "${max_pods}"
   remote_exec "${vm_ip}" 'sudo systemctl status aks-flex-node-agent.service --no-pager -l || true'
 
   log_info "Deleting Kubernetes Node ${vm_name} to trigger ${mode} repave"

pkg/aksmachine/client_armapi.go

@@ -47,7 +47,7 @@ func newARMClient(cfg *config.Config, logger *slog.Logger) (MachineClient, error
 }
 
 func (c *armMachineClient) Create(ctx context.Context, desired GoalState) (*Machine, error) {
-	if err := desired.validate(); err != nil {
+	if err := desired.Validate(); err != nil {
 		return nil, fmt.Errorf("validate goal state: %w", err)
 	}
 	params := armcontainerservice.Machine{
@@ -214,10 +214,14 @@ func clientCertificateCredentialOptions(clientOpts azcore.ClientOptions) *aziden
 func buildK8sProfile(goal GoalState) *armcontainerservice.MachineKubernetesProfile {
 	// FlexNode RP accepts the registration surface below; local kubelet defaults
 	// are consumed during node bootstrap and must not be sent as Machine fields.
-	maxPods := int32(goal.MaxPods) //nolint:gosec // validated non-negative and small
+	var maxPods *int32
+	if goal.MaxPods != nil {
+		value := int32(*goal.MaxPods) //nolint:gosec // validated non-negative and small
+		maxPods = &value
+	}
 	p := &armcontainerservice.MachineKubernetesProfile{
 		OrchestratorVersion: &goal.KubernetesVersion,
-		MaxPods:             &maxPods,
+		MaxPods:             maxPods,
 		NodeLabels:          stringPointerMap(goal.NodeLabels),
 		NodeTaints:          stringPointerSlice(goal.NodeTaints),
 	}
@@ -275,9 +279,16 @@ func machineFromARM(machine armcontainerservice.Machine, defaultID, defaultName
 		kubernetes := properties.Kubernetes
 		if kubernetes.OrchestratorVersion != nil {
 			result.Goal.KubernetesVersion = *kubernetes.OrchestratorVersion
+			if kubernetes.CurrentOrchestratorVersion != nil {
+				result.Goal.KubernetesVersion = resolveKubernetesVersionAlias(
+					result.Goal.KubernetesVersion,
+					*kubernetes.CurrentOrchestratorVersion,
+				)
+			}
 		}
 		if kubernetes.MaxPods != nil {
-			result.Goal.MaxPods = int(*kubernetes.MaxPods)
+			value := int(*kubernetes.MaxPods)
+			result.Goal.MaxPods = &value
 		}
 		if kubernetes.NodeLabels != nil {
 			result.Goal.NodeLabels = stringMapFromPointers(kubernetes.NodeLabels)
@@ -287,10 +298,12 @@ func machineFromARM(machine armcontainerservice.Machine, defaultID, defaultName
 		}
 		if kubernetes.KubeletConfig != nil {
 			if kubernetes.KubeletConfig.ImageGcHighThreshold != nil {
-				result.Goal.KubeletConfig.ImageGCHighThreshold = int(*kubernetes.KubeletConfig.ImageGcHighThreshold)
+				value := int(*kubernetes.KubeletConfig.ImageGcHighThreshold)
+				result.Goal.KubeletConfig.ImageGCHighThreshold = &value
 			}
 			if kubernetes.KubeletConfig.ImageGcLowThreshold != nil {
-				result.Goal.KubeletConfig.ImageGCLowThreshold = int(*kubernetes.KubeletConfig.ImageGcLowThreshold)
+				value := int(*kubernetes.KubeletConfig.ImageGcLowThreshold)
+				result.Goal.KubeletConfig.ImageGCLowThreshold = &value
 			}
 		}
 	}
@@ -303,6 +316,15 @@ func machineFromARM(machine armcontainerservice.Machine, defaultID, defaultName
 	return result
 }
 
+func resolveKubernetesVersionAlias(desired, current string) string {
+	desiredVersion := strings.TrimPrefix(strings.TrimSpace(desired), "v")
+	currentVersion := strings.TrimPrefix(strings.TrimSpace(current), "v")
+	if len(strings.Split(desiredVersion, ".")) == 2 && strings.HasPrefix(currentVersion, desiredVersion+".") {
+		return currentVersion
+	}
+	return desiredVersion
+}
+
 func stringMapFromPointers(values map[string]*string) map[string]string {
 	result := make(map[string]string, len(values))
 	for key, value := range values {