feat: tolerate HyperPod node-health-status taint in training-operator

[kethang-sm]

What's changing and why?

The kubeflow training-operator Deployment has no tolerations, so it cannot be scheduled onto nodes carrying the HyperPod deep health check taint sagemaker.amazonaws.com/node-health-status=Unschedulable:NoSchedule.

On clusters where every node is tainted during an on-start deep health check, the operator pod stays Pending and PyTorchJob-based checks such as MultiNodeTrainingValidation never get reconciled.

Other controllers and device plugins in this chart already tolerate the same taint (mpi-operator, neuron-device-plugin, nvidia-device-plugin, aws-efa-k8s-device-plugin).

Before/After UX

Before:

During an on-start deep health check, the operator pod cannot be placed:

$ kubectl -n kubeflow get pods
NAME READY STATUS RESTARTS AGE
training-operator-7d9f8c6b4d-x2n7q 0/1 Pending 0 4m

Events:
Warning FailedScheduling ... 0/3 nodes are available: 3 node(s) had untolerated
taint {sagemaker.amazonaws.com/node-health-status: Unschedulable}

PyTorchJob-based deep health checks are never reconciled.

After:
The operator pod schedules onto a tainted node and reconciles PyTorchJobs normally:

$ kubectl -n kubeflow get pods
NAME READY STATUS RESTARTS AGE
training-operator-7d9f8c6b4d-x2n7q 1/1 Running 0 30s

How was this change tested?

• helm lint helm_chart/HyperPodHelmChart/charts/training-operators — passes

  • Installed this chart on a live HyperPod EKS cluster (2 × ml.g6.8xlarge) with both nodes tainted
    sagemaker.amazonaws.com/node-health-status=Unschedulable:NoSchedule:

    NAME READY STATUS NODE
    tolerationtest-training-operators-86b89c5564-69t57 1/1 Running hyperpod-i-0e4270...

  • Negative control — removed the toleration from the same live Deployment, same tainted nodes:

    NAME STATUS NODE
    tolerationtest-training-operators-6c47bd8d5-bdw7h Pending

    Warning FailedScheduling default-scheduler 0/2 nodes are available: 2 node(s) had
    untolerated taint {sagemaker.amazonaws.com/node-health-status: Unschedulable}.

  • Separately, the equivalent toleration has been running in an internal HyperPod integration test that installs the training-operator from a standalone
    manifest: with the MultiNodeTraining deep health check enabled, burn-in completed and nodes were labeled Passed.

Are unit tests added?

Are integration tests added?

Reviewer Guidelines

‼️ Merge Requirements: PRs with failing integration tests cannot be merged without justification.

One of the following must be true:

• All automated PR checks pass
• Failed tests include local run results/screenshots proving they work
• Changes are documentation-only