feat(sandbox-env): opt-in node placeholder "balloon" for warm node capacity

[nicacioliveira]

Summary

warmPool pre-warms sandbox pods (image pull + kubelet start + daemon boot), but every warm-pool pod still needs a node. When the pool refills after a burst, its KEDA/HPA scaler grows it, or a cold claim lands beyond the pool, the new pods go Pending and wait on Karpenter to provision a node — tens of seconds of user-visible claim latency that warmPool alone can't remove.

This adds an opt-in node placeholder ("balloon") to the sandbox-env chart: low-priority pods that reserve warm node capacity on the sandbox NodePool, so the pod-level warm pool always has somewhere to land instantly.

Same pattern as the existing placeholder-pods on eks-serverless, but scoped to the taint-isolated sandbox NodePool (the general-pool balloon does nothing for it).

How it works

nodePlaceholder (default off) runs replicas low-priority pods that hold one sandbox-slot of requests each on already-running nodes. A real sandbox / warm-pool pod preempts one instantly (priority-based), takes the freed slot on the warm node, and Karpenter re-provisions a node for the evicted balloon in the background. Net: the user-facing claim never waits on node provisioning.

Design notes

• Placement inherits the sandbox pod's nodeSelector / tolerations by default (values.nodeSelector / values.tolerations) so warm capacity lands on the same dedicated NodePool. Override under nodePlaceholder.* only to pin elsewhere.
• priorityClassName must be low / negative (default placeholder-priority) so real sandbox pods (priority 0) preempt it — enforced at template time.
• terminationGracePeriodSeconds: 0 — balloons hold nothing durable (unlike real sandbox pods, which use 90s for the SIGTERM git-sync).
• pause image, locked-down securityContext (nonroot, drop ALL, RO rootfs), built-in soft AZ spread.
• Resource requests default to a full sandbox pod footprint (container 500m/2Gi + orgfs sidecar 50m/128Mi) so preempting one balloon frees exactly one slot.

Testing

• helm template default → no placeholder Deployment rendered.
• helm template with prod-like top-level nodeSelector/tolerations → balloon inherits them, renders placeholder-priority, terminationGracePeriodSeconds: 0, AZ spread.
• Validation fails at template time for enabled + replicas=0 and enabled + empty priorityClassName.
• helm lint clean.

Follow-up (separate PR, deco-apps-cd)

Enablement is a per-env values change (nodePlaceholder.enabled: true + replicas) plus a targetRevision bump to 0.9.51. Merge after this chart version publishes to OCI. Since prod and staging share the sandbox NodePool, size replicas once for the pool.

Recommend enabling in staging first and confirming via cluster events that the claim-latency tail is actually node provisioning (Pod-created → Node-assigned) before prod.

---

Summary by cubic

Adds an opt-in node placeholder “balloon” to the sandbox-env chart to keep warm node capacity on the sandbox NodePool so sandbox and warm-pool pods schedule instantly without waiting for Karpenter. Bumps the chart to 0.9.51. Default is off.

• New Features

  • Adds nodePlaceholder.* and a Deployment (studio-sandbox-placeholder-<env>) of low-priority pods that reserve node capacity; real pods preempt immediately.
  • Defaults placement to the sandbox nodeSelector/tolerations; requires a low/negative priorityClassName (e.g., placeholder-priority); terminationGracePeriodSeconds: 0.
  • Requests sized to one sandbox slot per replica using the pause image; secure context and soft AZ spread.
  • Template validations for replicas >= 1 and non-empty priority class; README and helpers updated; chart version 0.9.51.

• Migration

  • Enable with nodePlaceholder.enabled: true, set replicas, and ensure a low/negative PriorityClass exists (e.g., placeholder-priority).
  • Size replicas once for the shared sandbox NodePool (staging/prod share it) and bump your ArgoCD targetRevision to 0.9.51.

^{Written for commit 8b7f826. Summary will update on new commits.}