feat(sandbox): per-(org, repo) size tiers

[pedrofrxncx]

Why

Sandbox pods are one fixed size for every tenant — 500m/2Gi request, 2/4Gi limit. A repo that needs more than 4Gi OOMs, and today the only lever is raising the limit for the whole fleet, which we pay for on every pod.

Note the OOM is on the limit, not the request. Each tier moves both: values.yaml already records that raising only the limit is what produced prod's node memory-pressure evictions.

What

Chart — one SandboxTemplate per entry in values.tiers (small / medium / large ship by default), rendered from the same template body so tiers can't drift in security posture, netinit, or org-fs wiring. A tier owns resources and placement: nodeSelector, tolerations, affinity, topologySpreadConstraints, podLabels, podAnnotations, warmPoolSize, each wholesale-overriding the top-level value of the same name. That's the seam for "this tier runs on on-demand capacity only" (snippet included, commented — the labels need a NodePool that offers it) and for per-tier cost labels.

defaultTier renders at the unsuffixed name, so existing claims and an existing STUDIO_SANDBOX_TEMPLATE_NAME keep resolving. Adding tiers to a running environment is a pure addition.

Studio — the tier is picked per (org, repo) from STUDIO_SANDBOX_TIER_MAP, overrides only, keyed <orgSlug>/<owner>/<repo> or <orgSlug>, most specific first:

STUDIO_SANDBOX_TIER_MAP: '{"acme/acme/monorepo":"large","acme":"medium"}'

Deliberately not per-agent metadata: that blob is writable by any org member, so a tenant could assign itself the largest tier. Assignment stays operator-controlled and the default stays the cheapest option, which is what makes tiers priceable. No UI — that's the follow-up when the list outgrows an env var (ponytail: comment names the upgrade path: a column on organization_settings plus an /api/_admin tool).

The tier stays provider-agnostic: it rides on EnsureOptions.tier in the runner-agnostic contract, resolution lives in apps/api/src/sandbox/tier.ts, and only the agent-sandbox runner knows a tier maps to a SandboxTemplate name. Runners without tiering ignore it.

Decisions worth reviewing

• app.kubernetes.io/name is NOT tier-suffixed. The housekeeper's podSelector and any topologySpreadConstraints labelSelector key off it, so tiering it would silently drop the other tiers out of idle reaping. Tier gets its own studio.decocms.com/tier label, which also joins cAdvisor pod metrics to a tier for cost attribution.
• A warm pool is rendered per tier even at size: 0. Claims always carry warmpool: "default", so the pool must exist for the reference to resolve; an empty one falls through to a cold render — the same path a fully-claimed pool already takes in prod. The existing HPA stays scoped to the default tier.
• emptyDir size limits are now per-tier (values.emptyDirSizeLimits, previously hardcoded 4Gi/1Gi/5Gi). A large tier with a 40Gi ephemeral-storage ceiling but a 4Gi workdir cap would just fail on disk instead of memory.
• A tier is frozen for a sandbox's lifetime. The claim handle can't encode it — sandbox-proxy.ts derives the handle from the URL and has no tier context, so encoding it would 404 every proxy call. A re-assignment applies on the next cold provision; to resize now, SANDBOX_DELETE. The tier rides in the persisted ensureOpts, so resurrection and autonomous recovery re-ensure at the same tier.
• Tier names are shape-validated on both sides but membership can't be. Studio can't enumerate the chart's tiers, so a name with no matching tiers key fails claim creation for that (org, repo) only — the two sides have to move together. Called out in the chart README and both code comments.
• Malformed STUDIO_SANDBOX_TIER_MAP degrades to "no overrides" instead of throwing: it gates provisioning for every org, so a typo in one env var must not take the fleet's sandboxes down. Bad individual entries are dropped with a warning naming them.
• The chart rejects studio.decocms.com/* keys in a tier's podLabels/podAnnotations. That prefix is the runner's additionalPodMetadata namespace and the operator rejects claims redefining a key the template already set — every claim on that tier would fail. Banning the prefix rather than a key list keeps it in sync with the runner without duplicating LABEL_KEYS.

Testing

• Default tier render is unchanged. Diffed helm template for studio-sandbox-staging against main: identical apart from the chart-version label and the two new studio.decocms.com/tier: "small" labels. Resources, volumes, sizeLimits, netinit, org-fs all byte-identical.
• Per-tier render verified for medium/large: resources, emptyDir sizeLimits, and per-tier podLabels/podAnnotations land where expected.
• All four chart validations exercised and fail at template time: defaultTier not in tiers, empty tiers, non-DNS-label tier name, reserved studio.decocms.com/* pod label and annotation.
• Warm pools: with warmPool.size=5, default tier gets 5 and medium/large get an explicit 0 (uses hasKey, not default — default treats 0 as empty and would have leaked the global size).
• Unit tests: tier.test.ts (precedence, repo-key-without-org must not match, case-insensitivity), template-name.test.ts (unsuffixed default, suffixing, 8 malformed inputs fall back), plus 5 new resolve-config cases for the env-var parse.
• helm lint, bun run check, bun run lint (0 errors, same 16 pre-existing warnings as main), bun run fmt:check, and the sandbox-provider + settings + sandbox test files (232 pass) all clean.

Rollout

Chart upgrade is additive and prod-inert, but the chart is consumed by a pinned targetRevision in deco-apps-cd — that needs a bump to 0.10.0 or nothing ships. Order: chart → targetRevision → STUDIO_SANDBOX_TIER_MAP. Verify medium/large end-to-end on staging before assigning a real tenant.

Follow-ups

• Assignment UI / admin tool + DB column when the override list outgrows an env var.
• An on-demand NodePool if we want the large tier's spot opt-out (infra, not this chart).

---

Summary by cubic

Introduce per-(org, repo) sandbox size tiers to stop OOMs without raising limits for everyone. The chart renders per-tier SandboxTemplates; Studio assigns tiers via STUDIO_SANDBOX_TIER_MAP, and untiered sandboxes keep using the existing default.

• New Features

  • Helm: one SandboxTemplate per values.tiers with per-tier resources, placement, warmPoolSize, and emptyDirSizeLimits; defaultTier renders unsuffixed; adds a studio.decocms.com/tier label on templates, warm pools, and pods; tier validations added.
  • Studio/API: parses STUDIO_SANDBOX_TIER_MAP (lowercases keys, drops invalid entries, malformed JSON or absent map → no overrides), resolves overrides (<orgSlug>/<owner>/<repo> > <orgSlug>; repo-less uses org-wide), and passes EnsureOptions.tier; the agent-sandbox runner maps tiers to <template>-<tier> and falls back on invalid input; runners without tiering ignore it.

• Migration

  • Bump the sandbox-env chart to 0.10.0 and update the CD targetRevision.
  • Optionally set STUDIO_SANDBOX_TIER_MAP (JSON) in configMap.meshConfig; keys are case-insensitive; unlisted repos use defaultTier.
  • Verify non-default tiers on staging; keep defaultTier as the cheapest option.

^{Written for commit 8033b9f. Summary will update on new commits.}