Skip to content

feat(sandbox): log which cache tier served each dependency install - #5346

Merged
pedrofrxncx merged 1 commit into
mainfrom
feat/sandbox-deps-restore-log-line
Jul 28, 2026
Merged

feat(sandbox): log which cache tier served each dependency install#5346
pedrofrxncx merged 1 commit into
mainfrom
feat/sandbox-deps-restore-log-line

Conversation

@pedrofrxncx

@pedrofrxncx pedrofrxncx commented Jul 28, 2026

Copy link
Copy Markdown
Collaborator

Replaces #5339 as the answer to "what is the golden cache's hit rate". Opened after review of that PR pointed out it ships blocked machinery for a question that is answerable today.

What

One line per completed dependency step, next to the existing emitInstalledDeps call:

{"msg":"sandbox.deps.restore","source":"miss","repo_hash":"a1b2…","duration_ms":42319,"bootId":"01JB…"}

source is l1 (reflinked the golden, install skipped) or miss (ran a full install). stats by (source) count() in VictoriaLogs gives the hit rate, and repo_hash gives repo-concentration-per-node — the two inputs that size the EFS L2 tier.

Why a log line and not a metric

  • It's the only channel that leaves a sandbox pod. Egress is locked to 53/443, so no in-cluster OTLP collector is reachable. dep-metrics.ts already documents this two lines above the import.
  • bootId is free here and impossible as a metric attribute — it's unbounded cardinality. Without it a counter can't separate "one pod installed three times" from "three pods", and stepInstall does re-run within a boot (config change, reprovision). That would quietly bias the hit rate.
  • Raw durations beat pre-bucketed histograms when the range is what you're measuring. Picking bucket boundaries requires already knowing the answer.

Sized against the pipeline that drops big lines and samples bursts (the constraints #4271/#4273/#4275 established the hard way): ~100 bytes, one per boot — measured at 97 bytes in the orchestrator test. Both limits are orders of magnitude away, so no chunking and no sample_rate correction.

Timing covers the whole step

durationMs starts before the golden probe, not after. A failed probe (stat, partial reflink, cleanup) is real boot latency, and on a miss the cost L2 would replace is probe + install, not install alone.

Scope note

Only completed steps are recorded — failed installs, the fingerprint-skip path and the no-package-manager early return are not. So the denominator is "dependency steps that finished", which is the right one for "did the cache save a boot", but it is not "installs attempted".

Testing

  • deps-restore-line.test.ts — pure builder, no mocks. Asserts the line stays well under the byte budget, that a ghs_ token in the clone URL survives nowhere in the output, that credentials don't change repo_hash (a token rotation must not split the hit rate across two series), and that a repo-less boot still emits a countable line.
  • bun test packages/sandbox/daemon/setup/ → 89 pass. daemon.e2e.test.ts → 103 pass.
  • bun run fmt, bun run lint (0 errors), knip clean. No new dependencies; daemon bundle unchanged.

Summary by cubic

Add a small JSON log per completed dependency step to record which cache tier served it and how long it took. This lets us measure the golden cache hit rate and size the EFS L2 tier accurately.

  • New Features
    • Emit sandbox.deps.restore with source (l1 or miss), repo_hash, duration_ms, and bootId. Use VictoriaLogs stats by (source) count() to get hit rate.
    • Time covers the whole step (golden probe + install) to reflect real boot latency.
    • repo_hash is a credential‑stripped 16‑hex hash of the clone URL; emits "unknown" when no repo.
    • Sized for the log pipeline and guarded from failures; tests ensure small lines, no credential leaks, and stable hashing across credential changes.

Written for commit ef208e1. Summary will update on new commits.

Review in cubic

The golden node_modules cache runs in prod but nothing measures it. A hit
requires the pod to land on a node already warm for its repo, so the hit
rate is a property of fleet churn — not something the cache code can
report about itself. Without that number we can't size the cross-node
(EFS L2) tier that would remove the same-node dependency.

Emit one line per completed dependency step:

  {"msg":"sandbox.deps.restore","source":"miss","repo_hash":"…",
   "duration_ms":42319,"bootId":"…"}

`stats by (source) count()` in VictoriaLogs gives the hit rate today.

A log line rather than a metric. It is the only channel that leaves a
sandbox pod — egress is locked to 53/443, so no in-cluster OTLP collector
is reachable, which the sibling emitInstalledDeps already documents. It
also carries bootId, which a metric can't: as an attribute that is
unbounded cardinality, and without it a counter cannot separate "one pod
installed three times" from "three pods" — and stepInstall does re-run
within a boot on config change or reprovision. Raw durations also beat
pre-bucketed histograms when the range is what you're trying to discover.

Sized for the pipeline that drops large lines and samples bursts: ~100
bytes, one per boot. Both limits are orders of magnitude away, so unlike
the per-dep format this needs no chunking and no sample_rate correction.

Times the whole step, golden probe included — a failed probe is real boot
latency, and on a miss the cost L2 would replace is probe + install.
@pedrofrxncx
pedrofrxncx enabled auto-merge (squash) July 28, 2026 17:22
@pedrofrxncx
pedrofrxncx merged commit 5c22a6f into main Jul 28, 2026
16 of 17 checks passed
@pedrofrxncx
pedrofrxncx deleted the feat/sandbox-deps-restore-log-line branch July 28, 2026 17:28
decocms Bot pushed a commit that referenced this pull request Jul 28, 2026
PR: #5346 feat(sandbox): log which cache tier served each dependency install
Bump type: minor

- @decocms/sandbox (packages/sandbox/package.json): 1.24.4 -> 1.25.0
- deploy/helm/sandbox-env (chart 0.9.36) (deploy/helm/sandbox-env/values.yaml deploy/helm/sandbox-env/Chart.yaml): image.tag/appVersion -> 1.25.0

Deploy-Scope: both
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant