feat(sandbox): log which cache tier served each dependency install - #5346
Merged
Conversation
The golden node_modules cache runs in prod but nothing measures it. A hit
requires the pod to land on a node already warm for its repo, so the hit
rate is a property of fleet churn — not something the cache code can
report about itself. Without that number we can't size the cross-node
(EFS L2) tier that would remove the same-node dependency.
Emit one line per completed dependency step:
{"msg":"sandbox.deps.restore","source":"miss","repo_hash":"…",
"duration_ms":42319,"bootId":"…"}
`stats by (source) count()` in VictoriaLogs gives the hit rate today.
A log line rather than a metric. It is the only channel that leaves a
sandbox pod — egress is locked to 53/443, so no in-cluster OTLP collector
is reachable, which the sibling emitInstalledDeps already documents. It
also carries bootId, which a metric can't: as an attribute that is
unbounded cardinality, and without it a counter cannot separate "one pod
installed three times" from "three pods" — and stepInstall does re-run
within a boot on config change or reprovision. Raw durations also beat
pre-bucketed histograms when the range is what you're trying to discover.
Sized for the pipeline that drops large lines and samples bursts: ~100
bytes, one per boot. Both limits are orders of magnitude away, so unlike
the per-dep format this needs no chunking and no sample_rate correction.
Times the whole step, golden probe included — a failed probe is real boot
latency, and on a miss the cost L2 would replace is probe + install.
This was referenced Jul 28, 2026
pedrofrxncx
enabled auto-merge (squash)
July 28, 2026 17:22
decocms Bot
pushed a commit
that referenced
this pull request
Jul 28, 2026
PR: #5346 feat(sandbox): log which cache tier served each dependency install Bump type: minor - @decocms/sandbox (packages/sandbox/package.json): 1.24.4 -> 1.25.0 - deploy/helm/sandbox-env (chart 0.9.36) (deploy/helm/sandbox-env/values.yaml deploy/helm/sandbox-env/Chart.yaml): image.tag/appVersion -> 1.25.0 Deploy-Scope: both
This was referenced Jul 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces #5339 as the answer to "what is the golden cache's hit rate". Opened after review of that PR pointed out it ships blocked machinery for a question that is answerable today.
What
One line per completed dependency step, next to the existing
emitInstalledDepscall:{"msg":"sandbox.deps.restore","source":"miss","repo_hash":"a1b2…","duration_ms":42319,"bootId":"01JB…"}sourceisl1(reflinked the golden, install skipped) ormiss(ran a full install).stats by (source) count()in VictoriaLogs gives the hit rate, andrepo_hashgives repo-concentration-per-node — the two inputs that size the EFS L2 tier.Why a log line and not a metric
dep-metrics.tsalready documents this two lines above the import.bootIdis free here and impossible as a metric attribute — it's unbounded cardinality. Without it a counter can't separate "one pod installed three times" from "three pods", andstepInstalldoes re-run within a boot (config change, reprovision). That would quietly bias the hit rate.Sized against the pipeline that drops big lines and samples bursts (the constraints #4271/#4273/#4275 established the hard way): ~100 bytes, one per boot — measured at 97 bytes in the orchestrator test. Both limits are orders of magnitude away, so no chunking and no
sample_ratecorrection.Timing covers the whole step
durationMsstarts before the golden probe, not after. A failed probe (stat, partial reflink, cleanup) is real boot latency, and on a miss the cost L2 would replace is probe + install, not install alone.Scope note
Only completed steps are recorded — failed installs, the fingerprint-skip path and the no-package-manager early return are not. So the denominator is "dependency steps that finished", which is the right one for "did the cache save a boot", but it is not "installs attempted".
Testing
deps-restore-line.test.ts— pure builder, no mocks. Asserts the line stays well under the byte budget, that aghs_token in the clone URL survives nowhere in the output, that credentials don't changerepo_hash(a token rotation must not split the hit rate across two series), and that a repo-less boot still emits a countable line.bun test packages/sandbox/daemon/setup/→ 89 pass.daemon.e2e.test.ts→ 103 pass.bun run fmt,bun run lint(0 errors),knipclean. No new dependencies; daemon bundle unchanged.Summary by cubic
Add a small JSON log per completed dependency step to record which cache tier served it and how long it took. This lets us measure the golden cache hit rate and size the EFS L2 tier accurately.
sandbox.deps.restorewithsource(l1ormiss),repo_hash,duration_ms, andbootId. Use VictoriaLogsstats by (source) count()to get hit rate.repo_hashis a credential‑stripped 16‑hex hash of the clone URL; emits"unknown"when no repo.Written for commit ef208e1. Summary will update on new commits.