feat(metrics): add wizard metrics program (runs the context-mill metrics skill) - #1102
Conversation
Flat skill command mirroring ai-observability: no fixed skillId — the agent loads the context-mill `metrics` category (PostHog/context-mill#342) and picks the platform variant (python, nodejs, javascript, kubernetes, other/OTLP) from the project manifest. Ships a metrics-specific intro screen, registry + switchboard binding, and program tests. Generated-By: PostHog Desktop Task-Id: 495fbacb-61e8-4fdb-8c0f-7f69cd7a3fd6
🧙 Wizard CIRun the Wizard CI and test your changes against wizard-workbench example apps by replying with a GitHub comment using one of the following commands: Test all apps:
Test all apps in a directory:
Test an individual app:
Show more apps
Results will be posted here when complete. |
Adding a new intro screen requires registering it in two harness allow-lists; without them a headless run stalls on the intro forever (the flow trace repeated metrics-intro for all 40 guard iterations). Registers ScreenId.MetricsIntro in the action registry and decideE2eAction, declares the program's e2e path in test/e2e.json, and adds a metrics flow snapshot so the walkthrough is regression-covered. Pre-existing, unrelated: the action-registry exhaustiveness test still reports `task-notice` as uncovered on main; left alone here. Generated-By: PostHog Desktop Task-Id: 495fbacb-61e8-4fdb-8c0f-7f69cd7a3fd6
|
Hey @DanielVisca Will be looking at this today, sorry for being a little late. I'll be looking to do a few things on stacked PRs
|
… in profiles + switchboard)
…1129) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…iant machinery on the wizard side too Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…inear prompt pull the matching one themselves Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…lows them Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![veria-ai[bot]](https://avatars.githubusercontent.com/in/1723776?s=60&v=4){kind=small}
| 'publish_handoff', | ||
| ]; | ||
|
|
||
| const OPT_IN_WIZARD_TOOLS = ['wizard_ask', 'load_skill_menu', 'install_skill']; |
There was a problem hiding this comment.
Medium: Skill installation can escape the workspace
Registering install_skill here makes the existing unsafe destination calculation reachable from remote task prompts. The installer passes the registry-controlled skillEntry.id to path.join(installDir, '.claude', 'skills', skillEntry.id) without verifying that the result remains under .claude/skills; an attacker controlling the skill registry can use an ID such as ../../../../target to write the downloaded bundle outside the project, and a rejected scan subsequently calls rmSync on that escaped directory. Validate skill IDs against a strict format and resolve the destination before any directory creation, download extraction, scanning, or cleanup, rejecting paths that are not descendants of the intended skills root.
PR overviewThis pull request adds a One security issue remains open in the skill installation flow. A registry-controlled skill identifier can escape the workspace path, allowing an attacker who controls registry content to write or delete files outside the intended skills directory. No issues have yet been addressed. Open issues (1)
Fixed/addressed: 0 · PR risk: 7/10 |
Works great, added verify/install SDK steps + basic monorepo support
|
gewenyu99
left a comment
gewenyu99
left a comment
There was a problem hiding this comment.
This looks good. 1 last change before I merge
-> code owners
-> fixing handoff
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…trics skill) (PostHog#1102) Co-authored-by: Vincent (Wen Yu) Ge <vincent@posthog.com> Co-authored-by: Vincent (Wen Yu) Ge <29069505+gewenyu99@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2 participants
Problem
PostHog now has an application-metrics product (
posthog.metrics— counters, gauges, histograms, alpha), and context-mill is growing ametricsskill family (PostHog/context-mill#342) that teaches the agent where to place metrics calls. The wizard had no command to run it — users would have to know the exact variant id and typewizard skill metrics-pythonthemselves.Changes
Adds a flat
wizard metricsTUI program, the same shape asai-observability:src/lib/programs/metrics/index.ts—ProgramConfigwith no fixedskillId; thecustomPrompthas the agent callload_skill_menuwith themetricscategory and pick the platform variant (python / nodejs / javascript / kubernetes / other-OTLP) from the project manifest, asking viawizard_askwhen ambiguous. Additive-only: reuse an existing PostHog client rather than constructing a second one.MetricsIntroScreen— variant-picking programs have no pre-seededskillId, so the generic intro renders "Let's run the unknown skill"; this mirrorsAiObservabilityIntroScreen.bin.tswiring, switchboardDEFAULT_BINDING, program tests.ScreenId.MetricsIntroin the harness action registry anddecideE2eAction, declares the program's path intest/e2e.json, and adds a metrics flow snapshot. Without this a headless run stalls on the intro (see Test plan).Test plan
pnpm build && pnpm test && pnpm lint— 1858/1858 tests pass, 0 lint errors.Recorded walkthrough
A real terminal recording, not a re-render:
asciinemain a 116×30 PTY, the wizard driven with actual keystrokes.asciinema play wizard-metrics-demo.castWhat it shows, in order:
wizard --helplistsmetrics Add PostHog application metrics to your project.wizard metricson a Flask app: the new intro screen, ↓ + Enter into More info (which describes the program and the platform variants), Enter back, Continue, health check, then the auth screen. Interactively the wizard wants a browser OAuth round-trip or aphx_personal key.fetchSkillMenuresolves themetricscategory (all five variants) andinstallSkillById('metrics-python')extractsSKILL.md+ references, withtriage: undefined(the strict path where a warlock-flagged skill fails closed). Production wizard code, no model needed.posthog.metrics:The pass/fail verdict in the recording is computed from the returned rows (counters on summed delta, the gauge on its last value, the histogram on sample count) and exits non-zero on mismatch — it is not a hardcoded success line.
What the recording does not show
The agent's own model call. This sandbox holds a
pha_OAuth application token, andgateway.us.posthog.com/wizardrejects that app for thewizardproduct:The wizard requires
llm_gateway:readon aphx_key (or browser OAuth), and it deliberately strips a directANTHROPIC_API_KEYfrom the agent subprocess (agent-env-isolation.ts) so runs bill through the gateway — so there is no legitimate local bypass, and none was used. Everything up to and including the agent initializing and running this program's flow is real; writing the instrumentation is the one step that needs a wizard-authorised key. Under the repo's ci-mode harness (which injects credentials) the run reachesAgent initialized. Let's get cooking!/Setting up application metrics...before the same 403, and renders a structured error outro rather than a stack trace.The drivability bug this surfaced
Adding a new intro screen needs it registered in two harness allow-lists. Before the fix, the offline flow trace repeated
metrics-introfor all 40 guard iterations — a headless run would hang on the intro forever. The harness's own exhaustiveness test confirms the delta:['metrics-intro', 'task-notice']['task-notice']task-noticeis a pre-existing gap onmain, unrelated to metrics — left alone rather than silently changed. Note these harness tests are currently excluded frompnpm testbyvitest.config.ts("the e2e harness and its tests live in a separate stacked PR"), so they were run explicitly; that exclusion is why the stall wouldn't have failed CI.The committed metrics flow snapshot now pins the full path:
Skill content
Validated end-to-end separately: instrumenting a Flask app from the
metrics-pythonskill produced counter, gauge, and histogram values in PostHog that matched deterministic generated traffic exactly — see PostHog/context-mill#342.Note: a full agent run needs the
metricscategory live inskill-menu.json, i.e. context-mill#342 merged + released (or--local-mcpagainst a local build, as used here). Merge order: context-mill first.Created with PostHog Desktop