feat(analytics): runtime_type across the full event chain + BYOK run events

[elihahhahahah11]

Why

While drilling the user-behaviour funnel down by runtime environment (AMR / BYOK / CLI) I hit a wall: runtime_type (the only field that cleanly separates byok from amr_cloud) was emitted only on onboarding ui_click events. It was absent from run_created / run_finished / export, so the daemon-side run events could not be split by runtime, and BYOK runs were effectively invisible to analytics. The only workaround was hand-built static cohorts that need daily rebuilding and double-count config-switchers.

Two root causes, both fixed here:

1. runtime_type was never promoted to a global public param (unlike configure_type), so it didn't ride onto the full event chain.
2. BYOK runs (config.mode === 'api') stream client-side straight to the user's provider and never reach the daemon, so the daemon's authoritative run_created / run_finished were never emitted for them. The web already had trackRunCreated / trackRunFinished defined but no callers.

What users will see

No user-facing UI change. Internally, every analytics event now carries runtime_type (amr_cloud / byok / local_cli / none), and BYOK runs now emit run_created / run_finished like daemon runs do — so dashboards can finally split the run funnel by AMR / BYOK / CLI.

Surface area

• API / contract — packages/contracts: runtime_type added to AnalyticsConfigureGlobals; runtimeType added to ChatAnalyticsHints; run agent_provider_id widened to allow BYOK providers.

Not a user capability, so the UI/CLI dual-track rule doesn't apply — this is analytics instrumentation behind existing flows.

Bug fix verification

Red-spec first, per AGENTS.md:

• apps/daemon/tests/run-runtime-type-analytics.test.ts — pins that a client runtime_type='byok' hint overrides the daemon's BYOK-blind derivation. Went red (helper absent) → green.
• apps/web/tests/byok-run-analytics.test.ts — pins the BYOK run_created / run_finished prop builders (provider mapping, model-id bucketing, required fields).

Validation

• pnpm guard (61 pass)
• pnpm --filter @open-design/contracts build && pnpm --filter @open-design/contracts test (164 pass)
• pnpm --filter @open-design/daemon typecheck + daemon analytics tests (8 pass)
• pnpm --filter @open-design/web typecheck + web analytics tests (22 pass)

Notes / follow-ups

• Daemon derivation stays BYOK-blind by design (no key visibility); accuracy for BYOK comes from the web client's per-run hint.
• BYOK run_finished currently reports result, artifact_count, asked_user_question, total_duration_ms; token-usage and duration-breakdown fields are left as an honest follow-up (token_count_source: 'unknown') rather than faked.
• configure_type (incl. both) is intentionally left as-is; runtime_type is now the canonical runtime dimension.

🤖 Generated with Claude Code

[lefarcen]

Hi @elihahhahahah11! 👋

Thanks for wiring runtime_type through the analytics chain and filling the BYOK run-event gap — the problem statement in the PR body is clear, and I've routed this for maintainer review.

💡 To drive this PR to merge hands-free, paste this to your AI coding agent (Claude Code / Codex / opencode / Cursor …):
Take over nexu-io/open-design#4510 until it merges — read https://raw.githubusercontent.com/nexu-io/looper/main/skills/pr-takeover/SKILL.md and follow it.

[github-actions]

Visual regression review

Head: 0516f07 · Base: 48da58f

5 changed · 33 unchanged · 0 missing baseline · 0 failed

Changed cases

Case	Main	PR	Diff
visual-integrations-use-everywhere
visual-settings-byok
visual-settings-byok-model-dropdown
visual-settings-byok-openai
visual-settings-local-cli-model-dropdown

Unchanged cases

Case	Main	PR	Diff
visual-avatar-local-agent-list
visual-avatar-menu
visual-critical-settings
visual-critical-workspace
visual-critical-workspace-preview
visual-design-system-detail
visual-design-systems
visual-home
visual-home-catalog
visual-home-context-picker
visual-home-plugin-filter
visual-home-plugin-use-staged
visual-home-plugin-use-with-query
visual-home-staged-attachment
visual-integrations
visual-integrations-mcp
visual-new-project-modal
visual-onboarding-runtime
visual-plugin-details
visual-plugin-share-menu

Visual diff is advisory only and does not block merging.

[mrcfps]

Thanks for wiring this through — the overall direction makes sense, but I left two follow-ups that affect how trustworthy the new BYOK runtime funnel will be in practice.

_{🔁 Powered by Looper · runner=reviewer · agent=opencode · An autonomous AI dev team for your GitHub repos.}

[lefarcen]

Hey @elihahhahahah11 — I took another look after @mrcfps's pass.

The two follow-ups above are the right ones to address on this head:

• keep BYOK runtime_type pinned per run instead of inheriting mutable session state for run_finished
• make the success-path run_finished emit survive refreshProjectFiles() failures so the funnel can't get stuck at run_created

Once those are handled, this should be in a good place for another review pass.

[mrcfps]

Thanks for wiring this through — I found two follow-ups that still affect how trustworthy the new BYOK runtime funnel will be on this head.

_{🔁 Powered by Looper · runner=reviewer · agent=opencode · An autonomous AI dev team for your GitHub repos.}

[AmyShang-alt]

QA 验收记录

验收范围：

• runtime_type analytics 字段
• BYOK run_created / run_finished payload
• AMR/local CLI daemon runtime_type hint 覆盖
• contracts/web/daemon 相关自动化验证

已验证：

• PR head: 91d7eb2
• pnpm guard 通过，61 pass
• pnpm --filter @open-design/contracts build 通过
• pnpm --filter @open-design/contracts test 通过，164 pass
• pnpm --filter @open-design/web typecheck 通过
• web Vitest 通过，335 files / 3338 tests passed
• pnpm --filter @open-design/daemon typecheck 通过
• daemon runtime_type 目标测试通过，2 tests passed

未验证：

• PostHog 最近 24 小时未查到 run_created / run_finished。
• PostHog 最近 24 小时未查到 properties.runtime_type。
• 未在真实 BYOK provider 环境中完成一次成功/失败/取消 run 的 live payload 验证。

风险/关注点：

• BYOK 成功路径的 run_finished 依赖 refreshProjectFiles() 后异步发送；若 refreshProjectFiles() 失败，成功态 run_finished 可能漏报。
• 当前 GitHub PR checks 只返回 approve/label 两项，未看到完整 CI suite 状态。

结论：

• 需补验证：代码层自动化基本通过，但核心线上埋点尚未在 PostHog 观测到，不建议现在标 validated。

[lefarcen]

Hey @elihahhahahah11 — quick status check on this head:

• the same two follow-ups from @mrcfps still look like the right next steps before another review pass:
  • pin BYOK runtime_type on the run events themselves
  • make the success-path BYOK run_finished emit resilient when refreshProjectFiles() fails
• QA also noted that automated checks are in good shape, but they don't want to treat this as validated yet without a real PostHog / live BYOK payload verification.

Once those are addressed, this should be ready for another look.

[mrcfps]

Thanks for pushing this forward — I found two follow-ups that still affect how trustworthy the BYOK runtime funnel is on this head.

_{🔁 Powered by Looper · runner=reviewer · agent=opencode · An autonomous AI dev team for your GitHub repos.}

[lefarcen]

Hey @elihahhahahah11 — I checked the latest review on this head.

The same two follow-ups from @mrcfps are still the right things to close out before another pass:

• pin BYOK runtime_type on the emitted run events themselves instead of relying on mutable session-level analytics state
• make the success-path BYOK run_finished emit survive refreshProjectFiles() failures so successful runs can't get stuck at run_created

Once those are addressed, this should be ready for another look.

[mrcfps]

Thanks for pushing this forward — I found three follow-ups that still affect how trustworthy the new runtime analytics will be on this head.

_{🔁 Powered by Looper · runner=reviewer · agent=opencode · An autonomous AI dev team for your GitHub repos.}

[lefarcen]

Hey @elihahhahahah11 — I checked the latest review on this head.

The three follow-ups from @mrcfps look like the right things to close out before another pass:

• make BYOK runtime_type explicit on the emitted run events instead of relying on mutable session-level analytics state
• make mode === 'api' win over a remembered agentId === 'amr' so the active BYOK runtime is classified correctly
• make the success-path BYOK run_finished emit survive refreshProjectFiles() failures so successful runs can't get stuck at run_created

Once those are addressed, this should be ready for another look.

[elihahhahahah11]

Addressed all three follow-ups in 0516f07:

1. byok-run.ts — runtime_type='byok' is now stamped directly on the run_created/run_finished payloads (RunCreatedProps gained an optional per-event runtime_type), instead of relying on the mutable super-property. A mid-stream mode switch can no longer split a BYOK run across runtime buckets.
2. ProjectView — the artifact-count refreshProjectFiles() is now wrapped in try/catch; a rejected refetch emits run_finished with count 0 rather than swallowing it, so a successful BYOK turn never hangs at run_created.
3. deriveConfigureGlobals — reordered so mode === 'api' wins over a stale remembered agentId === 'amr', so the AMR→Use-API flow reports runtime_type=byok (added a regression test).

Validation: contracts build+test (164), web + daemon typecheck, web analytics (23) + daemon analytics (8) tests, pnpm guard (61) — all green.

[lefarcen]

Hey @elihahhahahah11 — thanks for the detailed update.

I see the three follow-ups from @mrcfps are addressed on this head, and the next maintainer review is already pending on the latest revision. Once that pass lands, we’ll have the up-to-date reviewer signal on this version.

[mrcfps]

@elihahhahahah11 I re-reviewed the updated runtime analytics chain on ad41825 and verified the earlier follow-ups are now covered in the diff: BYOK run events pin runtime_type per event, the success-path BYOK run_finished emit survives project-file refresh failures, and deriveConfigureGlobals now lets API mode win over a stale remembered AMR agent. Thanks for pushing the fixes through — this looks ready from my review side.

_{🔁 Powered by Looper · runner=reviewer · agent=opencode · An autonomous AI dev team for your GitHub repos.}