Add opt-in OpenTelemetry diagnostics groundwork

[calebeden]

Summary

Adds an opt-in OpenTelemetry diagnostics endpoint setting in the companion app and lays the groundwork for future OTel instrumentation.

This does not start exporting general app behavior yet. The only data sent today is a small diagnostic probe when the user explicitly configures an OTLP endpoint: one sample trace span, one sample metric, and one sample structured log. If no endpoint is configured, no OpenTelemetry export is enabled.

This is intended for observability and debugging against an endpoint the user controls, not background consumer tracking. OTLP/gRPC and OTLP/HTTP are both supported; HTTP/protobuf derives the signal-specific trace, metric, and log endpoints from the configured collector base URL.

A maintainer follow-up adds an explicit Send probe again action for retrying unchanged saved settings after a collector outage. Automatic startup/settings application remains deduplicated, and the UI continues to distinguish local SDK flush completion from collector acknowledgement.

Telemetry boundary

docs/TELEMETRY.md establishes the ongoing telemetry conventions for OpenClaw: export is opt-in, disabled by default, and sent only to the user-configured endpoint. Future instrumentation should stay limited to low-cardinality operational diagnostics, avoid user content and secrets, avoid arbitrary existing log export, and use the shared OpenClaw trace/metric/log conventions deliberately.

Screenshots

OpenClaw WinUI

OTLP/gRPC

OTLP/HTTP

Validation

• .\build.ps1 — passed
• dotnet test .\tests\OpenClaw.Shared.Tests\OpenClaw.Shared.Tests.csproj --no-restore — 2737 passed, 31 skipped
• dotnet test .\tests\OpenClaw.Tray.Tests\OpenClaw.Tray.Tests.csproj --no-restore — 1685 passed
• dotnet test .\tests\OpenClaw.Tray.Tests\OpenClaw.Tray.Tests.csproj --no-restore --filter "FullyQualifiedName~OpenTelemetryEndpointConnectionTests|FullyQualifiedName~DiagnosticsPageContractTests" — 68 passed
• Failed-job GitHub Actions rerun on 05bb0d78 — all jobs passed, including the previously failing accessibility and WSL network-recovery shards
• Rubber-duck review completed; warning styling/message polish applied
• Autoreview command: python .agents\skills\autoreview\scripts\autoreview --mode local --stream-engine-output with UTF-8 environment — clean, no accepted/actionable findings

Real behavior proof

• OTLP/gRPC and OTLP/HTTP dashboard screenshots above show the sample trace, metric, and structured log arriving at the configured collector.
• Export remains gated behind explicit user configuration: leaving the endpoint empty disables OpenTelemetry export.
• Current-head resend semantics are covered by the focused service test and diagnostics-page wiring contract above: explicit resend creates a new probe sink while later automatic application of unchanged settings still deduplicates.
• Current-head visual proof blocker: the existing live tray instance held the single-instance app mutex, so the newly built worktree executable did not expose a separate window for a fresh screenshot. The earlier screenshots remain valid for the unchanged endpoint and collector flows, but do not show the new Send probe again button.

[clawsweeper]

Codex review: needs real behavior proof before merge. Reviewed July 11, 2026, 12:10 AM ET / 04:10 UTC.

Summary
Adds default-off OTLP endpoint settings, trace/metric/log probes, shared telemetry helpers, WinUI diagnostics controls, documentation, localization, and tests.

Reproducibility: not applicable. This PR adds a new opt-in diagnostics capability rather than fixing broken established behavior.

Review metrics: 3 noteworthy metrics.

• Change surface: 31 files, +2759/-3. The PR establishes a cross-cutting contract across persisted settings, shared APIs, WinUI, documentation, localization, and tests.
• Runtime dependencies: 2 packages added. The tray gains the official OpenTelemetry SDK and OTLP exporter as runtime dependencies.
• Probe matrix: 3 signals across 2 protocols. The existing collector evidence covers traces, metrics, and structured logs over OTLP/gRPC and OTLP/HTTP.

Merge readiness
Overall: 🦐 gold shrimp
Proof: 🦐 gold shrimp
Patch quality: 🦞 diamond lobster
Result: blocked until stronger real behavior proof is added.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

• Capture a redacted isolated current-head screenshot or recording showing Send probe again and collector receipt of the additional probe.

Proof guidance:

• [P1] Needs stronger real behavior proof before merge: Real screenshots prove the original WinUI configuration and collector flows, but not the current-head Send probe again control or an additional collector event; add redacted exact-head UI plus collector evidence, update the PR body to trigger review, or ask a maintainer to comment @clawsweeper re-review.

Risk before merge

• [P1] The exact-head Send probe again control and the resulting additional trace, metric, and log probe have not been demonstrated against a real collector.
• [P1] This introduces a permanent user-facing diagnostics setting and network-export boundary; a maintainer should explicitly accept that product/privacy surface after the proof gate is satisfied.

Maintainer options:

1. Decide the mitigation before merge
   Ship the default-off, tray-owned diagnostics boundary only after a redacted isolated current-head run visibly shows Send probe again creating an additional collector event without changing saved settings.
2. Pause or close
   Do not merge this PR until maintainers decide whether the risk is worth taking.

Next step before merge

• [P1] The contributor must first provide exact-head resend proof; afterward a maintainer must accept or narrow the persistent telemetry product boundary.

Maintainer decision needed

• Question: After exact-head resend proof is supplied, should OpenClaw adopt this user-configurable OTLP diagnostics endpoint and shared telemetry conventions as a supported product surface?
• Rationale: The patch is technically sound, but adding persistent telemetry configuration, runtime exporters, and future instrumentation conventions is a product and privacy boundary that contributor evidence alone cannot authorize.
• Likely owner: shanselman — He has the strongest history on the diagnostics/settings surface and authored the current-head retry refinement.
• Options:
  • Accept after proof (recommended): Adopt the default-off, user-controlled diagnostics surface once current-head resend behavior is visibly proven against a collector.
  • Narrow the groundwork: Keep only approved documentation or shared primitives and defer the runtime endpoint setting and exporters to a separately sponsored change.
  • Pause the direction: Do not establish an in-app telemetry export surface until maintainers define a different observability strategy.

Security
Cleared: The official dependencies, restrictive endpoint validation, explicit opt-in, and allowlisted probe fields present no concrete security or supply-chain blocker.

Review details

Best possible solution:

Ship the default-off, tray-owned diagnostics boundary only after a redacted isolated current-head run visibly shows Send probe again creating an additional collector event without changing saved settings.

Do we have a high-confidence way to reproduce the issue?

Not applicable: this PR adds a new opt-in diagnostics capability rather than fixing broken established behavior.

Is this the best way to solve the issue?

Yes, technically: the exporter is default-off and tray-owned, shared code avoids SDK coupling, endpoint and data boundaries are explicit, and automatic application remains deduplicated; exact-head proof and product acceptance are still required.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 622c0e27c6b0.

Label changes

Label justifications:

• P2: This is a normal-priority optional diagnostics feature with limited default impact but a broad implementation and policy surface.
• rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦐 gold shrimp and patch quality is 🦞 diamond lobster.
• status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs stronger real behavior proof before merge: Real screenshots prove the original WinUI configuration and collector flows, but not the current-head Send probe again control or an additional collector event; add redacted exact-head UI plus collector evidence, update the PR body to trigger review, or ask a maintainer to comment @clawsweeper re-review.
• proof: 📸 screenshot: Contributor real behavior proof includes screenshot evidence. Real screenshots prove the original WinUI configuration and collector flows, but not the current-head Send probe again control or an additional collector event; add redacted exact-head UI plus collector evidence, update the PR body to trigger review, or ask a maintainer to comment @clawsweeper re-review.

Evidence reviewed

What I checked:

• Repository proof policy: The repository requires current-head visible proof of the active changed WinUI state; the newly added resend control therefore needs exact-head UI and collector evidence before merge. (AGENTS.md:56, 622c0e27c6b0)
• Current-head resend behavior: The latest head adds a user-visible Send probe again button wired to a forced probe path, so this behavior is materially newer than the posted WinUI screenshots. (src/OpenClaw.Tray.WinUI/Pages/DebugPage.xaml:354, c517dcdafff5)
• Real behavior proof gap: The inspected screenshots demonstrate the original endpoint UI and real collector receipt, while the PR body explicitly states that they do not show the current-head Send probe again button or the resulting additional collector event. (c517dcdafff5)
• Opt-in and data boundary: Endpoint validation rejects credentials, query strings, and fragments; exporter setup is disabled for an empty endpoint and emits only the bounded probe fields through an allowlisted logging category. (src/OpenClaw.Tray.WinUI/Services/OpenTelemetryEndpointOptions.cs:30, c517dcdafff5)
• Prior review continuity: Earlier OTLP routing, flush-status, and retry findings were addressed; the latest completed review cycle reported no remaining code finding and only requested exact-head resend proof. (src/OpenClaw.Tray.WinUI/Services/OpenTelemetryEndpointConnection.cs:105, c517dcdafff5)

Likely related people:

• shanselman: Scott Hanselman introduced much of the current diagnostics/settings surface on main and authored the current-head retry behavior. (role: introduced behavior and recent area contributor; confidence: high; commits: 4166e0fd63f8, c517dcdafff5; files: src/OpenClaw.Tray.WinUI/Pages/DebugPage.xaml.cs, src/OpenClaw.Tray.WinUI/Services/SettingsManager.cs, src/OpenClaw.Tray.WinUI/App.xaml.cs)
• calebeden: Caleb Eden authored the telemetry implementation and also has prior merged work in the central App and tray UX paths, making him relevant beyond merely opening this PR. (role: feature author and recent adjacent contributor; confidence: high; commits: 17e8ed434948, 561a9e9ec19a, c35a0ed; files: src/OpenClaw.Tray.WinUI/App.xaml.cs, src/OpenClaw.Tray.WinUI/Pages/DebugPage.xaml.cs, src/OpenClaw.Tray.WinUI/Services/OpenTelemetryEndpointConnection.cs)
• karkarl: Karen recently changed the same diagnostics page, SettingsManager, and application startup/settings paths for side-by-side development installs. (role: recent settings and diagnostics contributor; confidence: medium; commits: dae64b6c49e9; files: src/OpenClaw.Tray.WinUI/Pages/DebugPage.xaml.cs, src/OpenClaw.Tray.WinUI/Services/SettingsManager.cs, src/OpenClaw.Tray.WinUI/App.xaml.cs)

What the crustacean ranks mean

• 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
• 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
• 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
• 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
• 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
• 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
• 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

Review history (9 earlier review cycles; latest 8 shown)

• reviewed 2026-07-10T21:30:32.625Z sha 0b95b92 :: needs real behavior proof before merge. :: [P2] Do not mark an unacknowledged probe as connected
• reviewed 2026-07-10T21:49:25.705Z sha 020d655 :: needs changes before merge. :: [P2] Do not treat SDK flush as collector connectivity
• reviewed 2026-07-10T21:57:18.441Z sha 020d655 :: needs changes before merge. :: [P2] Do not treat SDK flush as collector connectivity
• reviewed 2026-07-10T22:12:06.861Z sha 3bffb1b :: needs changes before merge. :: [P2] Do not report SDK flush as collector connectivity
• reviewed 2026-07-10T22:52:05.586Z sha 05bb0d7 :: needs changes before merge. :: [P2] Allow retrying an unacknowledged probe
• reviewed 2026-07-10T23:07:28.013Z sha 05bb0d7 :: needs changes before merge. :: [P2] Allow retrying an unacknowledged probe
• reviewed 2026-07-11T03:58:34.863Z sha c517dcd :: needs real behavior proof before merge. :: none
• reviewed 2026-07-11T04:05:54.628Z sha c517dcd :: needs real behavior proof before merge. :: none

[shanselman]

@clawsweeper re-review

[clawsweeper]

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

• State: Complete
• Detail: The targeted re-review published an exact durable review comment and queued an executing target-wide serialized router scan.
• Run: https://github.com/openclaw/clawsweeper/actions/runs/29139096433
• Updated: 2026-07-11T04:12:12.998Z