feat: add operator and node gateway lifecycle telemetry

[calebeden]

Additional instructions

MUST: Keep Allow edits from maintainers enabled for this PR so maintainers
can help update the branch when needed.

What Problem This Solves

When the Windows tray cannot establish or maintain its gateway connections, existing diagnostics do not clearly show whether the operator or Windows node is failing during local preparation, WebSocket transport, gateway handshake, authentication or pairing, reconnect, or readiness transitions.

Why This Change Was Made

Adds opt-in operator-to-gateway and node-to-gateway lifecycle traces, metrics, and allowlisted structured logs through the existing tray-owned OpenTelemetry exporter. Instrumentation remains observational, exports only finite operational states, and does not change connection lifecycle decisions or introduce OpenTelemetry SDK dependencies into shared libraries.

User Impact

Developers and operators with an OpenTelemetry endpoint configured can diagnose operator and Windows node connection timing, reconnect attempts, authentication or pairing failures, superseded local attempts, and ready/degraded state transitions. Telemetry remains disabled when no endpoint is configured.

Evidence

• Current-head collector captures show operator and Windows node connection traces, structured connection-state logs, and all three lifecycle metrics.
• Operator and Windows node root traces distinguish connection and automatic reconnection attempts, with coarse preparation, WebSocket transport, and gateway handshake phases.
• Metrics cover attempt counts, attempt duration, and state transitions, with operator and node roles available for filtering.
• Node tests cover paired success, pairing required/rejected, classified failures, automatic reconnect, supersession, stale events, and retirement/disposal races.
• Full required local build, Shared tests, and Tray tests passed; affected focused tests passed after the final follow-up changes.

Change Type

• Bug fix
• Feature
• Refactor
• Docs or instructions
• Tests or validation
• Security hardening
• Chore or infrastructure

Scope

• Tray or WinUI UX
• Windows node capability
• Local MCP or winnode
• Gateway, connection, or pairing
• Setup or onboarding
• Permissions, privacy, or security
• Tests, CI, or docs

Validation

• .\build.ps1 — passed; 5 projects built successfully.
• dotnet test .\tests\OpenClaw.Shared.Tests\OpenClaw.Shared.Tests.csproj --no-restore — 2,763 passed, 31 skipped, 0 failed.
• dotnet test .\tests\OpenClaw.Tray.Tests\OpenClaw.Tray.Tests.csproj --no-restore — 1,709 passed, 0 failed.
• Focused WindowsNodeClientTests.HandleResponse_TerminalError_EmitsFiniteFailureClassification — 5 passed, 0 failed after final code changes.
• Focused node lifecycle telemetry tests in OpenClaw.Connection.Tests — 9 passed, 0 failed after final code changes.
• Stabilized endpoint concurrency test — passed once with build and 20 consecutive no-build repetitions; passed again after incorporating review findings.
• Hanselman dual-model adversarial reviews completed with no blocking findings; accepted findings were applied and final review found no significant issues.
• Current-head GitHub test and gateway/setup E2E jobs passed.

Real Behavior Proof

• Environment tested: Windows ARM64 tray with a manually configured OTLP collector/dashboard.
• PR head or commit tested: current PR head.
• Exact steps or command run: launched the tray, observed operator and Windows node gateway connections and reconnect behavior, then inspected exported traces, metrics, and structured logs in the configured OpenTelemetry dashboard.
• Evidence after fix: screenshots show operator and node lifecycle traces, ordered structured connection-state logs, and openclaw.connection.attempts, openclaw.connection.attempt.duration, and openclaw.connection.state.transitions metrics with role filtering.
• Observed result: exported signals explain staged operator and node connectivity and distinguish local preparation from transport, handshake, pairing, reconnect, and state-transition outcomes.
• Screenshot or artifact links verified? (Yes/No/N/A): Yes
• Not verified or blocked: None known. Exporter protocol behavior is unchanged by this PR.

Screenshots

Operator-Gateway Connection

Trace of operator connecting to gateway Trace of operator failing to connect to gateway Trace of operator reconnecting to gateway Trace of node connecting to gateway Structured logs documenting connection status changes Metrics (includes operator and node which can be filtered on)

Security Impact

• New permissions or capabilities? (Yes/No): No
• Secrets or tokens handling changed? (Yes/No): No
• New or changed network calls? (Yes/No): No; signals use the existing user-configured OTLP exporter.
• Command or tool execution surface changed? (Yes/No): No
• Data access scope changed? (Yes/No): Yes; opt-in export now includes finite operator and node gateway lifecycle states and durations.
• If any answer is Yes, explain the risk and mitigation: exported fields are restricted to role, operation, outcome, coarse error category, and finite operator/node/overall states. Gateway URLs and IDs, device IDs, request IDs, credentials, raw errors, and diagnostic text are excluded.

Compatibility and Migration

• Backward compatible? (Yes/No): Yes
• Config or environment changes? (Yes/No): No
• Migration needed? (Yes/No): No
• If yes, list the exact upgrade steps: N/A

Review Conversations

• I replied to or resolved every bot review conversation addressed by this PR.
• I left unresolved only conversations that still need maintainer judgment.

[clawsweeper]

Codex review: needs real behavior proof before merge. Reviewed July 13, 2026, 9:44 PM ET / July 14, 2026, 01:44 UTC.

Summary
Adds opt-in gateway lifecycle traces, three metrics, allowlisted structured state logs, telemetry documentation, and regression coverage through the existing tray-owned exporter.

Reproducibility: not applicable. This PR adds an observability capability rather than fixing a defined failing behavior. The relevant verification path is a real gateway connection observed through an external OTLP collector.

Review metrics: 2 noteworthy metrics.

• Patch scope: 10 files, +965/-10. The change spans shared telemetry helpers, gateway lifecycle code, tray export plumbing, documentation, and three test projects.
• Exported signal surface: 5 span names, 3 metric names, 1 log category. These identifiers become an operator-facing observability contract that should be proven together against the current head.

Merge readiness
Overall: 🦐 gold shrimp
Proof: 🦐 gold shrimp
Patch quality: 🐚 platinum hermit
Result: blocked until stronger real behavior proof is added.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

• Attach redacted current-head collector output that visibly demonstrates lifecycle traces, all three metrics, and ordered structured state logs.
• Resolve the failing GitHub test job or provide evidence that a rerun passes on the unchanged head.

Proof guidance:

• [P1] Needs stronger real behavior proof before merge: The screenshots are useful live evidence but were captured before two telemetry-specific fixes and omit metric output; add redacted current-head collector evidence for traces, metrics, and logs, then update the PR body to trigger re-review or ask a maintainer to comment @clawsweeper re-review.

Risk before merge

• [P1] Collector evidence predates the fixes that changed state ordering and trace context, so current-head trace parentage, event ordering, and log sequencing have not been demonstrated in a real exporter.
• [P1] No current-head collector evidence shows openclaw.connection.attempts, openclaw.connection.attempt.duration, or openclaw.connection.state.transitions, leaving the newly documented metrics externally unverified.
• [P1] The current-head GitHub test job is failing, and the available evidence does not establish whether it is unrelated or exposes a remaining patch problem.

Maintainer options:

1. Prove the current head (recommended)
   Run the current head against a real collector, attach redacted trace, metric, and structured-log output, and resolve the failing test workflow before requesting merge.
2. Pause pending collector access
   Keep the draft open until someone can reproduce the telemetry from the exact head in an external collector rather than accepting stale runtime evidence.

Next step before merge

• [P1] The contributor must supply current-head external collector proof and resolve the failing test workflow; no concrete automated code repair is established from the available evidence.

Security
Cleared: The patch uses the existing opt-in exporter, adds no dependency or permission surface, and restricts export to finite low-cardinality operational fields with focused sensitive-data exclusion coverage.

Review details

Best possible solution:

Keep the bounded opt-in instrumentation, then capture redacted current-head collector evidence showing the lifecycle traces, all three metrics, and ordered structured state logs, and resolve or explain the failing test job before merge.

Do we have a high-confidence way to reproduce the issue?

Not applicable: this PR adds an observability capability rather than fixing a defined failing behavior. The relevant verification path is a real gateway connection observed through an external OTLP collector.

Is this the best way to solve the issue?

Yes, the proposed ownership and data boundaries appear appropriate: instrumentation stays in the connection manager, uses shared diagnostics-only primitives, and exports through the existing opt-in tray pipeline. The remaining blocker is current-head runtime proof, not an identified alternative implementation.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 77c42440376e.

Label changes

Label justifications:

• P2: This is a bounded opt-in diagnostics feature with useful tests but no evidence of an urgent user-facing regression.
• merge-risk: 🚨 other: Merging without current-head collector proof could ship misleading or incomplete lifecycle observability even when unit tests pass.
• rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦐 gold shrimp and patch quality is 🐚 platinum hermit.
• status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs stronger real behavior proof before merge: The screenshots are useful live evidence but were captured before two telemetry-specific fixes and omit metric output; add redacted current-head collector evidence for traces, metrics, and logs, then update the PR body to trigger re-review or ask a maintainer to comment @clawsweeper re-review.
• proof: 📸 screenshot: Contributor real behavior proof includes screenshot evidence. The screenshots are useful live evidence but were captured before two telemetry-specific fixes and omit metric output; add redacted current-head collector evidence for traces, metrics, and logs, then update the PR body to trigger re-review or ask a maintainer to comment @clawsweeper re-review.

Evidence reviewed

What I checked:

• Bounded telemetry implementation: The connection manager defines five lifecycle span names, three metric instruments, and low-cardinality role, operation, state, outcome, and coarse-error tags through the shared diagnostics-only telemetry helpers. (src/OpenClaw.Connection/GatewayConnectionManager.cs:12, b458e100a793)
• Sensitive state excluded: The endpoint connection projects gateway snapshots into finite overall, operator, and node states; its tests explicitly populate sensitive IDs, URLs, and errors while asserting that only the finite projection reaches the sink. (tests/OpenClaw.Tray.Tests/OpenTelemetryEndpointConnectionTests.cs:28, b458e100a793)
• Prior finding resolved: The current head adds detached activity helpers and tests proving that starting or stopping cross-callback spans preserves the newer ambient Activity context, addressing the previous review finding. (tests/OpenClaw.Shared.Tests/Telemetry/OpenClawTelemetryTests.cs:69, b458e100a793)
• Proof is not current-head complete: The PR states that live collector screenshots were captured on 2e2de8e, before the state-ordering fix c3dd64c and ambient-context fix b458e10; the attached proof shows traces and logs but does not show exported metrics from the current head. (2e2de8ef69f5)
• Automated status unresolved: The GitHub test check is failing on the current head, while the build and other primary jobs were skipped; the supplied context does not identify a narrow failure cause. (b458e100a793)
• Telemetry policy alignment: The repository policy requires opt-in export, low-cardinality operational data, focused signal tests, and current-head collector proof; the patch follows the first three requirements, but the last proof requirement remains incomplete. (AGENTS.md:54, 77c42440376e)

Likely related people:

• ranjeshj: Authored the recent connection-stability work in fix: connection stability — stop node reconnect storms, fix bootstrap token handling #287 across the tray and shared WebSocket lifecycle paths now being instrumented. (role: recent connection-area contributor; confidence: high; commits: 5f2de90; files: src/OpenClaw.Tray.WinUI/App.xaml.cs, src/OpenClaw.Shared/WebSocketClientBase.cs)
• shanselman: Merged the recent connection-stability change and has adjacent history around gateway retry and pairing behavior, so is well positioned to assess lifecycle semantics and operational signal usefulness. (role: merger and adjacent connection contributor; confidence: medium; commits: 2fcfe76; files: src/OpenClaw.Tray.WinUI/App.xaml.cs, src/OpenClaw.Shared/OpenClawGatewayClient.cs, src/OpenClaw.Shared/WebSocketClientBase.cs)

What the crustacean ranks mean

• 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
• 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
• 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
• 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
• 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
• 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
• 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

Review history (5 earlier review cycles)

• reviewed 2026-07-13T23:34:42.019Z sha c3dd64c :: needs real behavior proof before merge. :: none
• reviewed 2026-07-13T23:45:32.151Z sha c3dd64c :: needs real behavior proof before merge. :: [P2] Preserve ambient context when detached spans stop
• reviewed 2026-07-14T00:03:02.021Z sha c3dd64c :: needs real behavior proof before merge. :: [P2] Preserve ambient context when detached spans stop
• reviewed 2026-07-14T00:23:18.377Z sha c3dd64c :: needs real behavior proof before merge. :: [P2] Preserve ambient context when detached spans stop
• reviewed 2026-07-14T01:27:05.081Z sha b458e10 :: needs real behavior proof before merge. :: none

[calebeden]

@clawsweeper re-review

[clawsweeper]

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

[calebeden]

@clawsweeper re-review

[clawsweeper]

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

• State: Complete
• Detail: The item is closed, terminal review leases were released, and no stale verdict was published or routed.
• Run: https://github.com/openclaw/clawsweeper/actions/runs/29398869633
• Updated: 2026-07-15T07:54:06.492Z

[calebeden]

@clawsweeper re-review