Port openframe-client updates from openframe-oss-tenant (round 2: Jul 6–22)

[mikhailm-coder]

What & why

Second sync round of the shared Rust agent from openframe-oss-tenant. The previous port (#1359) covered tenant history through 019691eba (Jul 2). This PR ports every client commit merged to tenant main since, through 922f664d7 (Jul 22) — 17 commits, 1:1, original authorship and messages preserved.

Ported commits (tenant → this branch)

tenant	here	title
60dd6e472	454b59f97	fix: nats consumer reconnect
e72734f39	478483dad	fix: revert heartbeat
f865c3ad3	25f2feb20	feat: add token_refresh_run_manager (#2016)
37e6b2100	279d5a232	docs(client): add trailing newline to PERMISSIONS.md (#2091)
b9e49451f	80438571b	fix(client): restore 0o644 on atomic_write (#2092)
75a242891	4b295f023	feat: add script lock result
ed49c0ee1	e5e07fb22	feat: scheduled scripts parallel run
d8c4b6c00	6b674dcb3	feat: add script id
2e755d901	1f090c43b	fix(mesh-self-heal): version-proof detection, real restart, silence detection (#2138)
d82668956	c17eda840	feat: listeners reconnect
7f63f6b3f	2b2e2fa6c	fix: nats message lost during rebind
4f6f97b42	882e18dce	client: disable tool-restart listener until backend support (#2163)
7026f76b1	47006c6ac	Hotfix/client update ordering and lkg ratchet v2 (#2169)
f4a2d1952	059c161e1	Remove tacticalrmm-agent-version feature flag (#2173)
908d42e12	dc55a832c	fix(client): ascii-only updater script, write with utf-8 bom
d1b51c944	7fcbd7210	Remove Redundant Folder (#2174) — client files only
922f664d7	de9455b90	chore: remove remaining Tactical RMM references (#2177) — client files only

Deliberately not ported: cf8d20296 (#2143) + c81348c83 (its revert) — verified to cancel exactly (empty combined diff); v2 (#2169) is included. The two "client files only" commits also touched tenant-wide files (docs, configs, pom.xml, .gitignore) that don't apply to this repo — only their clients/openframe-client/ changes are ported.

Plus one follow-up commit: clippy fix in the new LKG service (sort_by → sort_by_key(Reverse), clippy::unnecessary_sort_by under this repo's -D warnings gate; same fix applies upstream).

Deviations from verbatim tenant code

Same policy as #1359: cargo fmt folded per commit (pre-commit hook), extraction-era clippy idioms preserved (sorted module/import lists, derive(Default), no unused imports), and the one clippy fix above.

Verification (4 independent methods, same harness as #1359)

1. Fmt-normalized full-tree diff + delta invariance (fmt(tenant@922f664d7) vs this branch, compared against fmt(tenant@019691eba) vs current lib main): 234 files identical, 43 with constant lib-specific delta, 7 flagged — every flag explained (3 = extraction deltas on code #2169 deleted, 2 = the clippy fixes, 2 = sorted-order-only).
2. Per-commit file sets: 17/17 tenant/ported pairs touch identical in-scope file lists.
3. Symbol-level comparison: only the known extraction-era Default-idiom diffs.
4. Marker sweep: all new features present (TokenRefreshRunManager, LastKnownGoodService, client_update_gate, ToolRestartService, execution semaphore, ascii-script tests, …); removed code absent (tacticalrmm-agent-version feature, tactical-rmm infra scripts).

Gates

• cargo clippy --all-targets --features bin -- -D warnings ✅ (and hook variant ✅)
• cargo fmt --all -- --check ✅ every commit
• cargo test --features bin: 102 passed, 0 failed, 3 ignored (the test(client): ignore privileged/interactive tests that fail or hang in headless CI #1352 CI-safe ignores, intact after the port)

Note on #1360

The open follow-up #1360 (update/uninstall per-tool-lock + to_ascii_lowercase) targets files this PR also changes (tool_agent_update_listener, system_service); tenant still has neither fix. After this merges, #1360 needs a rebase onto the new listener shape — I'll handle it.

🤖 Generated with Claude Code

https://claude.ai/code/session_011z7MGH2qYeKt2ZQV88EaD8

Summary by CodeRabbit

• New Features
  • Added safer client update orchestration with boot-attempt tracking, crash-loop rollback protection, and last-known-good recovery.
  • Added proactive shared-token refresh and guarded tool restarts with improved self-heal behavior.
  • Execution requests/results now preserve optional script and schedule identifiers.
  • Improved NATS consumer handling: reconnection-aware listeners and bounded concurrent execution.
• Bug Fixes
  • Improved Windows service detection and installer resilience (service querying, leftover-process cleanup, safe retry/rollback).
  • Safer config/update-state persistence via atomic writes; improved temp-artifact cleanup.
  • Execution script launch now waits for temporary file readiness to avoid locked-file failures.
• Documentation
  • Updated support instructions to include permission-check output.
• Chores
  • Removed legacy MeshCentral and Tactical RMM installer scripts and related configuration/tool-version overrides.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: b3b6f6b2-0649-4859-bea4-7764f8d8d827

📥 Commits

Reviewing files that changed from the base of the PR and between 387f689 and d7770b9.

📒 Files selected for processing (1)

• clients/openframe-client/src/clients/registration_client.rs

---

📝 Walkthrough

Walkthrough

The client adds update verification and rollback handling, reconnect-aware NATS listeners, bounded execution concurrency, guarded tool restarts, proactive token refresh, atomic configuration writes, and MeshCentral self-healing changes. TacticalRMM wiring and installer scripts are removed.

Changes

Client lifecycle and update orchestration

Layer / File(s)	Summary
Contracts and shared foundations clients/openframe-client/Cargo.toml, src/models/*, src/utils/*, config/agent.toml, docs/PERMISSIONS.md, infrastructure/meshcentral/win.ps1	TacticalRMM overrides, integration settings, and the Windows MeshCentral installer are removed; execution metadata, restart messages, update phases, atomic writes, JWT expiry parsing, and support guidance are updated.
Durable update and rollback flow src/services/{last_known_good_service,openframe_client_update_service,update_handler_service,update_state_service}.rs, src/platform/update_scripts/*, src/platform/updater_launcher/*	Update state, last-known-good binaries, boot markers, transcript retention, canonical versions, downgrade checks, boot verification, and rollback paths are implemented for macOS and Windows.
Message consumption and execution src/services/nats_connection_manager.rs, src/listener/*, src/listener/execution_listener.rs	Listeners react to reconnects, park tool work during client updates with progress acknowledgements, and execute messages under a shared semaphore.
Tool restart and self-healing src/services/{tool_restart_service,tool_kill_service,mesh_self_heal_service}.rs, src/listener/tool_restart_message_listener.rs	Guarded restart outcomes, process matching, restart-message consumption, log-tail health seeding, cooldowns, silence detection, and restart fallback behavior are added.
Platform service and installation safety src/platform/system_service.rs, src/platform/installation_detector.rs, src/service.rs, src/service_adapter.rs, src/executor/windows/mod.rs	Windows service checks use shared status helpers, binary installation retries after terminating holders, and temporary PowerShell scripts wait for readability before execution.
Runtime services and startup wiring src/lib.rs, src/services/{agent_configuration_service,execution_service,shared_token_service,token_refresh_run_manager,tool_run_manager}.rs, src/services/mod.rs	Configuration writes become serialized and atomic, execution results preserve metadata, token refresh runs proactively, pending client updates coordinate tool messages, and new services are initialized and wired during startup.

Estimated code review effort: 5 (Critical) | ~120 minutes

Possibly related PRs

• flamingo-stack/openframe-oss-lib#1359: Extends the related tool uninstall listener with update parking and reconnect-aware JetStream handling.
• flamingo-stack/openframe-oss-lib#1480: Extends the related execution flow with server-side scriptId and scheduleId propagation.

Suggested reviewers: kirill-567

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 45.95% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately describes the PR as a second round of ported openframe-client updates from openframe-oss-tenant.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch hotfix/port-tenant-client-updates-2

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 4

🧹 Nitpick comments (2)

clients/openframe-client/src/services/openframe_client_info_service.rs (1)

68-81: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick win

Same read-modify-write hazard just fixed elsewhere in this PR is still present here.

reconcile_version (and the pre-existing update_version/set_update_status/set_binary_path) all do get() → mutate a field → save() with no synchronization — the identical lost-update pattern that this PR just fixed in AgentConfigurationService via a new write_lock: Arc<Mutex<()>> (see agent_configuration_service.rs). If two of these methods are ever called concurrently (e.g. update-flow status transitions racing with a version reconciliation), one write can silently clobber the other's field.

Consider applying the same Arc<Mutex<()>> guard pattern used in AgentConfigurationService here for consistency.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@clients/openframe-client/src/services/openframe_client_info_service.rs`
around lines 68 - 81, Protect the read-modify-write sequences in
OpenFrameClientInfoService methods reconcile_version, update_version,
set_update_status, and set_binary_path with the same Arc<Mutex<()>> write-lock
pattern used by AgentConfigurationService. Initialize the lock with the service
and acquire it before each get/mutation/save sequence, ensuring concurrent
updates cannot overwrite one another.

clients/openframe-client/src/services/token_refresh_run_manager.rs (1)

49-67: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win

Consider capped/backoff retry instead of a fixed 60s forever-retry.

On persistent reauthenticate() failure (e.g. revoked refresh token, prolonged auth-server outage), this inner loop retries every RETRY_INTERVAL indefinitely with no backoff or cap. That's a steady drumbeat of auth calls with no eventual escalation (e.g. falling back to re-registration).

♻️ Suggested backoff sketch

-                    sleep(RETRY_INTERVAL).await;
+                    sleep(next_backoff_delay(&mut consecutive_failures)).await;

Please confirm whether AgentAuthService::reauthenticate() already applies its own backoff/circuit-breaking internally — if so this is lower priority.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@clients/openframe-client/src/services/token_refresh_run_manager.rs` around
lines 49 - 67, Update the retry loop surrounding
AgentAuthService::reauthenticate() to avoid fixed-interval retries forever: add
bounded exponential backoff and a maximum retry duration or attempt count, then
escalate through the existing re-registration or failure path when the limit is
reached. Preserve the immediate success exit and distinguish timeout from
authentication errors; first verify whether reauthenticate() already provides
equivalent backoff or circuit breaking and avoid duplicating it.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@clients/openframe-client/src/services/openframe_client_update_service.rs`:
- Around line 104-111: Normalize the build-time OPENFRAME_VERSION value to
canonical semver before comparing it in the update flow. Apply the same
normalized value to the already-running check around canonical_version and
downstream target_version comparisons, preserving the short-circuit when
versions are semantically equal despite a v prefix or build metadata.

In `@clients/openframe-client/src/services/tool_kill_service.rs`:
- Around line 52-74: Update collect_matching_processes to ignore empty pattern
entries before applying cmdline or executable-path matching. Ensure empty
strings cannot satisfy contains and cause every process to be collected,
preserving normal matching for non-empty patterns and protecting callers such as
is_installed_tool_running and stop_tool_by_path.

In `@clients/openframe-client/src/utils/fs.rs`:
- Around line 16-21: The atomic write utility must not force replacement files
to world-readable 0644. Update atomic_write to accept an explicit file mode from
each caller, use 0600 for token-bearing agent configuration, and preserve
appropriate modes for non-sensitive callers; revise the affected test to assert
the secure caller-specific mode instead of 0644.

In `@clients/openframe-client/src/utils/jwt.rs`:
- Around line 12-18: Update the JWT segment validation around the token split to
reject empty header, payload, or signature components before decoding. Preserve
the existing three-segment-only requirement and return None for any empty
segment so malformed tokens use the existing fallback path.

---

Nitpick comments:
In `@clients/openframe-client/src/services/openframe_client_info_service.rs`:
- Around line 68-81: Protect the read-modify-write sequences in
OpenFrameClientInfoService methods reconcile_version, update_version,
set_update_status, and set_binary_path with the same Arc<Mutex<()>> write-lock
pattern used by AgentConfigurationService. Initialize the lock with the service
and acquire it before each get/mutation/save sequence, ensuring concurrent
updates cannot overwrite one another.

In `@clients/openframe-client/src/services/token_refresh_run_manager.rs`:
- Around line 49-67: Update the retry loop surrounding
AgentAuthService::reauthenticate() to avoid fixed-interval retries forever: add
bounded exponential backoff and a maximum retry duration or attempt count, then
escalate through the existing re-registration or failure path when the limit is
reached. Preserve the immediate success exit and distinguish timeout from
authentication errors; first verify whether reauthenticate() already provides
equivalent backoff or circuit breaking and avoid duplicating it.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 061ba8ac-ce81-4348-af1d-77ab2c09893d

📥 Commits

Reviewing files that changed from the base of the PR and between 8789907 and 387f689.

📒 Files selected for processing (53)

• clients/openframe-client/Cargo.toml
• clients/openframe-client/build.rs
• clients/openframe-client/config/agent.toml
• clients/openframe-client/docs/PERMISSIONS.md
• clients/openframe-client/infrastructure/meshcentral/mac.sh
• clients/openframe-client/infrastructure/meshcentral/win.ps1
• clients/openframe-client/infrastructure/tactical-rmm/mac_arm64.sh
• clients/openframe-client/infrastructure/tactical-rmm/win_amd64.ps1
• clients/openframe-client/src/config/update_config.rs
• clients/openframe-client/src/executor/windows/mod.rs
• clients/openframe-client/src/lib.rs
• clients/openframe-client/src/listener/client_update_gate.rs
• clients/openframe-client/src/listener/execution_listener.rs
• clients/openframe-client/src/listener/mod.rs
• clients/openframe-client/src/listener/openframe_client_update_listener.rs
• clients/openframe-client/src/listener/tool_agent_update_listener.rs
• clients/openframe-client/src/listener/tool_installation_message_listener.rs
• clients/openframe-client/src/listener/tool_restart_message_listener.rs
• clients/openframe-client/src/listener/tool_uninstall_message_listener.rs
• clients/openframe-client/src/models/execution.rs
• clients/openframe-client/src/models/mod.rs
• clients/openframe-client/src/models/tool_restart_message.rs
• clients/openframe-client/src/models/tool_version_overrides.rs
• clients/openframe-client/src/models/update_state.rs
• clients/openframe-client/src/platform/installation_detector.rs
• clients/openframe-client/src/platform/system_service.rs
• clients/openframe-client/src/platform/update_scripts/macos.rs
• clients/openframe-client/src/platform/update_scripts/mod.rs
• clients/openframe-client/src/platform/update_scripts/windows.rs
• clients/openframe-client/src/platform/updater_launcher/macos.rs
• clients/openframe-client/src/platform/updater_launcher/mod.rs
• clients/openframe-client/src/platform/updater_launcher/windows.rs
• clients/openframe-client/src/service.rs
• clients/openframe-client/src/service_adapter.rs
• clients/openframe-client/src/services/agent_configuration_service.rs
• clients/openframe-client/src/services/execution_service.rs
• clients/openframe-client/src/services/last_known_good_service.rs
• clients/openframe-client/src/services/mesh_self_heal_service.rs
• clients/openframe-client/src/services/mod.rs
• clients/openframe-client/src/services/nats_connection_manager.rs
• clients/openframe-client/src/services/openframe_client_info_service.rs
• clients/openframe-client/src/services/openframe_client_update_service.rs
• clients/openframe-client/src/services/shared_token_service.rs
• clients/openframe-client/src/services/token_refresh_run_manager.rs
• clients/openframe-client/src/services/tool_kill_service.rs
• clients/openframe-client/src/services/tool_restart_service.rs
• clients/openframe-client/src/services/tool_run_manager.rs
• clients/openframe-client/src/services/update_cleanup_service.rs
• clients/openframe-client/src/services/update_handler_service.rs
• clients/openframe-client/src/services/update_state_service.rs
• clients/openframe-client/src/utils.rs
• clients/openframe-client/src/utils/fs.rs
• clients/openframe-client/src/utils/jwt.rs

💤 Files with no reviewable changes (8)

• clients/openframe-client/infrastructure/meshcentral/mac.sh
• clients/openframe-client/infrastructure/tactical-rmm/mac_arm64.sh
• clients/openframe-client/Cargo.toml
• clients/openframe-client/src/models/tool_version_overrides.rs
• clients/openframe-client/config/agent.toml
• clients/openframe-client/infrastructure/meshcentral/win.ps1
• clients/openframe-client/infrastructure/tactical-rmm/win_amd64.ps1
• clients/openframe-client/build.rs