Skip to content

fix(rivetkit): prevent connection snapshot stalls - #5579

Open
NathanFlurry wants to merge 1 commit into
mainfrom
fix/connection-snapshot-stall
Open

fix(rivetkit): prevent connection snapshot stalls#5579
NathanFlurry wants to merge 1 commit into
mainfrom
fix/connection-snapshot-stall

Conversation

@NathanFlurry

Copy link
Copy Markdown
Member
  • Snapshot connection handles before exposing the iterator so retained reads cannot block connection map writers.
  • Preserve exact iterator sizing and document live-handle snapshot semantics.
  • Add regression coverage for disconnecting while a connection snapshot is retained.

@railway-app

railway-app Bot commented Aug 20, 2026

Copy link
Copy Markdown

🚅 Deployed to the actors-pr-5579 environment in rivet-frontend

Service Status Web Updated (UTC)
kitchen-sink 😴 Sleeping (View Logs) Web Aug 20, 2026 at 6:10 am
frontend-cloud 😴 Sleeping (View Logs) Web Aug 20, 2026 at 6:06 am
frontend-inspector 😴 Sleeping (View Logs) Web Aug 20, 2026 at 6:03 am
website ❌ Build Failed (View Logs) Web Aug 20, 2026 at 6:01 am
mcp-hub ✅ Success (View Logs) Web Aug 20, 2026 at 5:57 am
ladle ✅ Success (View Logs) Web Aug 20, 2026 at 5:57 am

@claude

claude Bot commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Review

Small, well-scoped fix: ConnHandles moves from a lock-backed iterator (holds a RwLockReadGuard until dropped) to an eager point-in-time snapshot (Vec<ConnHandle> cloned under the lock, then released). This removes a real foot-gun: any caller that iterates ctx.conns() and .awaits inside the loop body (or simply retains the returned iterator across an await point) used to hold the connection-map read lock the whole time, which could stall a concurrent insert_existing/remove_existing writer. The regression test in tests/connection.rs correctly reproduces this: it retains the unconsumed snapshot across a background remove_existing call and asserts the writer isn't blocked, which would have hung/failed under the old implementation. This also lines up with the repo's 'hold lock guards for as short as possible' guidance.

Ordering is preserved (BTreeMap::values() still iterates in key order like the old range-based cursor did), and ConnHandle clones are just Arc refcount bumps, so the eager collect is cheap per-item.

Performance: hot-path regression in can_arm_sleep_timer

rivetkit-rust/packages/rivetkit-core/src/actor/sleep.rs:263:

if !self.conns().is_empty() {
    return CanSleep::ActiveConnections;
}

Previously ConnHandles::is_empty() read guard.is_empty() directly, O(1), no cloning. With this change, conns()/iter_connections() unconditionally does self.0.connections.read().values().cloned().collect() before is_empty() even runs, so this now clones every ConnHandle in the map (Arc refcount bump per connection) just to answer a boolean. can_arm_sleep_timer runs on the actor sleep-check path, so for actors with many live connections this is a real, easily avoidable per-check cost that didn't exist before this PR.

active_connection_count() (connection.rs:474, pre-existing and unmodified by this PR) already provides the O(1) equivalent (self.0.connections.read().len()). Suggest swapping this call site to self.active_connection_count() > 0 (or similar) to avoid the new allocation on this path. Worth a quick grep for other .conns().is_empty() / .conns().len()-only call sites that could use the same fix (the task.rs:1575 one is only on a rare shutdown-timeout warning log, so lower priority there).

Minor nit

ConnHandles<'a> keeps a PhantomData<&'a ()> and lifetime parameter purely for API-signature compatibility now that it no longer borrows anything (it owns an owned Vec iterator). Not a bug, just worth a one-line comment explaining why the lifetime is retained, so a future reader doesn't wonder why an owned iterator still carries a lifetime param.

Test coverage

The new test directly targets the fixed race and is a good regression guard. No NAPI/TS-level test needed since this is pure rivetkit-core internal iteration logic with no wire/protocol implications.

Overall: solid, minimal fix for a real deadlock-shaped bug, with one easy efficiency follow-up on the sleep hot path.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant