fix(concurrency): split send/recv locks so heartbeat & recv loop never block each other

[olivier-babelcast]

Summary

Split the single per-plant self.lock into two locks — send_lock and
recv_lock — so the heartbeat send and the recv loop can never block each
other. This removes a class of self-inflicted keepalive starvation under load
without weakening any of the safety the single lock provided.

Background

Each plant runs three background coroutines on one event loop: _recv_loop,
_process_loop, _heartbeat_loop. Today they all contend for one
self.lock:

• _recv_loop holds it across await ws.recv() (bounded by listen_interval).
• _send_and_recv / _send_and_recv_immediate hold it across a full
  send→recv round-trip.
• _send / _send_heartbeat take it for each ws.send.

So a heartbeat (template_id=18) can queue behind an inline recv that is
itself waiting on the wire. When several plants share an event loop and traffic
is bursty, the heartbeat is delayed past the negotiated interval and Rithmic
drops the connection — triggering a reconnect storm that makes the contention
worse. The root issue is that two genuinely independent concerns — writing to
the socket and reading from it — are serialized against each other.

What changed

Two locks with clear, non-overlapping responsibilities:

• send_lock — serializes all websocket writes (ws.send), including the
  heartbeat. Concurrent ws.send() on one connection is unsafe, so every write
  still goes through one lock. This is the default lock for try_acquire_lock.
• recv_lock — serializes websocket reads (_recv_loop + the inline-recv
  in _send_and_recv / _send_and_recv_immediate) so reads never collide and
  never block a write.

try_acquire_lock(plant, ..., lock=None) now takes an optional lock and
defaults to plant.send_lock. The recv paths pass lock=self.recv_lock.

_send_and_recv_immediate holds recv_lock for the whole send+recv op (it must
serialize with _recv_loop — no two coroutines may ws.recv() at once) but
takes send_lock only for the actual ws.send. So:

• the heartbeat (send_lock) is never blocked by an inline recv (recv_lock);
• concurrent writes are still fully serialized (send_lock);
• concurrent reads are still fully serialized (recv_lock).

Deadlock safety

The only nesting is recv→send (inside _send_and_recv_immediate); there is
no send→recv path anywhere, so there is no lock-ordering cycle. The existing
reconnect-deadlock regression (test_no_deadlock_on_reconnect) is updated to
acquire recv_lock in its simulated reconnect (the lock the recv listener
holds) and still passes.

Tests

pytest -q green on top of v1.6.1. test_no_deadlock_on_reconnect updated
for the rename; all other tests unchanged.

Relationship to #59 / #53

This is independent of and complementary to the historical-data rewrite (#59).
With #59, history fetches no longer hold a lock across a recv, so the most
acute starvation source is already gone; this split is the structural
guarantee that send and recv can't serialize against each other on any path,
which keeps the heartbeat reliable under arbitrary load. It is also independent
of the order-terminal-ack fix in #53.

🤖 Generated with Claude Code

[olivier-bn]

we were tracking the same issue you did fix few weeks back on the history blocking and did this potential fix - our heart beat wwas getting stale. it is not strictly mandatory BUT it seems a correct fix to separate both send and recv locks. I let you assess if this is a good idea or not. updating my fork to latest solved my history hang issue.

[rundef]

Hi,

Thanks for reporting this and for looking into it.

I decided to address this at the architectural level in #63 instead of using the proposed approach here. The change removes the need for the inline send + recv operation in the normal flow, keeping it only for a specific pre-login / pre-heartbeat request in the ticker plant (get_system_info).

This has been released in version 1.6.2.

Thanks again!