Skip to content

Concurrent sendFile() calls can kill the reliable data channel #1995

Description

@eeegoloauq

Describe the bug

What I'm expecting

Calling sendFile() for a few files without awaiting in between should just work (maybe slowly).

What happens instead

In my app users can drop several files into chat at once. When they do, the reliable data channel dies mid-transfer and usually only one file out of N arrives, the rest stall forever. Console shows publisher data channel 'RELIABLE' closed unexpectedly.

I dug into it and I'm pretty sure it's a check-then-send race in RTCEngine.sendDataPacket:

await this.waitForBufferStatusLow(kind);
this.reliableMessageBuffer.push({ data: msg, sequence: packet.sequence });
if (this.attemptingReconnect) return;
dc.send(msg);

waitForBufferStatusLow resolves for every waiter once bufferedAmount drops below the threshold, but nothing reserves capacity — so N concurrent stream writers all see "buffer low" in the same tick and all call dc.send() together, blowing past the SCTP send buffer. The browser then aborts the channel. Your close handler comment even describes this signature ("A publisher DC closing while the session is up ... see livekit/rust-sdks#1137").

The per-writer Mutex in streamBytes only serializes chunks within one stream, not across streams. Sending files strictly one at a time never triggers it.
In a minimal standalone repro (logs below) it shows up as the other failure mode: all sender promises resolve fulfilled while the receiver gets files 1-4 truncated, silent data loss instead of the channel dying.

Reproduction

// several files, even small ones (1-20 MB each), no await between calls:
for (const file of files) {
  room.localParticipant.sendFile(file, { topic: 'files' });
}

Two participants (sender and receiver with a registerByteStreamHandler).
Sender picks 3-5 files at once. With 5 x 15 MB: file 0 arrives complete,
files 1-4 truncated. Sending strictly one at a time never triggers it;
that's my workaround (FIFO, one open writer at a time).

Logs

receiver:
RECEIVED blob-0.bin 15728640
RECEIVE FAILED blob-1.bin DataStreamError: Not enough chunk(s) received - expected 15728640 bytes of data total, only received 12536488 bytes
RECEIVE FAILED blob-2.bin DataStreamError: Not enough chunk(s) received - expected 15728640 bytes of data total, only received 9920768 bytes
RECEIVE FAILED blob-3.bin DataStreamError: Not enough chunk(s) received - expected 15728640 bytes of data total, only received 7964336 bytes
RECEIVE FAILED blob-4.bin DataStreamError: Not enough chunk(s) received - expected 15728640 bytes of data total, only received 6419336 bytes

sender (same run):
send 0 fulfilled
send 1 fulfilled
send 2 fulfilled
send 3 fulfilled
send 4 fulfilled

System Info

System:
  OS: Fedora Linux 44 (Workstation)
Browsers:
  Chrome: 150.0.7871.46 (Official Build) (64-bit)
npmPackages:
  livekit-client: 2.20.0
Server: self-hosted livekit/livekit-server v1.13.2 (reproduced browser-to-browser through the SFU)

Severity

serious, but I can work around it

Additional Information

Looked at the Data streams v2 branch (#1985), it doesn't touch RTCEngine.ts, and its write mutex in the new OutgoingDataStreamManager is still created per-writer inside streamBytes(), so the race should survive the rewrite as-is.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions