Skip to content

Address data channel thundering herd due to concurrent sendFile(...) calls#2010

Closed
1egoman wants to merge 4 commits into
mainfrom
data-channel-thundering-herd
Closed

Address data channel thundering herd due to concurrent sendFile(...) calls#2010
1egoman wants to merge 4 commits into
mainfrom
data-channel-thundering-herd

Conversation

@1egoman

@1egoman 1egoman commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Fixes #1995

Problem

Sending n files concurrently over a data stream can result in all of the sendFile calls attempting to send their first packet via the reliable data channel at the same time. When this happens, multiple writers can think the buffer status is low at the same time and all call dc.send(...), exhausting the data channel buffer and causing the buffer to overflow (which causes data to get dropped).

Pre-existing workaround

The reporter of the issue has had success with converting concurrent sendFile calls into sequential ones - so await Promise.all([sendFile(...), sendFile(...), sendFile(...)]) vs await sendFile(...); await sendFile(...); await sendFile(...);. The latter works because there's only one data stream actively writing to the data channel at a time, so there's no contention.

Solution

To address this, I've opted to sequentialize waitForBufferStatusLow(...) resolutions when the the data channel is not "low". So what will now happen:

  1. await waitForBufferStatusLow(...) works as it did before as long as it is in "low status", and by default has no sequentializing semantics.
  2. Once await waitForBufferStatusLow(...) gets called and the data channel is no longer in low status, then the resolve is cached in waitForBufferedStatusLowResolves. Once the data channel is no longer in "low status", instead of resolving all promises ASAP (ie, a "thundering herd"), instead resolve them in sequence and verify that the data channel buffer is still in "low status" between resolutions. This way events naturally resolve in the order they were called until the data channel buffer is full again, where they pause until the buffer is no longer full again and so on until the buffer exhausts.

Demo

Code added to demo app to test this

diff --git a/examples/demo/demo.ts b/examples/demo/demo.ts
index 3471049a..002d744e 100644
--- a/examples/demo/demo.ts
+++ b/examples/demo/demo.ts
@@ -168,6 +168,54 @@ const appActions = {
       onProgress: (progress) => console.log('sending file, progress', Math.ceil(progress * 100)),
     });
   },
+  // Reproduces https://github.com/livekit/client-sdk-js/issues/1995 — several large
+  // sendFile() calls fired in parallel without awaiting between them race on the
+  // reliable data channel's buffer-low signal, so the receiver sees truncated files
+  // or the channel closes with "publisher data channel 'RELIABLE' closed unexpectedly".
+  reproduceParallelSendFile: async () => {
+    const lp = currentRoom?.localParticipant;
+    if (!lp) {
+      appendLog('reproduce: not connected to a room');
+      return;
+    }
+    const fileCount = 5;
+    const fileSizeBytes = 15.7 * 1024 * 1024; // ~15.7 MB, matching the issue report
+    appendLog(
+      `reproduce #1995: firing ${fileCount} parallel sendFile() calls of ~${(
+        fileSizeBytes /
+        (1024 * 1024)
+      ).toFixed(1)} MB each (no await between calls)`,
+    );
+
+    const files = Array.from({ length: fileCount }, (_, i) => {
+      // Fill with a per-file byte pattern so the receiver can spot truncation/corruption.
+      const bytes = new Uint8Array(fileSizeBytes);
+      bytes.fill((i + 1) & 0xff);
+      return new File([bytes], `repro-${i + 1}.bin`, { type: 'application/octet-stream' });
+    });
+
+    // Intentionally do NOT await sequentially — this is the buggy usage pattern.
+    const sends = files.map((file, i) =>
+      lp
+        .sendFile(file, {
+          mimeType: file.type,
+          topic: 'files',
+          onProgress: (progress) =>
+            console.log(`reproduce: file ${i + 1} send progress`, Math.ceil(progress * 100)),
+        })
+        .then((info) => appendLog(`reproduce: file ${i + 1} send resolved (streamId ${info.id})`))
+        .catch((err) => appendLog(`reproduce: file ${i + 1} send FAILED: ${err}`)),
+    );
+
+    try {
+      await Promise.all(sends);
+      appendLog(
+        'reproduce #1995: all send promises resolved — check the receiver, files are likely truncated',
+      );
+    } catch (err) {
+      appendLog(`reproduce #1995: a send rejected: ${err}`);
+    }
+  },
   connectWithFormInput: async () => {
     const url = (<HTMLInputElement>$('url')).value;
     const token = (<HTMLInputElement>$('token')).value;
@@ -479,7 +527,17 @@ const appActions = {
         throw err;
       }
       const result = new Blob(byteContents, { type: info.mimeType });
-      appendLog(`Completely received file "${info.name}" from ${participant?.identity}`);
+      const receivedBytes = result.size;
+      const expectedBytes = info.size;
+      const sizeSuffix =
+        expectedBytes != null
+          ? ` (${receivedBytes} / ${expectedBytes} bytes${
+              receivedBytes !== expectedBytes ? ' — TRUNCATED!' : ''
+            })`
+          : ` (${receivedBytes} bytes)`;
+      appendLog(
+        `Completely received file "${info.name}" from ${participant?.identity}${sizeSuffix}`,
+      );
 
       streamReaderAbortController = undefined;
       (<HTMLButtonElement>$('cancel-chat-receive-button')).style.display = 'none';
diff --git a/examples/demo/index.html b/examples/demo/index.html
index 9ec439b5..d8d8a939 100644
--- a/examples/demo/index.html
+++ b/examples/demo/index.html
@@ -190,6 +190,12 @@
               </button>
               <input id="file" type="file" />
               <button onclick="appActions.sendFile()">Send file</button>
+              <button
+                onclick="appActions.reproduceParallelSendFile()"
+                title="Fires several large sendFile() calls in parallel to reproduce https://github.com/livekit/client-sdk-js/issues/1995"
+              >
+                Reproduce #1995 (parallel sendFile)
+              </button>
               <select
                 id="simulate-scenario"
                 class="custom-select"

Before, here was the behavior:

Screen.Recording.2026-07-13.at.5.08.04.PM.mov

Now, with this pull request:

Screen.Recording.2026-07-13.at.4.50.00.PM.mov

Note the decreased upload speed, because the bandwidth is being split between all file uploads.

@changeset-bot

changeset-bot Bot commented Jul 13, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 042e6d4

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
Name Type
livekit-client Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@github-actions

github-actions Bot commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

size-limit report 📦

Path Size
dist/livekit-client.esm.mjs 101.34 KB (+0.09% 🔺)
dist/livekit-client.umd.js 110.25 KB (-0.07% 🔽)

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 2 potential issues.

View 2 additional findings in Devin Review.

Open in Devin Review

Comment thread src/room/RTCEngine.ts Outdated
Comment thread src/room/RTCEngine.ts Outdated
@1egoman 1egoman force-pushed the data-channel-thundering-herd branch from e48e2b0 to d4a824c Compare July 13, 2026 21:16
@1egoman 1egoman changed the title Address data channel thundering herd Address data channel thundering herd due to concurrent sendFile(...) calls Jul 13, 2026
Comment thread src/room/RTCEngine.ts
if (this.isClosed) {
throw new UnexpectedConnectionState('engine closed');
}
if (this.isBufferStatusLow(kind)) {

@lukasIO lukasIO Jul 14, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue: this path here means ordering isn't guaranteed anymore.
it needs to live after acquiring the lock

Comment thread src/room/RTCEngine.ts
// updated bufferedAmount rather than the stale (still-low) value. The caller's continuation is
// scheduled when this promise settles; deferring the unlock to a later task lets that send run
// first.
setTimeout(unlock, 0);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how many packets will this affect worse case?

I'm not a huge fan of the setTimeout here as timeouts get heavily throttled under certain circumstances, e.g. in backgrounded tabs

@1egoman

1egoman commented Jul 15, 2026

Copy link
Copy Markdown
Contributor Author

Lukas and I have been going back and forth on this in the background - it's fairly tricky to get right and come up with a solution that generalizes both to data streams and data tracks. Closing this in favor of #2013, as this approach seems to generalize better.

@1egoman 1egoman closed this Jul 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Concurrent sendFile() calls can kill the reliable data channel

2 participants