feat(realtime): glass-to-glass latency via LiveKit frame metadata#185
Merged
Conversation
…data Replace the visible pixel-marker glass-to-glass measurement with LiveKit frame metadata (per-frame capture timestamps propagated through inference), surfaced as g2gMs / ttffMs on the stats and connectionQuality signals under the opt-in debugQuality flag. Browser-only; React Native rejects debugQuality.
commit: |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
TimeSyncUpdate emits `timestamp` as a DOMHighResTimeStamp relative to performance.timeOrigin, but it was compared against the publisher's epoch userTimestamp — making every latency negative, so all samples were dropped and g2gMs/ttffMs stayed null whenever debugQuality was on. Add timeOrigin to reach epoch ms. Fix the reader test, which masked this by feeding an already- epoch timestamp; it now drives a relative timestamp like the real SDK.
The server appends a packet trailer to every frame once frame_timing is advertised; a subscriber without the frame-metadata worker can't strip it and the VP8/VP9 decoder then fails on every frame. frameTiming was derived from debugQuality alone and sent in the join before the worker was created, so a worker that failed to construct left the server stamping trailers into a workerless room (broken video, not just missing metrics). Create the worker up front and gate frameTiming — and the diagnostics reader — on its success.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit bb8152a. Configure here.
onConnectionQuality / the connectionQuality event are debounced to verdict changes, so reading g2g/rtt through them goes stale between changes (the event looked like the latency API but wasn't). Add a connectionMetrics event + onConnectionMetrics callback that push the live ConnectionQualityMetrics every stats tick (~1/s); keep connectionQuality for the coarse verdict and document getConnectionQuality()?.metrics as the pull path.
…ctionMetrics Revert the dedicated connectionMetrics event/callback (92b21cb): the continuous g2g data is already on the `stats` event (`stats.glassToGlass`, every tick), so a parallel channel added surface without new capability. Instead document the `stats` event + `getConnectionQuality()` polling as the live path, and clarify that `onConnectionQuality` / the `connectionQuality` event are debounced to verdict changes (which is what made g2g look stale).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Network RTT hides the dominant cost in real-time video — model inference — so a session can report "good" while actually feeling laggy. This lets apps measure the real camera→display (glass-to-glass) latency their users actually experience, so they can surface honest connection quality and diagnose lag. It stays opt-in and diagnostic (browser-only), and replaces an earlier approach that painted a visible marker onto the video — the new measurement is invisible to end users. Startup (
ttffMs) and steady-state (g2gMs) latency are surfaced on the existingconnectionQualityandstatssignals and drive the latency verdict instead of RTT when available.Usage
Note
Medium Risk
Touches LiveKit publish/subscribe setup and server capability negotiation on an experimental encoded-transform path; mis-gating frame timing could break video decode, though the change is opt-in via
debugQualityand includes worker-availability guards.Overview
Replaces the opt-in pixel-marker glass-to-glass path (visible bottom-left stamp, outgoing frame transform,
pixel_latencyquery param) with LiveKit frame metadata: capture timestamps on publish, server propagation, and matching on remote playout viatimeSyncUpdate/lookupFrameMetadata.debugQuality: truestill surfacesttffMs/g2gMsonstatsandconnectionQuality, but measurement is invisible and no longer wraps the outgoing stream.When a frame-metadata worker can be created, the SDK advertises
client_capabilities.frame_timingon join (instead ofpixel_latency), wires the LiveKitframeMetadataworker andtimestamp: truepublish options, and only then enables diagnostics. If the worker fails (e.g. CSP), frame timing stays off so subscribers are not left with undecodable VP8/VP9 trailers.g2gDropRatiois documented and implemented asnulluntil frame IDs propagate end-to-end; seq-based drop inference is removed with the marker code. Docs and the demo page call out usingstats(or polling metrics) for live g2g numbers becauseconnectionQualityis debounced.Reviewed by Cursor Bugbot for commit e246dd4. Bugbot is set up for automated code reviews on this repo. Configure here.