FCE-3602: Create mobile custom source high level API#560
Open
MiloszFilimowski wants to merge 20 commits into
Open
FCE-3602: Create mobile custom source high level API#560MiloszFilimowski wants to merge 20 commits into
MiloszFilimowski wants to merge 20 commits into
Conversation
VisionCamera v5 frame outputs that publish camera frames to Fishjam — zero-copy forwarding and a WebGPU-rendered video source. - New package packages/react-native-vision-camera-source (useVisionCameraSource + /webgpu tier, forward/pooled track management, frame timestamp + orientation helpers). - react-client/mobile-client: useCustomSource gains a UseCustomSourceOptions argument so a source can publish its video as a regular camera track (videoType: 'camera' — a "virtual camera"). - webrtc-client: route createDataChannel through createNewConnection so a data-channel-first connection still wires ICE/track managers (tracks added later were silently never published). - Bump packages/react-native-webrtc to the custom-video-track native support. - Add the package to the release workflow + workspaces. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
MiloszFilimowski
force-pushed
the
feat/vision-camera-source
branch
from
July 9, 2026 08:33
d0356df to
8572830
Compare
A custom source is custom by definition — publishing its video as a `camera` track was a footgun. Drop UseCustomSourceOptions / CustomSourceTrackTypes and the videoType threading across react-client, mobile-client, and react-native-vision-camera-source. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
setStream was referentially unstable because its useCallback closed over the sources state, so it changed identity on every publish. Read sources through a ref instead; setStream is now stable. With that fixed, the useManagedCustomSource adapter (which existed only to work around the instability via a ref) is redundant — inline its publish effect into useVisionCameraSource and useVisionCameraWebGpuSource. Add useCustomSourceManager tests pinning setStream stability and the publish/unpublish lifecycle. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Prep for the TGSL shader port: JS-body TGSL needs the unplugin-typegpu build transform, which tsc can't run. Move the package build to bob with a babel.config.js that runs @babel/preset-typescript + unplugin-typegpu. - module target (configFile:true) so bob defers to babel.config.js; verified the built output preserves 'worklet' directives (for the consumer's react-native-worklets plugin) and resolves TGSL to WGSL. - typescript target emits .d.ts to dist/typescript. - exports/main/types repointed to dist/module + dist/typescript. - add typegpu, unplugin-typegpu, react-native-builder-bob, babel deps. No shader code changed yet; shaders still ship as raw WGSL strings. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Port the /webgpu passthrough shader from hand-written WGSL strings to TypeGPU: - FrameCropParams is now a d.struct — one source of truth for the WGSL struct decl and the 40-byte worklet packer (size derived via d.sizeOf). - vertexMain / fragmentMain are TGSL functions (tgpu.vertexFn/fragmentFn); the crop uniform is a tgpu.bindGroupLayout. - sampleCamera is a WGSL-bodied tgpu.fn the TGSL fragment calls. It stays WGSL because typegpu 0.11 can't resolve a texture_external inside a TGSL function; its texture+sampler bindings are declared as raw WGSL too. This is the only part not authored in TGSL. - Public API now exposes the sampleCamera TgpuFn + bindingDeclarations instead of a prebuilt shaderCode string (breaking). The per-frame worklet path stays raw react-native-webgpu — TGSL resolves to WGSL only at pipeline setup, so no tgpu proxy runs per frame. Verified: generated WGSL is functionally identical to the previous hand-written shader across ios/android x mirror (Android YUV decode byte-identical; only cosmetic naming + an intentional bind-group index swap differ), so GPU work is unchanged. CPU packing is untouched. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
tgpu['~unstable'].vertexFn/fragmentFn are @deprecated aliases (the feature graduated to stable); switch to the top-level tgpu.vertexFn/tgpu.fragmentFn. Same function reference, so the generated WGSL is unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Spell out that the WebGPU entry needs react-native-webgpu, the unplugin-typegpu Babel plugin (for the TGSL shaders), and iOS 17+ for the Metal camera-import path — and that the base camera-publishing tier requires none of these. The package sets no deployment target itself (pure JS); the app chooses its own. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This PR had bumped the mobile-client Expo plugin's default IPHONEOS_DEPLOYMENT_TARGET from 15.1 to 17.0, which forces iOS 17 on every @fishjam-cloud/react-native-client consumer — not just those using the WebGPU tier. Restore 15.1; apps that need a higher floor (e.g. for the /webgpu camera-import path) can still set it via options.ios.iphoneDeploymentTarget. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add warnIfIosVersionUnsupported(): a one-time console.warn when the /webgpu camera tier runs on iOS < 17, where Metal external-texture camera import may fail. Surfaces a clear, actionable reason before the cryptic device/feature error. Called automatically by useCameraWebGpuDevice (the choke point both shared and bring-your-own-device paths pass through) and exported for advanced callers that skip the hook. No-op off iOS. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… tier" This reverts commit a6c7e39.
Remove the custom-source manager test added for the setStream stability fix, per request. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The effect mixed a (async()=>{})().catch() IIFE, mutable pool/created vars
reassigned in three places, and cancellation re-checked after every await.
Extract the two-step allocation into allocatePooledTrack() (which owns the
"free the orphaned pool if the track fails" case), collapse the effect to a
single disposed flag + one allocation holder disposed exactly once, and pull
out named helpers (toWorkletBufferDescriptor, toError, disposePool,
disposeAllocation).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
useManagedForwardTrack was already clean, but tidy it for consistency with the pooled sibling: guard `disposed` (was `cancelled`) and only store the created track when not torn down, mirroring useManagedPooledTrack. Extract the repeated `cause instanceof Error ? cause : new Error(String(cause))` into a shared internal toError() (4 call sites across the two managed-track hooks and useCameraWebGpuDevice). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The vision-camera dep was used in only 3 files; the rest (the custom-video
track lifecycle hooks + the WebGPU camera toolkit) is source-agnostic.
Extract it into a new package @fishjam-cloud/react-native-custom-video-source
that publishes your own frames to Fishjam from any source, and make the
VisionCamera package a thin adapter on top of it.
- new package: internal/{useManagedForwardTrack,useManagedPooledTrack,toError}
+ webgpu/* (camera shader bindings, passthrough, crop, render context,
required features, device, texture resolver, runtime). Owns typegpu +
unplugin-typegpu (the TGSL shaders live here now).
- vision-camera-source keeps only the VC-coupled files (useVisionCameraSource,
useVisionCameraWebGpuSource, frameTimestamp, orientation), depends on the
generic package (workspace:*), and its /webgpu re-exports the generic
toolkit. Dropped its now-unused typegpu/unplugin-typegpu deps + babel plugin.
- register the new workspace in the root package.json.
Both packages typecheck, build (bob), and lint clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- release: publish + version-bump react-native-custom-video-source, gate vision-camera-source publish on it - vision-camera-source: react-native-worklets as peer dep (native module) - webgpu: guard shared-device slot against late device-lost callbacks - docs: fix stale package name in RNWebGPU warning, README outputView example, encodeCameraPassthrough single-crop constraint, timestamp fallback quirk, drop untrue "asserted in tests" claim - extract shared usePublishedStream + stopStreamTracks helpers - log frame-processing failures with the cause object Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
MiloszFilimowski
changed the title
MiloszFilimowski
requested review from
Magmusacy and
czerwiukk
and removed request for
czerwiukk
July 10, 2026 10:27
Export ManagedForwardTrack/ManagedPooledTrack — they are the return types of public hooks, so typedoc (treatWarningsAsErrors) failed on them being absent from the docs. Exclude external symbols in vision-camera-source docs: the options interfaces inherit VisionCamera members whose upstream doc comments carry @link tags unresolvable in our docs. Also turn the one cross-entry-point @link in webgpu.ts into a plain reference. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… and silent RNWebGPU failure - Lower the frame-timestamp magnitude threshold from 1e12 to 1e9: both platform clocks count from boot, so seconds can never reach 1e9 while nanoseconds pass it one second after boot. The old threshold misclassified Android frames captured within ~16.7 minutes of boot. - Catch setStream rejections in usePublishedStream (the same-tick removeTracks race rejects) and downgrade them to warnings. - Make getWebGpuRuntime throw a descriptive error when install() cannot bind the RNWebGPU global, and surface that through the WebGPU source hook's `error` result instead of crashing render or failing per frame. - Add vitest + a frameTimestamp regression suite (fails on the old threshold, passes on the fix). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
MiloszFilimowski
force-pushed
the
feat/vision-camera-source
branch
4 times, most recently
from
July 10, 2026 11:57
c208ac7 to
28f9e46
Compare
Replacing a published stream issues an unpublish and a publish in the same tick; run concurrently they raced to remove the same track IDs and the new stream was nondeterministically never published. setStream calls now run through a promise chain, with a synchronously-updated sources snapshot so queued calls see each other's effects. The connected effect also no longer leaves addTrack rejections unhandled. useVisionCameraSource now supplies timestampNs on every frame (interval-paced fallback via nextFrameTimestampNanoseconds, like the WebGPU tier), clamped to at least 1 ns: native reads an omitted timestamp — and the value 0, the timeline's first frame — as "stamp with my own clock", which mixed two timestamp domains on one track. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
MiloszFilimowski
force-pushed
the
feat/vision-camera-source
branch
from
July 10, 2026 12:33
28f9e46 to
7b1ea86
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
@fishjam-cloud/react-native-custom-video-source— generic custom video source package:useManagedPooledTrack/useManagedForwardTracklifecycle hooks/webgpuentry point with camera passthrough pipeline (TypeGPU shaders, crop utilities, device/feature setup)@fishjam-cloud/react-native-vision-camera-source— VisionCamera v5 → Fishjam adapter:useVisionCameraSource(frame-processor push path) with timestamp + orientation handling/webgputier:useVisionCameraWebGpuSourcefor GPU-processed camera frames (iOS 17+)react-client: stabilizeuseCustomSourcesetStream, dropuseManagedCustomSourcewebrtc-client: remove custom-source track type overridereact-native-webrtcsubmodule; wire both packages into release automationMotivation and Context
Productizes the custom video track primitive: lets apps publish VisionCamera (optionally WebGPU-processed) frames as a Fishjam video track, with the generic source layer split out for reuse beyond VisionCamera.
Documentation impact
Types of changes
not work as expected)