Native scrcpy Session

Native scrcpy Session

中文

Current contract

NativeScrcpySession.swift owns the app-managed primary screen path and uses the official, unmodified scrcpy 4.1 server. scrcpy-mask is a behavior/reference project only; its server extensions are not part of this protocol. The project bundles a verified server asset through scripts/prepare-scrcpy-server.sh, which downloads only the official v4.1 release if missing and rejects checksum mismatches.

Lifecycle and socket order

1. Resolve and SHA-256 verify the bundled server JAR.
2. Push it to /data/local/tmp/scrcpy-server.jar, create an ADB forward, and launch com.genymobile.scrcpy.Server with version 4.1.
3. Connect sockets serially: video -> optional audio -> control. Socket roles depend on accept order, so these connections must never be parallelized.
4. Read device metadata from video, retain the ADB forward for the lifetime of the media session, and consume media. Close sockets/process/decoders and remove the forward on stop or failure.

Codec discovery and selector pages

Remote encoders and local decoders are two different capability sets and must not be presented as one list:

• Video/audio encoder pages check the cache for every directly configured USB, TCP/IP, and mDNS serial before entering the detecting state or running any ADB command. A hit displays the list immediately. On a miss, the page resolves the active ADB target, checks that resolved serial once more, and only then queries the connected Android device through scrcpy list_encoders=true. The refresh button intentionally bypasses both cache checks. A successful query replaces that media kind and associates the result with the resolved target plus every configured connection address for the same session, so switching connection methods does not force a redundant probe. The page keeps Default and a custom encoder name as explicit options, then shows the detected device encoders. Search matches the encoder name, codec display name, and MIME type; opening the page does not focus the search field.
• Detected video encoder formats are grouped as H264, H265, AV1, VP8, and VP9 when the device reports them. Audio formats are grouped as OPUS, AAC, FLAC, and RAW when reported. A format filter is generated from the actual result instead of hiding a device-reported encoder behind a fixed allowlist.
• Video/audio decoder pages describe this Mac, not the Android device. Their capability lists are persisted with a macOS runtime signature and reused while that signature remains valid; refresh bypasses and replaces the cache. They provide search, a codec menu with All as its initial value, and hardware/software filters. The video page also filters low-latency, VideoToolbox, and FFmpeg routes. Menu controls size to their current labels, and opening a decoder page does not activate its search field.
• Video decoder rows cover H264, H265, AV1, VP8, and VP9 so that local capability and planned routes remain visible. H264 and H265 are selectable native VideoToolbox routes. AV1, VP8, and VP9 rows remain readable but cannot be selected until their corresponding native or FFmpeg session route is implemented.
• Audio decoder rows are enumerated from Core Audio at runtime for OPUS, AAC, and FLAC, with RAW as the uncompressed fallback. They are not a single hard-coded decoder entry.

All four codec selectors share the video-decoder page structure: the same title bar, icon refresh action, Done action, 12-point section rhythm, and viewport-constrained content with an 18-point inset on both visible edges. Encoder-only custom input remains a functional difference; its width adapts between 140 and 300 points instead of forcing the selector wider than its sheet. Search and filter controls may compress within compact sheets but must not erase the outer content inset.

The UI intentionally has no decorative decoder Test action. Such an action requires a real encoded sample, decoder creation, output validation, and error reporting; refreshing capability enumeration is not a decode test.

The persistent cache is stored in ~/Library/Application Support/Screen Remote/codec-capabilities.json. A normal launch and Xcode Run use this same file because both inherit the user's home directory; rebuilding the app does not clear it. SCREEN_REMOTE_CODEC_CACHE_DIRECTORY is an explicit test override, not a separate Xcode cache. Remote entries are isolated by resolved device serial and its configured connection aliases. Native session startup may reuse a complete cached remote list, but codec pairing still validates the selected remote encoder against the locally supported decoder route; cached capability never overrides negotiated stream metadata.

Codec pairing and fallback rules

The native session selects a codec, a remote encoder, and a compatible local decoder as one combination:

• Native video supports H265 first, then H264. A manually selected encoder or decoder constrains the remote stream format. An incompatible or unavailable pair is ignored and automatically reselected. If device probing fails, H264 with the server-default encoder is the safe baseline.
• Native audio prefers OPUS -> AAC -> FLAC -> RAW and only selects a compressed codec when both a remote encoder and a local Core Audio decoder exist. An incompatible manual pair is ignored and automatically reselected. RAW does not pass an audio encoder or compressed-audio bitrate to the server.
• The executable scrcpy launch path may forward device-supported H264, H265, AV1, VP8, or VP9 selections. This does not imply that the app-owned native decoder implements all five formats.

Current media and control state

Video stream metadata, session headers, configuration packets, and media packets are parsed. H264 and H265 Annex B streams are configured and decoded through VideoToolbox, then published to ScrcpyVideoSurface for SwiftUI.

When audio is enabled, OPUS, AAC, and FLAC configuration packets are converted to the Core Audio cookies required by AudioConverter; decoded stereo PCM is queued for playback. RAW packets carry uncompressed PCM, bypass compressed decoding, and are played directly. The selected audio volume is applied by the renderer.

The control socket is retained for the session, and pointer down/move/up events from the video surface are serialized as official scrcpy touch messages. Other control-message families, such as keyboard, scroll, clipboard, and device actions, are not yet a complete native routing layer and must not be documented as finished.

App-specific virtual-display launch remains temporarily executable-backed; it is not yet the same native primary-session pipeline.

Target split

Continue reducing NativeScrcpySession to orchestration only. Keep protocol messages/headers in Services/Scrcpy/Protocol, deployment/forward/server process in Session, VideoToolbox decode in Video, move the remaining embedded audio renderer into Audio, and move touch plus future input/control routing and coordinate mapping into Control. A state machine should own cancellation, reconnect, timeout, and cleanup transitions.

Regression checks

Verify server checksum and asset copy, failed push/forward/start paths, mandatory ordering with audio both enabled and disabled, metadata failure, video close, cancellation cleanup, and English diagnostic details. Also verify encoder-list parsing, alias handling, hardware preference, incompatible pair fallback, H264 fallback after probe failure, audio priority, and RAW fallback with:

swiftc -parse-as-library \
  Screen-Remote/Services/Scrcpy/Session/ScrcpySessionConfiguration.swift \
  scripts/check-codec-selection.swift \
  -o /tmp/check-codec-selection && /tmp/check-codec-selection

Run both the normal Xcode build and static analysis after changes to the media pipeline. Audio playback and touch routing must retain the ordered-connection contract and official protocol compatibility.