Skip to content

Add a WebRTC realtime voice provider on Android (full-duplex speech models) #69

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
s546126 wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Add a WebRTC realtime voice provider on Android (full-duplex speech models) #69

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ Gemini Live API (WebSocket)

**Key pieces:**
- **Gemini Live** -- real-time voice + vision AI over WebSocket (native audio, not STT-first)
- **OpenAI Realtime (Android)** -- optional alternate voice backend over WebRTC (Opus, built-in AEC, full-duplex barge-in); switch providers in Settings
- **OpenClaw** (optional) -- local gateway that gives Gemini access to 56+ tools and all your connected apps
- **Phone mode** -- test the full pipeline using your phone camera instead of glasses
- **WebRTC streaming** -- share your glasses POV live to a browser viewer
Expand Down Expand Up @@ -262,6 +263,8 @@ All source code is in `samples/CameraAccessAndroid/app/src/main/java/.../cameraa
| `gemini/GeminiLiveService.kt` | OkHttp WebSocket client for Gemini Live API |
| `gemini/AudioManager.kt` | AudioRecord (16kHz) + AudioTrack (24kHz) |
| `gemini/GeminiSessionViewModel.kt` | Session lifecycle, tool call wiring, UI state |
| `voice/RealtimeVoiceService.kt` | Provider-neutral voice session interface + factory |
| `voice/openai/OpenAIRealtimeService.kt` | OpenAI Realtime API over WebRTC (audio + data channel) |
| `openclaw/ToolCallModels.kt` | Tool declarations, data classes |
| `openclaw/OpenClawBridge.kt` | OkHttp HTTP client for OpenClaw gateway |
| `openclaw/ToolCallRouter.kt` | Routes Gemini tool calls to OpenClaw |
Expand All @@ -277,6 +280,14 @@ All source code is in `samples/CameraAccessAndroid/app/src/main/java/.../cameraa
- **iOS iPhone mode**: Uses `.voiceChat` audio session for echo cancellation + mic gating during AI speech
- **iOS Glasses mode**: Uses `.videoChat` audio session (mic is on glasses, speaker is on phone -- no echo)
- **Android**: Uses `VOICE_COMMUNICATION` audio source for built-in acoustic echo cancellation
- **Android, OpenAI Realtime provider**: WebRTC owns mic and speaker end to end (Opus, jitter buffer, echo cancellation inside the peer connection); the PCM pipeline above stays idle

#### Voice Providers (Android)

Settings -> Voice Provider switches between:

- **Gemini Live** (default) -- WebSocket, PCM audio pumped by `AudioManager`, continuous ~1fps vision frames
- **OpenAI Realtime** -- WebRTC via the ephemeral-key flow (`/v1/realtime/client_secrets` then SDP exchange with `/v1/realtime/calls`); model/tool events flow over the `oai-events` data channel. Vision is per-turn: the latest camera frame is attached as an image item when you start speaking, instead of a continuous stream. Requires an OpenAI API key in Settings.

### Video Pipeline

Expand Down
88 changes: 47 additions & 41 deletions ...in/java/com/meta/wearable/dat/externalsampleapps/cameraaccess/gemini/GeminiLiveService.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,9 @@ import android.util.Log
import com.meta.wearable.dat.externalsampleapps.cameraaccess.openclaw.GeminiToolCall
import com.meta.wearable.dat.externalsampleapps.cameraaccess.openclaw.GeminiToolCallCancellation
import com.meta.wearable.dat.externalsampleapps.cameraaccess.openclaw.ToolDeclarations
import com.meta.wearable.dat.externalsampleapps.cameraaccess.openclaw.ToolResult
import com.meta.wearable.dat.externalsampleapps.cameraaccess.voice.RealtimeVoiceService
import com.meta.wearable.dat.externalsampleapps.cameraaccess.voice.VoiceConnectionState
import java.io.ByteArrayOutputStream
import java.util.Timer
import java.util.TimerTask
Expand All @@ -23,33 +26,27 @@ import okio.ByteString
import org.json.JSONArray
import org.json.JSONObject

sealed class GeminiConnectionState {
data object Disconnected : GeminiConnectionState()
data object Connecting : GeminiConnectionState()
data object SettingUp : GeminiConnectionState()
data object Ready : GeminiConnectionState()
data class Error(val message: String) : GeminiConnectionState()
}

class GeminiLiveService {
class GeminiLiveService : RealtimeVoiceService {
companion object {
private const val TAG = "GeminiLiveService"
}

private val _connectionState = MutableStateFlow<GeminiConnectionState>(GeminiConnectionState.Disconnected)
val connectionState: StateFlow<GeminiConnectionState> = _connectionState.asStateFlow()
private val _connectionState = MutableStateFlow<VoiceConnectionState>(VoiceConnectionState.Disconnected)
override val connectionState: StateFlow<VoiceConnectionState> = _connectionState.asStateFlow()

private val _isModelSpeaking = MutableStateFlow(false)
val isModelSpeaking: StateFlow<Boolean> = _isModelSpeaking.asStateFlow()
override val isModelSpeaking: StateFlow<Boolean> = _isModelSpeaking.asStateFlow()

override val managesOwnAudio: Boolean = false

var onAudioReceived: ((ByteArray) -> Unit)? = null
var onTurnComplete: (() -> Unit)? = null
var onInterrupted: (() -> Unit)? = null
var onDisconnected: ((String?) -> Unit)? = null
var onInputTranscription: ((String) -> Unit)? = null
var onOutputTranscription: ((String) -> Unit)? = null
var onToolCall: ((GeminiToolCall) -> Unit)? = null
var onToolCallCancellation: ((GeminiToolCallCancellation) -> Unit)? = null
override var onAudioReceived: ((ByteArray) -> Unit)? = null
override var onTurnComplete: (() -> Unit)? = null
override var onInterrupted: (() -> Unit)? = null
override var onDisconnected: ((String?) -> Unit)? = null
override var onInputTranscription: ((String) -> Unit)? = null
override var onOutputTranscription: ((String) -> Unit)? = null
override var onToolCall: ((GeminiToolCall) -> Unit)? = null
override var onToolCallCancellation: ((GeminiToolCallCancellation) -> Unit)? = null

// Latency tracking
private var lastUserSpeechEnd: Long = 0
Expand All @@ -65,22 +62,22 @@ class GeminiLiveService {
.pingInterval(10, TimeUnit.SECONDS)
.build()

fun connect(callback: (Boolean) -> Unit) {
override fun connect(callback: (Boolean) -> Unit) {
val url = GeminiConfig.websocketURL()
if (url == null) {
_connectionState.value = GeminiConnectionState.Error("No API key configured")
_connectionState.value = VoiceConnectionState.Error("No API key configured")
callback(false)
return
}

_connectionState.value = GeminiConnectionState.Connecting
_connectionState.value = VoiceConnectionState.Connecting
connectCallback = callback

val request = Request.Builder().url(url).build()
webSocket = client.newWebSocket(request, object : WebSocketListener() {
override fun onOpen(webSocket: WebSocket, response: Response) {
Log.d(TAG, "WebSocket opened")
_connectionState.value = GeminiConnectionState.SettingUp
_connectionState.value = VoiceConnectionState.SettingUp
sendSetupMessage()
}

Expand All @@ -95,23 +92,23 @@ class GeminiLiveService {
override fun onFailure(webSocket: WebSocket, t: Throwable, response: Response?) {
val msg = t.message ?: "Unknown error"
Log.e(TAG, "WebSocket failure: $msg")
_connectionState.value = GeminiConnectionState.Error(msg)
_connectionState.value = VoiceConnectionState.Error(msg)
_isModelSpeaking.value = false
resolveConnect(false)
onDisconnected?.invoke(msg)
}

override fun onClosing(webSocket: WebSocket, code: Int, reason: String) {
Log.d(TAG, "WebSocket closing: $code $reason")
_connectionState.value = GeminiConnectionState.Disconnected
_connectionState.value = VoiceConnectionState.Disconnected
_isModelSpeaking.value = false
resolveConnect(false)
onDisconnected?.invoke("Connection closed (code $code: $reason)")
}

override fun onClosed(webSocket: WebSocket, code: Int, reason: String) {
Log.d(TAG, "WebSocket closed: $code $reason")
_connectionState.value = GeminiConnectionState.Disconnected
_connectionState.value = VoiceConnectionState.Disconnected
_isModelSpeaking.value = false
}
})
Expand All @@ -120,31 +117,31 @@ class GeminiLiveService {
timeoutTimer = Timer().apply {
schedule(object : TimerTask() {
override fun run() {
if (_connectionState.value == GeminiConnectionState.Connecting
|| _connectionState.value == GeminiConnectionState.SettingUp) {
if (_connectionState.value == VoiceConnectionState.Connecting
|| _connectionState.value == VoiceConnectionState.SettingUp) {
Log.e(TAG, "Connection timed out")
_connectionState.value = GeminiConnectionState.Error("Connection timed out")
_connectionState.value = VoiceConnectionState.Error("Connection timed out")
resolveConnect(false)
}
}
}, 15000)
}
}

fun disconnect() {
override fun disconnect() {
timeoutTimer?.cancel()
timeoutTimer = null
webSocket?.close(1000, null)
webSocket = null
onToolCall = null
onToolCallCancellation = null
_connectionState.value = GeminiConnectionState.Disconnected
_connectionState.value = VoiceConnectionState.Disconnected
_isModelSpeaking.value = false
resolveConnect(false)
}

fun sendAudio(data: ByteArray) {
if (_connectionState.value != GeminiConnectionState.Ready) return
override fun sendAudio(data: ByteArray) {
if (_connectionState.value != VoiceConnectionState.Ready) return
sendExecutor.execute {
val base64 = Base64.encodeToString(data, Base64.NO_WRAP)
val json = JSONObject().apply {
Expand All @@ -159,8 +156,8 @@ class GeminiLiveService {
}
}

fun sendVideoFrame(bitmap: Bitmap) {
if (_connectionState.value != GeminiConnectionState.Ready) return
override fun sendVideoFrame(bitmap: Bitmap) {
if (_connectionState.value != VoiceConnectionState.Ready) return
sendExecutor.execute {
val baos = ByteArrayOutputStream()
bitmap.compress(Bitmap.CompressFormat.JPEG, GeminiConfig.VIDEO_JPEG_QUALITY, baos)
Expand All @@ -177,14 +174,23 @@ class GeminiLiveService {
}
}

fun sendToolResponse(response: JSONObject) {
override fun sendToolResponse(callId: String, name: String, result: ToolResult) {
val response = JSONObject().apply {
put("toolResponse", JSONObject().apply {
put("functionResponses", JSONArray().put(JSONObject().apply {
put("id", callId)
put("name", name)
put("response", result.toJSON())
}))
})
}
sendExecutor.execute {
webSocket?.send(response.toString())
}
}

fun sendTextMessage(text: String) {
if (_connectionState.value != GeminiConnectionState.Ready) return
override fun sendTextMessage(text: String) {
if (_connectionState.value != VoiceConnectionState.Ready) return
sendExecutor.execute {
val json = JSONObject().apply {
put("clientContent", JSONObject().apply {
Expand Down Expand Up @@ -258,7 +264,7 @@ class GeminiLiveService {

// Setup complete
if (json.has("setupComplete")) {
_connectionState.value = GeminiConnectionState.Ready
_connectionState.value = VoiceConnectionState.Ready
resolveConnect(true)
return
}
Expand All @@ -267,7 +273,7 @@ class GeminiLiveService {
if (json.has("goAway")) {
val goAway = json.getJSONObject("goAway")
val seconds = goAway.optJSONObject("timeLeft")?.optInt("seconds", 0) ?: 0
_connectionState.value = GeminiConnectionState.Disconnected
_connectionState.value = VoiceConnectionState.Disconnected
_isModelSpeaking.value = false
onDisconnected?.invoke("Server closing (time left: ${seconds}s)")
return
Expand Down
Loading