Accessibility: Make VisionClaw usable for blind users
Context
I'm evaluating VisionClaw as a daily assistive tool for my blind grandfather
(Meta Ray-Bans + iPhone). The architecture (native audio via Gemini Live,
no STT→LLM→TTS pipeline) is exactly right for this use case — latency between
question and spoken answer is the single most important metric for blind users.
However, a few changes would turn this from a developer demo into a genuinely
usable accessibility tool. Happy to contribute PRs for some of these.
Proposed changes
1. Non-visual activation
"Tap the AI button" is the worst possible interaction for a blind user. Suggestions:
- Auto-start the Gemini session on app launch (opt-in setting)
- Trigger via the glasses' capture button and/or iPhone Action Button
- Add VoiceOver/TalkBack accessibility labels to all UI controls
2. Blind-user system prompt preset
A selectable prompt profile in GeminiConfig optimized for blind users:
- Spatial language using clock positions + distance ("cup at 2 o'clock, arm's length")
- Prioritized brevity: hazards first, details only on request
- Read text verbatim (mail, labels, expiry dates) instead of summarizing
- Express uncertainty explicitly instead of confident guessing (safety-critical
for medication labels etc.)
- Never say "as you can see"
3. Adaptive frame quality for text reading
1fps @ 50% JPEG is fine for scene description but too low for OCR — reading
text (mail, packaging, price tags) is the #1 use case for blind users.
Proposal: when a reading intent is detected, send a single full-resolution frame.
4. Audio state feedback (earcons)
Blind users can't see status indicators. Add short audio cues for: session
started, connection lost, reconnected, frame captured. Otherwise users talk
into a dead session without noticing.
5. Session resilience
Gemini Live sessions have time limits and WebSockets drop. Add automatic
reconnect with session resumption / context handover, requiring zero user action.
6. Offline fallback via local VLM
When connectivity drops (basement, train, rural areas) the glasses go
completely silent — exactly when descriptions are needed most. Proposal:
an on-device fallback (e.g. Gemma 4 E2B, as used by Envision for on-device
scene interpretation) that takes over on WebSocket disconnect. Slower and
simpler answers, but never silence.
7. Safety framing
At 1fps + cloud latency this is not a real-time safety system. The system
prompt should prevent the model from making navigation-safety claims
("the path is clear") — it complements a cane/guide dog, it doesn't replace them.
Priority
Items 1, 2 and 4 are low-effort / high-impact (~80% of the difference).
3 and 5 next. Item 6 is the most ambitious but the one that makes this
reliable for daily use.
Accessibility: Make VisionClaw usable for blind users
Context
I'm evaluating VisionClaw as a daily assistive tool for my blind grandfather
(Meta Ray-Bans + iPhone). The architecture (native audio via Gemini Live,
no STT→LLM→TTS pipeline) is exactly right for this use case — latency between
question and spoken answer is the single most important metric for blind users.
However, a few changes would turn this from a developer demo into a genuinely
usable accessibility tool. Happy to contribute PRs for some of these.
Proposed changes
1. Non-visual activation
"Tap the AI button" is the worst possible interaction for a blind user. Suggestions:
2. Blind-user system prompt preset
A selectable prompt profile in
GeminiConfigoptimized for blind users:for medication labels etc.)
3. Adaptive frame quality for text reading
1fps @ 50% JPEG is fine for scene description but too low for OCR — reading
text (mail, packaging, price tags) is the #1 use case for blind users.
Proposal: when a reading intent is detected, send a single full-resolution frame.
4. Audio state feedback (earcons)
Blind users can't see status indicators. Add short audio cues for: session
started, connection lost, reconnected, frame captured. Otherwise users talk
into a dead session without noticing.
5. Session resilience
Gemini Live sessions have time limits and WebSockets drop. Add automatic
reconnect with session resumption / context handover, requiring zero user action.
6. Offline fallback via local VLM
When connectivity drops (basement, train, rural areas) the glasses go
completely silent — exactly when descriptions are needed most. Proposal:
an on-device fallback (e.g. Gemma 4 E2B, as used by Envision for on-device
scene interpretation) that takes over on WebSocket disconnect. Slower and
simpler answers, but never silence.
7. Safety framing
At 1fps + cloud latency this is not a real-time safety system. The system
prompt should prevent the model from making navigation-safety claims
("the path is clear") — it complements a cane/guide dog, it doesn't replace them.
Priority
Items 1, 2 and 4 are low-effort / high-impact (~80% of the difference).
3 and 5 next. Item 6 is the most ambitious but the one that makes this
reliable for daily use.