Skip to content

Client-side stereo detection transfer via ONNX (first pass)#1709

Open
mattdawkins wants to merge 1 commit into
dev/add-interactive-seg-and-stereofrom
dev/stereo-onnx-transfer
Open

Client-side stereo detection transfer via ONNX (first pass)#1709
mattdawkins wants to merge 1 commit into
dev/add-interactive-seg-and-stereofrom
dev/stereo-onnx-transfer

Conversation

@mattdawkins

Copy link
Copy Markdown
Member

First pass at warping a detection from one camera to the other entirely
client-side (web + Electron renderer), with no backend, by running VIAME's
epipolar template-matching stereo model (stereo measurement "method 1")
exported to ONNX and executed with onnxruntime-web.

This is the client counterpart to the desktop backend stereo service: the
desktop ViewerLoader warps via native IPC (stereoTransferLine /
stereoTransferPoints); this does the equivalent correspondence search in the
browser so it also works on the web.

What's here (client/dive-common/use/stereo/)

  • StereoOnnxMatcher — loads the match ONNX model and warps source points to
    target points (epipolar candidate generation + NCC along the curve).
  • calibration.ts / npz.ts — parse .npz/.json calibration in-browser
    (mirrors read_stereo_rig), plus invertRig to swap source/target camera.
  • image.ts — RGBA to BT.601 grayscale (matches OpenCV BGR2GRAY used by the NCC).
  • frameSource.ts — read full-res frame pixels from a GeoJS viewer.
  • useStereoOnnxTransfer — on stereo-annotation-complete, warp a box (corners)
    or head/tail line (keypoints) to the other camera and write the feature.
  • Web glue: platform/web-girder/useStereoOnnxWeb.ts, bound to the Viewer's
    stereo-annotation-complete event in the web ViewerLoader.vue.

onnxruntime-web is added as a dependency. The exported model is served as a
static asset (default /models/stereo_match.onnx).

Testing

  • Core is unit-tested (__tests__/stereoOnnx.spec.ts, runs under npm test):
    calibration parsing, grayscale, and point warping validated against the VIAME
    C++/Python reference (matches to ~0.25 px) via onnxruntime-web in Node.
  • The web glue (frame-pixel read, ViewerLoader binding, calibration retrieval) is
    type-checked and lint-clean but has not been live-tested in a running web
    viewer with a real stereo dataset.

Notes

  • Disparity search range is currently a permissive default; should become a user
    setting.
  • DINO-augmented method 2 exists on the VIAME side but is not wired here yet.

@mattdawkins mattdawkins force-pushed the dev/stereo-onnx-transfer branch from eb3573d to 42ca8d8 Compare June 22, 2026 16:25
Runs VIAME's epipolar template-matching model (stereo method 1) in the browser
with onnxruntime-web to warp a detection (box / head-tail keypoints) from one
camera to the other, with no backend. Includes calibration (.npz/.json) and
grayscale helpers, a platform-agnostic transfer composable, and web ViewerLoader
wiring. Core is validated against the VIAME reference in unit tests.
@mattdawkins mattdawkins force-pushed the dev/stereo-onnx-transfer branch from 42ca8d8 to a55690f Compare June 22, 2026 17:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant