SORT (Simple Online and Realtime Tracking)

This repo contains:

• A modern websocket “bridge” that assigns stable track IDs to per-frame CV detections and re-broadcasts the streams.
• Image-space and world-space tracking modes built around the SORT idea.
• A legacy MOTChallenge demo (kept for reference, not used by the bridge).

Companion docs:

• docs/WORLD_SPACE_CONSTANT_VELOCITY_TRACKER_GUIDE.md for the full frame-by-frame walkthrough of the world-space constant-velocity tracker
• docs/ASTAR_EGO_RELATIVE_PATH_PLANNER_GUIDE.md for the full runtime walkthrough of the A* ego-relative path planner
• docs/TRACKER_PARAMETERS_GUIDE.md for world-space tuning behavior
• docs/TRACK_STATE_MACHINE_GUIDE.md for lifecycle and status semantics

How SORT works (high level)

SORT is a minimal multi-object tracker that combines:

• A motion model (Kalman Filter per track) and
• A data association step (Hungarian assignment) to match detections ↔ tracks each frame.

Per-frame loop

For each video frame:

• 1) Predict: each track’s Kalman Filter predicts the next state (where the object should be now).
• 2) Associate: compute a cost between each detection and each predicted track.
  • Classic SORT uses IOU between 2D boxes as the matching signal.
  • Solve the assignment with the Hungarian algorithm (min-cost matching).
  • Apply gating (e.g., minimum IOU) so bad pairs don’t match.
• 3) Update:
  • matched tracks run a Kalman update with the matched detection
  • unmatched detections spawn new tracks
  • unmatched tracks age; tracks exceeding max_age are removed
• 4) Confirm tracks: only output IDs after a track has enough consecutive hits (min_hits).

Kalman Filter internals

The Kalman Filter (KF) is the motion model that powers each track. It maintains a state vector and an uncertainty covariance, and alternates between two steps: predict and update. Both steps modify the state vector.

Variables and matrices

Symbol	Name	Shape (CV / CA)	Description
x	State vector	4×1 / 6×1	The tracked quantities. CV: [east, north, vₑ, vₙ]. CA: [east, north, vₑ, vₙ, aₑ, aₙ].
P	State covariance	4×4 / 6×6	Uncertainty of the state estimate. Diagonal = variance of each state variable; off-diagonal = cross-correlations (e.g., how position uncertainty relates to velocity uncertainty).
F	State transition (process) matrix	4×4 / 6×6	Encodes the physics model. For CV: position += velocity × dt. For CA: position += velocity × dt + ½ × accel × dt², velocity += accel × dt.
Q	Process noise covariance	4×4 / 6×6	Models how much the real world deviates from the idealized physics. Larger Q → the filter trusts predictions less and relies more on measurements.
z	Measurement vector	2×1	What the detector actually observes: [east, north] (position only — no direct velocity or acceleration measurement).
H	Observation (measurement) matrix	2×4 / 2×6	Maps the full state to the measurement space. H = [[1,0,0,0,...],[0,1,0,0,...]] — it "picks out" just the position components from x.
R	Measurement noise covariance	2×2	Uncertainty of the detector's position measurements. Larger R → the filter trusts measurements less.
y	Innovation (residual)	2×1	y = z − H·x⁻. The difference between what we measured and what we predicted we'd measure.
S	Innovation covariance	2×2	S = H·P⁻·Hᵀ + R. Combined uncertainty of prediction and measurement in measurement space.
K	Kalman gain	4×2 / 6×2	K = P⁻·Hᵀ·S⁻¹. The "correction weight" that maps a 2D position error into corrections for ALL state variables. This is the key matrix.

Step 1: Predict

Extrapolate the state forward in time using the physics model:

x⁻ = F · x       (predicted state)
P⁻ = F · P · Fᵀ + Q   (predicted covariance)

For our CV model with dt = 1 frame, F looks like:

F = | 1  0  dt  0 |     x⁻ = | east + vₑ·dt  |
    | 0  1  0  dt |           | north + vₙ·dt |
    | 0  0  1   0 |           | vₑ             |
    | 0  0  0   1 |           | vₙ             |

Position moves by velocity × dt. Velocity stays the same (constant-velocity assumption). The CA model adds acceleration terms similarly.

After predict, we apply kinematic clamping (_clamp_state()) to cap velocity and acceleration magnitudes.

Step 2: Update

Correct the prediction using the actual measurement. This step modifies ALL state variables, not just the measured ones:

y = z − H · x⁻           (innovation: position error)
S = H · P⁻ · Hᵀ + R      (innovation covariance)
K = P⁻ · Hᵀ · S⁻¹        (Kalman gain)
x⁺ = x⁻ + K · y          (corrected state)
P⁺ = (I − K · H) · P⁻    (corrected covariance)

Why the update changes velocity (and acceleration): The Kalman gain K is a 4×2 matrix (CV) or 6×2 matrix (CA). It maps the 2D position innovation y into corrections for every dimension of the state vector. Concretely:

K · y = | K₀₀·yₑ + K₀₁·yₙ |   ← east position correction
        | K₁₀·yₑ + K₁₁·yₙ |   ← north position correction
        | K₂₀·yₑ + K₂₁·yₙ |   ← east velocity correction   ← !!
        | K₃₀·yₑ + K₃₁·yₙ |   ← north velocity correction  ← !!

If the predicted position was too far east compared to the measurement, K will not only correct the position but also reduce the eastward velocity. This is how the KF infers velocity from position-only measurements: it uses the cross-correlations in P to distribute the position error across all state variables.

The magnitude of velocity correction depends on:

• P cross-terms between position and velocity — if the filter is uncertain about velocity, it allows larger corrections
• R — if measurements are trusted (small R), the correction is larger
• Q — if process noise is large, the filter gives more weight to measurements

After update, we apply kinematic clamping again to ensure the corrected velocity/acceleration still respects the configured caps.

Why clamping is needed after both steps

• After predict: prevents velocity from accumulating beyond physical limits during occlusions (no measurements → predict-only → velocity can grow unbounded in CA, or drift in CV).
• After update: the Kalman gain can inject velocity corrections that exceed the cap. For example, if you clamp velocity to 0 after predict, then the update step computes x⁺ = x⁻ + K·y and the K matrix re-introduces non-zero velocity from the measurement innovation. Without post-update clamping, the cap would be violated for matched tracks.

Common KF questions for the world-space tracker

Q: What is the difference between state covariance and process noise covariance?

• self.kf.P is the state covariance: the filter's current uncertainty about the estimated state x.
• self.kf.Q is the process noise covariance: how much unmodeled motion/disturbance we expect beyond the ideal CV/CA physics model.
• self.kf.R is the measurement noise covariance: how noisy we think the detector's measured position is.

In short:

• bigger P means "I am less sure about this track right now"
• bigger Q means "the real world may deviate more from my motion model"
• bigger R means "the measurements may be noisier"

Q: Where in the code do I change process noise covariance or measurement noise covariance?

In sort_ws/world_space_tracker.py:

• CV model (KalmanCVPointTracker)
  • process noise intensity: self._q_intensity
  • process covariance matrix build: _rebuild_dynamics() where self.kf.Q is rebuilt from dt
  • measurement covariance: _measurement_covariance_from_rel_enu() rebuilds self.kf.R from the current relative ego-to-target direction, measurement_noise_cross_var_m2, and measurement_noise_radial_scale
  • initial state covariance: self.kf.P
• CA model (KalmanCAPointTracker)
  • process noise intensity: self._q_intensity
  • process covariance matrix build: _rebuild_dynamics() where self.kf.Q is rebuilt from dt
  • measurement covariance: _measurement_covariance_from_rel_enu() rebuilds self.kf.R from the current relative ego-to-target direction, measurement_noise_cross_var_m2, and measurement_noise_radial_scale
  • initial state covariance: self.kf.P

Current world-space measurement covariance logic:

• cross_var = measurement_noise_cross_var_m2
• radial_var = cross_var * measurement_noise_radial_scale
• build unit vectors for the current radial and cross-range directions from world_rel_ego_east_m/world_rel_ego_north_m
• form R = radial_var * (radial radial^T) + cross_var * (cross cross^T) in ENU space

Q: What does "over time" mean when we say the KF trusts prediction or measurement more?

The trust split is not fixed. It changes every cycle because it depends on the current covariance:

P⁻ = F · P · Fᵀ + Q
K = P⁻ · Hᵀ · (H · P⁻ · Hᵀ + R)⁻¹
x⁺ = x⁻ + K · (z − H · x⁻)

• the predict step propagates uncertainty forward and adds Q
• the update step computes K from the current predicted covariance P⁻ and R
• larger K means more measurement trust; smaller K means more prediction trust

So yes, there is effectively an initial trust behavior and a later trust behavior:

• early in a track, initial P is large (especially for velocity/acceleration), so the filter is more uncertain
• after repeated updates, P often shrinks and K settles toward a more stable value
• if measurements are missed for several frames, P grows again during predict-only steps, so K may increase when a measurement finally arrives

Because this tracker rebuilds F and Q from the real dt every frame, K can vary with timing, missed detections, and maneuvers.

Q: What is the current input into the world-space tracker?

The world-space Kalman Filter does not directly ingest image-space x/y/w/h as its measurement.

Current pipeline:

• upstream detections typically include image-space fields such as x, y, width/w, height/h, distance, heading, category, and confidence
• in sort_ws/bridge.py, _track_world_space() uses the detection's horizontal position (x + width/2) plus distance, ego pose, and camera parameters to project the detection into local ENU meters
• that projection writes world_east_m and world_north_m into the detection dict
• WorldSpaceSort.assign() then uses [world_east_m, world_north_m] as the KF measurement z

What each field is used for:

• world_east_m, world_north_m: direct Kalman measurement input
• heading, confidence: used in association cost / metadata, not as KF state measurements
• category: used for speed/acceleration caps
• y, width, height: mainly preserved for unmatched-track back-projection and output formatting

Q: If distance is unstable/jumpy, does that affect each track's P/Q/R/K?

Jumpy distance causes jumpy ENU measurements, because distance is part of the image-to-world projection. That mainly affects the measurement z.

• Q is a tunable process-noise family set by q_intensity
• R is rebuilt every update from the current relative ego-to-target direction plus the tunables measurement_noise_cross_var_m2 and measurement_noise_radial_scale
• noisy distance does make z noisier, which can cause larger innovations y = z - Hx⁻
• that can make the corrected state wiggle more if the filter is trusting measurements strongly
• if the noise is bad enough to cause missed associations, the tracker may do several predict-only steps, which grows P
• once P is larger, the next matched update often produces a larger K, so the measurement can pull the state harder

So the short version is:

• noisy distance directly corrupts the measurement z
• it does not directly rewrite Q or R
• it can indirectly change P and therefore K over subsequent frames

Q: If I scale both Q and R up by the same factor, but keep their ratio the same, does behavior change?

Usually yes, especially during startup and other non-steady situations.

• in an ideal linear time-invariant system, scaling both Q and R by the same factor can lead to a similar steady-state gain
• in this tracker, behavior is not perfectly invariant because initial P is fixed, dt varies, association/gating is nonlinear, missed updates happen, and kinematic clamping is applied outside the pure KF equations

So if you double, triple, or quadruple both Q and R while keeping the same ratio:

• steady-state behavior may look somewhat similar
• transient behavior, convergence speed, recovery after misses, and even association outcomes can still change
• therefore "same Q:R ratio" does not guarantee identical tracker behavior in this pipeline

What this repo is used for (CV detections over websockets)

We are not using MOTChallenge evaluation in day-to-day work here. Instead, we use this repository to:

• consume a video stream with detection metadata (per frame),
• run a tracker to turn “frame detections” into “stable object IDs”, and
• re-broadcast the same stream with stable track_id on each bbox and obj_id preserved from upstream (original detection ID).

The entrypoint is sort-3d.py which runs sort_ws/bridge.py.

Installation

Python + venv

This repo expects you to use a local virtual environment in venv/.

cd /path/to/sort
python -m venv venv
source venv/bin/activate
pip install -r requirements.txt

Notes on dependencies

• lap is used for fast Hungarian assignment; if it can’t import, the code falls back to SciPy’s linear_sum_assignment.
• Core runtime deps for the bridge/tracker are already captured in requirements.txt (notably websockets, numpy, scipy, filterpy).

Running the websocket bridge

Upstream / downstream ports

The bridge connects to upstream websocket servers (typically a replay process) and re-broadcasts downstream:

• Upstream video: ws://127.0.0.1:5001 (binary payload: [4-byte len][json][jpeg])
• Upstream NMEA: ws://127.0.0.1:3636 (JSON text)
• Upstream control: ws://127.0.0.1:6001 (JSON/text)

Downstream defaults:

• video: ws://0.0.0.0:5002
• nmea: ws://0.0.0.0:3637
• control: ws://0.0.0.0:6002

Enter venv

You can either:

• activate the venv: source venv/bin/activate, then run python ..., or
• call the interpreter directly: venv/bin/python ...

Image-space mode

Image-space mode tracks directly in pixel coordinates using IOU (plus optional extra terms).

venv/bin/python sort-3d.py \
  --mode image_space \
  --image-space-max-age 40 \
  --image-space-min-hits 300 \
  --image-space-iou-threshold 0.10

Detection input (per bbox): expects dicts like:

• required: x, y, width, height (pixels)
• optional: confidence, distance, heading, category

Output behavior:

• both matched and unmatched confirmed tracks are output every frame (unconfirmed detections are dropped)
• each bbox has track_id (stable tracker ID) and, for matched bboxes, obj_id (original upstream detection ID)
• outgoing metadata includes tracked_space="image_space" and tracked_status ("matched" or "unmatched")

World-space mode

World-space mode uses ego NMEA (lat/lon/heading) + camera parameters to project each detection into local ENU meters, then tracks in that world space.

Why local ENU meters (not absolute lat/lon) for KF + Hungarian?

We track in a local tangent plane (ENU = East/North in meters) instead of directly in latitude/longitude because:

• Linear units: Kalman Filters assume a (locally) linear state and measurement model. ENU meters are approximately linear over small areas; lat/lon are angular degrees with non-uniform scaling (longitude depends on cos(latitude)).
• Meaningful distances/costs: Hungarian association needs a cost with consistent units. Euclidean distance in ENU meters is meaningful; “distance” in degrees is not (and varies with latitude).
• Numerical conditioning: meter-scale values (tens/hundreds) are better conditioned than tiny degree deltas (~1e-5) and avoid mixing lat/lon scales.

Current limitations:

• Current local ENU ref_latlon approach loses accuracy as journey range increases: our ENU frame is anchored to a single fixed ref_latlon (the first ego GPS fix). This “flat plane” (equirectangular) approximation is typically accurate over small/local regions (rule of thumb: up to a few km); we do not currently re-anchor it as the boat travels, so accuracy can degrade over longer distances.
• Why we need the first ego fix before outputting world-space detections: projecting a detection into world space requires a reference pose (ego lat/lon + heading) to convert pixel bearing + range into ENU meters. Until we receive the first valid ego state, the bridge cannot compute world_east_m/world_north_m, so in --mode world_space it drops bboxes rather than emitting incorrect world coordinates/IDs.

Where the ENU conversion happens

• Detection → ENU (relative meters): sort_ws/image_to_world.py:
  • pixel_x_to_bearing_rad() and camera_relative_xy_m() convert pixel x + range to (right_m, forward_m)
  • right_fwd_to_enu_m() converts (right/forward) into ENU using ego heading
  • detection_to_world_latlon() returns:
    • enu_rel_ego (meters relative to ego), and
    • enu_from_ref (meters relative to a fixed local reference point)
• Bridge populates ENU fields used by the world tracker: sort_ws/bridge.py in _track_world_space():
  • writes world_east_m / world_north_m into each detection dict
  • after association, enriches every output track with enu_ref_lat / enu_ref_lon and full KF kinematic state (vel_east_mps, vel_north_mps, speed_mps, course_deg, accel_east_mps2, accel_north_mps2)
• World tracker consumes ENU meters and outputs full kinematic state: sort_ws/world_space_tracker.py:
  • WorldSpaceSort reads world_east_m / world_north_m and associates tracks using meter distances
  • assign() returns a 3-tuple: (assigned_ids, unmatched_tracks, matched_track_states) where matched_track_states maps each matched track ID to its full KF state (position, velocity, acceleration, speed, course)
  • Each KF tracker (KalmanCVPointTracker, KalmanCAPointTracker) exposes get_full_state() returning the complete kinematic state as a dict

venv/bin/python sort-3d.py \
  --mode world_space \
  --world-space-cv-width-px 1920 \
  --world-space-fov-x-deg 55.3 \
  --world-space-camera-yaw-offset-deg 0.0 \
  --world-space-max-distance-m 50.0

Requirements:

• upstream NMEA stream must provide ego Latitude/Longitude/Heading
• video metadata bboxes must include distance (meters) and pixel x/width so the bridge can compute bearing

Output behavior:

• both matched and unmatched confirmed tracks are output every frame
• each bbox has track_id (stable tracker ID) and, for matched bboxes, obj_id (original upstream detection ID)
• world fields are added (e.g. world_latitude/world_longitude, world_east_m/world_north_m, etc.); unmatched tracks get back-projected bbox fields for AR view
• outgoing metadata includes tracked_space="world_space" and tracked_status ("matched" or "unmatched")
• every track includes the ENU reference point (enu_ref_lat, enu_ref_lon) — the fixed lat/lon origin of the ENU plane (first ego GPS fix), enabling downstream consumers to convert ENU positions/velocities without a separate handshake
• every track includes full Kalman Filter kinematic state: velocity (vel_east_mps, vel_north_mps), derived speed and course (speed_mps, course_deg), and acceleration (accel_east_mps2, accel_north_mps2). Acceleration is null for the CV (constant-velocity) model and populated for the CA (constant-acceleration) model

Tracker output structure

Both trackers now output matched and unmatched confirmed tracks every frame. Unmatched tracks are confirmed tracks that weren't associated with a detection this frame but are still within max_age. This allows downstream consumers (e.g., lookout-V2 AR view) to continue displaying tracks even when detections are temporarily missing.

Output mode selection logic

The --mode parameter controls which tracker's output is used:

• image_space: Only outputs image-space tracks (matched + unmatched). World-space tracker still runs internally but its output is discarded.
• world_space: Only outputs world-space tracks (matched + unmatched). May output empty list until first ego fix arrives.
• auto (default): If world-space tracks are non-empty, outputs only world-space tracks; otherwise outputs only image-space tracks. This provides a seamless fallback when ego data is unavailable.

Output fields by tracker and match status

Each output bbox includes a tracked_status field:

• "matched" — track was associated with a detection this frame
• "unmatched" — confirmed track with no detection this frame (using predicted/stored values)

Track output decision tree

The tracker uses two counters to decide whether to output a track:

• hits: total lifetime matches (cumulative, never resets)
• hit_streak: consecutive matches (resets to 0 on any missed frame)

                        Track exists
                             |
            +----------------+----------------+
            |                                 |
    Detection matched?                 No detection matched
            |                                 |
   hit_streak >= min_hits?            hits >= min_hits?
            |                                 |
      +-----+-----+                     +-----+-----+
      |           |                     |           |
     YES         NO                    YES         NO
      |           |                     |           |
   OUTPUT    hits > min_hits?        OUTPUT      NOT
   MATCHED        |                  UNMATCHED   OUTPUT
            +-----+-----+            (ghost)
            |           |
           YES         NO
            |           |
         OUTPUT       NOT
        UNMATCHED    OUTPUT
        (re-warm)   (warming)

State definitions:

State	Condition	Meaning
Matched	hit_streak >= min_hits	Track has enough consecutive matches to be fully confirmed
Re-warming	hit_streak < min_hits but hits > min_hits	Track was previously confirmed but lost consecutive matches; still outputs as unmatched using KF world position plus current detection image-space fields
Unmatched (ghost)	No detection this frame, but hits >= min_hits	Confirmed track with no detection; outputs using Kalman prediction
Warming	hits < min_hits	New track building confidence; not yet output

Position and kinematic source by state:

Track State	Position Source	Velocity / Acceleration Source
Matched	KF state after update for world-space fields; current detection for image-space bbox fields	KF state (post-update)
Re-warming	KF prediction for world-space fields; current detection for image-space bbox fields	KF state (post-update, while status remains rewarming)
Unmatched (ghost)	Kalman filter prediction	KF state (post-predict)

Why the outputs are split between world-space and image-space fields:

1. Matched world tracks now emit the KF-corrected ENU/latlon position so the displayed world track reflects filter state rather than raw measurement truth.
2. Re-warming world tracks keep the current detection image-space bbox fields, but leave world-space position on the KF-prediction side so downstream can compare "fresh bbox" vs "not fully re-confirmed world state."
3. Ghost world tracks use predict-only KF world state and back-project image-space bbox values only for AR rendering.

Track lifecycle and deletion

A track is created when an unmatched detection spawns a new tracker. The track uses a third counter:

• time_since_update: frames since last matched detection (resets to 0 on match, increments on each predict)

Track deletion rule: A track is permanently deleted when time_since_update > max_age.

Track created (unmatched detection)
         |
         v
    [Warming phase: hits < min_hits]
         |
    hit_streak >= min_hits?
         |
    +----+----+
    |         |
   YES       NO (missed frames)
    |         |
    v         v
 [Confirmed] time_since_update > max_age?
    |              |
    |         +----+----+
    |         |         |
    |        NO        YES
    |         |         |
    |    [Continue]  [DELETED]
    |    (as ghost)
    v
 [Outputs as matched/unmatched/ghost]

Summary: when is a track NOT output?

Condition	Result
hits < min_hits (warming)	Not output — still building confidence
time_since_update > max_age	Track deleted — gone forever

Summary: when IS a track output?

Once hits >= min_hits (i.e., the track was confirmed at some point), the track will be output every frame until it's deleted. The output status depends on the current frame:

• Matched: detection matched AND hit_streak >= min_hits
• Unmatched (re-warming): detection matched BUT hit_streak < min_hits
• Unmatched (ghost): no detection matched this frame

1. Image-space tracker, matched

All original detection fields are preserved, plus tracker metadata:

Field	Source
x, y, width, height	From detection (pixels)
confidence, distance, heading, category	From detection
obj_id	From detection (original upstream ID)
track_id	Stable tracker ID (1-based)
tracked_space	"image_space"
tracked_status	"matched"

2. Image-space tracker, unmatched

Uses Kalman-predicted bbox and last known metadata:

Field	Source
x, y, width, height	Kalman-predicted bbox (pixels)
confidence, distance, heading, category	Last known values from most recent match
track_id	Stable tracker ID (1-based)
tracked_space	"image_space"
tracked_status	"unmatched"

3. World-space tracker, matched

All original detection fields plus computed world fields and KF kinematic state:

Field	Source
x, y, width, height	From detection (pixels)
confidence, distance, heading, category	From detection
obj_id	From detection (original upstream ID)
world_latitude, world_longitude	Converted from the KF-corrected ENU state
world_east_m, world_north_m	KF-corrected ENU position (post-update)
world_rel_ego_east_m, world_rel_ego_north_m	Recomputed from KF ENU position relative to current ego
enu_ref_lat, enu_ref_lon	Fixed ENU origin (first ego GPS fix, degrees)
vel_east_mps, vel_north_mps	KF velocity in ENU (m/s)
speed_mps	Derived speed: hypot(vel_east, vel_north) (m/s)
course_deg	Course over ground: atan2(vel_east, vel_north) (0=N, 90=E, degrees)
accel_east_mps2, accel_north_mps2	KF acceleration in ENU (m/s²); null for CV model
track_id	Stable tracker ID (1-based)
tracked_space	"world_space"
tracked_status	"matched"

4. World-space tracker, unmatched

Tracks with tracked_status="unmatched" include two sub-cases with different position sources:

4a. Ghost tracks (no detection this frame)

Uses Kalman-predicted world position + stored bbox geometry + full KF kinematic state:

Field	Source
world_east_m, world_north_m	Kalman-predicted position (ENU meters)
x	Back-projected from predicted world position
y, width, height	Stored from most recent matched detection
distance	Back-projected from predicted world position
world_latitude, world_longitude	Converted from predicted ENU
world_rel_ego_east_m, world_rel_ego_north_m	Recomputed relative to current ego
enu_ref_lat, enu_ref_lon	Fixed ENU origin (first ego GPS fix, degrees)
vel_east_mps, vel_north_mps	KF velocity in ENU (m/s)
speed_mps	Derived speed (m/s)
course_deg	Course over ground (degrees)
accel_east_mps2, accel_north_mps2	KF acceleration in ENU (m/s²); null for CV model
confidence, heading, category	Last known values
track_id	Stable tracker ID (1-based)
tracked_space	"world_space"
tracked_status	"unmatched"

4b. Re-warming tracks (detection matched, but hit_streak < min_hits)

Uses the current detection for image-space bbox/confidence/category fields, but keeps the KF-predicted world position:

Field	Source
x, y, width, height	Directly from current detection (exact values)
distance	Directly from current detection
confidence, heading, category	Directly from current detection
obj_id	Directly from current detection (preserved!)
world_east_m, world_north_m	KF-predicted world position
world_latitude, world_longitude	Converted from KF-predicted ENU
world_rel_ego_east_m, world_rel_ego_north_m	Recomputed from KF-predicted ENU relative to ego
enu_ref_lat, enu_ref_lon	Fixed ENU origin (first ego GPS fix, degrees)
vel_east_mps, vel_north_mps	KF velocity in ENU (m/s)
speed_mps	Derived speed (m/s)
course_deg	Course over ground (degrees)
accel_east_mps2, accel_north_mps2	KF acceleration in ENU (m/s²); null for CV model
track_id	Stable tracker ID (1-based)
tracked_space	"world_space"
tracked_status	"unmatched"

Why re-warming is mixed-source: Re-warming tracks have a fresh detection this frame, so the bridge preserves the current image-space bbox, obj_id, confidence, heading, and category. But their world-space position remains on the KF-prediction side so matched/rewarming/ghost states can all be compared in the same ENU frame without snapping world position back to raw measurement truth.

Why compute artificial bbox for ghost tracks?

Ghost tracks (true unmatched — no detection this frame) need artificial bbox computation because the world tracker only maintains ENU position, not pixel coordinates. The lookout-V2 AR view needs bbox coordinates to display overlays synchronized with the Aerial View.

For ghost tracks only, we back-project world position to image space:

1. world_to_image.py::world_to_image_space() computes x_center_px and distance from the Kalman-predicted world position
2. Stored y, width, height from the last matched detection are reused

For re-warming tracks, no back-projection is needed — we pass through the detection data directly.

This allows the AR view to continue showing a bbox overlay for the track even when the detector completely misses the object (ghost tracks).

Parameters (what they mean)

Image-space tracker parameters

• --image-space-max-age: frames a track can miss before being deleted
• --image-space-min-hits: consecutive matched frames required before emitting an ID
• --image-space-iou-threshold: minimum IOU to allow a detection↔track match
• --image-space-alpha-distance: weight for distance-difference cost term (if distance present)
• --image-space-beta-heading: weight for heading-difference cost term (if heading present)
• --image-space-gamma-confidence: weight for confidence penalty term (prefers high-confidence detections)
• --image-space-new-track-min-confidence: minimum confidence to spawn a new track

World-space camera parameters

• --world-space-cv-width-px: image width in pixels (used to convert x-pixel → bearing)
• --world-space-fov-x-deg: horizontal field-of-view in degrees
• --world-space-camera-yaw-offset-deg: yaw offset between camera forward and boat forward

World-space tracker parameters

• --world-space-max-age: frames a track can miss before being deleted
• --world-space-min-hits: consecutive matched frames required before emitting an ID
• --world-space-max-distance-m: max ENU distance allowed for association (gating)
• --world-space-beta-heading: weight for heading-difference cost term (if target heading exists)
• --world-space-gamma-confidence: weight for confidence penalty term
• --world-space-new-track-min-confidence: minimum confidence to spawn a new track
• --world-space-q-intensity: process-noise intensity for the world-space KF motion model; larger values make motion more maneuver-friendly but noisier
• --world-space-measurement-cross-var-m2: baseline measurement covariance in the cross-range direction
• --world-space-measurement-radial-scale: scales radial measurement covariance relative to the cross-range baseline, producing anisotropic R

Per-category kinematic caps

These cap the KF state to physically reasonable values, preventing tracks from "flying away" during occlusions or noisy detections. Clamping is applied after both the KF predict and update steps, preserving direction while capping magnitude.

The clamp is always active. 0 = clamp to zero (e.g., force stationary), positive = cap at that value. Set a large value (e.g. 50 for speed, 20 for acceleration) if you effectively don't want a constraint.

• --world-space-max-speed-boat-mps: max speed for boat category tracks (m/s). Velocity vector magnitude is clamped to this value. Default: 50.
• --world-space-max-speed-other-mps: max speed for non-boat category tracks (m/s). Default: 50.
• --world-space-max-accel-boat-mps2: max acceleration for boat category tracks (m/s²). Only effective with the CA model. Default: 20.
• --world-space-max-accel-other-mps2: max acceleration for non-boat category tracks (m/s²). Only effective with the CA model. Default: 20.

File guide (what each file does)

Websocket bridge + pipeline

• sort-3d.py: entrypoint; calls sort_ws.bridge.cli_main().
• sort_ws/bridge.py: connects to upstream websockets, runs tracking, re-broadcasts downstream.
• sort_ws/codec.py: encodes/decodes the binary video websocket payload: [len][json][jpeg].

Tracking + transforms

• sort_ws/image_space_tracker.py: image-space SORT-style tracker (ImageSpaceSort) + IOU association + Hungarian assignment (with optional extra cost terms).
• sort_ws/image_to_world.py: math utilities to project pixels + distance into ENU meters and back to lat/lon.
• sort_ws/ego.py: ego state smoothing and storage; consumes NMEA JSON, maintains a smoothed EgoState (lat/lon/heading and local ENU).
• sort_ws/world_space_tracker.py: world-space SORT-style tracker (WorldSpaceSort) that tracks points in ENU meters. Supports constant-velocity (CV) and constant-acceleration (CA) Kalman Filter models. Each tracker exposes get_full_state() returning ENU position, velocity, acceleration, speed, and course.

Legacy / unused: MOTChallenge demo (not used by the bridge)

The original upstream SORT demo is preserved under sort_mot/:

• sort_mot/sort.py: the original demo that reads MOT-format detections from data/ and can optionally display results.

Run the legacy demo (uses data/train/*/det/det.txt):

venv/bin/python sort_mot/sort.py

To display results, you must also have the MOT images available and symlinked to mot_benchmark/ as expected by the script:

ln -s /path/to/MOT2015_challenge/data/2DMOT2015 mot_benchmark
venv/bin/python sort_mot/sort.py --display

License

SORT is released under the GPL License (refer to LICENSE).

Citing SORT

If you find SORT useful, please consider citing:

@inproceedings{Bewley2016_sort,
  author={Bewley, Alex and Ge, Zongyuan and Ott, Lionel and Ramos, Fabio and Upcroft, Ben},
  booktitle={2016 IEEE International Conference on Image Processing (ICIP)},
  title={Simple online and realtime tracking},
  year={2016},
  pages={3464-3468},
  keywords={Benchmark testing;Complexity theory;Detectors;Kalman filters;Target tracking;Visualization;Computer Vision;Data Association;Detection;Multiple Object Tracking},
  doi={10.1109/ICIP.2016.7533003}
}

See also

• DeepSORT
• Local Metrics for Multi-Object Tracking