feat: video analytics subsystem — DeepStream 9.0 + Triton (live RTSP, batch segments, tracking, semantic search)

[davidamacey]

Summary

Add a public, production-grade video analytics subsystem: NVIDIA DeepStream 9.0 pipelines (decode → detect → track → crop) fed by Triton over gRPC, with results stored durably and queryable (time/camera/class filters, text semantic search, reverse-image search) for an external UI. Runs on a single multi-GPU server via docker compose and ships a Kubernetes Helm chart with KEDA autoscaling and HA hygiene.

📄 Full implementation plan (4 documents): https://gist.github.com/davidamacey/aa2e01e46894f5437176b231caeb0645

1. 01-overview-context-decisions.md — context, verified foundations on main, locked decisions, reconciled naming, risk register
2. 02-core-pipeline-design.md — repo layout, DeepStream pipeline + PGIE/tracker configs, crop sampling/embedding, event schemas, track stitching, Postgres/OpenSearch/MinIO data model, query API, job orchestration, performance plan, testing
3. 03-deployment-kubernetes-design.md — compose overlay, container images + Trivy gates, Helm chart, KEDA, GPU scheduling, HA checklist, scaling formulas, CI/CD, ops runbook
4. 04-implementation-phases.md — validation spikes S1–S7 (riskiest-first, go/no-go), PR1–PR10 breakdown with definitions of done, end-to-end verification script

The implementation-guide comments below this issue contain ground rules, environment facts, spike details, and per-PR acceptance criteria for the implementing agent/developer.

Capabilities

• Live RTSP feeds — 1..N third-party IP cameras (we only receive; no edge control). Runtime camera add/remove without pipeline restarts (nvmultiurisrcbin + a Postgres-backed assigner and a compacted Kafka control topic).
• Batch video segments — 5/10/15-minute chunks recorded from live feeds but delivered late; processed faster-than-realtime (K=8 concurrent file sources per GPU, ~20×+ realtime on an A6000), queue-driven, KEDA scale-to-zero.
• Ad-hoc uploads — arbitrary videos via POST /v1/video/jobs (priority lane).
• Detection + tracking — existing yolov11_small_trt_end2end engine (EfficientNMS in-engine, 80 COCO classes, vehicles prioritized) as nvinferserver PGIE via Triton gRPC; NvDCF tracker (perf preset live / accuracy preset batch).
• Search — 3–10 representative crops per track embedded with mobileclip2_s2_image_encoder (512-d); OpenSearch kNN indexes for crops/tracks/scenes; text queries via the existing mobileclip2_s2_text_encoder path; cross-segment track stitching (spatiotemporal + appearance gates) yields global_track_id.
• Storage — PostgreSQL (source of truth: cameras, jobs, assets, tracks), OpenSearch (vectors/search), MinIO/S3 (segments, crops, keyframes, model repo), Kafka-protocol events (Redpanda locally; portable to Apache Kafka / AWS MSK incl. IAM auth).

Architecture

RTSP cams ──► video-worker-live  (DeepStream: NVDEC → PGIE yolov11 @ Triton gRPC → NvDCF → probes) ─┐
5-15 min files ► MinIO ─ S3 events ─► job topics ─► video-worker-batch (same pipeline, K=8 files)   ├─►
ad-hoc upload ► POST /v1/video/jobs ────────────────────────────────────────────────────────────────┘
        │  crop JPEGs (nvjpeg → MinIO)  +  track/segment events (Kafka JSON)
        ▼
video-writer (Kafka → Postgres upserts → MobileCLIP embeddings via AsyncTritonPool
              → OpenSearch bulk → track stitching → rollups)     video-janitor (leases, intake, GC)
        ▼
video-api (/v1/video/*: cameras, jobs, tracks, text/image search, presigned media) ─► external UI

Built on assets already on main: the hardened DeepStream 9.0 container (docker/hardened/deepstream/, CVE-gated, verified nvinferserver → Triton gRPC at ~951 FPS), the hardened Triton 26.06 image, existing TensorRT engines, AsyncTritonPool, and the OpenSearch kNN layer. Guiding rule: pixels stay on the GPU — only metadata, embeddings, and selected JPEGs leave the pipeline (NVIDIA VSS-blueprint pattern).

Scope

In: everything above + compose overlay (docker-compose.video.yml, ports 4620–4629), Helm chart (deploy/helm/openprocessor-video/) with bundled↔external switches (Redpanda↔MSK, CNPG↔RDS, MinIO↔S3, OpenSearch↔managed), CI workflows (image Trivy gates, helm lint/kind fake-mode smoke, compose validate), docs + runbook, demo profile (mediamtx fake cameras — zero-camera out-of-box demo).

Out (documented for later): VLM/summarization tiers, ReID tracker models (config-swap path documented), Avro/schema registry (JSON envelope now), DINO-style visual reranker, edge components (cameras are third-party).

Phase checklist

• Spikes S1–S7 (S1 pyds-on-DS9 go/no-go; S2 nvmultiurisrcbin file-EOS; S3 bbox parser; S4 obj_enc; S5 SGIE tensor-meta; S6 NVDEC saturation; S7 stitch threshold)
• PR1 — design docs, VideoSettings, compose infra (redpanda/postgres/minio/video-init), Makefile targets
• PR2 — data layer (SQLAlchemy models, Alembic 0001_video_schema, s3.py, kafka.py)
• PR3 — event schemas + writer skeleton (idempotent upserts, DLQ)
• PR4 — DeepStream worker image + batch pipeline (first end-to-end)
• PR5 — live mode + camera assigner + control topic + demo profile
• PR6 — job orchestration (upload/S3-event intake, claim/lease, janitor)
• PR7 — crop embeddings + track stitching + search endpoints
• PR8 — perf bench + dashboards + activity rollups
• PR9 — Helm chart + KEDA + kind fake-mode smoke
• PR10 — CI/CD workflows + release + runbook

Acceptance criteria

1. make video-up + make video-demo on a GPU box: fake RTSP cameras register, tracks accumulate, camera removal detaches cleanly.
2. Ad-hoc job on a known clip: succeeded, track count within ±20% of golden, every track has ≥1 crop in MinIO and docs in OpenSearch.
3. One clip split into two overlapping segments → the boundary vehicle carries a single global_track_id.
4. POST /v1/video/search/text {"query":"red truck"} ranks the seeded truck track first; reverse-image search returns the same vehicle's other crops.
5. Writer event replay changes no row/doc counts (idempotency).
6. make video-bench: >400 aggregate FPS on an A6000-class GPU; a 5-minute 1080p segment processes in <60 s.
7. Trivy 0 CRIT / 0 HIGH (with documented VEX) on all published images; helm lint + kubeconform + kind fake-mode install green.
8. All Kafka client code runs unmodified against plain Apache Kafka (SASL/TLS) — verified by the release smoke.

[davidamacey]

Implementation guide (1/4): ground rules for the implementing agent

Branch & workflow

• Base everything on main. Create the feature branch in a separate git worktree so the primary checkout (which runs live services) is never disturbed:

  git worktree add /mnt/nvm/repos/triton-api-video -b feat/video-analytics main

• One PR per phase (see the checklist in the issue body and gist file 04-implementation-phases.md). Conventional commits (feat(video): ..., build(video): ..., docs(video): ..., test(video): ...). Merge with merge commits (--no-ff) — never squash. Never bypass hooks or signing (--no-verify is forbidden); if a hook fails, fix the cause.
• Run .venv/bin/pre-commit run --all-files before every commit — call venv binaries directly, never source the env. Same rule for tests: .venv/bin/python -m pytest tests/video/ -v.
• The feature is fully public: no references to private branches, private model names, or non-public infrastructure anywhere in code, docs, comments, or commit messages.

Code conventions

• Reuse before writing: AsyncTritonPool (src/clients/triton_pool.py), EmbeddingService (src/services/embedding.py), letterbox_cpu (src/services/cpu_preprocess.py), OpenSearch index-builder patterns (src/clients/opensearch.py), pydantic-settings pattern (src/config/settings.py), /v1 router aggregation (src/routers/v1.py), /live+/ready probes (src/routers/health.py).
• New shared code lives in src/video/ (importable by api/writer/janitor). DeepStream-container code lives in services/deepstream/ (its own Python env — never imported by the API image). The DS container may import only src.video.{events,settings,sampling} — add the unit test that enforces this boundary early (PR3).
• All configuration via environment → VideoSettings. No secrets in code, images, or compose files — .env (gitignored) + env.template placeholders; K8s uses existingSecret values.
• SQL through SQLAlchemy 2 async with bound parameters; Alembic for every schema change (expand/contract — each migration must be backward-compatible one release).
• Comments only for non-obvious why; match the surrounding style.

Definition of done (every PR)

• Unit + relevant integration tests green; pre-commit clean; compose config -q still valid; no new image pushed except via the release workflow (local builds tag :local only — latest moves ONLY on release tags).

[davidamacey]

Implementation guide (2/4): environment facts (primary dev server)

GPUs (all Ampere sm86 — one TensorRT engine set serves all three; engines are NOT portable across SM families or TensorRT major versions):

Slot	Card	VRAM	Default video role
0	RTX A6000	48 GB	VIDEO_BATCH_GPU (batch worker — treat as the preemptible tenant; other workloads on this box claim GPUs 0/2 at times)
1	RTX 3080 Ti	12 GB	Typically hosts Triton via the existing deployment overlays
2	RTX A6000	48 GB	VIDEO_LIVE_GPU (live worker — the protected tenant)

• GPU pinning is .env-driven (VIDEO_LIVE_GPU / VIDEO_BATCH_GPU interpolated into device_ids) — check nvidia-smi before starting GPU services and point workers at free GPUs. Don't hardcode GPU ids anywhere.
• Host driver: CUDA 13.x line. DS 9.0 validates driver ≥590; 580-series works in-container via CUDA forward-compat (already proven by docker/hardened/test/test_deepstream.sh).
• 2× NVDEC per GPU, no decode session caps; decode (not inference) is the live-stream bottleneck.

Ports — 4600–4612 are taken by the existing stack (Triton 4600–4602, yolo-api 4603, monitoring 4604–4606, OpenSearch 4607–4608, others through 4612). The video feature owns 4620–4629 (assignments in gist file 03, §1.2). Internal metrics port for workers/writer: :9108.

Compose — always docker compose (space). The video stack is docker compose -f docker-compose.yml -f docker-compose.video.yml <cmd>; add --profile demo for fake cameras. The overlay must re-declare triton-server.command to append the three video --load-model flags (a re-declared command replaces the base list — this is the established overlay technique in this repo).

Existing services reused in dev — triton-server (gRPC :8001 in-network), opensearch (:9200 in-network), prometheus/grafana/dcgm-exporter. New scrape targets go in monitoring/prometheus.yml; dashboard JSON in monitoring/dashboards/ (same JSON ships as Grafana ConfigMaps in the Helm chart).

Watchtower caution — this host runs watchtower against public :latest tags. Never push a test/dev build to a public latest; local experiments use :local tags only. latest moves exclusively via the tag-triggered release workflow.

Model assets — models/yolov11_small_trt_end2end/ (PGIE; EfficientNMS in-engine; boxes normalized [0,1] — the custom parser must denormalize), models/mobileclip2_s2_image_encoder/ (SGIE-equivalent, writer-side), models/mobileclip2_s2_text_encoder/. Re-export tooling in export/ if engines need rebuilding for a new TRT version (Dockerfile.triton asserts TRT 11.1.0.106 lockstep).

Test media — do NOT commit video files. make download-test-videos fetches two license-clean 10–15 s 1080p H.264 clips with known object content (mirror the existing download-test-images pattern); golden counts live beside the integration tests.

[davidamacey]

Implementation guide (3/4): validation spikes — run BEFORE the dependent PR

Throwaway scripts under services/deepstream/spikes/. Ordered riskiest-first. Full table in gist file 04.

S1 — pyds build + Triton gRPC compat (go/no-go for the whole Python approach; gates PR4)
Build the pyds wheel from source (pinned DS-9.0-compatible tag of deepstream_python_apps/bindings) inside the unhardened nvcr.io/nvidia/deepstream:9.0-triton-multiarch builder (the hardened runtime has no compiler — toolchain was purged for the CVE gate). Then run videotestsrc → nvstreammux → nvinferserver(gRPC → tritonserver 26.06) → probe and confirm NvDsObjectMeta detections arrive. This validates BOTH the binding build AND the bundled Triton client (26.01) ↔ server (26.06) gRPC pairing.
Fallback: thin C++ probe/emitter in services/deepstream/native/ emitting the identical JSON events; Python keeps job/stream management.

S2 — nvmultiurisrcbin dynamics (gates PR4 batch + PR5 live)
REST POST /api/v1/stream/add|remove under load; file:// URIs; reliable per-source EOS detection (batch mode's job-completion signal); tracker state across add/remove; watch for the Ubuntu-24 glibc memory surge on frequent add/remove (mitigate with MALLOC_ARENA_MAX=4).
Fallback: custom source-bin manager on old nvstreammux (runtime_source_add_delete reference app pattern).

S3 — custom bbox parser (gates PR4)
NvDsInferParseYoloE2E must multiply normalized [0,1] boxes by networkInfo.width/height (the repo's export bakes normalize_boxes=true, so DS's shipped EfficientNMS parsers are wrong as-is) and tolerate det_scores as FP16 or FP32. Verify with rendered overlays on a known clip + numeric comparison against the existing /detect API on the same frame.

S4 — nvds_obj_enc from Python (gates PR4)
Sustain ~500 crops/s of JPEG encodes from a pad probe with no memory growth; verify JPEG validity and padding (~15%).
Fallback: CPU-side encode in the emitter thread (acceptable at ≤10 crops/track).

S5 — SGIE tensor-meta (informational; does NOT gate anything)
Test mobileclip2_s2_image_encoder as SGIE in CAPI mode with output_tensor_meta: true, and separately over gRPC, comparing embeddings against the API-side embedding of the same crop. This decides whether the flagged-off SGIE optimization is ever safe — the primary design embeds in the writer and does not depend on this.

S6 — batch throughput (gates PR8 targets)
K=8 concurrent file:// sources: expect ≥400 aggregate FPS on an A6000 and NVDEC saturation in nvidia-smi dmon. Tune K upward if NVDEC is idle.

S7 — stitching threshold calibration (gates PR7)
Embed crops of same/different vehicles across a synthetic segment cut; sweep the cosine threshold; pick the value with <2% false merges at >80% correct-merge recall. Start point: 0.75, per-class overrides allowed. Record the calibration set + result in docs/design/video_pipeline.md.

[davidamacey]

Implementation guide (4/4): per-PR definition of done + final verification

PR	Definition of done
PR1 infra	docker compose -f docker-compose.yml -f docker-compose.video.yml config -q passes (± --profile demo); make video-up brings redpanda/postgres/minio healthy; video-init idempotently creates topics + buckets; design docs land in docs/design/
PR2 data layer	Alembic upgrade/downgrade round-trips; unit tests: S3 key layout, job claim/lease expiry, kafka config matrix (PLAINTEXT / SASL_SSL / AWS_MSK_IAM)
PR3 events+writer	Replaying a captured event batch twice yields identical Postgres row and OpenSearch doc counts; poison event → video.dlq.writer without stalling the consumer; DS import-boundary test in place
PR4 DS batch e2e	Ad-hoc job on a golden clip → succeeded; track count ±20% of golden; classes ⊆ expected; every track ≥1 crop in MinIO + docs in OpenSearch; worker image passes the Trivy 0/0 gate (with inherited VEX)
PR5 live	Fake RTSP cams (mediamtx demo profile) register → streams running → tracks flow → camera removal detaches cleanly; killing the live worker re-attaches its cameras on restart via the compacted control topic
PR6 jobs	File dropped into video-raw/segments/... auto-creates asset+job and processes; killing a worker mid-job → janitor lease-requeues → job completes on retry; exhausted retries land in video.dlq.jobs
PR7 search	Split-clip stitching yields ONE global_track_id for the boundary vehicle; text search "red truck" ranks the seeded truck first; reverse-image search returns the same vehicle's other crops
PR8 perf	make video-bench recorded in benchmarks/results/: >400 aggregate FPS (A6000), 5-min 1080p segment <60 s; Grafana panels render pipeline FPS / lag / GPU
PR9 helm	helm lint + helm template | kubeconform -strict clean for values{,-minimal,-production}; kind install with gpu.enabled=false + WORKER_MODE=fake passes the api→kafka→writer→PG/OS e2e
PR10 ci/cd	PR CI green incl. image Trivy gates; tag dry-run produces vX.Y.Z/X.Y/latest images + packaged chart (appVersion == tag); runbook covers bootstrap order, topic plan, engine re-export, upgrade/drain, backup

Final release gate (manual, on a GPU box — GPU paths are not CI-testable):

make video-up && make video-status
make download-test-videos
curl -F file=@test_videos/traffic_a.mp4 localhost:4620/v1/video/jobs     # batch path
make video-demo                                                          # live path (fake cams)
curl -X POST localhost:4620/v1/video/search/text -d '{"query":"red truck","k":5}'
curl -X POST localhost:4620/v1/video/search/image -F file=@crop.jpg
make video-bench

All issue acceptance criteria green + Trivy 0/0 (documented VEX only) on openprocessor, openprocessor-deepstream, openprocessor-video-worker → tag the release.

Where to look when stuck: gist file 02 §2 (pipeline configs), §3 (event schemas + stitching), §5 (claim semantics); gist file 03 §1.3 (compose sketch), §3.3 (assigner loop), §3.5 (KEDA); docker/hardened/README.md (DeepStream↔Triton wiring already proven in this repo); docker/hardened/deepstream/Dockerfile (why the GStreamer registry cache must never be baked without a GPU).