From 91eb12218ede1d9741b92951845b8eec6146258b Mon Sep 17 00:00:00 2001 From: Shakirov Renat Date: Thu, 23 Jul 2026 10:36:47 +0300 Subject: [PATCH 01/13] robots_ego_cvat: egocentric video -> SAM3 text-prompt segmentation -> CVAT example New example: drop a long first-person video into a folder; the pipeline samples frames (ffmpeg fps=1 -> perceptual-hash dedup), runs SAM3 open-vocabulary text-prompt segmentation (person), and uploads box+polygon pre-annotations to CVAT for review. Reuses the SAM3->CVAT tail from sam_cvat verbatim; the new part is the video ingest + frame sampling front stage. --- examples/robots_ego_cvat/.gitignore | 10 + examples/robots_ego_cvat/README.md | 87 +++++ examples/robots_ego_cvat/__init__.py | 1 + examples/robots_ego_cvat/app.py | 114 +++++++ examples/robots_ego_cvat/config.py | 64 ++++ examples/robots_ego_cvat/data.py | 132 ++++++++ examples/robots_ego_cvat/models.py | 120 +++++++ examples/robots_ego_cvat/pyproject.toml | 33 ++ .../robots_ego_cvat/scripts/fetch_video.py | 74 +++++ examples/robots_ego_cvat/steps.py | 313 ++++++++++++++++++ 10 files changed, 948 insertions(+) create mode 100644 examples/robots_ego_cvat/.gitignore create mode 100644 examples/robots_ego_cvat/README.md create mode 100644 examples/robots_ego_cvat/__init__.py create mode 100644 examples/robots_ego_cvat/app.py create mode 100644 examples/robots_ego_cvat/config.py create mode 100644 examples/robots_ego_cvat/data.py create mode 100644 examples/robots_ego_cvat/models.py create mode 100644 examples/robots_ego_cvat/pyproject.toml create mode 100644 examples/robots_ego_cvat/scripts/fetch_video.py create mode 100644 examples/robots_ego_cvat/steps.py diff --git a/examples/robots_ego_cvat/.gitignore b/examples/robots_ego_cvat/.gitignore new file mode 100644 index 00000000..708821e8 --- /dev/null +++ b/examples/robots_ego_cvat/.gitignore @@ -0,0 +1,10 @@ +.env +.venv/ +.frames/ +videos/ +*.webm +*.mp4 +*.mkv +*.mov +__pycache__/ +*.pyc diff --git a/examples/robots_ego_cvat/README.md b/examples/robots_ego_cvat/README.md new file mode 100644 index 00000000..79f49446 --- /dev/null +++ b/examples/robots_ego_cvat/README.md @@ -0,0 +1,87 @@ +# Egocentric video → SAM3 → CVAT ("robots" demo) + +Drop a long first-person video into a folder — the pipeline samples frames, segments them with a +**text prompt** (SAM3), and hands you ready-to-review pre-annotations in **CVAT**. Built for a +robotics audience: *"here is the inbox folder, put your 20-hour egocentric recording in it, we +process it for you."* + +It reuses the whole SAM3→CVAT tail from [`../sam_cvat`](../sam_cvat); the only thing added on the +front is turning a long video into a deduplicated set of frames. + +## Pipeline + +``` +stage=video list_videos folder INPUT_VIDEO_DIR -> video +stage=sample extract_frames ffmpeg fps=SAMPLE_FPS -> frames +stage=sample dedup_frames perceptual-hash dedup -> local_images +stage=ingest list_sam_config SAM_TEXT_PROMPT -> sam_config +stage=sam sam_inference SAM3 image-mode -> sam_predictions +stage=sam sam_to_cvat_xml -> sam_cvat_xml +stage=cvat prepare_cvat_input / CVATStep / parse_cvat_annotations -> image__annotations +``` + +`local_images (image_id, image_path)` is exactly what the SAM→CVAT tail consumes, so everything from +`sam_inference` onward is identical to `sam_cvat`. + +## Why sample at 1 fps then dedup (not "every frame", not 1/3 fps) + +Walking POV footage is extremely redundant — you move ~1 m/s and consecutive frames are near +duplicates, so **1 fps and 1/3 fps are not meaningfully different** in what ends up annotated. The +real lever is content-awareness. So: extract at `SAMPLE_FPS=1` for good event coverage, then drop +near-duplicates with a perceptual hash (`PHASH_MAX_DISTANCE`, Hamming). A 24h source at 1 fps is +~86k frames; after dedup it collapses to a few thousand genuinely different frames — enough coverage +without flooding CVAT. Both are ordinary incremental datapipe stages: add another video and only its +frames are processed. + +## Prerequisites + +- **GPU** with >8 GB VRAM for SAM3 (native 1008px; `DEVICE` auto-selects `cuda:0`). +- **`ffmpeg`** on `PATH` (frame extraction). +- **SAM3 is a gated HuggingFace model** — accept the license on the SAM3 model page, create a token, + set `HF_TOKEN` in `.env`. +- **CVAT** deployed at `CVAT_URL` (see [`../datapipe_cvat/simple_project`](../datapipe_cvat/simple_project/README.md) + for a local Docker setup), a project created (`CVAT_PROJECT_ID`) with labels matching + `CVAT_BOX_LABEL` / `CVAT_POLYGON_LABEL` (defaults `person_box` / `person_mask`). + +## Get a video + +```bash +# built-in ~24h set of busy IN/JP/US city walks (720p), into $INPUT_VIDEO_DIR: +python scripts/fetch_video.py --height 720 +# or your own: +python scripts/fetch_video.py --dir videos "https://youtu.be/VIDEO_ID" +# or just a clip (needs ffmpeg): +python scripts/fetch_video.py --section 00:10:00-00:20:00 "https://youtu.be/VIDEO_ID" +``` + +`fetch_video.py` needs `yt-dlp` and, for many YouTube videos, a JS runtime to solve YouTube's +n-challenge — install **`deno`** (or `node`); the EJS solver script is auto-fetched via +`--remote-components ejs:github`. Videos that demand sign-in ("confirm you're not a bot") also need +browser cookies — the script passes `--cookies-from-browser chrome` by default (be logged into +YouTube in that browser; use `--cookies-from-browser ""` to disable). + +Internal-demo only — downloading violates YouTube ToS; do not redistribute the videos or the frames. +For license-clean footage use stock (Pexels/Mixkit) or a dataset clip (EPIC-KITCHENS / Ego4D). + +## Run + +```bash +cp .env.example .env # set DB_URL, HF_TOKEN, INPUT_VIDEO_DIR, CVAT_*, SAM_TEXT_PROMPT +uv sync +datapipe db create-all +datapipe run +``` + +Run a single stage: `datapipe step --labels stage=sample run`, `... stage=sam run`, +`... stage=cvat run`. + +## Live demo + +Drop a ~10s clip into `INPUT_VIDEO_DIR` and `datapipe run`: it is sampled, deduped, segmented, and a +new CVAT task with box+polygon pre-annotations appears within a few seconds. Change `SAM_TEXT_PROMPT` +(e.g. `person` → `car`) to show open-vocabulary segmentation without retraining. + +## Annotate in CVAT + +Open CVAT, find the task, review/fix the SAM pre-annotations, save, mark completed. Re-run the +pipeline to pull the reviewed annotations back into `image__annotations`. diff --git a/examples/robots_ego_cvat/__init__.py b/examples/robots_ego_cvat/__init__.py new file mode 100644 index 00000000..b764ee30 --- /dev/null +++ b/examples/robots_ego_cvat/__init__.py @@ -0,0 +1 @@ +"""Egocentric video -> frame sampling -> SAM3 -> CVAT annotation pipeline example.""" diff --git a/examples/robots_ego_cvat/app.py b/examples/robots_ego_cvat/app.py new file mode 100644 index 00000000..31d8f34e --- /dev/null +++ b/examples/robots_ego_cvat/app.py @@ -0,0 +1,114 @@ +from __future__ import annotations + +from dotenv import load_dotenv + +load_dotenv() + +from datapipe.compute import Catalog, DatapipeApp, Pipeline +from datapipe.datatable import DataStore +from datapipe.executor import ExecutorConfig +from datapipe.step.batch_generate import BatchGenerate +from datapipe.step.batch_transform import BatchTransform +from datapipe_cvat.cvat_step import CVATStep + +import data +import steps +from config import ( + CVAT_ORGANIZATION, + CVAT_PASSWORD, + CVAT_PROJECT_ID, + CVAT_URL, + CVAT_USERNAME, + DBCONN, + FILES_BATCH, + PRIMARY_KEYS, +) + +pipeline = Pipeline( + [ + # --- video -> frames -------------------------------------------------------------------- + BatchGenerate( + steps.list_videos, + outputs=[data.video_tbl], + labels=[("stage", "video")], + ), + BatchTransform( + func=steps.extract_frames, + inputs=[data.video_tbl], + outputs=[data.frames_tbl], + transform_keys=["video_id"], + chunk_size=1, + labels=[("stage", "sample")], + ), + BatchTransform( + func=steps.dedup_frames, + inputs=[data.frames_tbl], + outputs=[data.local_images_tbl], + transform_keys=["video_id"], + chunk_size=1, + labels=[("stage", "sample")], + ), + # --- SAM3 text-prompt inference --------------------------------------------------------- + BatchGenerate( + steps.list_sam_config, + outputs=[data.sam_config_tbl], + labels=[("stage", "ingest")], + ), + BatchTransform( + func=steps.sam_inference, + inputs=[data.local_images_tbl, data.sam_config_tbl], + outputs=[data.sam_predictions_tbl], + transform_keys=["image_id", "config_id"], + chunk_size=1, + labels=[("stage", "sam")], + executor_config=ExecutorConfig(parallelism=0), + ), + BatchTransform( + func=steps.sam_to_cvat_xml, + inputs=[data.sam_predictions_tbl], + outputs=[data.sam_cvat_xml_tbl], + transform_keys=["image_id"], + labels=[("stage", "sam")], + ), + # --- CVAT upload + sync-back ------------------------------------------------------------ + BatchTransform( + func=steps.prepare_cvat_input, + inputs=[data.local_images_tbl, data.sam_cvat_xml_tbl], + outputs=[data.image_tbl], + transform_keys=["image_id"], + labels=[("stage", "cvat")], + ), + CVATStep( + input=data.image_tbl, + output__input_batches="image_batches", + output__cvat_task="cvat_task", + output__cvat_files="cvat_images", + task_sync_table="cvat_task_sync_table", + output__cvat_annotation="cvat_annotation", + file_path_column="image_path", + labels=[("stage", "cvat")], + minimum_files_in_job=1, + files_batch=FILES_BATCH, + cvat_url=CVAT_URL, + cvat_credentials=(CVAT_USERNAME, CVAT_PASSWORD), + cvat_project_id=CVAT_PROJECT_ID, + cvat_organization=CVAT_ORGANIZATION, + primary_keys=PRIMARY_KEYS, + cloud_storage_bucket=None, + delete_unannotated_tasks_only_on_update=False, + task_queue_id__name="task_queue_id", + task_name_format="[{date:%Y-%m-%d}] TaskQueue:{task_queue_id} batch:{inner_task_id}", + create_table=True, + ), + BatchTransform( + func=steps.parse_cvat_annotations, + inputs=["cvat_annotation"], + outputs=[data.image_annotations_tbl], + transform_keys=["image_id", "task_queue_id", "inner_task_id"], + labels=[("stage", "cvat")], + ), + ] +) + +ds = DataStore(DBCONN, create_meta_table=True) +app = DatapipeApp(ds, Catalog({}), pipeline) diff --git a/examples/robots_ego_cvat/config.py b/examples/robots_ego_cvat/config.py new file mode 100644 index 00000000..be5069d2 --- /dev/null +++ b/examples/robots_ego_cvat/config.py @@ -0,0 +1,64 @@ +from __future__ import annotations + +import os +from pathlib import Path + +import torch +from datapipe.store.database import DBConn + +# --- video ingest / frame sampling --------------------------------------------------------------- + +# Folder you drop long egocentric videos into. Every video in here is processed. +_INPUT_VIDEO_DIR_RAW = os.environ.get("INPUT_VIDEO_DIR") +INPUT_VIDEO_DIR = Path(_INPUT_VIDEO_DIR_RAW).resolve() if _INPUT_VIDEO_DIR_RAW else None +VIDEO_SUFFIXES = {".mp4", ".mkv", ".mov", ".webm", ".m4v"} + +# Where extracted frames are written (one subfolder per video_id). +_FRAMES_DIR_RAW = os.environ.get("FRAMES_DIR") +FRAMES_DIR = ( + Path(_FRAMES_DIR_RAW).resolve() + if _FRAMES_DIR_RAW + else Path(__file__).resolve().parent / ".frames" +) + +# ffmpeg extraction rate. 1 fps gives good event coverage; 1/3fps is not meaningfully different for +# walking POV footage (see README). Dedup below removes the near-duplicates either way. +SAMPLE_FPS = float(os.environ.get("SAMPLE_FPS", "1")) + +# Perceptual-hash near-duplicate threshold (Hamming distance between consecutive frames). A frame is +# kept only if it differs from the last kept frame by more than this. Higher = more aggressive dedup. +PHASH_MAX_DISTANCE = int(os.environ.get("PHASH_MAX_DISTANCE", "10")) +PHASH_SIZE = int(os.environ.get("PHASH_SIZE", "8")) # phash hash_size (bits per side) + +IMAGE_SUFFIXES = {".jpg", ".jpeg", ".png"} + +# --- SAM3 ----------------------------------------------------------------------------------------- + +DEVICE = "cuda:0" if torch.cuda.is_available() else "cpu" + +HF_TOKEN = os.environ.get("HF_TOKEN", "") +SAM_TEXT_PROMPT = os.environ.get("SAM_TEXT_PROMPT", "person") +SAM_SCORE_THRESHOLD = float(os.environ.get("SAM_SCORE_THRESHOLD", "0.5")) +SAM_MAX_DETECTIONS = int(os.environ.get("SAM_MAX_DETECTIONS", "20")) + +# --- CVAT ----------------------------------------------------------------------------------------- + +TASK_QUEUE_ID = os.environ.get("TASK_QUEUE_ID", "queue1") +FILES_BATCH = int(os.environ.get("FILES_BATCH", "500")) + +CVAT_URL = os.environ.get("CVAT_URL", "http://localhost:8080") +CVAT_USERNAME = os.environ.get("CVAT_USERNAME", "admin") +CVAT_PASSWORD = os.environ.get("CVAT_PASSWORD", "admin") +CVAT_PROJECT_ID = int(os.environ.get("CVAT_PROJECT_ID", "1")) +CVAT_ORGANIZATION = os.environ.get("CVAT_ORGANIZATION", "") +CVAT_PROJECT_NAME = os.environ.get("CVAT_PROJECT_NAME", "datapipe-robots-ego-cvat") + +CVAT_BOX_LABEL = os.environ.get("CVAT_BOX_LABEL", "person_box") +CVAT_POLYGON_LABEL = os.environ.get("CVAT_POLYGON_LABEL", "person_mask") + +PRIMARY_KEYS = ["image_id", "task_queue_id"] + +# --- datapipe ------------------------------------------------------------------------------------- + +DB_URL = os.environ.get("DB_URL") +DBCONN = DBConn(DB_URL, None) diff --git a/examples/robots_ego_cvat/data.py b/examples/robots_ego_cvat/data.py new file mode 100644 index 00000000..23ddf552 --- /dev/null +++ b/examples/robots_ego_cvat/data.py @@ -0,0 +1,132 @@ +from __future__ import annotations + +from datapipe.compute import Table +from datapipe.store.database import TableStoreDB +from sqlalchemy import Column, Float, Integer, JSON, String + +from config import DBCONN + +# One row per video file found in INPUT_VIDEO_DIR. +video_tbl = Table( + name="video", + store=TableStoreDB( + dbconn=DBCONN, + name="video", + data_sql_schema=[ + Column("video_id", String, primary_key=True), + Column("video_path", String), + ], + create_table=True, + ), +) + +# One row per frame extracted from a video (before dedup). video_id is part of the PK so the +# per-video extract/dedup transforms (transform_keys=["video_id"]) can explode/reduce cleanly. +frames_tbl = Table( + name="frames", + store=TableStoreDB( + dbconn=DBCONN, + name="frames", + data_sql_schema=[ + Column("video_id", String, primary_key=True), + Column("frame_id", String, primary_key=True), + Column("ts_sec", Float), + Column("frame_path", String), + ], + create_table=True, + ), +) + +# Survivors of perceptual-hash dedup. image_id feeds the SAM->CVAT tail; video_id stays in the PK so +# dedup (grouped per video) can delete+reinsert a video's survivors, and is reduced away downstream. +local_images_tbl = Table( + name="local_images", + store=TableStoreDB( + dbconn=DBCONN, + name="local_images", + data_sql_schema=[ + Column("video_id", String, primary_key=True), + Column("image_id", String, primary_key=True), + Column("image_path", String), + ], + create_table=True, + ), +) + +sam_config_tbl = Table( + name="sam_config", + store=TableStoreDB( + dbconn=DBCONN, + name="sam_config", + data_sql_schema=[ + Column("config_id", String, primary_key=True), + Column("text_prompt", String), + ], + create_table=True, + ), +) + +sam_predictions_tbl = Table( + name="sam_predictions", + store=TableStoreDB( + dbconn=DBCONN, + name="sam_predictions", + data_sql_schema=[ + Column("image_id", String, primary_key=True), + Column("detection_id", String, primary_key=True), + Column("score", Float), + Column("x_min", Float), + Column("y_min", Float), + Column("x_max", Float), + Column("y_max", Float), + Column("polygon_points", JSON), + ], + create_table=True, + ), +) + +sam_cvat_xml_tbl = Table( + name="sam_cvat_xml", + store=TableStoreDB( + dbconn=DBCONN, + name="sam_cvat_xml", + data_sql_schema=[ + Column("image_id", String, primary_key=True), + Column("annotations", String), + ], + create_table=True, + ), +) + +image_tbl = Table( + name="image", + store=TableStoreDB( + dbconn=DBCONN, + name="image", + data_sql_schema=[ + Column("image_id", String, primary_key=True), + Column("task_queue_id", String, primary_key=True), + Column("image_path", String), + Column("annotations", String), + ], + create_table=True, + ), +) + +image_annotations_tbl = Table( + name="image__annotations", + store=TableStoreDB( + dbconn=DBCONN, + name="image__annotations", + data_sql_schema=[ + Column("image_id", String, primary_key=True), + Column("task_queue_id", String, primary_key=True), + Column("inner_task_id", Integer, primary_key=True), + Column("boxes", JSON), + Column("polygons", JSON), + Column("box_labels", JSON), + Column("polygon_labels", JSON), + ], + create_table=True, + ), +) diff --git a/examples/robots_ego_cvat/models.py b/examples/robots_ego_cvat/models.py new file mode 100644 index 00000000..f5465e4a --- /dev/null +++ b/examples/robots_ego_cvat/models.py @@ -0,0 +1,120 @@ +from __future__ import annotations + +import logging +from typing import List, Optional + +import cv2 +import numpy as np +import torch +from cv_pipeliner import BboxData +from PIL import Image + +from config import DEVICE, HF_TOKEN, SAM_MAX_DETECTIONS, SAM_SCORE_THRESHOLD + +logger = logging.getLogger(__name__) + +_processor = None + + +def ensure_hf_login() -> None: + from huggingface_hub import login + + if HF_TOKEN: + login(token=HF_TOKEN) + return + login() + + +def get_processor(): + global _processor + if _processor is None: + ensure_hf_login() + from sam3.model.sam3_image_processor import Sam3Processor + from sam3.model_builder import build_sam3_image_model + + model = build_sam3_image_model() + _processor = Sam3Processor(model) + logger.info("Loaded SAM3 model on device %s", DEVICE) + return _processor + + +def _mask_to_polygon(mask: np.ndarray) -> Optional[np.ndarray]: + mask_uint8 = (mask.astype(np.uint8) * 255) if mask.max() <= 1 else mask.astype(np.uint8) + contours, _ = cv2.findContours(mask_uint8, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE) + if not contours: + return None + + contour = max(contours, key=cv2.contourArea) + epsilon = 0.005 * cv2.arcLength(contour, True) + approx = cv2.approxPolyDP(contour, epsilon, True) + if approx.shape[0] < 3: + return None + return approx.reshape(-1, 2).astype(np.int32) + + +def _to_numpy(value) -> np.ndarray: + return np.asarray(value.detach().cpu().float().numpy()) + + +def _to_scalar(value) -> float: + array = _to_numpy(value) + return float(array.reshape(-1)[0]) + + +def infer_image(image: Image.Image, text_prompt: str) -> List[BboxData]: + processor = get_processor() + device_type = "cuda" if DEVICE.startswith("cuda") else "cpu" + + if device_type == "cuda": + with torch.autocast(device_type, dtype=torch.bfloat16): + inference_state = processor.set_image(image) + output = processor.set_text_prompt(state=inference_state, prompt=text_prompt) + else: + inference_state = processor.set_image(image) + output = processor.set_text_prompt(state=inference_state, prompt=text_prompt) + + masks = output.get("masks", []) + boxes = output.get("boxes", []) + scores = output.get("scores", []) + + if masks is None or boxes is None or scores is None: + return [] + + num_detections = min(len(masks), len(boxes), len(scores)) + detections: List[BboxData] = [] + + indexed_scores = [(idx, _to_scalar(scores[idx])) for idx in range(num_detections)] + indexed_scores.sort(key=lambda item: item[1], reverse=True) + + kept = 0 + for rank, (idx, score) in enumerate(indexed_scores): + if score < SAM_SCORE_THRESHOLD: + continue + if kept >= SAM_MAX_DETECTIONS: + break + + box = _to_numpy(boxes[idx]).reshape(-1) + if box.size < 4: + continue + + x_min, y_min, x_max, y_max = [float(v) for v in box[:4]] + mask = _to_numpy(masks[idx]) + if mask.ndim == 3: + mask = mask[0] + polygon = _mask_to_polygon(mask) + mask_polygons = [polygon] if polygon is not None else [] + + detections.append( + BboxData( + xmin=x_min, + ymin=y_min, + xmax=x_max, + ymax=y_max, + detection_score=score, + mask=mask_polygons, + additional_info={"detection_id": str(rank)}, + ) + ) + kept += 1 + + return detections diff --git a/examples/robots_ego_cvat/pyproject.toml b/examples/robots_ego_cvat/pyproject.toml new file mode 100644 index 00000000..ec1875e8 --- /dev/null +++ b/examples/robots_ego_cvat/pyproject.toml @@ -0,0 +1,33 @@ +[project] +name = "robots-ego-cvat-example" +version = "0" +requires-python = ">=3.10,<3.13" +dependencies = [ + "torch==2.6.0", + "torchvision==0.21.0", + "setuptools<82", + "sam3", + "datapipe-core", + "datapipe-cvat", + "python-dotenv", + "opencv-python", + "huggingface-hub==1.20.1", + "cv-pipeliner", + "einops", + "pycocotools", + "imagehash", + "psutil" +] + +[[tool.uv.index]] +name = "pytorch-cu124" +url = "https://download.pytorch.org/whl/cu124" +explicit = true + +[tool.uv.sources] +torch = { index = "pytorch-cu124" } +torchvision = { index = "pytorch-cu124" } +datapipe-core = { path = "../../libs/datapipe-core", editable = true } +datapipe-cvat = { path = "../../libs/datapipe-cvat", editable = true } +cv-pipeliner = { git = "https://github.com/epoch8/cv-pipeliner", rev = "5724f8d54e4df64013fad85d41129799bc143293" } +sam3 = { git = "https://github.com/facebookresearch/sam3.git", rev = "5dd401d1c5c1d5c3eedff06d41b77af824517619" } diff --git a/examples/robots_ego_cvat/scripts/fetch_video.py b/examples/robots_ego_cvat/scripts/fetch_video.py new file mode 100644 index 00000000..ab76eb77 --- /dev/null +++ b/examples/robots_ego_cvat/scripts/fetch_video.py @@ -0,0 +1,74 @@ +#!/usr/bin/env python3 +"""Download egocentric source videos into INPUT_VIDEO_DIR with yt-dlp. + +Internal-demo use only: downloading violates YouTube ToS, so do not redistribute the videos or the +frames sampled from them. For license-clean footage use stock (Pexels/Mixkit) or a dataset clip. + +Usage: + python scripts/fetch_video.py # the default ~24h walk set (IN/JP/US), 720p + python scripts/fetch_video.py --height 480 # smaller download + python scripts/fetch_video.py URL [URL ...] # your own videos + python scripts/fetch_video.py --section 00:10:00-00:20:00 URL # a clip only (needs ffmpeg) +""" +from __future__ import annotations + +import argparse +import os +import subprocess +import sys +from pathlib import Path + +# Verified (yt-dlp) ~24h of busy first-person city walks: Japan 8h, USA 8h, India ~8h (5 clips). +DEFAULT_VIDEOS = [ + "https://youtu.be/BsiHD4m6_BU", # Tokyo, 9 districts, 8:06:59, 4K + "https://youtu.be/27Pv4Cg4EV4", # New York full city walk, 8:04:31, 4K + "https://youtu.be/60Q5E0KZb38", # Mumbai markets, 2:41:51, 4K + "https://youtu.be/qskdzPj39hE", # New Delhi Paharganj, 1:57:15, 4K + "https://youtu.be/8W4ZTX1z02E", # Mumbai busy streets, 1:36:11, 4K + "https://youtu.be/7wBNtsgqNOI", # New Delhi crowds, 0:58:59, 4K + "https://youtu.be/Lteooc0BHtk", # New Delhi streets, 0:39:41, 4K +] + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + parser.add_argument("urls", nargs="*", help="video URLs (default: the built-in ~24h walk set)") + parser.add_argument("--dir", default=os.environ.get("INPUT_VIDEO_DIR"), help="target dir (default: $INPUT_VIDEO_DIR)") + parser.add_argument("--height", type=int, default=720, help="max video height, e.g. 480/720/1080 (default 720)") + parser.add_argument("--section", default=None, help="download only a section, e.g. 00:10:00-00:20:00") + parser.add_argument("--cookies-from-browser", default="chrome", + help="browser to read cookies from (chrome/safari/firefox/...); '' to disable") + args = parser.parse_args() + + if not args.dir: + parser.error("set --dir or the INPUT_VIDEO_DIR env var") + out_dir = Path(args.dir).resolve() + out_dir.mkdir(parents=True, exist_ok=True) + + urls = args.urls or DEFAULT_VIDEOS + fmt = f"bv*[height<={args.height}]+ba/b[height<={args.height}]/b[height<={args.height}]" + cmd = [ + "yt-dlp", + "-N", "4", + "--retries", "infinite", + "--fragment-retries", "infinite", + "--sleep-interval", "3", "--max-sleep-interval", "12", + # Some videos require sign-in ("confirm you're not a bot") -> pass browser cookies. Solving + # YouTube's JS n-challenge needs a JS runtime (install `deno` or `node`) plus the EJS solver + # script, fetched by --remote-components; without it only storyboard images are returned. + "--remote-components", "ejs:github", + "-f", fmt, + "-o", str(out_dir / "%(id)s.%(ext)s"), + ] + if args.cookies_from_browser: + cmd += ["--cookies-from-browser", args.cookies_from_browser] + if args.section: + cmd += ["--download-sections", f"*{args.section}", "--force-keyframes-at-cuts"] + cmd += urls + + print(f"Downloading {len(urls)} video(s) -> {out_dir} (<= {args.height}p)", file=sys.stderr) + return subprocess.run(cmd).returncode + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/examples/robots_ego_cvat/steps.py b/examples/robots_ego_cvat/steps.py new file mode 100644 index 00000000..41f5071f --- /dev/null +++ b/examples/robots_ego_cvat/steps.py @@ -0,0 +1,313 @@ +from __future__ import annotations + +import subprocess +import xml.etree.ElementTree as ET +from pathlib import Path +from typing import Iterator + +import imagehash +import pandas as pd +from cv_pipeliner import BboxData +from PIL import Image + +from config import ( + CVAT_BOX_LABEL, + CVAT_POLYGON_LABEL, + FRAMES_DIR, + INPUT_VIDEO_DIR, + PHASH_MAX_DISTANCE, + PHASH_SIZE, + SAM_TEXT_PROMPT, + SAMPLE_FPS, + TASK_QUEUE_ID, + VIDEO_SUFFIXES, +) +from models import infer_image + +SAM_CONFIG_ID = "default" + +_LOCAL_IMAGES_COLUMNS = ["video_id", "image_id", "image_path"] +_FRAMES_COLUMNS = ["video_id", "frame_id", "ts_sec", "frame_path"] +_SAM_PRED_COLUMNS = [ + "image_id", + "detection_id", + "score", + "x_min", + "y_min", + "x_max", + "y_max", + "polygon_points", +] + + +# --- video -> frames ----------------------------------------------------------------------------- + + +def list_videos() -> Iterator[pd.DataFrame]: + records: list[dict[str, str]] = [] + if INPUT_VIDEO_DIR is not None and INPUT_VIDEO_DIR.exists(): + for path in sorted(INPUT_VIDEO_DIR.iterdir()): + if path.is_file() and path.suffix.lower() in VIDEO_SUFFIXES: + records.append({"video_id": path.stem, "video_path": str(path)}) + + if records: + yield pd.DataFrame(records) + else: + yield pd.DataFrame(columns=["video_id", "video_path"]) + + +def _extract_one_video(video_id: str, video_path: str) -> list[dict]: + out_dir = FRAMES_DIR / video_id + out_dir.mkdir(parents=True, exist_ok=True) + + existing = sorted(out_dir.glob("f_*.jpg")) + if not existing: + # -vf fps=N samples N frames per second; VFR keeps timestamps honest. + cmd = [ + "ffmpeg", + "-hide_banner", + "-loglevel", + "error", + "-i", + video_path, + "-vf", + f"fps={SAMPLE_FPS}", + "-q:v", + "2", + str(out_dir / "f_%06d.jpg"), + ] + subprocess.run(cmd, check=True) + existing = sorted(out_dir.glob("f_*.jpg")) + + records = [] + for path in existing: + n = int(path.stem.split("_")[-1]) # f_000123 -> 123 (1-based) + records.append( + { + "frame_id": f"{video_id}_{n:06d}", + "video_id": video_id, + "ts_sec": (n - 1) / SAMPLE_FPS, + "frame_path": str(path), + } + ) + return records + + +def extract_frames(df_video: pd.DataFrame) -> pd.DataFrame: + if df_video.empty: + return pd.DataFrame(columns=_FRAMES_COLUMNS) + + records: list[dict] = [] + for _, row in df_video.iterrows(): + records.extend(_extract_one_video(str(row["video_id"]), str(row["video_path"]))) + + if not records: + return pd.DataFrame(columns=_FRAMES_COLUMNS) + return pd.DataFrame(records, columns=_FRAMES_COLUMNS) + + +def dedup_frames(df_frames: pd.DataFrame) -> pd.DataFrame: + """Perceptual-hash dedup per video: keep a frame only if it differs from the last kept frame by + more than PHASH_MAX_DISTANCE (Hamming). Walking POV frames are highly redundant, so a sequential + compare-to-last-kept pass removes near-duplicates cheaply while preserving genuine scene change.""" + if df_frames.empty: + return pd.DataFrame(columns=_LOCAL_IMAGES_COLUMNS) + + records: list[dict] = [] + for video_id, group in df_frames.groupby("video_id"): + ordered = group.sort_values(["ts_sec", "frame_id"]) + last_hash = None + for _, row in ordered.iterrows(): + try: + with Image.open(row["frame_path"]) as img: + current = imagehash.phash(img.convert("RGB"), hash_size=PHASH_SIZE) + except (OSError, ValueError): + continue + if last_hash is not None and (current - last_hash) <= PHASH_MAX_DISTANCE: + continue + last_hash = current + records.append( + { + "video_id": str(video_id), + "image_id": row["frame_id"], + "image_path": row["frame_path"], + } + ) + + if not records: + return pd.DataFrame(columns=_LOCAL_IMAGES_COLUMNS) + return pd.DataFrame(records, columns=_LOCAL_IMAGES_COLUMNS) + + +# --- SAM config + inference (reused from sam_cvat) ----------------------------------------------- + + +def list_sam_config() -> Iterator[pd.DataFrame]: + yield pd.DataFrame([{"config_id": SAM_CONFIG_ID, "text_prompt": SAM_TEXT_PROMPT}]) + + +def _polygon_points_from_bbox(bbox: BboxData) -> list[list[float]] | None: + if not isinstance(bbox.mask, list) or not bbox.mask: + return None + polygon = max(bbox.mask, key=lambda points: len(points)).reshape(-1, 2) + if len(polygon) < 3: + return None + return [[float(x), float(y)] for x, y in polygon] + + +def sam_inference(df_local_images: pd.DataFrame, df_sam_config: pd.DataFrame) -> pd.DataFrame: + if df_local_images.empty: + return pd.DataFrame(columns=_SAM_PRED_COLUMNS) + + if df_sam_config.empty: + text_prompt = SAM_TEXT_PROMPT + else: + text_prompt = str(df_sam_config.iloc[0]["text_prompt"]) + + records = [] + for _, row in df_local_images.iterrows(): + image = Image.open(row["image_path"]).convert("RGB") + detections = infer_image(image, text_prompt) + for detection in detections: + records.append( + { + "image_id": row["image_id"], + "detection_id": detection.additional_info.get("detection_id", "0"), + "score": detection.detection_score, + "x_min": detection.xmin, + "y_min": detection.ymin, + "x_max": detection.xmax, + "y_max": detection.ymax, + "polygon_points": _polygon_points_from_bbox(detection), + } + ) + + if not records: + return pd.DataFrame(columns=_SAM_PRED_COLUMNS) + return pd.DataFrame(records) + + +# --- SAM predictions -> CVAT XML -> CVAT input (reused from sam_cvat) ----------------------------- + + +def _polygon_points_to_cvat_attr(points: list[list[float]]) -> str: + return ";".join(f"{x:.2f},{y:.2f}" for x, y in points) + + +def sam_to_cvat_xml(df_sam_predictions: pd.DataFrame) -> pd.DataFrame: + if df_sam_predictions.empty: + return pd.DataFrame(columns=["image_id", "annotations"]) + + records = [] + for image_id, group in df_sam_predictions.groupby("image_id"): + lines = [""] + for _, row in group.iterrows(): + lines.append( + f' ' + ) + lines.append(" ") + + polygon_points = row.get("polygon_points") + if isinstance(polygon_points, list) and len(polygon_points) >= 3: + points_attr = _polygon_points_to_cvat_attr(polygon_points) + lines.append( + f' ' + ) + lines.append(" ") + lines.append("") + records.append({"image_id": image_id, "annotations": "\n".join(lines)}) + + return pd.DataFrame(records, columns=["image_id", "annotations"]) + + +def prepare_cvat_input(df_local_images: pd.DataFrame, df_sam_cvat_xml: pd.DataFrame) -> pd.DataFrame: + if df_local_images.empty: + return pd.DataFrame(columns=["image_id", "task_queue_id", "image_path", "annotations"]) + + df = pd.merge(df_local_images, df_sam_cvat_xml, on="image_id", how="left") + df["task_queue_id"] = TASK_QUEUE_ID + df["image_path"] = df["image_path"].apply(lambda path: str(Path(path))) + df["annotations"] = df["annotations"].fillna("") + return df[["image_id", "task_queue_id", "image_path", "annotations"]] + + +# --- CVAT annotations -> datapipe (reused from sam_cvat) ------------------------------------------ + + +def _parse_points_attr(points_attr: str) -> list[list[float]]: + points: list[list[float]] = [] + for pair in points_attr.split(";"): + if not pair.strip(): + continue + x_str, y_str = pair.split(",", 1) + points.append([float(x_str), float(y_str)]) + return points + + +def parse_cvat_annotations(df_cvat_annotation: pd.DataFrame) -> pd.DataFrame: + columns = [ + "image_id", + "task_queue_id", + "inner_task_id", + "boxes", + "polygons", + "box_labels", + "polygon_labels", + ] + if df_cvat_annotation.empty: + return pd.DataFrame(columns=columns) + + records = [] + for _, row in df_cvat_annotation.iterrows(): + annotation_xml = row.get("annotations") or "" + if not annotation_xml.strip(): + continue + + try: + image_element = ET.fromstring(annotation_xml) + except ET.ParseError: + continue + + boxes = [] + box_labels = [] + for box in image_element.findall("box"): + boxes.append( + { + "xtl": float(box.attrib.get("xtl", 0)), + "ytl": float(box.attrib.get("ytl", 0)), + "xbr": float(box.attrib.get("xbr", 0)), + "ybr": float(box.attrib.get("ybr", 0)), + } + ) + box_labels.append(box.attrib.get("label", "")) + + polygons = [] + polygon_labels = [] + for polygon in image_element.findall("polygon"): + points_attr = polygon.attrib.get("points", "") + if not points_attr: + continue + polygons.append(_parse_points_attr(points_attr)) + polygon_labels.append(polygon.attrib.get("label", "")) + + if not boxes and not polygons: + continue + + records.append( + { + "image_id": row["image_id"], + "task_queue_id": row["task_queue_id"], + "inner_task_id": int(row["inner_task_id"]), + "boxes": boxes, + "polygons": polygons, + "box_labels": box_labels, + "polygon_labels": polygon_labels, + } + ) + + if not records: + return pd.DataFrame(columns=columns) + return pd.DataFrame(records) From 22a7175bf16cfaeee3036445cf86e8150fe76914 Mon Sep 17 00:00:00 2001 From: Shakirov Renat Date: Thu, 23 Jul 2026 10:47:09 +0300 Subject: [PATCH 02/13] robots_ego_cvat: resolve ffmpeg via imageio-ffmpeg fallback (run without system ffmpeg) --- examples/robots_ego_cvat/config.py | 19 +++++++++++++++++++ examples/robots_ego_cvat/pyproject.toml | 1 + examples/robots_ego_cvat/steps.py | 3 ++- 3 files changed, 22 insertions(+), 1 deletion(-) diff --git a/examples/robots_ego_cvat/config.py b/examples/robots_ego_cvat/config.py index be5069d2..ccb8fddd 100644 --- a/examples/robots_ego_cvat/config.py +++ b/examples/robots_ego_cvat/config.py @@ -25,6 +25,25 @@ # walking POV footage (see README). Dedup below removes the near-duplicates either way. SAMPLE_FPS = float(os.environ.get("SAMPLE_FPS", "1")) + +def _resolve_ffmpeg() -> str: + # Prefer a system ffmpeg; fall back to the binary bundled by the imageio-ffmpeg package so the + # example runs on hosts without ffmpeg on PATH. + import shutil + + exe = os.environ.get("FFMPEG_BIN") or shutil.which("ffmpeg") + if exe: + return exe + try: + import imageio_ffmpeg + + return imageio_ffmpeg.get_ffmpeg_exe() + except Exception: + return "ffmpeg" + + +FFMPEG_BIN = _resolve_ffmpeg() + # Perceptual-hash near-duplicate threshold (Hamming distance between consecutive frames). A frame is # kept only if it differs from the last kept frame by more than this. Higher = more aggressive dedup. PHASH_MAX_DISTANCE = int(os.environ.get("PHASH_MAX_DISTANCE", "10")) diff --git a/examples/robots_ego_cvat/pyproject.toml b/examples/robots_ego_cvat/pyproject.toml index ec1875e8..367b6956 100644 --- a/examples/robots_ego_cvat/pyproject.toml +++ b/examples/robots_ego_cvat/pyproject.toml @@ -16,6 +16,7 @@ dependencies = [ "einops", "pycocotools", "imagehash", + "imageio-ffmpeg", "psutil" ] diff --git a/examples/robots_ego_cvat/steps.py b/examples/robots_ego_cvat/steps.py index 41f5071f..d865adc1 100644 --- a/examples/robots_ego_cvat/steps.py +++ b/examples/robots_ego_cvat/steps.py @@ -13,6 +13,7 @@ from config import ( CVAT_BOX_LABEL, CVAT_POLYGON_LABEL, + FFMPEG_BIN, FRAMES_DIR, INPUT_VIDEO_DIR, PHASH_MAX_DISTANCE, @@ -64,7 +65,7 @@ def _extract_one_video(video_id: str, video_path: str) -> list[dict]: if not existing: # -vf fps=N samples N frames per second; VFR keeps timestamps honest. cmd = [ - "ffmpeg", + FFMPEG_BIN, "-hide_banner", "-loglevel", "error", From 81ae3a5e8e6028d3557a06725b145665d5e1d1b9 Mon Sep 17 00:00:00 2001 From: Shakirov Renat Date: Thu, 23 Jul 2026 13:37:57 +0300 Subject: [PATCH 03/13] CVATStep: optional segment_size (CVAT jobs per task); robots_ego_cvat: one task per video split into jobs of N frames Adds a backward-compatible segment_size param to CVATStep, passed to the CVAT TaskWriteRequest so a task is split into jobs of N frames. robots_ego_cvat scopes the CVAT batch by video_id (one task per video), sets SEGMENT_SIZE (jobs) and a large FILES_BATCH, and defaults SAMPLE_FPS to 0.2 (1 frame/5s). --- examples/robots_ego_cvat/app.py | 5 ++++- examples/robots_ego_cvat/config.py | 9 ++++++++- examples/robots_ego_cvat/steps.py | 3 ++- libs/datapipe-cvat/datapipe_cvat/cvat_step.py | 6 ++++++ 4 files changed, 20 insertions(+), 3 deletions(-) diff --git a/examples/robots_ego_cvat/app.py b/examples/robots_ego_cvat/app.py index 31d8f34e..ddd1c0e1 100644 --- a/examples/robots_ego_cvat/app.py +++ b/examples/robots_ego_cvat/app.py @@ -21,7 +21,9 @@ CVAT_USERNAME, DBCONN, FILES_BATCH, + MIN_FILES_IN_JOB, PRIMARY_KEYS, + SEGMENT_SIZE, ) pipeline = Pipeline( @@ -87,8 +89,9 @@ output__cvat_annotation="cvat_annotation", file_path_column="image_path", labels=[("stage", "cvat")], - minimum_files_in_job=1, + minimum_files_in_job=MIN_FILES_IN_JOB, files_batch=FILES_BATCH, + segment_size=SEGMENT_SIZE, cvat_url=CVAT_URL, cvat_credentials=(CVAT_USERNAME, CVAT_PASSWORD), cvat_project_id=CVAT_PROJECT_ID, diff --git a/examples/robots_ego_cvat/config.py b/examples/robots_ego_cvat/config.py index ccb8fddd..6e181c1f 100644 --- a/examples/robots_ego_cvat/config.py +++ b/examples/robots_ego_cvat/config.py @@ -63,7 +63,14 @@ def _resolve_ffmpeg() -> str: # --- CVAT ----------------------------------------------------------------------------------------- TASK_QUEUE_ID = os.environ.get("TASK_QUEUE_ID", "queue1") -FILES_BATCH = int(os.environ.get("FILES_BATCH", "500")) +# One CVAT task per video: task_queue_id is set to video_id (see steps.prepare_cvat_input), and +# FILES_BATCH is large so all of a video's frames land in a single task. +FILES_BATCH = int(os.environ.get("FILES_BATCH", "100000")) +# Minimum frames to open a task — 1 so short videos still get a task. +MIN_FILES_IN_JOB = int(os.environ.get("MIN_FILES_IN_JOB", "1")) +# CVAT jobs per task: each task is split into jobs of this many frames (None = single job). +_SEGMENT_SIZE_RAW = os.environ.get("SEGMENT_SIZE", "200") +SEGMENT_SIZE = int(_SEGMENT_SIZE_RAW) if _SEGMENT_SIZE_RAW.strip() else None CVAT_URL = os.environ.get("CVAT_URL", "http://localhost:8080") CVAT_USERNAME = os.environ.get("CVAT_USERNAME", "admin") diff --git a/examples/robots_ego_cvat/steps.py b/examples/robots_ego_cvat/steps.py index d865adc1..bfc03c99 100644 --- a/examples/robots_ego_cvat/steps.py +++ b/examples/robots_ego_cvat/steps.py @@ -229,7 +229,8 @@ def prepare_cvat_input(df_local_images: pd.DataFrame, df_sam_cvat_xml: pd.DataFr return pd.DataFrame(columns=["image_id", "task_queue_id", "image_path", "annotations"]) df = pd.merge(df_local_images, df_sam_cvat_xml, on="image_id", how="left") - df["task_queue_id"] = TASK_QUEUE_ID + # One CVAT task per video: scope the CVAT batch by video_id (falls back to a constant if absent). + df["task_queue_id"] = df["video_id"] if "video_id" in df.columns else TASK_QUEUE_ID df["image_path"] = df["image_path"].apply(lambda path: str(Path(path))) df["annotations"] = df["annotations"].fillna("") return df[["image_id", "task_queue_id", "image_path", "annotations"]] diff --git a/libs/datapipe-cvat/datapipe_cvat/cvat_step.py b/libs/datapipe-cvat/datapipe_cvat/cvat_step.py index 5a98bc58..2074f1e2 100644 --- a/libs/datapipe-cvat/datapipe_cvat/cvat_step.py +++ b/libs/datapipe-cvat/datapipe_cvat/cvat_step.py @@ -444,6 +444,7 @@ def get_or_create_task( task_queue_id: Any, max_attempts: int, attempt_poll_s: int, + segment_size: Optional[int] = None, ) -> Tuple[Task, pd.DataFrame]: """ Creates a new task in CVAT or returns an existing one, and also associates it with images. @@ -509,6 +510,7 @@ def get_or_create_task( spec=TaskWriteRequest( name=new_task_name, project_id=project_id, + **({"segment_size": segment_size} if segment_size else {}), ), resources=resources, resource_type=resource_type, @@ -834,6 +836,7 @@ def upload_batches_to_cvat( task_name_format: str, max_attempts: int, attempt_poll_s: int, + segment_size: Optional[int] = None, failure_hook: Optional[CVATFailureHook] = None, ) -> Tuple[pd.DataFrame, pd.DataFrame]: """ @@ -1011,6 +1014,7 @@ def upload_batches_to_cvat( task_queue_id__name=task_queue_id__name, max_attempts=max_attempts, attempt_poll_s=attempt_poll_s, + segment_size=segment_size, ) task_record = { @@ -1140,6 +1144,7 @@ class CVATStep(PipelineStep): file_type: Literal["image", "video"] = "image" files_batch: Union[int, dict[Any, int]] = 100 minimum_files_in_job: Union[int, dict[Any, int]] = 50 + segment_size: Optional[int] = None # CVAT jobs per task: split each task into jobs of N frames task_queue_id__name: str = "task_queue_id" task_name_format: str = "[{date:%Y-%m-%d}] {task_queue_id} batch={inner_task_id}" sampling_order: Literal["default", "random"] = "default" @@ -1335,6 +1340,7 @@ def _mk(dt_name: str, schema: List[Column]): task_queue_id__name=self.task_queue_id__name, max_attempts=self.max_attempts, attempt_poll_s=self.attempt_poll_s, + segment_size=self.segment_size, ), executor_config=ExecutorConfig(parallelism=0), ), From b78a9ba165b1abc677c9c2a32ec460ec1e63fef5 Mon Sep 17 00:00:00 2001 From: Shakirov Renat Date: Thu, 23 Jul 2026 13:45:38 +0300 Subject: [PATCH 04/13] rename example robots_ego_cvat -> video_segmentation_cvat (video -> SAM3 segmentation -> CVAT) --- examples/{robots_ego_cvat => video_segmentation_cvat}/.gitignore | 0 examples/{robots_ego_cvat => video_segmentation_cvat}/README.md | 0 examples/{robots_ego_cvat => video_segmentation_cvat}/__init__.py | 0 examples/{robots_ego_cvat => video_segmentation_cvat}/app.py | 0 examples/{robots_ego_cvat => video_segmentation_cvat}/config.py | 0 examples/{robots_ego_cvat => video_segmentation_cvat}/data.py | 0 examples/{robots_ego_cvat => video_segmentation_cvat}/models.py | 0 .../{robots_ego_cvat => video_segmentation_cvat}/pyproject.toml | 0 .../scripts/fetch_video.py | 0 examples/{robots_ego_cvat => video_segmentation_cvat}/steps.py | 0 10 files changed, 0 insertions(+), 0 deletions(-) rename examples/{robots_ego_cvat => video_segmentation_cvat}/.gitignore (100%) rename examples/{robots_ego_cvat => video_segmentation_cvat}/README.md (100%) rename examples/{robots_ego_cvat => video_segmentation_cvat}/__init__.py (100%) rename examples/{robots_ego_cvat => video_segmentation_cvat}/app.py (100%) rename examples/{robots_ego_cvat => video_segmentation_cvat}/config.py (100%) rename examples/{robots_ego_cvat => video_segmentation_cvat}/data.py (100%) rename examples/{robots_ego_cvat => video_segmentation_cvat}/models.py (100%) rename examples/{robots_ego_cvat => video_segmentation_cvat}/pyproject.toml (100%) rename examples/{robots_ego_cvat => video_segmentation_cvat}/scripts/fetch_video.py (100%) rename examples/{robots_ego_cvat => video_segmentation_cvat}/steps.py (100%) diff --git a/examples/robots_ego_cvat/.gitignore b/examples/video_segmentation_cvat/.gitignore similarity index 100% rename from examples/robots_ego_cvat/.gitignore rename to examples/video_segmentation_cvat/.gitignore diff --git a/examples/robots_ego_cvat/README.md b/examples/video_segmentation_cvat/README.md similarity index 100% rename from examples/robots_ego_cvat/README.md rename to examples/video_segmentation_cvat/README.md diff --git a/examples/robots_ego_cvat/__init__.py b/examples/video_segmentation_cvat/__init__.py similarity index 100% rename from examples/robots_ego_cvat/__init__.py rename to examples/video_segmentation_cvat/__init__.py diff --git a/examples/robots_ego_cvat/app.py b/examples/video_segmentation_cvat/app.py similarity index 100% rename from examples/robots_ego_cvat/app.py rename to examples/video_segmentation_cvat/app.py diff --git a/examples/robots_ego_cvat/config.py b/examples/video_segmentation_cvat/config.py similarity index 100% rename from examples/robots_ego_cvat/config.py rename to examples/video_segmentation_cvat/config.py diff --git a/examples/robots_ego_cvat/data.py b/examples/video_segmentation_cvat/data.py similarity index 100% rename from examples/robots_ego_cvat/data.py rename to examples/video_segmentation_cvat/data.py diff --git a/examples/robots_ego_cvat/models.py b/examples/video_segmentation_cvat/models.py similarity index 100% rename from examples/robots_ego_cvat/models.py rename to examples/video_segmentation_cvat/models.py diff --git a/examples/robots_ego_cvat/pyproject.toml b/examples/video_segmentation_cvat/pyproject.toml similarity index 100% rename from examples/robots_ego_cvat/pyproject.toml rename to examples/video_segmentation_cvat/pyproject.toml diff --git a/examples/robots_ego_cvat/scripts/fetch_video.py b/examples/video_segmentation_cvat/scripts/fetch_video.py similarity index 100% rename from examples/robots_ego_cvat/scripts/fetch_video.py rename to examples/video_segmentation_cvat/scripts/fetch_video.py diff --git a/examples/robots_ego_cvat/steps.py b/examples/video_segmentation_cvat/steps.py similarity index 100% rename from examples/robots_ego_cvat/steps.py rename to examples/video_segmentation_cvat/steps.py From 7a01de2ab2cb89179c150de4ff5a0769bd0fc7d1 Mon Sep 17 00:00:00 2001 From: Shakirov Renat Date: Thu, 23 Jul 2026 14:01:29 +0300 Subject: [PATCH 05/13] video_segmentation_cvat: wire datapipe-app UI (DatapipeAPI) so the front renders the pipeline graph, table browser and per-stage run triggers --- examples/video_segmentation_cvat/app.py | 8 ++++++-- examples/video_segmentation_cvat/pyproject.toml | 4 ++++ 2 files changed, 10 insertions(+), 2 deletions(-) diff --git a/examples/video_segmentation_cvat/app.py b/examples/video_segmentation_cvat/app.py index ddd1c0e1..ad68f2ba 100644 --- a/examples/video_segmentation_cvat/app.py +++ b/examples/video_segmentation_cvat/app.py @@ -4,11 +4,12 @@ load_dotenv() -from datapipe.compute import Catalog, DatapipeApp, Pipeline +from datapipe.compute import Catalog, Pipeline from datapipe.datatable import DataStore from datapipe.executor import ExecutorConfig from datapipe.step.batch_generate import BatchGenerate from datapipe.step.batch_transform import BatchTransform +from datapipe_app import DatapipeAPI from datapipe_cvat.cvat_step import CVATStep import data @@ -114,4 +115,7 @@ ) ds = DataStore(DBCONN, create_meta_table=True) -app = DatapipeApp(ds, Catalog({}), pipeline) +# DatapipeAPI (not plain DatapipeApp) so the datapipe-app UI front (`datapipe --pipeline app api`) +# renders the pipeline graph, table browser, and per-stage run triggers. run_logs_backend defaults +# to None -> no ClickHouse needed. +app = DatapipeAPI(ds, Catalog({}), pipeline) diff --git a/examples/video_segmentation_cvat/pyproject.toml b/examples/video_segmentation_cvat/pyproject.toml index 367b6956..d9c618ac 100644 --- a/examples/video_segmentation_cvat/pyproject.toml +++ b/examples/video_segmentation_cvat/pyproject.toml @@ -9,6 +9,8 @@ dependencies = [ "sam3", "datapipe-core", "datapipe-cvat", + "datapipe-app[ui]", + "datapipe-ui", "python-dotenv", "opencv-python", "huggingface-hub==1.20.1", @@ -30,5 +32,7 @@ torch = { index = "pytorch-cu124" } torchvision = { index = "pytorch-cu124" } datapipe-core = { path = "../../libs/datapipe-core", editable = true } datapipe-cvat = { path = "../../libs/datapipe-cvat", editable = true } +datapipe-app = { path = "../../libs/datapipe-app", editable = true } +datapipe-ui = { path = "../../libs/datapipe-ui", editable = true } cv-pipeliner = { git = "https://github.com/epoch8/cv-pipeliner", rev = "5724f8d54e4df64013fad85d41129799bc143293" } sam3 = { git = "https://github.com/facebookresearch/sam3.git", rev = "5dd401d1c5c1d5c3eedff06d41b77af824517619" } From f28db1aaabef1b2e8e8af311a6fe7c0a1941ca39 Mon Sep 17 00:00:00 2001 From: Shakirov Renat Date: Thu, 23 Jul 2026 18:19:42 +0300 Subject: [PATCH 06/13] video_segmentation_cvat: fit SAM3 on tight GPUs, read-only-home fix, setup skill - infer_image: downscale frames to SAM_MAX_INFER_SIDE (default 640) before SAM3 and map detections back to full-res coords (CVAT still gets the full-res frame); on OOM retry at 512/384 and skip the frame as a last resort so one frame never kills the run; empty_cache between frames - config: set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True before importing torch so it applies to both CLI and UI-triggered runs; add SAM_MAX_INFER_SIDE knob - models: take HF_TOKEN from the env instead of persisting it to disk (works on read-only home dirs) - README: GPU-memory/OOM section, HF_HOME note, and a datapipe-api-vs-CLI run-log collision warning; ffmpeg now optional (imageio-ffmpeg fallback) - add setup-video-segmentation-cvat skill - commit the .env.example template (was gitignored, unlike sibling examples) --- .../setup-video-segmentation-cvat/SKILL.md | 112 ++++++++++++++++++ examples/video_segmentation_cvat/.env.example | 40 +++++++ examples/video_segmentation_cvat/README.md | 30 ++++- examples/video_segmentation_cvat/config.py | 9 ++ examples/video_segmentation_cvat/models.py | 91 ++++++++++++-- 5 files changed, 266 insertions(+), 16 deletions(-) create mode 100644 .claude/skills/setup-video-segmentation-cvat/SKILL.md create mode 100644 examples/video_segmentation_cvat/.env.example diff --git a/.claude/skills/setup-video-segmentation-cvat/SKILL.md b/.claude/skills/setup-video-segmentation-cvat/SKILL.md new file mode 100644 index 00000000..9ab071a4 --- /dev/null +++ b/.claude/skills/setup-video-segmentation-cvat/SKILL.md @@ -0,0 +1,112 @@ +--- +name: setup-video-segmentation-cvat +description: > + Use when working in examples/video_segmentation_cvat, or when setting up / running / debugging the + datapipe video → SAM3 → CVAT example (long egocentric video → sampled frames → text-prompt + segmentation → CVAT pre-annotations), on the built-in city-walk videos or your own footage. +--- + +# video_segmentation_cvat (video → SAM3 → CVAT) + +This skill = turn a long video into review-ready CVAT pre-annotations. Front stage samples + dedups +frames; the SAM3→CVAT tail is identical to [`../sam_cvat`](../sam_cvat). Set the knobs below first. + +**Ask first — don't assume (only the unresolved):** which videos (built-in city-walk set via +`scripts/fetch_video.py`, or the user's own) → `INPUT_VIDEO_DIR`? **`SAMPLE_FPS`** (how densely to +sample — the single biggest lever on frame count / run time) and **`SEGMENT_SIZE`** (frames per CVAT +job)? **which Postgres + which database** for `DB_URL` — never point at an existing DB or use a +default without confirming; external CVAT ready or provision it? reuse an existing venv / `uv` env or +create fresh? which GPU (VRAM + FlashAttention)? surface stage logs or run quiet? + +**How to work:** read the setup, propose a short plan, get a go-ahead before touching anything. +Prepare `.env` and **pause for the user to verify it** before running. Run stages with logs shown; +after each, say what you did and what changed — don't run silently. If a stage fails and the cause +isn't clear, re-run with `datapipe --debug … run` (`--debug-sql` for SQL); debug is very verbose, so +send it to a file and `grep` (`datapipe --debug run > /tmp/dp.log 2>&1; grep -nEi "error|traceback" /tmp/dp.log`). + +## Pipeline +``` +stage=video list_videos folder INPUT_VIDEO_DIR -> video +stage=sample extract_frames ffmpeg fps=SAMPLE_FPS -> frames +stage=sample dedup_frames perceptual-hash dedup -> local_images +stage=ingest list_sam_config SAM_TEXT_PROMPT -> sam_config +stage=sam sam_inference SAM3 image-mode -> sam_predictions +stage=sam sam_to_cvat_xml -> sam_cvat_xml +stage=cvat prepare_cvat_input / CVATStep / parse_cvat_annotations -> image__annotations +``` +`local_images (video_id, image_id, image_path)` is what the SAM→CVAT tail consumes; everything from +`sam_inference` on is identical to `sam_cvat`. **One CVAT task per video** (`task_queue_id=video_id`, +`FILES_BATCH` huge), split into jobs of `SEGMENT_SIZE` frames. + +## Sampling knobs (the front) +- **`SAMPLE_FPS`** (default 0.2 = 1 frame/5 s). Walking POV is highly redundant — 1 fps vs 1/3 fps + barely differ in what gets annotated; sample sparser to cut annotation load. **Ask before a run.** +- **`PHASH_MAX_DISTANCE`** (Hamming, default 10) — higher = more aggressive near-duplicate dedup. +- **Gotcha:** `extract_frames` reuses frames already in `FRAMES_DIR`, so **changing `SAMPLE_FPS` has + no effect until you clear `FRAMES_DIR`** (`rm -rf /`). +- Align class across all three: `SAM_TEXT_PROMPT` == `CVAT_BOX_LABEL`/`CVAT_POLYGON_LABEL` — mismatch + runs clean but yields 0 useful annotations. Levers: `SAM_SCORE_THRESHOLD` (0.5), `SAM_MAX_DETECTIONS` (20). + +## GPU memory / OOM +SAM3 emits masks at the **input** resolution, so a raw 720p frame OOMs an 8 GB card. +- **`SAM_MAX_INFER_SIDE`** (default 640): frame is downscaled so its longest side ≤ this before + inference, then detections are scaled back to the original frame's coords (CVAT gets the full-res + frame). Raise it on a roomy GPU for sharper masks; lower it if you still OOM; `0` disables. +- On OOM, `infer_image` retries at 512 → 384 and only skips the frame as a last resort — **one frame + never kills the run** (datapipe logs the skip and moves on). +- `config.py` sets `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` before importing torch, so it + applies to both CLI and UI-triggered runs. +- *Field note:* GTX 1070 (Pascal, 8 GB, no FlashAttention → math kernel) runs at 640 px, ~3 s/frame, + VRAM ~8 GB (tight). Ada/Ampere+ with FlashAttention is far lighter and faster — prefer it. + +## Prerequisites +- **GPU** (see above). **HF token** — SAM3 is gated: accept the license at huggingface.co/facebook/sam3, + set `HF_TOKEN` (gated read access). **Read-only home?** set `HF_HOME=/writable/path` so weights + + token cache there (the example already avoids persisting the token to disk when `HF_TOKEN` is set). +- **`ffmpeg`** on `PATH`, else the `imageio-ffmpeg` bundled binary is used automatically. +- **External PostgreSQL** at `DB_URL` (tables auto-create via `datapipe db create-all`). +- **External CVAT** at `CVAT_URL` (user/pass, optional org), project `CVAT_PROJECT_ID` whose labels + match `CVAT_BOX_LABEL` / `CVAT_POLYGON_LABEL` (defaults `person_box` / `person_mask`). +- **`uv` + Python ≥3.10,<3.13** → `uv sync`. Pins cu124 torch, editable local libs + (`../../libs/datapipe-*`, monorepo-only), builds `sam3` from a pinned git rev (+`imagehash`). After + `uv sync`, on a pre-AVX2 host re-apply `uv pip install polars-lts-cpu==1.33.1`. + +## Get a video +```bash +python scripts/fetch_video.py --height 720 # built-in ~24h city-walk set +python scripts/fetch_video.py --dir videos "https://youtu.be/ID" # your own (needs yt-dlp + deno) +python scripts/fetch_video.py --section 00:10:00-00:20:00 "https://youtu.be/ID" # just a clip +``` +Needs `yt-dlp` + a JS runtime (`deno`/`node`) for YouTube's n-challenge; sign-in-gated videos need +`--cookies-from-browser chrome`. Internal-demo only (YouTube ToS) — don't redistribute. + +## Run +```bash +cp .env.example .env # DB_URL, HF_TOKEN, INPUT_VIDEO_DIR, CVAT_*, SAM_TEXT_PROMPT, SAMPLE_FPS, SEGMENT_SIZE +uv sync && source .venv/bin/activate # else prefix each command with `uv run` +datapipe db create-all && datapipe run +# by stage: datapipe step --labels stage=sample run (then stage=sam, stage=cvat) +``` +Run from `examples/video_segmentation_cvat/` (`app.py` `load_dotenv()`s before importing config). +**Live demo:** drop a ~10 s clip in `INPUT_VIDEO_DIR` → `datapipe run` → a CVAT task appears in seconds. + +## Debug UI (`datapipe api`) +`datapipe api` (default `:8000`) serves the pipeline graph, table browser, and per-stage run triggers. +**Do not run `datapipe api` and a separate CLI `datapipe run` on the same DB at once** — both write +run logs to the shared Postgres and collide on `uq_run_log_seq` (the API's orphan-run reconciler +adopts the live CLI run), which kills the run. Either trigger runs from the UI, or stop the UI while a +CLI run is in flight. (Radical fix: give the API a dedicated ClickHouse `RunLogsBackend`.) + +## Annotate (human-in-the-loop) +In CVAT fix the pre-annotations, **mark the task completed**, re-run `datapipe run` → edits land in +`image__annotations`. CVAT tables (for wipes): `image_batches`, `cvat_task`, `cvat_images`, +`cvat_task_sync_table`. `CVATStep` does NOT push new pre-annotations to EXISTING tasks (only on image +add/remove/path change) — to change `SAM_TEXT_PROMPT`, wipe old tasks + datapipe CVAT tables, re-run. + +## Troubleshooting (may already be fixed — verify against current files) +- **0 detections** → class misaligned; align `SAM_TEXT_PROMPT` with the CVAT labels. +- **OOM** → lower `SAM_MAX_INFER_SIDE`; confirm `expandable_segments` took effect. +- **Changing `SAMPLE_FPS` did nothing** → clear `FRAMES_DIR` (frames are reused). +- **`uq_run_log_seq` duplicate key** → UI + CLI ran together; run one at a time. +- **Read-only filesystem on HF cache** → set `HF_HOME` to a writable path. +- **CVAT rejects label** → project needs labels named `CVAT_BOX_LABEL`/`CVAT_POLYGON_LABEL`. diff --git a/examples/video_segmentation_cvat/.env.example b/examples/video_segmentation_cvat/.env.example new file mode 100644 index 00000000..4f50bc14 --- /dev/null +++ b/examples/video_segmentation_cvat/.env.example @@ -0,0 +1,40 @@ +DB_URL=postgresql+psycopg2://postgres:postgres@localhost:5432/postgres + +# --- video ingest / frame sampling --- +# Folder you drop long egocentric videos into. Every video here is sampled and segmented. +INPUT_VIDEO_DIR=videos +# Where extracted frames are written (one subfolder per video). Defaults to ./.frames if unset. +# FRAMES_DIR=/tmp/robots-frames +# ffmpeg extraction rate (frames per second) -- ASK how densely to sample before a run. +# Walking POV barely differs between 1 and 1/3 fps; sample sparser to cut annotation load: +# 1 = 1 frame/s, 0.2 = 1 frame/5s, 0.1 = 1 frame/10s. Dedup below removes near-duplicates on top. +SAMPLE_FPS=0.2 +# Perceptual-hash near-duplicate threshold (Hamming). Higher = more aggressive dedup. +PHASH_MAX_DISTANCE=10 + +# --- SAM3 (gated model: accept the license on HuggingFace and set a token) --- +HF_TOKEN=replace-me +SAM_TEXT_PROMPT=person +SAM_SCORE_THRESHOLD=0.5 +SAM_MAX_DETECTIONS=20 +# SAM3 returns masks at the input resolution, so full 720p+ frames need a big GPU (README asks for +# >8GB). Downscale each frame so its longest side is at most this before inference (detections are +# scaled back to original coordinates); lower it if you OOM, set 0 to disable. 768 fits an 8GB card. +SAM_MAX_INFER_SIDE=640 +# HF caches the gated model + token under $HF_HOME (default ~/.cache/huggingface). Point it at a +# writable path if the home dir is read-only. +# HF_HOME=/var/tmp/hf_home + +# --- CVAT --- +CVAT_URL=http://localhost:8080 +CVAT_USERNAME=admin +CVAT_PASSWORD=admin +CVAT_PROJECT_ID=1 +CVAT_ORGANIZATION= +CVAT_BOX_LABEL=person_box +CVAT_POLYGON_LABEL=person_mask +TASK_QUEUE_ID=queue1 +# One CVAT task per video (task_queue_id=video_id) with jobs of SEGMENT_SIZE frames each. +FILES_BATCH=100000 +MIN_FILES_IN_JOB=1 +SEGMENT_SIZE=200 diff --git a/examples/video_segmentation_cvat/README.md b/examples/video_segmentation_cvat/README.md index 79f49446..888a39f7 100644 --- a/examples/video_segmentation_cvat/README.md +++ b/examples/video_segmentation_cvat/README.md @@ -35,10 +35,15 @@ frames are processed. ## Prerequisites -- **GPU** with >8 GB VRAM for SAM3 (native 1008px; `DEVICE` auto-selects `cuda:0`). -- **`ffmpeg`** on `PATH` (frame extraction). +- **GPU** for SAM3 (`DEVICE` auto-selects `cuda:0`). A FlashAttention-capable card (Ada/Ampere+) with + >8 GB is comfortable at native resolution. On a tight 8 GB card, or a Pascal card with no + FlashAttention (e.g. GTX 1070 → O(n²) math-attention), lower `SAM_MAX_INFER_SIDE` — see + [GPU memory](#gpu-memory-oom). +- **`ffmpeg`** on `PATH` (frame extraction). Not required: the example falls back to the binary + bundled by `imageio-ffmpeg` when no system `ffmpeg` is found. - **SAM3 is a gated HuggingFace model** — accept the license on the SAM3 model page, create a token, - set `HF_TOKEN` in `.env`. + set `HF_TOKEN` in `.env`. If the home dir is read-only, point `HF_HOME` at a writable path so the + gated weights + token cache there. - **CVAT** deployed at `CVAT_URL` (see [`../datapipe_cvat/simple_project`](../datapipe_cvat/simple_project/README.md) for a local Docker setup), a project created (`CVAT_PROJECT_ID`) with labels matching `CVAT_BOX_LABEL` / `CVAT_POLYGON_LABEL` (defaults `person_box` / `person_mask`). @@ -75,6 +80,25 @@ datapipe run Run a single stage: `datapipe step --labels stage=sample run`, `... stage=sam run`, `... stage=cvat run`. +## GPU memory (OOM) + +SAM3 returns masks at the input resolution, so full 720p+ frames need a lot of VRAM — an 8 GB card +OOMs on a raw 1280×720 frame. The example downscales each frame so its longest side is at most +`SAM_MAX_INFER_SIDE` (default **640**) before inference, then maps detections back to the original +frame's coordinates (CVAT still gets the full-res frame). A crowded frame can still spike, so on OOM +inference retries at progressively smaller sizes and only skips the frame as a last resort — one +frame never kills the run. Raise `SAM_MAX_INFER_SIDE` (or set `0` to disable) on a roomy GPU for +sharper masks; lower it if you still OOM. `config.py` also sets +`PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` to keep fragmentation from causing spurious OOMs. + +## Debug UI (`datapipe api`) + +`app.py` exposes a `DatapipeAPI`, so `datapipe api` (default `:8000`) serves the pipeline graph, table +browser, and per-stage run triggers. **Do not run `datapipe api` and a separate CLI `datapipe run` +against the same database at the same time** — both write run logs to the shared Postgres and collide +on `uq_run_log_seq` (the API's orphan-run reconciler adopts the live CLI run), which kills the run. +Either trigger runs from the UI, or stop the UI while a CLI run is in flight. + ## Live demo Drop a ~10s clip into `INPUT_VIDEO_DIR` and `datapipe run`: it is sampled, deduped, segmented, and a diff --git a/examples/video_segmentation_cvat/config.py b/examples/video_segmentation_cvat/config.py index 6e181c1f..6cdfbe9d 100644 --- a/examples/video_segmentation_cvat/config.py +++ b/examples/video_segmentation_cvat/config.py @@ -3,6 +3,11 @@ import os from pathlib import Path +# Ask PyTorch's CUDA allocator for expandable segments before torch is imported, so the setting +# applies no matter how the pipeline is launched (CLI `datapipe run` or a run triggered from the UI +# server). This keeps fragmentation from turning a tight 8GB card into spurious OOMs. +os.environ.setdefault("PYTORCH_CUDA_ALLOC_CONF", "expandable_segments:True") + import torch from datapipe.store.database import DBConn @@ -59,6 +64,10 @@ def _resolve_ffmpeg() -> str: SAM_TEXT_PROMPT = os.environ.get("SAM_TEXT_PROMPT", "person") SAM_SCORE_THRESHOLD = float(os.environ.get("SAM_SCORE_THRESHOLD", "0.5")) SAM_MAX_DETECTIONS = int(os.environ.get("SAM_MAX_DETECTIONS", "20")) +# SAM3 returns masks at the input image's resolution, so full 720p/1080p frames blow past small +# GPUs (an 8GB card OOMs on 1280x720). Downscale the frame so its longest side is at most this many +# pixels before inference, then scale detections back to original coordinates. 0 disables downscaling. +SAM_MAX_INFER_SIDE = int(os.environ.get("SAM_MAX_INFER_SIDE", "640")) # --- CVAT ----------------------------------------------------------------------------------------- diff --git a/examples/video_segmentation_cvat/models.py b/examples/video_segmentation_cvat/models.py index f5465e4a..d0f10e84 100644 --- a/examples/video_segmentation_cvat/models.py +++ b/examples/video_segmentation_cvat/models.py @@ -1,6 +1,7 @@ from __future__ import annotations import logging +import os from typing import List, Optional import cv2 @@ -9,7 +10,13 @@ from cv_pipeliner import BboxData from PIL import Image -from config import DEVICE, HF_TOKEN, SAM_MAX_DETECTIONS, SAM_SCORE_THRESHOLD +from config import ( + DEVICE, + HF_TOKEN, + SAM_MAX_DETECTIONS, + SAM_MAX_INFER_SIDE, + SAM_SCORE_THRESHOLD, +) logger = logging.getLogger(__name__) @@ -17,11 +24,15 @@ def ensure_hf_login() -> None: - from huggingface_hub import login - + # huggingface_hub reads HF_TOKEN from the environment for gated-model downloads, so when a token + # is configured we just make sure it is exported and skip login() -- login() persists the token to + # ~/.cache/huggingface, which fails on hosts with a read-only home dir. Fall back to interactive + # login only when no token is set. if HF_TOKEN: - login(token=HF_TOKEN) + os.environ["HF_TOKEN"] = HF_TOKEN return + from huggingface_hub import login + login() @@ -61,17 +72,69 @@ def _to_scalar(value) -> float: return float(array.reshape(-1)[0]) -def infer_image(image: Image.Image, text_prompt: str) -> List[BboxData]: +def _is_oom(exc: BaseException) -> bool: + return isinstance(exc, torch.cuda.OutOfMemoryError) or ( + isinstance(exc, RuntimeError) and "out of memory" in str(exc).lower() + ) + + +def _run_sam(image: Image.Image, text_prompt: str, device_type: str) -> dict: processor = get_processor() + try: + if device_type == "cuda": + with torch.autocast(device_type, dtype=torch.bfloat16): + inference_state = processor.set_image(image) + return processor.set_text_prompt(state=inference_state, prompt=text_prompt) + inference_state = processor.set_image(image) + return processor.set_text_prompt(state=inference_state, prompt=text_prompt) + finally: + # Release cached blocks after every attempt so fragmentation doesn't accumulate over a long + # run and a failed attempt frees its memory before the next (smaller) retry. + if device_type == "cuda": + torch.cuda.empty_cache() + + +def infer_image(image: Image.Image, text_prompt: str) -> List[BboxData]: device_type = "cuda" if DEVICE.startswith("cuda") else "cpu" - if device_type == "cuda": - with torch.autocast(device_type, dtype=torch.bfloat16): - inference_state = processor.set_image(image) - output = processor.set_text_prompt(state=inference_state, prompt=text_prompt) - else: - inference_state = processor.set_image(image) - output = processor.set_text_prompt(state=inference_state, prompt=text_prompt) + # SAM3 emits masks at the input resolution, so a full 720p frame needs far more VRAM than a small + # GPU has. Downscale before inference (detections are mapped back to the original frame's + # coordinates, since the full-res frame is what goes to CVAT). A crowded frame can still spike + # past a tight card, so on OOM we retry at progressively smaller sizes rather than fail the run. + orig_w, orig_h = image.size + longest = max(orig_w, orig_h) + target = SAM_MAX_INFER_SIDE if SAM_MAX_INFER_SIDE else longest + caps: List[int] = [] + for cap in (target, 512, 384): + cap = min(cap, longest) + if cap not in caps: + caps.append(cap) + + output = None + inv_scale = 1.0 + for cap in caps: + scale = cap / longest + if scale < 1.0: + sized = image.resize( + (max(1, round(orig_w * scale)), max(1, round(orig_h * scale))), + Image.BILINEAR, + ) + else: + sized = image + try: + output = _run_sam(sized, text_prompt, device_type) + inv_scale = 1.0 / scale + break + except Exception as exc: # noqa: BLE001 - only OOM is retried, everything else re-raises + if not _is_oom(exc): + raise + logger.warning( + "SAM OOM on a %dx%d frame at longest-side<=%d; retrying smaller", orig_w, orig_h, cap + ) + + if output is None: + logger.warning("SAM OOM on a %dx%d frame at every size; skipping frame", orig_w, orig_h) + return [] masks = output.get("masks", []) boxes = output.get("boxes", []) @@ -97,11 +160,13 @@ def infer_image(image: Image.Image, text_prompt: str) -> List[BboxData]: if box.size < 4: continue - x_min, y_min, x_max, y_max = [float(v) for v in box[:4]] + x_min, y_min, x_max, y_max = [float(v) * inv_scale for v in box[:4]] mask = _to_numpy(masks[idx]) if mask.ndim == 3: mask = mask[0] polygon = _mask_to_polygon(mask) + if polygon is not None and scale != 1.0: + polygon = (polygon.astype(np.float32) * inv_scale).astype(np.int32) mask_polygons = [polygon] if polygon is not None else [] detections.append( From 5fbdf2d88a16c64abda1b06f0e3bb3b3ef178dec Mon Sep 17 00:00:00 2001 From: Shakirov Renat Date: Fri, 24 Jul 2026 12:34:26 +0300 Subject: [PATCH 07/13] video_segmentation_cvat: move GPU downscaling into a dedicated step; restore verbatim sam_cvat tail The 8GB-GPU OOM fix used to live inside infer_image (downscale + coord rescale + retry), which diverged models.py from sam_cvat and broke the example's "tail identical to sam_cvat" contract. Move that concern out into an explicit pipeline step instead: - new steps.downscale_frames (stage=sample): resizes each deduped frame so its longest side <= SAM_MAX_INFER_SIDE (default 640) into IMAGES_DIR; the resized frame is what both SAM and CVAT use, so detections need no coordinate rescaling. 0 passes the original frame through. - dedup_frames now outputs deduped_frames (full-res); downscale_frames -> local_images - data.py: add deduped_frames table; local_images is now the resize output - config.py: add IMAGES_DIR; SAM_MAX_INFER_SIDE moves to the sampling knobs; drop the PYTORCH_CUDA_ALLOC_CONF hack (unneeded once frames are pre-sized) - models.py: revert to a verbatim copy of sam_cvat/models.py (HF read-only-home is handled by HF_HOME in .env, not by patching ensure_hf_login) - README/.env.example/setup skill: document the downscale_frames step and IMAGES_DIR --- .../setup-video-segmentation-cvat/SKILL.md | 38 ++++---- examples/video_segmentation_cvat/.env.example | 13 ++- examples/video_segmentation_cvat/README.md | 30 +++--- examples/video_segmentation_cvat/app.py | 8 ++ examples/video_segmentation_cvat/config.py | 24 +++-- examples/video_segmentation_cvat/data.py | 21 ++++- examples/video_segmentation_cvat/models.py | 91 +++---------------- examples/video_segmentation_cvat/steps.py | 43 ++++++++- 8 files changed, 139 insertions(+), 129 deletions(-) diff --git a/.claude/skills/setup-video-segmentation-cvat/SKILL.md b/.claude/skills/setup-video-segmentation-cvat/SKILL.md index 9ab071a4..b0945b07 100644 --- a/.claude/skills/setup-video-segmentation-cvat/SKILL.md +++ b/.claude/skills/setup-video-segmentation-cvat/SKILL.md @@ -26,17 +26,19 @@ send it to a file and `grep` (`datapipe --debug run > /tmp/dp.log 2>&1; grep -nE ## Pipeline ``` -stage=video list_videos folder INPUT_VIDEO_DIR -> video -stage=sample extract_frames ffmpeg fps=SAMPLE_FPS -> frames -stage=sample dedup_frames perceptual-hash dedup -> local_images -stage=ingest list_sam_config SAM_TEXT_PROMPT -> sam_config -stage=sam sam_inference SAM3 image-mode -> sam_predictions -stage=sam sam_to_cvat_xml -> sam_cvat_xml +stage=video list_videos folder INPUT_VIDEO_DIR -> video +stage=sample extract_frames ffmpeg fps=SAMPLE_FPS -> frames +stage=sample dedup_frames perceptual-hash dedup -> deduped_frames +stage=sample downscale_frames resize to SAM_MAX_INFER_SIDE -> local_images +stage=ingest list_sam_config SAM_TEXT_PROMPT -> sam_config +stage=sam sam_inference SAM3 image-mode -> sam_predictions +stage=sam sam_to_cvat_xml -> sam_cvat_xml stage=cvat prepare_cvat_input / CVATStep / parse_cvat_annotations -> image__annotations ``` `local_images (video_id, image_id, image_path)` is what the SAM→CVAT tail consumes; everything from -`sam_inference` on is identical to `sam_cvat`. **One CVAT task per video** (`task_queue_id=video_id`, -`FILES_BATCH` huge), split into jobs of `SEGMENT_SIZE` frames. +`sam_inference` on (incl. `models.py`) is a **verbatim copy of `sam_cvat`** — keep it that way. The +video front's only job is video → deduped, GPU-sized frames. **One CVAT task per video** +(`task_queue_id=video_id`, `FILES_BATCH` huge), split into jobs of `SEGMENT_SIZE` frames. ## Sampling knobs (the front) - **`SAMPLE_FPS`** (default 0.2 = 1 frame/5 s). Walking POV is highly redundant — 1 fps vs 1/3 fps @@ -48,16 +50,13 @@ stage=cvat prepare_cvat_input / CVATStep / parse_cvat_annotations -> image__a runs clean but yields 0 useful annotations. Levers: `SAM_SCORE_THRESHOLD` (0.5), `SAM_MAX_DETECTIONS` (20). ## GPU memory / OOM -SAM3 emits masks at the **input** resolution, so a raw 720p frame OOMs an 8 GB card. -- **`SAM_MAX_INFER_SIDE`** (default 640): frame is downscaled so its longest side ≤ this before - inference, then detections are scaled back to the original frame's coords (CVAT gets the full-res - frame). Raise it on a roomy GPU for sharper masks; lower it if you still OOM; `0` disables. -- On OOM, `infer_image` retries at 512 → 384 and only skips the frame as a last resort — **one frame - never kills the run** (datapipe logs the skip and moves on). -- `config.py` sets `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` before importing torch, so it - applies to both CLI and UI-triggered runs. -- *Field note:* GTX 1070 (Pascal, 8 GB, no FlashAttention → math kernel) runs at 640 px, ~3 s/frame, - VRAM ~8 GB (tight). Ada/Ampere+ with FlashAttention is far lighter and faster — prefer it. +SAM3 emits masks at the **input** resolution, so a raw 720p frame OOMs an 8 GB card. Instead of +touching the shared `infer_image`, the **`downscale_frames` step** resizes each deduped frame so its +longest side ≤ **`SAM_MAX_INFER_SIDE`** (default 640) and writes it under `IMAGES_DIR`. That resized +frame is what BOTH SAM and CVAT use, so coordinates line up with no rescaling. Raise it on a roomy GPU +for sharper masks; lower it if you still OOM; `0` passes the original frame through unchanged. +- *Field note:* GTX 1070 (Pascal, 8 GB, no FlashAttention → math kernel) at 640 px → ~3 s/frame, + VRAM ~3.8 GB (comfortable). Ada/Ampere+ with FlashAttention is far lighter and faster — prefer it. ## Prerequisites - **GPU** (see above). **HF token** — SAM3 is gated: accept the license at huggingface.co/facebook/sam3, @@ -105,8 +104,9 @@ add/remove/path change) — to change `SAM_TEXT_PROMPT`, wipe old tasks + datapi ## Troubleshooting (may already be fixed — verify against current files) - **0 detections** → class misaligned; align `SAM_TEXT_PROMPT` with the CVAT labels. -- **OOM** → lower `SAM_MAX_INFER_SIDE`; confirm `expandable_segments` took effect. +- **OOM** → lower `SAM_MAX_INFER_SIDE`, then clear `IMAGES_DIR` so frames re-resize. - **Changing `SAMPLE_FPS` did nothing** → clear `FRAMES_DIR` (frames are reused). +- **Changing `SAM_MAX_INFER_SIDE` did nothing** → clear `IMAGES_DIR` (resized frames are reused). - **`uq_run_log_seq` duplicate key** → UI + CLI ran together; run one at a time. - **Read-only filesystem on HF cache** → set `HF_HOME` to a writable path. - **CVAT rejects label** → project needs labels named `CVAT_BOX_LABEL`/`CVAT_POLYGON_LABEL`. diff --git a/examples/video_segmentation_cvat/.env.example b/examples/video_segmentation_cvat/.env.example index 4f50bc14..977e7178 100644 --- a/examples/video_segmentation_cvat/.env.example +++ b/examples/video_segmentation_cvat/.env.example @@ -5,24 +5,27 @@ DB_URL=postgresql+psycopg2://postgres:postgres@localhost:5432/postgres INPUT_VIDEO_DIR=videos # Where extracted frames are written (one subfolder per video). Defaults to ./.frames if unset. # FRAMES_DIR=/tmp/robots-frames +# Where downscale_frames writes the resized frames the SAM->CVAT tail consumes. Defaults to ./.images +# IMAGES_DIR=/tmp/robots-images # ffmpeg extraction rate (frames per second) -- ASK how densely to sample before a run. # Walking POV barely differs between 1 and 1/3 fps; sample sparser to cut annotation load: # 1 = 1 frame/s, 0.2 = 1 frame/5s, 0.1 = 1 frame/10s. Dedup below removes near-duplicates on top. SAMPLE_FPS=0.2 # Perceptual-hash near-duplicate threshold (Hamming). Higher = more aggressive dedup. PHASH_MAX_DISTANCE=10 +# downscale_frames resizes each frame so its longest side is at most this before SAM (SAM3 emits masks +# at input resolution, so raw 720p+ frames OOM a small GPU). CVAT gets the same resized frame, so +# coordinates line up. Lower it if you OOM, raise for sharper masks, 0 = pass original through. +# 640 fits an 8GB card with headroom. +SAM_MAX_INFER_SIDE=640 # --- SAM3 (gated model: accept the license on HuggingFace and set a token) --- HF_TOKEN=replace-me SAM_TEXT_PROMPT=person SAM_SCORE_THRESHOLD=0.5 SAM_MAX_DETECTIONS=20 -# SAM3 returns masks at the input resolution, so full 720p+ frames need a big GPU (README asks for -# >8GB). Downscale each frame so its longest side is at most this before inference (detections are -# scaled back to original coordinates); lower it if you OOM, set 0 to disable. 768 fits an 8GB card. -SAM_MAX_INFER_SIDE=640 # HF caches the gated model + token under $HF_HOME (default ~/.cache/huggingface). Point it at a -# writable path if the home dir is read-only. +# writable path if the home dir is read-only (else login()/weights download fail on such hosts). # HF_HOME=/var/tmp/hf_home # --- CVAT --- diff --git a/examples/video_segmentation_cvat/README.md b/examples/video_segmentation_cvat/README.md index 888a39f7..ddcfde7a 100644 --- a/examples/video_segmentation_cvat/README.md +++ b/examples/video_segmentation_cvat/README.md @@ -11,17 +11,20 @@ front is turning a long video into a deduplicated set of frames. ## Pipeline ``` -stage=video list_videos folder INPUT_VIDEO_DIR -> video -stage=sample extract_frames ffmpeg fps=SAMPLE_FPS -> frames -stage=sample dedup_frames perceptual-hash dedup -> local_images -stage=ingest list_sam_config SAM_TEXT_PROMPT -> sam_config -stage=sam sam_inference SAM3 image-mode -> sam_predictions -stage=sam sam_to_cvat_xml -> sam_cvat_xml +stage=video list_videos folder INPUT_VIDEO_DIR -> video +stage=sample extract_frames ffmpeg fps=SAMPLE_FPS -> frames +stage=sample dedup_frames perceptual-hash dedup -> deduped_frames +stage=sample downscale_frames resize to SAM_MAX_INFER_SIDE -> local_images +stage=ingest list_sam_config SAM_TEXT_PROMPT -> sam_config +stage=sam sam_inference SAM3 image-mode -> sam_predictions +stage=sam sam_to_cvat_xml -> sam_cvat_xml stage=cvat prepare_cvat_input / CVATStep / parse_cvat_annotations -> image__annotations ``` `local_images (image_id, image_path)` is exactly what the SAM→CVAT tail consumes, so everything from -`sam_inference` onward is identical to `sam_cvat`. +`sam_inference` onward is identical to `sam_cvat` — `models.py` is a verbatim copy. The only thing the +video front adds is turning a long video into deduplicated, GPU-sized frames (the last two `sample` +steps). ## Why sample at 1 fps then dedup (not "every frame", not 1/3 fps) @@ -83,13 +86,12 @@ Run a single stage: `datapipe step --labels stage=sample run`, `... stage=sam ru ## GPU memory (OOM) SAM3 returns masks at the input resolution, so full 720p+ frames need a lot of VRAM — an 8 GB card -OOMs on a raw 1280×720 frame. The example downscales each frame so its longest side is at most -`SAM_MAX_INFER_SIDE` (default **640**) before inference, then maps detections back to the original -frame's coordinates (CVAT still gets the full-res frame). A crowded frame can still spike, so on OOM -inference retries at progressively smaller sizes and only skips the frame as a last resort — one -frame never kills the run. Raise `SAM_MAX_INFER_SIDE` (or set `0` to disable) on a roomy GPU for -sharper masks; lower it if you still OOM. `config.py` also sets -`PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` to keep fragmentation from causing spurious OOMs. +OOMs on a raw 1280×720 frame. Rather than touch the (shared-with-`sam_cvat`) inference code, the +`downscale_frames` step resizes each deduped frame so its longest side is at most `SAM_MAX_INFER_SIDE` +(default **640**, fits an 8 GB card with headroom) and writes it under `IMAGES_DIR`. That resized +frame is what both SAM and CVAT use, so detection coordinates line up with no rescaling. Raise +`SAM_MAX_INFER_SIDE` on a roomy GPU for sharper masks, or set `0` to pass the original frame straight +through. ## Debug UI (`datapipe api`) diff --git a/examples/video_segmentation_cvat/app.py b/examples/video_segmentation_cvat/app.py index ad68f2ba..601bcae5 100644 --- a/examples/video_segmentation_cvat/app.py +++ b/examples/video_segmentation_cvat/app.py @@ -46,6 +46,14 @@ BatchTransform( func=steps.dedup_frames, inputs=[data.frames_tbl], + outputs=[data.deduped_frames_tbl], + transform_keys=["video_id"], + chunk_size=1, + labels=[("stage", "sample")], + ), + BatchTransform( + func=steps.downscale_frames, + inputs=[data.deduped_frames_tbl], outputs=[data.local_images_tbl], transform_keys=["video_id"], chunk_size=1, diff --git a/examples/video_segmentation_cvat/config.py b/examples/video_segmentation_cvat/config.py index 6cdfbe9d..d9317247 100644 --- a/examples/video_segmentation_cvat/config.py +++ b/examples/video_segmentation_cvat/config.py @@ -3,11 +3,6 @@ import os from pathlib import Path -# Ask PyTorch's CUDA allocator for expandable segments before torch is imported, so the setting -# applies no matter how the pipeline is launched (CLI `datapipe run` or a run triggered from the UI -# server). This keeps fragmentation from turning a tight 8GB card into spurious OOMs. -os.environ.setdefault("PYTORCH_CUDA_ALLOC_CONF", "expandable_segments:True") - import torch from datapipe.store.database import DBConn @@ -26,6 +21,15 @@ else Path(__file__).resolve().parent / ".frames" ) +# Where downscale_frames writes the resized frames the SAM->CVAT tail consumes (one subfolder per +# video_id). Separate from FRAMES_DIR so the full-res extracts stay intact. +_IMAGES_DIR_RAW = os.environ.get("IMAGES_DIR") +IMAGES_DIR = ( + Path(_IMAGES_DIR_RAW).resolve() + if _IMAGES_DIR_RAW + else Path(__file__).resolve().parent / ".images" +) + # ffmpeg extraction rate. 1 fps gives good event coverage; 1/3fps is not meaningfully different for # walking POV footage (see README). Dedup below removes the near-duplicates either way. SAMPLE_FPS = float(os.environ.get("SAMPLE_FPS", "1")) @@ -54,6 +58,12 @@ def _resolve_ffmpeg() -> str: PHASH_MAX_DISTANCE = int(os.environ.get("PHASH_MAX_DISTANCE", "10")) PHASH_SIZE = int(os.environ.get("PHASH_SIZE", "8")) # phash hash_size (bits per side) +# Longest-side cap (px) for the downscale_frames step. SAM3 returns masks at the input resolution, so +# full 720p+ frames blow past a small GPU (an 8GB card OOMs on 1280x720). Frames are resized down to +# this before SAM sees them and CVAT gets the same resized frame, so detection coordinates line up. +# 0 disables downscaling (use only on a roomy GPU). 640 fits an 8GB card with headroom. +SAM_MAX_INFER_SIDE = int(os.environ.get("SAM_MAX_INFER_SIDE", "640")) + IMAGE_SUFFIXES = {".jpg", ".jpeg", ".png"} # --- SAM3 ----------------------------------------------------------------------------------------- @@ -64,10 +74,6 @@ def _resolve_ffmpeg() -> str: SAM_TEXT_PROMPT = os.environ.get("SAM_TEXT_PROMPT", "person") SAM_SCORE_THRESHOLD = float(os.environ.get("SAM_SCORE_THRESHOLD", "0.5")) SAM_MAX_DETECTIONS = int(os.environ.get("SAM_MAX_DETECTIONS", "20")) -# SAM3 returns masks at the input image's resolution, so full 720p/1080p frames blow past small -# GPUs (an 8GB card OOMs on 1280x720). Downscale the frame so its longest side is at most this many -# pixels before inference, then scale detections back to original coordinates. 0 disables downscaling. -SAM_MAX_INFER_SIDE = int(os.environ.get("SAM_MAX_INFER_SIDE", "640")) # --- CVAT ----------------------------------------------------------------------------------------- diff --git a/examples/video_segmentation_cvat/data.py b/examples/video_segmentation_cvat/data.py index 23ddf552..5aa94bb0 100644 --- a/examples/video_segmentation_cvat/data.py +++ b/examples/video_segmentation_cvat/data.py @@ -37,8 +37,25 @@ ), ) -# Survivors of perceptual-hash dedup. image_id feeds the SAM->CVAT tail; video_id stays in the PK so -# dedup (grouped per video) can delete+reinsert a video's survivors, and is reduced away downstream. +# Survivors of perceptual-hash dedup, still at the source frame resolution. video_id stays in the PK +# so dedup (grouped per video) can delete+reinsert a video's survivors. +deduped_frames_tbl = Table( + name="deduped_frames", + store=TableStoreDB( + dbconn=DBCONN, + name="deduped_frames", + data_sql_schema=[ + Column("video_id", String, primary_key=True), + Column("image_id", String, primary_key=True), + Column("frame_path", String), + ], + create_table=True, + ), +) + +# Deduped frames resized down to SAM_MAX_INFER_SIDE by downscale_frames (see steps.py) — this is what +# the SAM->CVAT tail consumes, so it stays identical to sam_cvat's `local_images`. video_id lingers in +# the PK for the per-video CVAT task split and is reduced away by sam_inference. local_images_tbl = Table( name="local_images", store=TableStoreDB( diff --git a/examples/video_segmentation_cvat/models.py b/examples/video_segmentation_cvat/models.py index d0f10e84..f5465e4a 100644 --- a/examples/video_segmentation_cvat/models.py +++ b/examples/video_segmentation_cvat/models.py @@ -1,7 +1,6 @@ from __future__ import annotations import logging -import os from typing import List, Optional import cv2 @@ -10,13 +9,7 @@ from cv_pipeliner import BboxData from PIL import Image -from config import ( - DEVICE, - HF_TOKEN, - SAM_MAX_DETECTIONS, - SAM_MAX_INFER_SIDE, - SAM_SCORE_THRESHOLD, -) +from config import DEVICE, HF_TOKEN, SAM_MAX_DETECTIONS, SAM_SCORE_THRESHOLD logger = logging.getLogger(__name__) @@ -24,15 +17,11 @@ def ensure_hf_login() -> None: - # huggingface_hub reads HF_TOKEN from the environment for gated-model downloads, so when a token - # is configured we just make sure it is exported and skip login() -- login() persists the token to - # ~/.cache/huggingface, which fails on hosts with a read-only home dir. Fall back to interactive - # login only when no token is set. - if HF_TOKEN: - os.environ["HF_TOKEN"] = HF_TOKEN - return from huggingface_hub import login + if HF_TOKEN: + login(token=HF_TOKEN) + return login() @@ -72,69 +61,17 @@ def _to_scalar(value) -> float: return float(array.reshape(-1)[0]) -def _is_oom(exc: BaseException) -> bool: - return isinstance(exc, torch.cuda.OutOfMemoryError) or ( - isinstance(exc, RuntimeError) and "out of memory" in str(exc).lower() - ) - - -def _run_sam(image: Image.Image, text_prompt: str, device_type: str) -> dict: - processor = get_processor() - try: - if device_type == "cuda": - with torch.autocast(device_type, dtype=torch.bfloat16): - inference_state = processor.set_image(image) - return processor.set_text_prompt(state=inference_state, prompt=text_prompt) - inference_state = processor.set_image(image) - return processor.set_text_prompt(state=inference_state, prompt=text_prompt) - finally: - # Release cached blocks after every attempt so fragmentation doesn't accumulate over a long - # run and a failed attempt frees its memory before the next (smaller) retry. - if device_type == "cuda": - torch.cuda.empty_cache() - - def infer_image(image: Image.Image, text_prompt: str) -> List[BboxData]: + processor = get_processor() device_type = "cuda" if DEVICE.startswith("cuda") else "cpu" - # SAM3 emits masks at the input resolution, so a full 720p frame needs far more VRAM than a small - # GPU has. Downscale before inference (detections are mapped back to the original frame's - # coordinates, since the full-res frame is what goes to CVAT). A crowded frame can still spike - # past a tight card, so on OOM we retry at progressively smaller sizes rather than fail the run. - orig_w, orig_h = image.size - longest = max(orig_w, orig_h) - target = SAM_MAX_INFER_SIDE if SAM_MAX_INFER_SIDE else longest - caps: List[int] = [] - for cap in (target, 512, 384): - cap = min(cap, longest) - if cap not in caps: - caps.append(cap) - - output = None - inv_scale = 1.0 - for cap in caps: - scale = cap / longest - if scale < 1.0: - sized = image.resize( - (max(1, round(orig_w * scale)), max(1, round(orig_h * scale))), - Image.BILINEAR, - ) - else: - sized = image - try: - output = _run_sam(sized, text_prompt, device_type) - inv_scale = 1.0 / scale - break - except Exception as exc: # noqa: BLE001 - only OOM is retried, everything else re-raises - if not _is_oom(exc): - raise - logger.warning( - "SAM OOM on a %dx%d frame at longest-side<=%d; retrying smaller", orig_w, orig_h, cap - ) - - if output is None: - logger.warning("SAM OOM on a %dx%d frame at every size; skipping frame", orig_w, orig_h) - return [] + if device_type == "cuda": + with torch.autocast(device_type, dtype=torch.bfloat16): + inference_state = processor.set_image(image) + output = processor.set_text_prompt(state=inference_state, prompt=text_prompt) + else: + inference_state = processor.set_image(image) + output = processor.set_text_prompt(state=inference_state, prompt=text_prompt) masks = output.get("masks", []) boxes = output.get("boxes", []) @@ -160,13 +97,11 @@ def infer_image(image: Image.Image, text_prompt: str) -> List[BboxData]: if box.size < 4: continue - x_min, y_min, x_max, y_max = [float(v) * inv_scale for v in box[:4]] + x_min, y_min, x_max, y_max = [float(v) for v in box[:4]] mask = _to_numpy(masks[idx]) if mask.ndim == 3: mask = mask[0] polygon = _mask_to_polygon(mask) - if polygon is not None and scale != 1.0: - polygon = (polygon.astype(np.float32) * inv_scale).astype(np.int32) mask_polygons = [polygon] if polygon is not None else [] detections.append( diff --git a/examples/video_segmentation_cvat/steps.py b/examples/video_segmentation_cvat/steps.py index bfc03c99..5bcbcd9f 100644 --- a/examples/video_segmentation_cvat/steps.py +++ b/examples/video_segmentation_cvat/steps.py @@ -15,9 +15,11 @@ CVAT_POLYGON_LABEL, FFMPEG_BIN, FRAMES_DIR, + IMAGES_DIR, INPUT_VIDEO_DIR, PHASH_MAX_DISTANCE, PHASH_SIZE, + SAM_MAX_INFER_SIDE, SAM_TEXT_PROMPT, SAMPLE_FPS, TASK_QUEUE_ID, @@ -28,6 +30,7 @@ SAM_CONFIG_ID = "default" _LOCAL_IMAGES_COLUMNS = ["video_id", "image_id", "image_path"] +_DEDUPED_FRAMES_COLUMNS = ["video_id", "image_id", "frame_path"] _FRAMES_COLUMNS = ["video_id", "frame_id", "ts_sec", "frame_path"] _SAM_PRED_COLUMNS = [ "image_id", @@ -112,7 +115,7 @@ def dedup_frames(df_frames: pd.DataFrame) -> pd.DataFrame: more than PHASH_MAX_DISTANCE (Hamming). Walking POV frames are highly redundant, so a sequential compare-to-last-kept pass removes near-duplicates cheaply while preserving genuine scene change.""" if df_frames.empty: - return pd.DataFrame(columns=_LOCAL_IMAGES_COLUMNS) + return pd.DataFrame(columns=_DEDUPED_FRAMES_COLUMNS) records: list[dict] = [] for video_id, group in df_frames.groupby("video_id"): @@ -131,12 +134,48 @@ def dedup_frames(df_frames: pd.DataFrame) -> pd.DataFrame: { "video_id": str(video_id), "image_id": row["frame_id"], - "image_path": row["frame_path"], + "frame_path": row["frame_path"], } ) if not records: + return pd.DataFrame(columns=_DEDUPED_FRAMES_COLUMNS) + return pd.DataFrame(records, columns=_DEDUPED_FRAMES_COLUMNS) + + +def downscale_frames(df_deduped: pd.DataFrame) -> pd.DataFrame: + """Resize each deduped frame so its longest side is at most SAM_MAX_INFER_SIDE, writing the result + under IMAGES_DIR. SAM3 emits masks at the input resolution, so full 720p+ frames blow past a small + GPU; the resized frame is what both SAM and CVAT use, so detection coordinates line up. With + SAM_MAX_INFER_SIDE=0 the original frame is passed through unchanged (use only on a roomy GPU).""" + if df_deduped.empty: return pd.DataFrame(columns=_LOCAL_IMAGES_COLUMNS) + + records: list[dict] = [] + for _, row in df_deduped.iterrows(): + frame_path = str(row["frame_path"]) + if not SAM_MAX_INFER_SIDE: + image_path = frame_path + else: + out_dir = IMAGES_DIR / str(row["video_id"]) + out_dir.mkdir(parents=True, exist_ok=True) + out_path = out_dir / f"{row['image_id']}.jpg" + if not out_path.exists(): + with Image.open(frame_path) as img: + img = img.convert("RGB") + longest = max(img.size) + if longest > SAM_MAX_INFER_SIDE: + scale = SAM_MAX_INFER_SIDE / longest + img = img.resize( + (max(1, round(img.width * scale)), max(1, round(img.height * scale))), + Image.BILINEAR, + ) + img.save(out_path, quality=95) + image_path = str(out_path) + records.append( + {"video_id": str(row["video_id"]), "image_id": row["image_id"], "image_path": image_path} + ) + return pd.DataFrame(records, columns=_LOCAL_IMAGES_COLUMNS) From d3f11b73984e86ae6c54fa9640f2cf73863edc17 Mon Sep 17 00:00:00 2001 From: Shakirov Renat Date: Fri, 24 Jul 2026 17:34:14 +0300 Subject: [PATCH 08/13] video_segmentation_cvat: run SAM3 at native resolution + deploy/research notes - drop the downscale_frames step and SAM_MAX_INFER_SIDE/IMAGES_DIR: SAM3 runs at the frame's native resolution (a FlashAttention GPU fits full 720p in ~5GB). dedup_frames feeds local_images directly again; models.py stays a verbatim copy of sam_cvat - README/.env.example/setup skill: drop the resize knobs; add a measured GPU field note and a "bare GPU pod (no Docker)" deploy section - add video-native-cvat.md: researched notes on annotating video natively in CVAT (tracks/interpolation, SAM2 tracker, export formats) with sources --- .../setup-video-segmentation-cvat/SKILL.md | 58 ++++--- examples/video_segmentation_cvat/.env.example | 7 - examples/video_segmentation_cvat/README.md | 27 ++-- examples/video_segmentation_cvat/app.py | 8 - examples/video_segmentation_cvat/config.py | 15 -- examples/video_segmentation_cvat/data.py | 22 +-- examples/video_segmentation_cvat/steps.py | 43 +----- .../video-native-cvat.md | 142 ++++++++++++++++++ 8 files changed, 196 insertions(+), 126 deletions(-) create mode 100644 examples/video_segmentation_cvat/video-native-cvat.md diff --git a/.claude/skills/setup-video-segmentation-cvat/SKILL.md b/.claude/skills/setup-video-segmentation-cvat/SKILL.md index b0945b07..c017cae4 100644 --- a/.claude/skills/setup-video-segmentation-cvat/SKILL.md +++ b/.claude/skills/setup-video-segmentation-cvat/SKILL.md @@ -26,19 +26,18 @@ send it to a file and `grep` (`datapipe --debug run > /tmp/dp.log 2>&1; grep -nE ## Pipeline ``` -stage=video list_videos folder INPUT_VIDEO_DIR -> video -stage=sample extract_frames ffmpeg fps=SAMPLE_FPS -> frames -stage=sample dedup_frames perceptual-hash dedup -> deduped_frames -stage=sample downscale_frames resize to SAM_MAX_INFER_SIDE -> local_images -stage=ingest list_sam_config SAM_TEXT_PROMPT -> sam_config -stage=sam sam_inference SAM3 image-mode -> sam_predictions -stage=sam sam_to_cvat_xml -> sam_cvat_xml +stage=video list_videos folder INPUT_VIDEO_DIR -> video +stage=sample extract_frames ffmpeg fps=SAMPLE_FPS -> frames +stage=sample dedup_frames perceptual-hash dedup -> local_images +stage=ingest list_sam_config SAM_TEXT_PROMPT -> sam_config +stage=sam sam_inference SAM3 image-mode -> sam_predictions +stage=sam sam_to_cvat_xml -> sam_cvat_xml stage=cvat prepare_cvat_input / CVATStep / parse_cvat_annotations -> image__annotations ``` `local_images (video_id, image_id, image_path)` is what the SAM→CVAT tail consumes; everything from `sam_inference` on (incl. `models.py`) is a **verbatim copy of `sam_cvat`** — keep it that way. The -video front's only job is video → deduped, GPU-sized frames. **One CVAT task per video** -(`task_queue_id=video_id`, `FILES_BATCH` huge), split into jobs of `SEGMENT_SIZE` frames. +video front's only job is video → deduped frames (SAM runs at native resolution). **One CVAT task per +video** (`task_queue_id=video_id`, `FILES_BATCH` huge), split into jobs of `SEGMENT_SIZE` frames. ## Sampling knobs (the front) - **`SAMPLE_FPS`** (default 0.2 = 1 frame/5 s). Walking POV is highly redundant — 1 fps vs 1/3 fps @@ -49,14 +48,11 @@ video front's only job is video → deduped, GPU-sized frames. **One CVAT task p - Align class across all three: `SAM_TEXT_PROMPT` == `CVAT_BOX_LABEL`/`CVAT_POLYGON_LABEL` — mismatch runs clean but yields 0 useful annotations. Levers: `SAM_SCORE_THRESHOLD` (0.5), `SAM_MAX_DETECTIONS` (20). -## GPU memory / OOM -SAM3 emits masks at the **input** resolution, so a raw 720p frame OOMs an 8 GB card. Instead of -touching the shared `infer_image`, the **`downscale_frames` step** resizes each deduped frame so its -longest side ≤ **`SAM_MAX_INFER_SIDE`** (default 640) and writes it under `IMAGES_DIR`. That resized -frame is what BOTH SAM and CVAT use, so coordinates line up with no rescaling. Raise it on a roomy GPU -for sharper masks; lower it if you still OOM; `0` passes the original frame through unchanged. -- *Field note:* GTX 1070 (Pascal, 8 GB, no FlashAttention → math kernel) at 640 px → ~3 s/frame, - VRAM ~3.8 GB (comfortable). Ada/Ampere+ with FlashAttention is far lighter and faster — prefer it. +## GPU +SAM3 runs at the frame's **native resolution** (no in-pipeline resize). *Field note (measured):* a +FlashAttention GPU (Ada/Ampere+, 16 GB) does full 1280×720 in **~5.3 GB, ~0.27 s/frame**. A +non-FlashAttention (Pascal-class) 8 GB card OOMs on raw 720p — use a newer/bigger GPU, or shrink +frames upstream (e.g. add a `scale` filter to `extract_frames`) rather than editing the shared tail. ## Prerequisites - **GPU** (see above). **HF token** — SAM3 is gated: accept the license at huggingface.co/facebook/sam3, @@ -89,6 +85,30 @@ datapipe db create-all && datapipe run Run from `examples/video_segmentation_cvat/` (`app.py` `load_dotenv()`s before importing config). **Live demo:** drop a ~10 s clip in `INPUT_VIDEO_DIR` → `datapipe run` → a CVAT task appears in seconds. +## Deploy on a bare GPU pod (no Docker) — field-tested +The example itself needs **no Docker** — it's a plain `uv` venv. Docker is only for the infra deps +(Postgres + CVAT), so on a Docker-less pod either point `DB_URL`/`CVAT_URL` at a host that runs them +(needs network reach to their ports) or, for a SAM-only smoke test, skip them. + +Setup that worked, in order: +1. **Get the code on the branch** without disturbing an existing checkout: from the repo dir, + `git fetch origin ` then `git worktree add ` (reuses the host's git creds, + isolated working tree). +2. **`uv sync`** in the example dir. `uv` auto-fetches a Python 3.10–3.12 even if the base interpreter + is older. Takes a few min (torch cu124 ~2.5 GB + builds `sam3`/`cv-pipeliner` from git). **`sam3` + builds with no `nvcc`/CUDA toolkit** — it's pure PyTorch, no compiled CUDA ext (the usual build + worry is a non-issue). +3. **`.env`** — minimum for a SAM run: `HF_TOKEN` (gated), `HF_HOME=/writable/path` (the default home + cache may be small/read-only; weights are ~3.3 GB), `SAM_TEXT_PROMPT`, and **a `DB_URL` even if + unused** — `config.py` builds `DBConn(DB_URL)` at import and crashes on `None` (a dummy + `postgresql+psycopg2://x:x@localhost:5432/x` is enough when you only call `infer_image`). +4. **Run scripts from the example dir** (or set `PYTHONPATH` to it) — `import models`/`config` are + top-level modules, not a package. + +**GPU field note (measured):** on a FlashAttention GPU (Ada/Ampere+, 16 GB), **full 1280×720 SAM3 = +~5.3 GB peak, ~0.27 s/frame** — comfortable, ~10× faster than a non-FA card. First run pays a one-time +~250 s model load + weight download. + ## Debug UI (`datapipe api`) `datapipe api` (default `:8000`) serves the pipeline graph, table browser, and per-stage run triggers. **Do not run `datapipe api` and a separate CLI `datapipe run` on the same DB at once** — both write @@ -104,9 +124,9 @@ add/remove/path change) — to change `SAM_TEXT_PROMPT`, wipe old tasks + datapi ## Troubleshooting (may already be fixed — verify against current files) - **0 detections** → class misaligned; align `SAM_TEXT_PROMPT` with the CVAT labels. -- **OOM** → lower `SAM_MAX_INFER_SIDE`, then clear `IMAGES_DIR` so frames re-resize. +- **OOM** → non-FlashAttention / too-small GPU for native-res SAM3; use a bigger/newer card or shrink + frames upstream (`scale` filter in `extract_frames`). - **Changing `SAMPLE_FPS` did nothing** → clear `FRAMES_DIR` (frames are reused). -- **Changing `SAM_MAX_INFER_SIDE` did nothing** → clear `IMAGES_DIR` (resized frames are reused). - **`uq_run_log_seq` duplicate key** → UI + CLI ran together; run one at a time. - **Read-only filesystem on HF cache** → set `HF_HOME` to a writable path. - **CVAT rejects label** → project needs labels named `CVAT_BOX_LABEL`/`CVAT_POLYGON_LABEL`. diff --git a/examples/video_segmentation_cvat/.env.example b/examples/video_segmentation_cvat/.env.example index 977e7178..02bd0ba5 100644 --- a/examples/video_segmentation_cvat/.env.example +++ b/examples/video_segmentation_cvat/.env.example @@ -5,19 +5,12 @@ DB_URL=postgresql+psycopg2://postgres:postgres@localhost:5432/postgres INPUT_VIDEO_DIR=videos # Where extracted frames are written (one subfolder per video). Defaults to ./.frames if unset. # FRAMES_DIR=/tmp/robots-frames -# Where downscale_frames writes the resized frames the SAM->CVAT tail consumes. Defaults to ./.images -# IMAGES_DIR=/tmp/robots-images # ffmpeg extraction rate (frames per second) -- ASK how densely to sample before a run. # Walking POV barely differs between 1 and 1/3 fps; sample sparser to cut annotation load: # 1 = 1 frame/s, 0.2 = 1 frame/5s, 0.1 = 1 frame/10s. Dedup below removes near-duplicates on top. SAMPLE_FPS=0.2 # Perceptual-hash near-duplicate threshold (Hamming). Higher = more aggressive dedup. PHASH_MAX_DISTANCE=10 -# downscale_frames resizes each frame so its longest side is at most this before SAM (SAM3 emits masks -# at input resolution, so raw 720p+ frames OOM a small GPU). CVAT gets the same resized frame, so -# coordinates line up. Lower it if you OOM, raise for sharper masks, 0 = pass original through. -# 640 fits an 8GB card with headroom. -SAM_MAX_INFER_SIDE=640 # --- SAM3 (gated model: accept the license on HuggingFace and set a token) --- HF_TOKEN=replace-me diff --git a/examples/video_segmentation_cvat/README.md b/examples/video_segmentation_cvat/README.md index ddcfde7a..528e99ae 100644 --- a/examples/video_segmentation_cvat/README.md +++ b/examples/video_segmentation_cvat/README.md @@ -13,8 +13,7 @@ front is turning a long video into a deduplicated set of frames. ``` stage=video list_videos folder INPUT_VIDEO_DIR -> video stage=sample extract_frames ffmpeg fps=SAMPLE_FPS -> frames -stage=sample dedup_frames perceptual-hash dedup -> deduped_frames -stage=sample downscale_frames resize to SAM_MAX_INFER_SIDE -> local_images +stage=sample dedup_frames perceptual-hash dedup -> local_images stage=ingest list_sam_config SAM_TEXT_PROMPT -> sam_config stage=sam sam_inference SAM3 image-mode -> sam_predictions stage=sam sam_to_cvat_xml -> sam_cvat_xml @@ -23,8 +22,7 @@ stage=cvat prepare_cvat_input / CVATStep / parse_cvat_annotations -> image__ `local_images (image_id, image_path)` is exactly what the SAM→CVAT tail consumes, so everything from `sam_inference` onward is identical to `sam_cvat` — `models.py` is a verbatim copy. The only thing the -video front adds is turning a long video into deduplicated, GPU-sized frames (the last two `sample` -steps). +video front adds is turning a long video into a deduplicated set of frames. ## Why sample at 1 fps then dedup (not "every frame", not 1/3 fps) @@ -38,10 +36,10 @@ frames are processed. ## Prerequisites -- **GPU** for SAM3 (`DEVICE` auto-selects `cuda:0`). A FlashAttention-capable card (Ada/Ampere+) with - >8 GB is comfortable at native resolution. On a tight 8 GB card, or a Pascal card with no - FlashAttention (e.g. GTX 1070 → O(n²) math-attention), lower `SAM_MAX_INFER_SIDE` — see - [GPU memory](#gpu-memory-oom). +- **GPU** for SAM3 (`DEVICE` auto-selects `cuda:0`). SAM3 runs at the frame's native resolution — a + FlashAttention-capable card (Ada/Ampere+) handles full 720p in ~5 GB. A non-FlashAttention + (Pascal-class) card falls back to O(n²) math-attention and OOMs 8 GB on 720p; use a bigger/newer GPU + or sample smaller frames. - **`ffmpeg`** on `PATH` (frame extraction). Not required: the example falls back to the binary bundled by `imageio-ffmpeg` when no system `ffmpeg` is found. - **SAM3 is a gated HuggingFace model** — accept the license on the SAM3 model page, create a token, @@ -83,15 +81,10 @@ datapipe run Run a single stage: `datapipe step --labels stage=sample run`, `... stage=sam run`, `... stage=cvat run`. -## GPU memory (OOM) - -SAM3 returns masks at the input resolution, so full 720p+ frames need a lot of VRAM — an 8 GB card -OOMs on a raw 1280×720 frame. Rather than touch the (shared-with-`sam_cvat`) inference code, the -`downscale_frames` step resizes each deduped frame so its longest side is at most `SAM_MAX_INFER_SIDE` -(default **640**, fits an 8 GB card with headroom) and writes it under `IMAGES_DIR`. That resized -frame is what both SAM and CVAT use, so detection coordinates line up with no rescaling. Raise -`SAM_MAX_INFER_SIDE` on a roomy GPU for sharper masks, or set `0` to pass the original frame straight -through. +SAM3 runs at the frame's native resolution. **Field note (measured):** on a FlashAttention GPU +(Ada/Ampere+, 16 GB) full 1280×720 SAM3 peaks at only **~5.3 GB, ~0.27 s/frame**. A non-FlashAttention +(Pascal-class) 8 GB card OOMs on 720p — use a newer/bigger GPU, or reduce frame size upstream (e.g. an +`ffmpeg` `scale` filter in `extract_frames`). ## Debug UI (`datapipe api`) diff --git a/examples/video_segmentation_cvat/app.py b/examples/video_segmentation_cvat/app.py index 601bcae5..ad68f2ba 100644 --- a/examples/video_segmentation_cvat/app.py +++ b/examples/video_segmentation_cvat/app.py @@ -46,14 +46,6 @@ BatchTransform( func=steps.dedup_frames, inputs=[data.frames_tbl], - outputs=[data.deduped_frames_tbl], - transform_keys=["video_id"], - chunk_size=1, - labels=[("stage", "sample")], - ), - BatchTransform( - func=steps.downscale_frames, - inputs=[data.deduped_frames_tbl], outputs=[data.local_images_tbl], transform_keys=["video_id"], chunk_size=1, diff --git a/examples/video_segmentation_cvat/config.py b/examples/video_segmentation_cvat/config.py index d9317247..6e181c1f 100644 --- a/examples/video_segmentation_cvat/config.py +++ b/examples/video_segmentation_cvat/config.py @@ -21,15 +21,6 @@ else Path(__file__).resolve().parent / ".frames" ) -# Where downscale_frames writes the resized frames the SAM->CVAT tail consumes (one subfolder per -# video_id). Separate from FRAMES_DIR so the full-res extracts stay intact. -_IMAGES_DIR_RAW = os.environ.get("IMAGES_DIR") -IMAGES_DIR = ( - Path(_IMAGES_DIR_RAW).resolve() - if _IMAGES_DIR_RAW - else Path(__file__).resolve().parent / ".images" -) - # ffmpeg extraction rate. 1 fps gives good event coverage; 1/3fps is not meaningfully different for # walking POV footage (see README). Dedup below removes the near-duplicates either way. SAMPLE_FPS = float(os.environ.get("SAMPLE_FPS", "1")) @@ -58,12 +49,6 @@ def _resolve_ffmpeg() -> str: PHASH_MAX_DISTANCE = int(os.environ.get("PHASH_MAX_DISTANCE", "10")) PHASH_SIZE = int(os.environ.get("PHASH_SIZE", "8")) # phash hash_size (bits per side) -# Longest-side cap (px) for the downscale_frames step. SAM3 returns masks at the input resolution, so -# full 720p+ frames blow past a small GPU (an 8GB card OOMs on 1280x720). Frames are resized down to -# this before SAM sees them and CVAT gets the same resized frame, so detection coordinates line up. -# 0 disables downscaling (use only on a roomy GPU). 640 fits an 8GB card with headroom. -SAM_MAX_INFER_SIDE = int(os.environ.get("SAM_MAX_INFER_SIDE", "640")) - IMAGE_SUFFIXES = {".jpg", ".jpeg", ".png"} # --- SAM3 ----------------------------------------------------------------------------------------- diff --git a/examples/video_segmentation_cvat/data.py b/examples/video_segmentation_cvat/data.py index 5aa94bb0..62e010ed 100644 --- a/examples/video_segmentation_cvat/data.py +++ b/examples/video_segmentation_cvat/data.py @@ -37,25 +37,9 @@ ), ) -# Survivors of perceptual-hash dedup, still at the source frame resolution. video_id stays in the PK -# so dedup (grouped per video) can delete+reinsert a video's survivors. -deduped_frames_tbl = Table( - name="deduped_frames", - store=TableStoreDB( - dbconn=DBCONN, - name="deduped_frames", - data_sql_schema=[ - Column("video_id", String, primary_key=True), - Column("image_id", String, primary_key=True), - Column("frame_path", String), - ], - create_table=True, - ), -) - -# Deduped frames resized down to SAM_MAX_INFER_SIDE by downscale_frames (see steps.py) — this is what -# the SAM->CVAT tail consumes, so it stays identical to sam_cvat's `local_images`. video_id lingers in -# the PK for the per-video CVAT task split and is reduced away by sam_inference. +# Survivors of perceptual-hash dedup — this is what the SAM->CVAT tail consumes, so it stays identical +# to sam_cvat's `local_images`. video_id lingers in the PK for the per-video CVAT task split (and so +# dedup, grouped per video, can delete+reinsert a video's survivors); it is reduced away by sam_inference. local_images_tbl = Table( name="local_images", store=TableStoreDB( diff --git a/examples/video_segmentation_cvat/steps.py b/examples/video_segmentation_cvat/steps.py index 5bcbcd9f..bfc03c99 100644 --- a/examples/video_segmentation_cvat/steps.py +++ b/examples/video_segmentation_cvat/steps.py @@ -15,11 +15,9 @@ CVAT_POLYGON_LABEL, FFMPEG_BIN, FRAMES_DIR, - IMAGES_DIR, INPUT_VIDEO_DIR, PHASH_MAX_DISTANCE, PHASH_SIZE, - SAM_MAX_INFER_SIDE, SAM_TEXT_PROMPT, SAMPLE_FPS, TASK_QUEUE_ID, @@ -30,7 +28,6 @@ SAM_CONFIG_ID = "default" _LOCAL_IMAGES_COLUMNS = ["video_id", "image_id", "image_path"] -_DEDUPED_FRAMES_COLUMNS = ["video_id", "image_id", "frame_path"] _FRAMES_COLUMNS = ["video_id", "frame_id", "ts_sec", "frame_path"] _SAM_PRED_COLUMNS = [ "image_id", @@ -115,7 +112,7 @@ def dedup_frames(df_frames: pd.DataFrame) -> pd.DataFrame: more than PHASH_MAX_DISTANCE (Hamming). Walking POV frames are highly redundant, so a sequential compare-to-last-kept pass removes near-duplicates cheaply while preserving genuine scene change.""" if df_frames.empty: - return pd.DataFrame(columns=_DEDUPED_FRAMES_COLUMNS) + return pd.DataFrame(columns=_LOCAL_IMAGES_COLUMNS) records: list[dict] = [] for video_id, group in df_frames.groupby("video_id"): @@ -134,48 +131,12 @@ def dedup_frames(df_frames: pd.DataFrame) -> pd.DataFrame: { "video_id": str(video_id), "image_id": row["frame_id"], - "frame_path": row["frame_path"], + "image_path": row["frame_path"], } ) if not records: - return pd.DataFrame(columns=_DEDUPED_FRAMES_COLUMNS) - return pd.DataFrame(records, columns=_DEDUPED_FRAMES_COLUMNS) - - -def downscale_frames(df_deduped: pd.DataFrame) -> pd.DataFrame: - """Resize each deduped frame so its longest side is at most SAM_MAX_INFER_SIDE, writing the result - under IMAGES_DIR. SAM3 emits masks at the input resolution, so full 720p+ frames blow past a small - GPU; the resized frame is what both SAM and CVAT use, so detection coordinates line up. With - SAM_MAX_INFER_SIDE=0 the original frame is passed through unchanged (use only on a roomy GPU).""" - if df_deduped.empty: return pd.DataFrame(columns=_LOCAL_IMAGES_COLUMNS) - - records: list[dict] = [] - for _, row in df_deduped.iterrows(): - frame_path = str(row["frame_path"]) - if not SAM_MAX_INFER_SIDE: - image_path = frame_path - else: - out_dir = IMAGES_DIR / str(row["video_id"]) - out_dir.mkdir(parents=True, exist_ok=True) - out_path = out_dir / f"{row['image_id']}.jpg" - if not out_path.exists(): - with Image.open(frame_path) as img: - img = img.convert("RGB") - longest = max(img.size) - if longest > SAM_MAX_INFER_SIDE: - scale = SAM_MAX_INFER_SIDE / longest - img = img.resize( - (max(1, round(img.width * scale)), max(1, round(img.height * scale))), - Image.BILINEAR, - ) - img.save(out_path, quality=95) - image_path = str(out_path) - records.append( - {"video_id": str(row["video_id"]), "image_id": row["image_id"], "image_path": image_path} - ) - return pd.DataFrame(records, columns=_LOCAL_IMAGES_COLUMNS) diff --git a/examples/video_segmentation_cvat/video-native-cvat.md b/examples/video_segmentation_cvat/video-native-cvat.md new file mode 100644 index 00000000..3973f8fb --- /dev/null +++ b/examples/video_segmentation_cvat/video-native-cvat.md @@ -0,0 +1,142 @@ +# Video-native annotation in CVAT — research notes + +**Question:** can we annotate long videos *as video* in CVAT (tracks with keyframes/interpolation) +and combine that with an automated pre-annotation pipeline, instead of this example's current +"extract + dedup frames → SAM3 per-frame → upload images" approach? + +**Bottom line:** Yes. CVAT supports true video-native annotation, and a `video → SAM → CVAT tracks` +workflow is feasible and, for video, better than per-frame image tasks. CVAT ingests a video file +directly (decodes frame chunks on demand), gives annotators **Track Mode** with keyframe +interpolation, exports interpolated tracks natively, and already ships a **native SAM2 tracker** that +propagates a mask/polygon across frames. Two caveats dominate the integration: the built-in +auto-annotation `detect` protocol is **per-frame only** (tracks need a separate import type), and +uploading annotations **replaces** rather than merges existing ones. + +> Sourced from CVAT official docs + GitHub issues (see [Sources](#sources)). Confidence flags reflect +> adversarial verification; time-sensitive because CVAT docs URLs move and SAM2/SAM3 features are new. +> This deployment is self-hosted CVAT ~v2.65 — verify Enterprise-only features against it. + +--- + +## 1. Native video vs pre-extracted frames + +- CVAT ingests a **video file as a video task** using **"data on the fly"**: at task creation it + collects only minimal meta/manifest info, then decodes & caches frame chunks on demand — it does + *not* pre-extract every frame upfront (unlike our extract→upload-images flow). *(high)* + — https://docs.cvat.ai/docs/dataset_management/data-on-fly/ +- The **dataset manifest** is JSONL, one entry per keyframe `{number, pts, checksum(md5)}`, enabling + random-access seeking (seek to keyframe PTS → decode forward). *(medium — seeking framing is + inferred from code, not verbatim in docs)* + — https://docs.cvat.ai/docs/dataset_management/dataset_manifest/ +- **Track Mode** is the video-native workflow: annotators set keyframes (`K` / star) where a shape + changes; CVAT **linearly interpolates** the shape (box, polygon, …) across intermediate frames. *(high)* + — https://docs.cvat.ai/docs/annotation/manual-annotation/modes/track-mode-basics/ + — https://www.cvat.ai/academy/track-mode + +## 2. Video-task limits & gotchas + +- **Data-on-fly is not universal:** if a video has too few keyframes for smooth decoding, CVAT falls + back to full pre-extraction at task-creation time (slow) — a real risk for long/egocentric / + action-cam footage. First access is also slower. *(high)* + — https://docs.cvat.ai/docs/dataset_management/data-on-fly/ + — issues: https://github.com/cvat-ai/cvat/issues/1507 · https://github.com/cvat-ai/cvat/issues/7425 + · https://github.com/cvat-ai/cvat/issues/9519 · https://github.com/cvat-ai/cvat/issues/8913 + · https://github.com/openvinotoolkit/cvat/issues/2694 +- Reported in issues (**not independently verified**): a stall every ~36 frames at chunk boundaries + (default chunk size); heavy buffering on 4K / remote 1080p; non-zip-chunk mode is faster (`D`/`F` + hotkeys) but slightly degrades quality; **OpenH264 caps resolution at ~9.4 MP (~4K)**. +- ⚠️ **No source firmly quantified hard limits** (max length/size, codec allow-list, multi-hour + performance). Treat these as open — measure on our build. + +## 3. Tracks vs shapes, and export + +- Data model cleanly separates **`LabeledShape`** (per-frame) from **`Track`/`TrackedShape`**; the + `keyframe` and `outside` flags on a tracked shape mark keyframes and interpolation/absence + boundaries. *(high)* — https://docs.cvat.ai/docs/contributing/new-annotation-format/ +- **CVAT for video 1.1 (.xml)** represents each object as a `` whose shapes carry + `frame`/`keyframe`/`outside` — the structural basis of interpolation. *(high)* + — https://docs.cvat.ai/docs/dataset_management/formats/format-cvat/ +- Formats that **support tracks**: CVAT-for-video 1.1, CVAT-for-images 1.1, COCO, **MOT** (bbox tracks + only), **MOTS** (mask tracks), **Datumaro** (via `track_id`), Ultralytics YOLO variants. *(high)* + — https://docs.cvat.ai/docs/dataset_management/formats/ + — https://docs.cvat.ai/docs/dataset_management/formats/format-mot/ + — https://docs.cvat.ai/docs/dataset_management/formats/format-datumaro/ +- ⚠️ **Per-frame export (`group_by_frame()`) flattens tracks into shapes** — object identity is lost. + Track-aware formats iterate tracks directly. *(high)* + — https://docs.cvat.ai/docs/contributing/new-annotation-format/ + +## 4. Auto-annotation / API (most relevant to integration) + +- The SDK's built-in **`detect` protocol is per-image**: it returns per-frame shapes/tags, **not + interpolated tracks** — exactly this example's per-frame SAM3 model. *(high)* + — https://docs.cvat.ai/docs/api_sdk/sdk/auto-annotation/ +- A **separate tracking protocol** (`init_tracking_state` + `track`) propagates shapes onto subsequent + frames. Tracks are uploaded via a direct annotation-import type (`LabeledTrackRequest`), not `detect`. *(high)* + — https://docs.cvat.ai/docs/api_sdk/sdk/auto-annotation/ +- ⚠️ **Uploading annotations is destructive** — "CVAT removes the existing ones"; import is + replace, not merge, per job/task. Merge locally before upload. *(high)* + — https://docs.cvat.ai/docs/dataset_management/import-datasets/ + +## 5. SAM for video + +- CVAT ships a **native SAM2 tracker**: it propagates an **existing** polygon/mask forward across + frames (tracking, not detection), works with **polygons/masks only** (not boxes/skeletons), and has + an optional **"convert polygon shapes to tracks"**. *(high)* + — https://docs.cvat.ai/docs/annotation/auto-annotation/segment-anything-2-tracker/ + — https://www.cvat.ai/resources/changelog/video-annotation-sam-2 +- Two deployment forms: a **Nuclio serverless function (self-hosted Enterprise)** and an **"AI Agent" + worker** run on your own hardware (SAM2 tracking for CVAT Online, no server GPU/Nuclio). *(high)* + — https://www.cvat.ai/resources/blog/sam2-ai-agent-tracking +- **SAM3:** two claims that CVAT's SAM3 is visual-prompt / image-only with no text prompt were + **refuted (0-3)** — text prompts appear to be supported — but the exact scope (native video / + temporal in the UI) was **not positively confirmed**. Open question. *(low)* + — https://www.cvat.ai/resources/changelog/sam-3-image-segmentation + +## 6. Options for this pipeline + +| Approach | How | Pros | Cons | +|---|---|---|---| +| **Current** (this example) | extract+dedup frames → SAM3 per-frame → image task | Simple, already works, per-frame masks | No temporal tracks; object identity lost; more manual review | +| **Video-native, tracks from us** | upload video as video task; push SAM masks as `LabeledTrackRequest` | True tracks + interpolation for reviewers | SAM3 is per-frame with **no track_id** → needs an object-association step we don't have; destructive upload | +| **Video-native, CVAT tracks** | upload video; seed shapes; CVAT **SAM2 tracker** propagates | Least code on our side; temporal logic in CVAT | SAM2 Nuclio is **Enterprise** (check our ~v2.65 license) or run an AI-Agent worker | + +**Migration checklist:** (1) tracks must be uploaded via direct import, not `detect`; (2) upload +replaces — structure as one-shot pre-annotation before review, or merge locally; (3) confirm our +videos don't fall back to slow pre-extraction (keyframe density); (4) confirm SAM3's real scope on our +build. + +**Open questions:** concrete CVAT video limits; real SAM3 scope in v2.65; whether track import +round-trips cleanly. + +--- + +## Sources + +Primary (CVAT docs): +- data-on-fly — https://docs.cvat.ai/docs/dataset_management/data-on-fly/ +- dataset manifest — https://docs.cvat.ai/docs/dataset_management/dataset_manifest/ +- track mode — https://docs.cvat.ai/docs/annotation/manual-annotation/modes/track-mode-basics/ +- annotation formats (overview) — https://docs.cvat.ai/docs/dataset_management/formats/ +- CVAT format — https://docs.cvat.ai/docs/dataset_management/formats/format-cvat/ +- MOT format — https://docs.cvat.ai/docs/dataset_management/formats/format-mot/ +- Datumaro format — https://docs.cvat.ai/docs/dataset_management/formats/format-datumaro/ +- annotation data model — https://docs.cvat.ai/docs/contributing/new-annotation-format/ +- auto-annotation SDK — https://docs.cvat.ai/docs/api_sdk/sdk/auto-annotation/ +- import datasets — https://docs.cvat.ai/docs/dataset_management/import-datasets/ +- SAM2 tracker — https://docs.cvat.ai/docs/annotation/auto-annotation/segment-anything-2-tracker/ + +CVAT blog / academy / changelog: +- Track Mode academy — https://www.cvat.ai/academy/track-mode +- SAM2 video annotation changelog — https://www.cvat.ai/resources/changelog/video-annotation-sam-2 +- SAM2 AI-Agent tracking — https://www.cvat.ai/resources/blog/sam2-ai-agent-tracking +- SAM3 image segmentation changelog — https://www.cvat.ai/resources/changelog/sam-3-image-segmentation + +GitHub issues (video limits/perf, forum-quality): +- https://github.com/cvat-ai/cvat/issues/1507 +- https://github.com/cvat-ai/cvat/issues/7425 +- https://github.com/cvat-ai/cvat/issues/9519 +- https://github.com/cvat-ai/cvat/issues/8913 +- https://github.com/openvinotoolkit/cvat/issues/2694 + +_Method: multi-agent web research — 5 search angles, 22 sources fetched, 93 claims extracted, top 25 +adversarially verified (23 confirmed / 2 refuted). Confidence flags above reflect the vote._ From fc81d76a4edc023ec46066072919e294bb0acadd Mon Sep 17 00:00:00 2001 From: Shakirov Renat Date: Mon, 27 Jul 2026 12:12:29 +0300 Subject: [PATCH 09/13] video_segmentation_cvat: rename stage ingest -> prompt; skill: CVAT/Postgres stand-up - app.py/README/skill: the SAM-config stage was labelled "ingest" (a leftover from sam_cvat where ingest = image loading). Here frames come from the video front, so the stage only declares the text prompt -> rename to "prompt" (graph reads video -> sample -> prompt -> sam -> cvat) - setup skill: add a "Stand up CVAT + Postgres" section (docker run postgres, clone+compose CVAT v2.65.0, create admin + project/labels) and a one-line caveat for very new GPUs whose CUDA arch the pinned torch predates --- .../skills/setup-video-segmentation-cvat/SKILL.md | 14 ++++++++++++-- examples/video_segmentation_cvat/README.md | 2 +- examples/video_segmentation_cvat/app.py | 2 +- 3 files changed, 14 insertions(+), 4 deletions(-) diff --git a/.claude/skills/setup-video-segmentation-cvat/SKILL.md b/.claude/skills/setup-video-segmentation-cvat/SKILL.md index c017cae4..7ef73ecf 100644 --- a/.claude/skills/setup-video-segmentation-cvat/SKILL.md +++ b/.claude/skills/setup-video-segmentation-cvat/SKILL.md @@ -29,7 +29,7 @@ send it to a file and `grep` (`datapipe --debug run > /tmp/dp.log 2>&1; grep -nE stage=video list_videos folder INPUT_VIDEO_DIR -> video stage=sample extract_frames ffmpeg fps=SAMPLE_FPS -> frames stage=sample dedup_frames perceptual-hash dedup -> local_images -stage=ingest list_sam_config SAM_TEXT_PROMPT -> sam_config +stage=prompt list_sam_config SAM_TEXT_PROMPT -> sam_config stage=sam sam_inference SAM3 image-mode -> sam_predictions stage=sam sam_to_cvat_xml -> sam_cvat_xml stage=cvat prepare_cvat_input / CVATStep / parse_cvat_annotations -> image__annotations @@ -64,7 +64,17 @@ frames upstream (e.g. add a `scale` filter to `extract_frames`) rather than edit match `CVAT_BOX_LABEL` / `CVAT_POLYGON_LABEL` (defaults `person_box` / `person_mask`). - **`uv` + Python ≥3.10,<3.13** → `uv sync`. Pins cu124 torch, editable local libs (`../../libs/datapipe-*`, monorepo-only), builds `sam3` from a pinned git rev (+`imagehash`). After - `uv sync`, on a pre-AVX2 host re-apply `uv pip install polars-lts-cpu==1.33.1`. + `uv sync`, on a pre-AVX2 host re-apply `uv pip install polars-lts-cpu==1.33.1`. On a very new GPU + whose CUDA arch the pinned torch predates (`CUDA error: no kernel image is available`), reinstall a + matching build: `uv pip install --python .venv --reinstall torch torchvision --index-url https://download.pytorch.org/whl/cuXXX`. + +## Stand up CVAT + Postgres (if you don't already have them) +Both run in Docker; the example just points `DB_URL` / `CVAT_URL` at them (no code in it manages them). +- **Postgres:** `docker run -d --name dp_pg -e POSTGRES_PASSWORD= -e POSTGRES_DB=postgres -p 5432:5432 postgres:15` → `DB_URL=postgresql+psycopg2://postgres:@localhost:5432/postgres`. +- **CVAT v2.65.0** (matches `cvat-sdk==2.65.0`): `git clone --depth 1 --branch v2.65.0 https://github.com/cvat-ai/cvat.git /opt/cvat && cd /opt/cvat && docker compose up -d` (UI on `:8080`; ~15 containers, needs a few GB RAM + a reliable registry uplink — a flaky link fails the image pulls). The repo's `libs/datapipe-cvat/tests/start-cvat.sh` does the same but `docker compose down`s on exit (test helper), so for a persistent instance run `up -d` directly. +- **Admin** (first run): `docker exec cvat_server bash -lc "DJANGO_SUPERUSER_PASSWORD=admin python3 ~/manage.py createsuperuser --username admin --email a@e.com --noinput"`. +- **Project + labels:** create a project with labels named `CVAT_BOX_LABEL` / `CVAT_POLYGON_LABEL` (default `person_box` / `person_mask`, type `any` accepts box+polygon) and set `CVAT_PROJECT_ID` to its id. Via API: `POST /api/projects` with `{"name":...,"labels":[{"name":"person_box","type":"any"},{"name":"person_mask","type":"any"}]}`. +- Docker-less pod → keep CVAT + Postgres on another host and point `DB_URL`/`CVAT_URL` at it (see the bare-pod section). ## Get a video ```bash diff --git a/examples/video_segmentation_cvat/README.md b/examples/video_segmentation_cvat/README.md index 528e99ae..2e45a6bd 100644 --- a/examples/video_segmentation_cvat/README.md +++ b/examples/video_segmentation_cvat/README.md @@ -14,7 +14,7 @@ front is turning a long video into a deduplicated set of frames. stage=video list_videos folder INPUT_VIDEO_DIR -> video stage=sample extract_frames ffmpeg fps=SAMPLE_FPS -> frames stage=sample dedup_frames perceptual-hash dedup -> local_images -stage=ingest list_sam_config SAM_TEXT_PROMPT -> sam_config +stage=prompt list_sam_config SAM_TEXT_PROMPT -> sam_config stage=sam sam_inference SAM3 image-mode -> sam_predictions stage=sam sam_to_cvat_xml -> sam_cvat_xml stage=cvat prepare_cvat_input / CVATStep / parse_cvat_annotations -> image__annotations diff --git a/examples/video_segmentation_cvat/app.py b/examples/video_segmentation_cvat/app.py index ad68f2ba..32ff90a9 100644 --- a/examples/video_segmentation_cvat/app.py +++ b/examples/video_segmentation_cvat/app.py @@ -55,7 +55,7 @@ BatchGenerate( steps.list_sam_config, outputs=[data.sam_config_tbl], - labels=[("stage", "ingest")], + labels=[("stage", "prompt")], ), BatchTransform( func=steps.sam_inference, From 98d25deae6a61b76ebf3c331ddee295b8d847da0 Mon Sep 17 00:00:00 2001 From: Shakirov Renat Date: Mon, 27 Jul 2026 12:21:30 +0300 Subject: [PATCH 10/13] setup skill: ask demo-vs-own-data first; add two-stage demo choreography (bulk via CLI, smoke live via UI) --- .../setup-video-segmentation-cvat/SKILL.md | 31 ++++++++++++++----- 1 file changed, 24 insertions(+), 7 deletions(-) diff --git a/.claude/skills/setup-video-segmentation-cvat/SKILL.md b/.claude/skills/setup-video-segmentation-cvat/SKILL.md index 7ef73ecf..ce1d3224 100644 --- a/.claude/skills/setup-video-segmentation-cvat/SKILL.md +++ b/.claude/skills/setup-video-segmentation-cvat/SKILL.md @@ -11,12 +11,14 @@ description: > This skill = turn a long video into review-ready CVAT pre-annotations. Front stage samples + dedups frames; the SAM3→CVAT tail is identical to [`../sam_cvat`](../sam_cvat). Set the knobs below first. -**Ask first — don't assume (only the unresolved):** which videos (built-in city-walk set via -`scripts/fetch_video.py`, or the user's own) → `INPUT_VIDEO_DIR`? **`SAMPLE_FPS`** (how densely to -sample — the single biggest lever on frame count / run time) and **`SEGMENT_SIZE`** (frames per CVAT -job)? **which Postgres + which database** for `DB_URL` — never point at an existing DB or use a -default without confirming; external CVAT ready or provision it? reuse an existing venv / `uv` env or -create fresh? which GPU (VRAM + FlashAttention)? surface stage logs or run quiet? +**Ask first — don't assume (only the unresolved):** **is this a live demo on the built-in test +videos, or a real run on the user's own data?** (demo → the two-stage choreography below); which videos +(built-in city-walk set via `scripts/fetch_video.py`, or the user's own) → `INPUT_VIDEO_DIR`? +**`SAMPLE_FPS`** (how densely to sample — the single biggest lever on frame count / run time) and +**`SEGMENT_SIZE`** (frames per CVAT job)? **which Postgres + which database** for `DB_URL` — never +point at an existing DB or use a default without confirming; external CVAT ready or provision it? +reuse an existing venv / `uv` env or create fresh? which GPU (VRAM + FlashAttention)? surface stage +logs or run quiet? **How to work:** read the setup, propose a short plan, get a go-ahead before touching anything. Prepare `.env` and **pause for the user to verify it** before running. Run stages with logs shown; @@ -93,7 +95,22 @@ datapipe db create-all && datapipe run # by stage: datapipe step --labels stage=sample run (then stage=sam, stage=cvat) ``` Run from `examples/video_segmentation_cvat/` (`app.py` `load_dotenv()`s before importing config). -**Live demo:** drop a ~10 s clip in `INPUT_VIDEO_DIR` → `datapipe run` → a CVAT task appears in seconds. + +## Demo choreography — two stages (built-in test set) +For a live demo, split the built-in videos into a pre-baked bulk and one small live clip, so the +audience sees a run finish in seconds instead of waiting hours: + +- **Stage 1 — bulk, YOU run it fully ahead of time (CLI).** Put every built-in video **except** + `smoke_shibuya_3min` in `INPUT_VIDEO_DIR` and `datapipe run` to completion. This is the slow part + (the multi-hour city-walk videos) and pre-fills CVAT with the heavy tasks. Do it before the demo, + not on stage. Keep the UI **off** during this CLI run (see the `datapipe api` collision below). +- **Stage 2 — smoke clip, LIVE via the UI.** During the demo, start `datapipe api`, drop + `smoke_shibuya_3min` into `INPUT_VIDEO_DIR`, and trigger the run from the **UI's per-stage run + buttons**. It's tiny, so frames → SAM3 → a fresh CVAT task appear within seconds, live — and running + from the UI (single writer) avoids the run-log collision that a concurrent CLI run would cause. + +So: bulk is proof-of-scale (done offline), smoke is the live "watch it work" moment. On the user's own +data there's no split — just one `datapipe run` (or per-video, incrementally). ## Deploy on a bare GPU pod (no Docker) — field-tested The example itself needs **no Docker** — it's a plain `uv` venv. Docker is only for the infra deps From 11444cf6b2066ccf201d3fdbb3b8fbed08ee405a Mon Sep 17 00:00:00 2001 From: Shakirov Renat Date: Mon, 27 Jul 2026 12:30:57 +0300 Subject: [PATCH 11/13] video_segmentation_cvat: fetch built-in videos from the demo bucket instead of YouTube fetch_video.py now curls pre-encoded 720p .webm from $VIDEO_BUCKET_URL (default the e8-demo bucket) by key (drops yt-dlp/deno/cookies/n-challenge). Keys split into the city-walk set + smoke clip so the two-stage demo can fetch them separately. README + setup skill updated; the YouTube-ToS provenance note stays. --- .../setup-video-segmentation-cvat/SKILL.md | 12 ++- examples/video_segmentation_cvat/README.md | 24 +++-- .../scripts/fetch_video.py | 96 ++++++++++--------- 3 files changed, 68 insertions(+), 64 deletions(-) diff --git a/.claude/skills/setup-video-segmentation-cvat/SKILL.md b/.claude/skills/setup-video-segmentation-cvat/SKILL.md index ce1d3224..ce9263c4 100644 --- a/.claude/skills/setup-video-segmentation-cvat/SKILL.md +++ b/.claude/skills/setup-video-segmentation-cvat/SKILL.md @@ -80,12 +80,14 @@ Both run in Docker; the example just points `DB_URL` / `CVAT_URL` at them (no co ## Get a video ```bash -python scripts/fetch_video.py --height 720 # built-in ~24h city-walk set -python scripts/fetch_video.py --dir videos "https://youtu.be/ID" # your own (needs yt-dlp + deno) -python scripts/fetch_video.py --section 00:10:00-00:20:00 "https://youtu.be/ID" # just a clip +python scripts/fetch_video.py # whole built-in test set (city walks + smoke clip) +python scripts/fetch_video.py smoke_shibuya_3min # only the smoke clip (stage-2 live demo) +python scripts/fetch_video.py 27Pv4Cg4EV4 ... # specific bucket keys ``` -Needs `yt-dlp` + a JS runtime (`deno`/`node`) for YouTube's n-challenge; sign-in-gated videos need -`--cookies-from-browser chrome`. Internal-demo only (YouTube ToS) — don't redistribute. +Pulls pre-encoded 720p `.webm` via `curl` (resumable) from the demo bucket (`$VIDEO_BUCKET_URL`, +default the `e8-demo` bucket). Internal-demo only (source under YouTube ToS) — don't redistribute. +For the two-stage demo: stage 1 fetches the city walks, stage 2 fetches only `smoke_shibuya_3min`. +Own footage → just drop any video file into `INPUT_VIDEO_DIR`. ## Run ```bash diff --git a/examples/video_segmentation_cvat/README.md b/examples/video_segmentation_cvat/README.md index 2e45a6bd..9d0f21f6 100644 --- a/examples/video_segmentation_cvat/README.md +++ b/examples/video_segmentation_cvat/README.md @@ -52,22 +52,20 @@ frames are processed. ## Get a video ```bash -# built-in ~24h set of busy IN/JP/US city walks (720p), into $INPUT_VIDEO_DIR: -python scripts/fetch_video.py --height 720 -# or your own: -python scripts/fetch_video.py --dir videos "https://youtu.be/VIDEO_ID" -# or just a clip (needs ffmpeg): -python scripts/fetch_video.py --section 00:10:00-00:20:00 "https://youtu.be/VIDEO_ID" +# whole built-in ~24h test set (busy IN/JP/US city walks + a 3-min smoke clip), into $INPUT_VIDEO_DIR: +python scripts/fetch_video.py +# just the smoke clip (for the live demo): +python scripts/fetch_video.py smoke_shibuya_3min +# specific videos (bucket keys): +python scripts/fetch_video.py 27Pv4Cg4EV4 BsiHD4m6_BU ``` -`fetch_video.py` needs `yt-dlp` and, for many YouTube videos, a JS runtime to solve YouTube's -n-challenge — install **`deno`** (or `node`); the EJS solver script is auto-fetched via -`--remote-components ejs:github`. Videos that demand sign-in ("confirm you're not a bot") also need -browser cookies — the script passes `--cookies-from-browser chrome` by default (be logged into -YouTube in that browser; use `--cookies-from-browser ""` to disable). +`fetch_video.py` pulls the pre-encoded 720p `.webm` files with `curl` (resumable) from the demo +object-storage bucket — override the base URL with `$VIDEO_BUCKET_URL`. For your own footage, drop any +video file into `INPUT_VIDEO_DIR` directly (`.mp4/.mkv/.mov/.webm/.m4v`). -Internal-demo only — downloading violates YouTube ToS; do not redistribute the videos or the frames. -For license-clean footage use stock (Pexels/Mixkit) or a dataset clip (EPIC-KITCHENS / Ego4D). +Internal-demo only — the source footage is under YouTube ToS; do not redistribute the videos or the +frames. For license-clean footage use stock (Pexels/Mixkit) or a dataset clip (EPIC-KITCHENS / Ego4D). ## Run diff --git a/examples/video_segmentation_cvat/scripts/fetch_video.py b/examples/video_segmentation_cvat/scripts/fetch_video.py index ab76eb77..816f8729 100644 --- a/examples/video_segmentation_cvat/scripts/fetch_video.py +++ b/examples/video_segmentation_cvat/scripts/fetch_video.py @@ -1,14 +1,16 @@ #!/usr/bin/env python3 -"""Download egocentric source videos into INPUT_VIDEO_DIR with yt-dlp. +"""Download the built-in egocentric test videos into INPUT_VIDEO_DIR from the demo bucket. -Internal-demo use only: downloading violates YouTube ToS, so do not redistribute the videos or the -frames sampled from them. For license-clean footage use stock (Pexels/Mixkit) or a dataset clip. +The ~24h city-walk set (Japan ~8h, USA ~8h, India ~8h across 5 clips) plus a 3-min smoke clip live in +an object-storage bucket as pre-encoded 720p `.webm`. Override the base URL with $VIDEO_BUCKET_URL. + +Internal-demo use only: the source footage is under YouTube ToS — do not redistribute the videos or +the frames sampled from them. Usage: - python scripts/fetch_video.py # the default ~24h walk set (IN/JP/US), 720p - python scripts/fetch_video.py --height 480 # smaller download - python scripts/fetch_video.py URL [URL ...] # your own videos - python scripts/fetch_video.py --section 00:10:00-00:20:00 URL # a clip only (needs ffmpeg) + python scripts/fetch_video.py # the whole built-in test set + python scripts/fetch_video.py smoke_shibuya_3min # just the 3-min smoke clip (live demo) + python scripts/fetch_video.py 27Pv4Cg4EV4 BsiHD4m6_BU # specific bucket keys """ from __future__ import annotations @@ -18,26 +20,39 @@ import sys from pathlib import Path -# Verified (yt-dlp) ~24h of busy first-person city walks: Japan 8h, USA 8h, India ~8h (5 clips). -DEFAULT_VIDEOS = [ - "https://youtu.be/BsiHD4m6_BU", # Tokyo, 9 districts, 8:06:59, 4K - "https://youtu.be/27Pv4Cg4EV4", # New York full city walk, 8:04:31, 4K - "https://youtu.be/60Q5E0KZb38", # Mumbai markets, 2:41:51, 4K - "https://youtu.be/qskdzPj39hE", # New Delhi Paharganj, 1:57:15, 4K - "https://youtu.be/8W4ZTX1z02E", # Mumbai busy streets, 1:36:11, 4K - "https://youtu.be/7wBNtsgqNOI", # New Delhi crowds, 0:58:59, 4K - "https://youtu.be/Lteooc0BHtk", # New Delhi streets, 0:39:41, 4K +# Base URL of the demo bucket holding the built-in test videos (public-read). Override via env. +BUCKET_URL = os.environ.get( + "VIDEO_BUCKET_URL", "https://storage.yandexcloud.net/e8-demo/robots-ego-video" +) + +# Built-in test set as bucket keys (without .webm). The 3-min smoke clip is listed separately so it +# can be fetched on its own for the live-demo stage (see the setup skill's two-stage choreography). +CITY_WALKS = [ + "BsiHD4m6_BU", # Tokyo, 9 districts, ~8h + "27Pv4Cg4EV4", # New York full city walk, ~8h + "60Q5E0KZb38", # Mumbai markets, ~2h40 + "qskdzPj39hE", # New Delhi Paharganj, ~2h + "8W4ZTX1z02E", # Mumbai busy streets, ~1h35 + "7wBNtsgqNOI", # New Delhi crowds, ~1h + "Lteooc0BHtk", # New Delhi streets, ~40m ] +SMOKE = "smoke_shibuya_3min" # 3-min live-demo clip +DEFAULT_KEYS = CITY_WALKS + [SMOKE] def main() -> int: - parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) - parser.add_argument("urls", nargs="*", help="video URLs (default: the built-in ~24h walk set)") - parser.add_argument("--dir", default=os.environ.get("INPUT_VIDEO_DIR"), help="target dir (default: $INPUT_VIDEO_DIR)") - parser.add_argument("--height", type=int, default=720, help="max video height, e.g. 480/720/1080 (default 720)") - parser.add_argument("--section", default=None, help="download only a section, e.g. 00:10:00-00:20:00") - parser.add_argument("--cookies-from-browser", default="chrome", - help="browser to read cookies from (chrome/safari/firefox/...); '' to disable") + parser = argparse.ArgumentParser( + description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter + ) + parser.add_argument( + "keys", nargs="*", help="bucket keys to fetch, without .webm (default: the whole built-in set)" + ) + parser.add_argument( + "--dir", default=os.environ.get("INPUT_VIDEO_DIR"), help="target dir (default: $INPUT_VIDEO_DIR)" + ) + parser.add_argument( + "--base-url", default=BUCKET_URL, help="bucket base URL (default: $VIDEO_BUCKET_URL)" + ) args = parser.parse_args() if not args.dir: @@ -45,29 +60,18 @@ def main() -> int: out_dir = Path(args.dir).resolve() out_dir.mkdir(parents=True, exist_ok=True) - urls = args.urls or DEFAULT_VIDEOS - fmt = f"bv*[height<={args.height}]+ba/b[height<={args.height}]/b[height<={args.height}]" - cmd = [ - "yt-dlp", - "-N", "4", - "--retries", "infinite", - "--fragment-retries", "infinite", - "--sleep-interval", "3", "--max-sleep-interval", "12", - # Some videos require sign-in ("confirm you're not a bot") -> pass browser cookies. Solving - # YouTube's JS n-challenge needs a JS runtime (install `deno` or `node`) plus the EJS solver - # script, fetched by --remote-components; without it only storyboard images are returned. - "--remote-components", "ejs:github", - "-f", fmt, - "-o", str(out_dir / "%(id)s.%(ext)s"), - ] - if args.cookies_from_browser: - cmd += ["--cookies-from-browser", args.cookies_from_browser] - if args.section: - cmd += ["--download-sections", f"*{args.section}", "--force-keyframes-at-cuts"] - cmd += urls - - print(f"Downloading {len(urls)} video(s) -> {out_dir} (<= {args.height}p)", file=sys.stderr) - return subprocess.run(cmd).returncode + keys = args.keys or DEFAULT_KEYS + base = args.base_url.rstrip("/") + rc = 0 + for key in keys: + key = key[:-5] if key.endswith(".webm") else key + url = f"{base}/{key}.webm" + dst = out_dir / f"{key}.webm" + print(f"-> {dst} <- {url}", file=sys.stderr) + # -f fail on HTTP errors, -S show errors, -L follow redirects, -C - resume a partial download. + ret = subprocess.run(["curl", "-fSL", "-C", "-", "-o", str(dst), url]).returncode + rc = rc or ret + return rc if __name__ == "__main__": From fe4815909aecccf6400a42272039502ada127054 Mon Sep 17 00:00:00 2001 From: Shakirov Renat Date: Mon, 27 Jul 2026 13:33:01 +0300 Subject: [PATCH 12/13] video_segmentation_cvat: raise SAM_MAX_DETECTIONS default 20->50; drop research note ~25% of frames in the 720p run hit the old 20-detection cap (busy street scenes have >20 people), so raise the default to 50. Remove video-native-cvat.md (a one-off CVAT-video research note, not part of the example). --- examples/video_segmentation_cvat/.env.example | 4 +- examples/video_segmentation_cvat/config.py | 2 +- .../video-native-cvat.md | 142 ------------------ 3 files changed, 4 insertions(+), 144 deletions(-) delete mode 100644 examples/video_segmentation_cvat/video-native-cvat.md diff --git a/examples/video_segmentation_cvat/.env.example b/examples/video_segmentation_cvat/.env.example index 02bd0ba5..68772a8e 100644 --- a/examples/video_segmentation_cvat/.env.example +++ b/examples/video_segmentation_cvat/.env.example @@ -16,7 +16,9 @@ PHASH_MAX_DISTANCE=10 HF_TOKEN=replace-me SAM_TEXT_PROMPT=person SAM_SCORE_THRESHOLD=0.5 -SAM_MAX_DETECTIONS=20 +# Max detections kept per frame (top-N by score). Crowded street scenes often have >20 people, so a +# low cap drops real detections — 50 suits busy egocentric footage; raise/lower per your scenes. +SAM_MAX_DETECTIONS=50 # HF caches the gated model + token under $HF_HOME (default ~/.cache/huggingface). Point it at a # writable path if the home dir is read-only (else login()/weights download fail on such hosts). # HF_HOME=/var/tmp/hf_home diff --git a/examples/video_segmentation_cvat/config.py b/examples/video_segmentation_cvat/config.py index 6e181c1f..197317f2 100644 --- a/examples/video_segmentation_cvat/config.py +++ b/examples/video_segmentation_cvat/config.py @@ -58,7 +58,7 @@ def _resolve_ffmpeg() -> str: HF_TOKEN = os.environ.get("HF_TOKEN", "") SAM_TEXT_PROMPT = os.environ.get("SAM_TEXT_PROMPT", "person") SAM_SCORE_THRESHOLD = float(os.environ.get("SAM_SCORE_THRESHOLD", "0.5")) -SAM_MAX_DETECTIONS = int(os.environ.get("SAM_MAX_DETECTIONS", "20")) +SAM_MAX_DETECTIONS = int(os.environ.get("SAM_MAX_DETECTIONS", "50")) # --- CVAT ----------------------------------------------------------------------------------------- diff --git a/examples/video_segmentation_cvat/video-native-cvat.md b/examples/video_segmentation_cvat/video-native-cvat.md deleted file mode 100644 index 3973f8fb..00000000 --- a/examples/video_segmentation_cvat/video-native-cvat.md +++ /dev/null @@ -1,142 +0,0 @@ -# Video-native annotation in CVAT — research notes - -**Question:** can we annotate long videos *as video* in CVAT (tracks with keyframes/interpolation) -and combine that with an automated pre-annotation pipeline, instead of this example's current -"extract + dedup frames → SAM3 per-frame → upload images" approach? - -**Bottom line:** Yes. CVAT supports true video-native annotation, and a `video → SAM → CVAT tracks` -workflow is feasible and, for video, better than per-frame image tasks. CVAT ingests a video file -directly (decodes frame chunks on demand), gives annotators **Track Mode** with keyframe -interpolation, exports interpolated tracks natively, and already ships a **native SAM2 tracker** that -propagates a mask/polygon across frames. Two caveats dominate the integration: the built-in -auto-annotation `detect` protocol is **per-frame only** (tracks need a separate import type), and -uploading annotations **replaces** rather than merges existing ones. - -> Sourced from CVAT official docs + GitHub issues (see [Sources](#sources)). Confidence flags reflect -> adversarial verification; time-sensitive because CVAT docs URLs move and SAM2/SAM3 features are new. -> This deployment is self-hosted CVAT ~v2.65 — verify Enterprise-only features against it. - ---- - -## 1. Native video vs pre-extracted frames - -- CVAT ingests a **video file as a video task** using **"data on the fly"**: at task creation it - collects only minimal meta/manifest info, then decodes & caches frame chunks on demand — it does - *not* pre-extract every frame upfront (unlike our extract→upload-images flow). *(high)* - — https://docs.cvat.ai/docs/dataset_management/data-on-fly/ -- The **dataset manifest** is JSONL, one entry per keyframe `{number, pts, checksum(md5)}`, enabling - random-access seeking (seek to keyframe PTS → decode forward). *(medium — seeking framing is - inferred from code, not verbatim in docs)* - — https://docs.cvat.ai/docs/dataset_management/dataset_manifest/ -- **Track Mode** is the video-native workflow: annotators set keyframes (`K` / star) where a shape - changes; CVAT **linearly interpolates** the shape (box, polygon, …) across intermediate frames. *(high)* - — https://docs.cvat.ai/docs/annotation/manual-annotation/modes/track-mode-basics/ - — https://www.cvat.ai/academy/track-mode - -## 2. Video-task limits & gotchas - -- **Data-on-fly is not universal:** if a video has too few keyframes for smooth decoding, CVAT falls - back to full pre-extraction at task-creation time (slow) — a real risk for long/egocentric / - action-cam footage. First access is also slower. *(high)* - — https://docs.cvat.ai/docs/dataset_management/data-on-fly/ - — issues: https://github.com/cvat-ai/cvat/issues/1507 · https://github.com/cvat-ai/cvat/issues/7425 - · https://github.com/cvat-ai/cvat/issues/9519 · https://github.com/cvat-ai/cvat/issues/8913 - · https://github.com/openvinotoolkit/cvat/issues/2694 -- Reported in issues (**not independently verified**): a stall every ~36 frames at chunk boundaries - (default chunk size); heavy buffering on 4K / remote 1080p; non-zip-chunk mode is faster (`D`/`F` - hotkeys) but slightly degrades quality; **OpenH264 caps resolution at ~9.4 MP (~4K)**. -- ⚠️ **No source firmly quantified hard limits** (max length/size, codec allow-list, multi-hour - performance). Treat these as open — measure on our build. - -## 3. Tracks vs shapes, and export - -- Data model cleanly separates **`LabeledShape`** (per-frame) from **`Track`/`TrackedShape`**; the - `keyframe` and `outside` flags on a tracked shape mark keyframes and interpolation/absence - boundaries. *(high)* — https://docs.cvat.ai/docs/contributing/new-annotation-format/ -- **CVAT for video 1.1 (.xml)** represents each object as a `` whose shapes carry - `frame`/`keyframe`/`outside` — the structural basis of interpolation. *(high)* - — https://docs.cvat.ai/docs/dataset_management/formats/format-cvat/ -- Formats that **support tracks**: CVAT-for-video 1.1, CVAT-for-images 1.1, COCO, **MOT** (bbox tracks - only), **MOTS** (mask tracks), **Datumaro** (via `track_id`), Ultralytics YOLO variants. *(high)* - — https://docs.cvat.ai/docs/dataset_management/formats/ - — https://docs.cvat.ai/docs/dataset_management/formats/format-mot/ - — https://docs.cvat.ai/docs/dataset_management/formats/format-datumaro/ -- ⚠️ **Per-frame export (`group_by_frame()`) flattens tracks into shapes** — object identity is lost. - Track-aware formats iterate tracks directly. *(high)* - — https://docs.cvat.ai/docs/contributing/new-annotation-format/ - -## 4. Auto-annotation / API (most relevant to integration) - -- The SDK's built-in **`detect` protocol is per-image**: it returns per-frame shapes/tags, **not - interpolated tracks** — exactly this example's per-frame SAM3 model. *(high)* - — https://docs.cvat.ai/docs/api_sdk/sdk/auto-annotation/ -- A **separate tracking protocol** (`init_tracking_state` + `track`) propagates shapes onto subsequent - frames. Tracks are uploaded via a direct annotation-import type (`LabeledTrackRequest`), not `detect`. *(high)* - — https://docs.cvat.ai/docs/api_sdk/sdk/auto-annotation/ -- ⚠️ **Uploading annotations is destructive** — "CVAT removes the existing ones"; import is - replace, not merge, per job/task. Merge locally before upload. *(high)* - — https://docs.cvat.ai/docs/dataset_management/import-datasets/ - -## 5. SAM for video - -- CVAT ships a **native SAM2 tracker**: it propagates an **existing** polygon/mask forward across - frames (tracking, not detection), works with **polygons/masks only** (not boxes/skeletons), and has - an optional **"convert polygon shapes to tracks"**. *(high)* - — https://docs.cvat.ai/docs/annotation/auto-annotation/segment-anything-2-tracker/ - — https://www.cvat.ai/resources/changelog/video-annotation-sam-2 -- Two deployment forms: a **Nuclio serverless function (self-hosted Enterprise)** and an **"AI Agent" - worker** run on your own hardware (SAM2 tracking for CVAT Online, no server GPU/Nuclio). *(high)* - — https://www.cvat.ai/resources/blog/sam2-ai-agent-tracking -- **SAM3:** two claims that CVAT's SAM3 is visual-prompt / image-only with no text prompt were - **refuted (0-3)** — text prompts appear to be supported — but the exact scope (native video / - temporal in the UI) was **not positively confirmed**. Open question. *(low)* - — https://www.cvat.ai/resources/changelog/sam-3-image-segmentation - -## 6. Options for this pipeline - -| Approach | How | Pros | Cons | -|---|---|---|---| -| **Current** (this example) | extract+dedup frames → SAM3 per-frame → image task | Simple, already works, per-frame masks | No temporal tracks; object identity lost; more manual review | -| **Video-native, tracks from us** | upload video as video task; push SAM masks as `LabeledTrackRequest` | True tracks + interpolation for reviewers | SAM3 is per-frame with **no track_id** → needs an object-association step we don't have; destructive upload | -| **Video-native, CVAT tracks** | upload video; seed shapes; CVAT **SAM2 tracker** propagates | Least code on our side; temporal logic in CVAT | SAM2 Nuclio is **Enterprise** (check our ~v2.65 license) or run an AI-Agent worker | - -**Migration checklist:** (1) tracks must be uploaded via direct import, not `detect`; (2) upload -replaces — structure as one-shot pre-annotation before review, or merge locally; (3) confirm our -videos don't fall back to slow pre-extraction (keyframe density); (4) confirm SAM3's real scope on our -build. - -**Open questions:** concrete CVAT video limits; real SAM3 scope in v2.65; whether track import -round-trips cleanly. - ---- - -## Sources - -Primary (CVAT docs): -- data-on-fly — https://docs.cvat.ai/docs/dataset_management/data-on-fly/ -- dataset manifest — https://docs.cvat.ai/docs/dataset_management/dataset_manifest/ -- track mode — https://docs.cvat.ai/docs/annotation/manual-annotation/modes/track-mode-basics/ -- annotation formats (overview) — https://docs.cvat.ai/docs/dataset_management/formats/ -- CVAT format — https://docs.cvat.ai/docs/dataset_management/formats/format-cvat/ -- MOT format — https://docs.cvat.ai/docs/dataset_management/formats/format-mot/ -- Datumaro format — https://docs.cvat.ai/docs/dataset_management/formats/format-datumaro/ -- annotation data model — https://docs.cvat.ai/docs/contributing/new-annotation-format/ -- auto-annotation SDK — https://docs.cvat.ai/docs/api_sdk/sdk/auto-annotation/ -- import datasets — https://docs.cvat.ai/docs/dataset_management/import-datasets/ -- SAM2 tracker — https://docs.cvat.ai/docs/annotation/auto-annotation/segment-anything-2-tracker/ - -CVAT blog / academy / changelog: -- Track Mode academy — https://www.cvat.ai/academy/track-mode -- SAM2 video annotation changelog — https://www.cvat.ai/resources/changelog/video-annotation-sam-2 -- SAM2 AI-Agent tracking — https://www.cvat.ai/resources/blog/sam2-ai-agent-tracking -- SAM3 image segmentation changelog — https://www.cvat.ai/resources/changelog/sam-3-image-segmentation - -GitHub issues (video limits/perf, forum-quality): -- https://github.com/cvat-ai/cvat/issues/1507 -- https://github.com/cvat-ai/cvat/issues/7425 -- https://github.com/cvat-ai/cvat/issues/9519 -- https://github.com/cvat-ai/cvat/issues/8913 -- https://github.com/openvinotoolkit/cvat/issues/2694 - -_Method: multi-agent web research — 5 search angles, 22 sources fetched, 93 claims extracted, top 25 -adversarially verified (23 confirmed / 2 refuted). Confidence flags above reflect the vote._ From 01e1087dbcde7a2d8f47d8151907b9fb89808fa3 Mon Sep 17 00:00:00 2001 From: Shakirov Renat Date: Mon, 27 Jul 2026 21:22:21 +0300 Subject: [PATCH 13/13] setup skill: fix gaps from cold QA run (cu128 for Blackwell, verify-checkout-current, flaky torch index) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit From a from-scratch subagent dry-run of stage 1: (1) name cu128 explicitly for Blackwell/50-series + note the index can 503 (retry); (2) tell the operator to verify the checkout is current — a stale one ships the old YouTube fetch_video.py instead of the bucket-fetch version. --- .claude/skills/setup-video-segmentation-cvat/SKILL.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/.claude/skills/setup-video-segmentation-cvat/SKILL.md b/.claude/skills/setup-video-segmentation-cvat/SKILL.md index ce9263c4..0fc503f6 100644 --- a/.claude/skills/setup-video-segmentation-cvat/SKILL.md +++ b/.claude/skills/setup-video-segmentation-cvat/SKILL.md @@ -67,8 +67,10 @@ frames upstream (e.g. add a `scale` filter to `extract_frames`) rather than edit - **`uv` + Python ≥3.10,<3.13** → `uv sync`. Pins cu124 torch, editable local libs (`../../libs/datapipe-*`, monorepo-only), builds `sam3` from a pinned git rev (+`imagehash`). After `uv sync`, on a pre-AVX2 host re-apply `uv pip install polars-lts-cpu==1.33.1`. On a very new GPU - whose CUDA arch the pinned torch predates (`CUDA error: no kernel image is available`), reinstall a - matching build: `uv pip install --python .venv --reinstall torch torchvision --index-url https://download.pytorch.org/whl/cuXXX`. + whose CUDA arch the pinned torch predates (`CUDA error: no kernel image is available` on a tiny cuda + matmul), reinstall a matching build — for **Blackwell / RTX 50-series use `cu128`** (torch ≥2.7): + `uv pip install --python .venv --reinstall torch torchvision --index-url https://download.pytorch.org/whl/cu128`. + That index can flake (503s) — wrap it in a retry loop. ## Stand up CVAT + Postgres (if you don't already have them) Both run in Docker; the example just points `DB_URL` / `CVAT_URL` at them (no code in it manages them). @@ -122,7 +124,9 @@ The example itself needs **no Docker** — it's a plain `uv` venv. Docker is onl Setup that worked, in order: 1. **Get the code on the branch** without disturbing an existing checkout: from the repo dir, `git fetch origin ` then `git worktree add ` (reuses the host's git creds, - isolated working tree). + isolated working tree). **Verify it's current** (`git log origin/..HEAD` empty) — a stale + checkout can ship the old YouTube `fetch_video.py` (needs yt-dlp) instead of the current + bucket-fetch one, and an out-of-date pipeline; `git pull`/re-checkout the file if behind. 2. **`uv sync`** in the example dir. `uv` auto-fetches a Python 3.10–3.12 even if the base interpreter is older. Takes a few min (torch cu124 ~2.5 GB + builds `sam3`/`cv-pipeliner` from git). **`sam3` builds with no `nvcc`/CUDA toolkit** — it's pure PyTorch, no compiled CUDA ext (the usual build