Hardware-Aware Benchmarking of Single & Dual Visual Backbones

When is Hybrid Better? Hardware-Aware Benchmarking of Single and Dual Visual Backbones for Real-Time Driving Scenario Video Understanding
Reference implementation — IEEE JETCAS, 2026

This repository profiles five frozen visual foundation backbones and their pairwise hybrids for real-time driving-scenario video understanding, pairing downstream accuracy with directly measured throughput, power, memory, and energy. It is organized into three self-contained components:

Directory	What it does	Frames	Focus
carla-benchmark/	H100 / server hardware-aware benchmark of 5 singles + 10 hybrids at FP32/FP16/INT8 on the CARLA corpus (a DAAD T=3 preset is also included).	T=3 early window (0.0/0.5/1.0 s @ 2 Hz)	Accuracy + measured hardware cost
daad-benchmark/	Accuracy-focused DAAD classifier that uses all 16 frames of each clip with a deeper temporal-transformer head.	all 16 frames (full clip)	Accuracy
jetson/	Jetson Orin Nano edge validation (Dockerized JetPack PyTorch hardware measurements).	—	Edge hardware cost

---

The two data recipes

The two driving benchmarks differ deliberately in how each clip is read:

• CARLA (carla-benchmark/) — T=3 early-anticipation window. Only the first three frames (the opening second at 2 Hz) of every clip are used; the maneuver tail is bypassed. This is the setting profiled against measured throughput / power / energy in the paper.

• DAAD (daad-benchmark/) — all 16 frames. The DAAD front-camera export stores exactly 16 frames per clip. This pipeline feeds all 16 as one sequence → one maneuver label, spanning the whole maneuver, and pairs them with a larger DeepTemporalHead for higher accuracy.

Both keep the same fair frozen-backbone probing protocol: features cached per split, z-score fit on train only, early stopping on validation macro-F1, and the test split touched exactly once.

---

Installation

Requires Python 3.10+ and (for hardware measurements) an NVIDIA GPU with a working CUDA toolchain; the accuracy paths also run on CPU. Dependencies are shared across all components:

git clone <your-fork-url>
cd video-encoder-benchmark

python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

Credentials & dataset roots

DINOv3 and Qwen3-VL are gated on the Hugging Face Hub. Put secrets in a .env at the repo root (loaded automatically by both benchmarks) — never commit it:

cp .env.example .env
# then edit .env:
#   HF_TOKEN=hf_xxx
#   CARLA_DATA_ROOT=/path/to/carla
#   DAAD_DATA_ROOT=/path/to/daad

---

Quick start

# CARLA hardware-aware grid (singles + hybrids + tables):
cd carla-benchmark
python train.py --dataset carla --data_root /path/to/carla --out_dir ./hw_out_carla

# DAAD 16-frame accuracy run:
cd ../daad-benchmark
python train.py --backbone dinov3 --data_root /path/to/daad --out_dir ./out_daad16

See each component's README for the full command-line reference, SLURM runner (run.sh), outputs, and the reproducibility / fairness protocol:

• carla-benchmark/README.md
• daad-benchmark/README.md
• jetson/README.md

---

Backbones

Key	Checkpoint	Params	Feat. dim	Paradigm
dinov2	facebook/dinov2-base	86.6 M	768	Self-supervised distillation
dinov3	facebook/dinov3-vitb16-pretrain-lvd1689m	85.7 M	768	Self-supervised + Gram
clip	openai/clip-vit-base-patch16	87.5 M	768	Contrastive image–text
siglip2	google/siglip2-base-patch16-224	92.9 M	768	Sigmoid contrastive
qwen3vl	Qwen/Qwen3-VL-2B-Instruct	2.0 B	2048	Full multimodal VLM

---

Project layout

.
├── README.md              # this overview
├── requirements.txt       # shared, pinned dependency floors
├── .env.example           # template for HF_TOKEN / dataset roots
├── LICENSE                # MIT
├── carla-benchmark/       # H100/server T=3 hardware-aware benchmark
│   ├── train.py
│   ├── run.sh             # SLURM submission wrapper
│   └── README.md
├── daad-benchmark/        # DAAD all-16-frames accuracy pipeline
│   ├── train.py
│   ├── run.sh
│   └── README.md
└── jetson/                # Jetson Orin Nano edge-validation pipeline
    ├── main.py
    ├── run.sh
    ├── Dockerfile
    ├── docker-compose.yaml
    └── README.md

---

License

Released under the MIT License.

This project received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 101006664.