diff --git a/.gitignore b/.gitignore index 1b53178..2314085 100644 --- a/.gitignore +++ b/.gitignore @@ -62,3 +62,4 @@ nul # Claude AI .claude/ CLAUDE.md +.DS_Store diff --git a/README.md b/README.md index 24377b2..5d8d0da 100644 --- a/README.md +++ b/README.md @@ -43,7 +43,7 @@ This aims to be a camera-agnostic ROS2 wrapper for Depth Anything 3 (DA3), provi - **Docker Support**: Pre-configured Docker and Docker Compose files - **Example Images**: Sample test images and benchmark scripts included - **Performance Profiling**: Built-in benchmarking and profiling tools -- **TensorRT Support**: Validated 6.8x speedup on Jetson (35.3 FPS) - see [TensorRT Status](#tensorrt-status-validated) +- **TensorRT Support**: Validated 7.7x speedup on Jetson (40 FPS @ 518x518, 93 FPS @ 308x308) - see [TensorRT Status](#tensorrt-status-validated) - **Post-Processing**: Depth map filtering, hole filling, and enhancement - **INT8 Quantization**: Model compression for faster inference - **ONNX Export**: Deploy to various platforms and runtimes @@ -516,7 +516,7 @@ bash scripts/demo.sh The demo script will: 1. **Auto-detect cameras** (USB and CSI) and let you select if multiple are found 2. **Build TensorRT engine** on first run (~2 minutes) -3. **Start TRT inference service** for 35+ FPS performance +3. **Start TRT inference service** for 40+ FPS performance (93+ FPS at 308x308) 4. **Launch Docker container** with ROS2 depth estimation node 5. **Open performance monitor** showing live FPS, latency, and GPU stats 6. **Optionally launch RViz2** for visualization @@ -577,8 +577,8 @@ The performance monitor displays real-time metrics: TensorRT Inference Service ---------------------------------------- Status: Running - FPS: 35.2 - Latency: 28.4 ms + FPS: 40.1 + Latency: 25.0 ms Frames: 1024 GPU Resources @@ -1120,11 +1120,11 @@ Measured on Jetson Orin NX 16GB (JetPack 6.0, L4T r36.2.0): |-------|---------|------------|-----|----------------| | DA3-SMALL | PyTorch FP32 | 518x518 | ~5.2 | ~193ms | -**Update (2026-01-31)**: TensorRT acceleration now validated with 6.8x speedup. See [TensorRT Status](#tensorrt-status-validated) below. +**Update (2026-02-02)**: TensorRT acceleration validated with up to 17.8x speedup (93 FPS @ 308x308). See [TensorRT Status](#tensorrt-status-validated) and [Benchmarks](docs/JETSON_BENCHMARKS.md) for details. ### TensorRT Status: VALIDATED (Host-Container Split) -TensorRT acceleration validated on Jetson Orin NX 16GB with **6.8x speedup** (35.3 FPS vs 5.2 FPS baseline). +TensorRT acceleration validated on Jetson Orin NX 16GB with **7.7x speedup** (40 FPS @ 518x518, 93 FPS @ 308x308). #### Architecture: Host-Container Split @@ -1157,18 +1157,23 @@ Due to broken TensorRT Python bindings in available Jetson containers ([dusty-nv - Container TRT 8.6 cannot build DA3 engines (DINOv2 incompatibility) - Host TRT 10.3 works perfectly (validated at 29.8ms latency) -#### Validated Performance (2026-01-31) +#### Validated Performance (2026-02-02) | Metric | Value | |--------|-------| | Platform | Jetson Orin NX 16GB | -| JetPack | 6.2.1 (L4T R36.4.7) | +| JetPack | 6.2 (L4T R36.4) | | TensorRT | 10.3.0.30 (host) | -| Model | DA3-SMALL @ 518x518 FP16 | -| Throughput | 35.3 FPS | -| Latency (median) | 26.4ms | -| Engine Size | 58MB | -| Speedup | 6.8x vs PyTorch | +| CUDA | 12.6 | + +| Configuration | FPS | Latency | Speedup | +|--------------|-----|---------|---------| +| DA3-Small @ 518x518 FP16 | **40 FPS** | 25.0ms | 7.7x | +| DA3-Small @ 400x400 FP16 | **64 FPS** | 15.8ms | 12.2x | +| DA3-Small @ 308x308 FP16 | **93 FPS** | 10.9ms | 17.8x | +| DA3-Small @ 256x256 FP16 | **110 FPS** | 9.1ms | 21.2x | + +Thermal stability validated: 10-minute sustained load at 40.79 FPS with no throttling. #### Quick Start @@ -1196,12 +1201,19 @@ See [docs/JETSON_DEPLOYMENT_GUIDE.md](docs/JETSON_DEPLOYMENT_GUIDE.md) for compl ### Validated TensorRT Performance -Measured on Jetson Orin NX 16GB with TensorRT 10.3 (2026-01-31): +Measured on Jetson Orin NX 16GB with TensorRT 10.3 (2026-02-02): + +| Model | Backend | Resolution | FPS | Latency | Speedup | +|-------|---------|------------|-----|---------|---------| +| DA3-Small | TensorRT FP16 | 518x518 | **40.1** | 25.0ms | 7.7x | +| DA3-Small | TensorRT FP16 | 400x400 | **63.6** | 15.8ms | 12.2x | +| DA3-Small | TensorRT FP16 | 308x308 | **92.6** | 10.9ms | 17.8x | +| DA3-Small | TensorRT FP16 | 256x256 | **110.2** | 9.1ms | 21.2x | +| DA3-Base | TensorRT FP16 | 518x518 | **19.2** | 51.4ms | - | +| DA3-Large | TensorRT FP16 | 518x518 | **7.5** | 132.2ms | - | +| DA3-Small | PyTorch FP32 | 518x518 | ~5.2 | ~193ms | Baseline | -| Model | Backend | Resolution | FPS | GPU Latency | Speedup vs PyTorch | -|-------|---------|------------|-----|-------------|-------------------| -| DA3-SMALL | TensorRT FP16 | 518x518 | 35.3 | 26.4ms (median) | 6.8x | -| DA3-SMALL | PyTorch FP32 | 518x518 | 5.2 | ~193ms | Baseline | +See [docs/JETSON_BENCHMARKS.md](docs/JETSON_BENCHMARKS.md) for comprehensive benchmarks. ### Optimization Tips (Current) diff --git a/TODO.md b/TODO.md index 01494d3..9976d54 100644 --- a/TODO.md +++ b/TODO.md @@ -2,14 +2,14 @@ ## Executive Summary -| Metric | PyTorch Baseline | TensorRT 10.3 FP16 | -|--------|------------------|-------------------| -| FPS | 5.2 | **35.3** | -| Latency | 193ms | **26.4ms** | -| Speedup | 1x | **6.8x** | -| Engine | N/A | 58MB | +| Metric | PyTorch Baseline | TensorRT 10.3 FP16 (518x518) | TensorRT 10.3 FP16 (308x308) | +|--------|------------------|------------------------------|------------------------------| +| FPS | 5.2 | **40.1** | **92.6** | +| Latency | 193ms | **25.0ms** | **10.9ms** | +| Speedup | 1x | **7.7x** | **17.8x** | +| Engine | N/A | 64MB | 60MB | -**Platform:** Jetson Orin NX 16GB, JetPack 6.2.1, TensorRT 10.3.0.30 +**Platform:** Jetson Orin NX 16GB, JetPack 6.2, TensorRT 10.3 --- @@ -22,7 +22,7 @@ --- -## Phase 2: Host-Container Split Architecture [IN PROGRESS] +## Phase 2: Host-Container Split Architecture [COMPLETE] **Problem:** Container TensorRT Python bindings are broken: - `dustynv/l4t-pytorch:r36.4.0` - TRT import fails ([Issue #714](https://github.com/dusty-nv/jetson-containers/issues/714)) @@ -40,36 +40,74 @@ HOST (TRT 10.3) CONTAINER (ROS2) +------------------+ +------------------+ ``` -### Files to Create (Claude Code) -- [ ] `scripts/trt_inference_service.py` - Host TRT service -- [ ] Update `da3_inference.py` - Add SharedMemoryInference class -- [ ] Update `deploy_jetson.sh` - Start host service + container +### Implementation (Complete) +- [x] `scripts/trt_inference_service.py` - Host TRT service with file-based IPC +- [x] `depth_anything_3_ros2/da3_inference.py` - SharedMemoryInference class with PyTorch fallback +- [x] `scripts/deploy_jetson.sh --host-trt` - Orchestrates host service + container startup ### Communication Protocol | File | Direction | Format | |------|-----------|--------| | `/tmp/da3_shared/input.npy` | Container -> Host | float32 [1,1,3,518,518] | | `/tmp/da3_shared/output.npy` | Host -> Container | float32 [1,518,518] | -| `/tmp/da3_shared/request.flag` | Container -> Host | Signal file | -| `/tmp/da3_shared/ready.flag` | Host -> Container | Signal file | +| `/tmp/da3_shared/request` | Container -> Host | Timestamp signal | +| `/tmp/da3_shared/status` | Host -> Container | "ready", "complete:time", "error:msg" | + +### Deployment +```bash +# Fresh Jetson deployment +cd ~ +rm -rf ~/depth_anything_3_ros2 ~/ros2_ws ~/da3_fresh_test +git clone https://github.com/GerdsenAI/Depth-Anything-3-ROS2-Wrapper.git depth_anything_3_ros2 +cd depth_anything_3_ros2 +pip3 install pycuda --break-system-packages +bash scripts/deploy_jetson.sh --host-trt +``` --- -## Phase 3: Resolution Tuning [PENDING] +## Phase 3: Performance Benchmarking [COMPLETE] + +### Resolution Benchmarks (DA3-Small) + +| Resolution | Throughput | Latency | Speedup | +|------------|------------|---------|---------| +| 518x518 | 40.1 FPS | 25.0ms | 1.0x | +| 400x400 | 63.6 FPS | 15.8ms | 1.6x | +| 308x308 | 92.6 FPS | 10.9ms | 2.3x | +| 256x256 | 110.2 FPS | 9.1ms | 2.7x | + +### Model Size Benchmarks (518x518) -| Resolution | Expected FPS | -|------------|--------------| -| 518x518 | 35 (validated) | -| 400x400 | ~45 | -| 308x308 | ~55 | +| Model | Parameters | Throughput | Latency | Engine Size | +|-------|------------|------------|---------|-------------| +| DA3-Small | ~24M | 40.0 FPS | 25.0ms | 64MB | +| DA3-Base | ~97M | 19.2 FPS | 51.4ms | 211MB | +| DA3-Large | ~335M | 7.5 FPS | 132.2ms | 674MB | + +**Recommendation:** DA3-Small @ 308-400px for real-time robotics (64-93 FPS) + +See `docs/JETSON_BENCHMARKS.md` for full benchmark documentation. --- -## Phase 4: Thermal/Stability Validation [PENDING] +## Phase 4: Thermal/Stability Validation [COMPLETE] + +### 10-Minute Sustained Load Test Results + +| Metric | Value | +|--------|-------| +| Duration | 600.06 seconds | +| Status | **PASSED** | +| Throughput | 40.79 FPS (stable) | +| Latency (mean) | 24.73ms | +| Latency (min) | 24.25ms | +| Latency (max) | 27.88ms | +| Latency (p99) | 25.19ms | -- [ ] 10-minute sustained load test -- [ ] GPU temp monitoring (<80C target) -- [ ] FPS stability check +- [x] 10-minute sustained load test - **PASSED** +- [x] FPS stability check - **PASSED** (variance < 5%) +- [x] No thermal throttling detected (consistent performance throughout) --- @@ -82,4 +120,4 @@ HOST (TRT 10.3) CONTAINER (ROS2) --- -**Last Updated:** 2026-01-31 +**Last Updated:** 2026-02-02 diff --git a/docs/BASELINES.md b/docs/BASELINES.md index 9cb30dc..7013984 100644 --- a/docs/BASELINES.md +++ b/docs/BASELINES.md @@ -1,183 +1,85 @@ # Performance Baselines -This document records measured performance baselines for future reference and comparison. +This document records measured performance baselines for the Depth Anything 3 ROS2 Wrapper. --- -## Jetson Orin NX 16GB +## Jetson Orin NX 16GB - Validated Results -**Test Date**: 2026-01-30 -**JetPack**: 6.0 (L4T r36.2.0) -**CUDA**: 12.2 -**PyTorch**: 2.3.0 -**torchvision**: 0.18.0 (built from source) -**TensorRT**: 8.6.2 (not currently usable - opset incompatibility) - -### DA3-SMALL (PyTorch FP32) - -| Metric | Value | -|--------|-------| -| Input Resolution | 518x518 | -| Inference Time | ~193ms | -| FPS | ~5.2 | -| GPU Memory | ~2GB (estimated) | -| Backend | PyTorch FP32 | - -### Test Conditions - -- **Image Source**: Static test images -- **Warm-up**: 10 frames discarded before measurement -- **Measurement**: Average over 100 frames -- **Power Mode**: MAXN (15W) -- **Cooling**: Stock heatsink with fan - -### Docker Build Statistics +**Test Date**: 2026-02-02 +**JetPack**: 6.2 (L4T r36.4) +**TensorRT**: 10.3.0.30 +**CUDA**: 12.6 -| Metric | Value | -|--------|-------| -| Base Image | dustynv/ros:humble-ros-base-l4t-r36.2.0 | -| Final Image Size | ~14.9GB | -| Build Time | ~45 minutes | +### TensorRT FP16 Performance ---- +#### Resolution Benchmarks (DA3-Small) -## TensorRT 10.3 - Validated Performance (2026-01-31) +| Resolution | Throughput | Latency (mean) | Latency (p99) | Speedup vs PyTorch | +|------------|------------|----------------|---------------|-------------------| +| 518x518 | **40.1 FPS** | 25.0ms | 25.6ms | 7.7x | +| 400x400 | **63.6 FPS** | 15.8ms | 16.4ms | 12.2x | +| 308x308 | **92.6 FPS** | 10.9ms | 11.1ms | 17.8x | +| 256x256 | **110.2 FPS** | 9.1ms | 9.3ms | 21.2x | -### Solution Validation +#### Model Size Benchmarks (518x518) -| Aspect | Details | -|--------|---------| -| **Previous Issue** | TensorRT 8.6.2 incompatible with DINOv2 Einsum ops | -| **Solution** | Docker image updated to L4T r36.4.0 (TensorRT 10.3) | -| **Status** | Validated - Phase 1 complete | -| **Validated FPS** | 35.3 FPS with TensorRT FP16 | +| Model | Parameters | Throughput | Latency (mean) | Engine Size | +|-------|------------|------------|----------------|-------------| +| DA3-Small | ~24M | **40.0 FPS** | 25.0ms | 64MB | +| DA3-Base | ~97M | **19.2 FPS** | 51.4ms | 211MB | +| DA3-Large | ~335M | **7.5 FPS** | 132.2ms | 674MB | -### Validated Performance Metrics - -**Test Date**: 2026-01-31 -**Platform**: Jetson Orin NX 16GB -**JetPack**: 6.2 (L4T r36.4.0) -**TensorRT**: 10.3 -**CUDA**: 12.6 -**Model**: DA3-SMALL -**Resolution**: 518x518 -**Precision**: FP16 +#### Thermal Stability (10-Minute Sustained Load) | Metric | Value | |--------|-------| -| Throughput | 35.3 FPS | -| GPU Latency (median) | 26.4ms | -| GPU Latency (min) | 25.5ms | -| Engine Size | 58MB | -| Speedup vs PyTorch | 6.8x | -| Test Method | Host validation script | - -**Key Technical Details:** -- Base image: `dustynv/ros:humble-pytorch-l4t-r36.4.0` -- Build command: `--memPoolSize=workspace:2048MiB` (TRT 10.x syntax) -- ONNX export: 5D input shape `pixel_values:1x1x3x518x518` -- Validation script: `scripts/test_trt10.3_host.sh` - -### torchvision Build Requirement - -| Aspect | Details | -|--------|---------| -| **Issue** | NVIDIA PyTorch wheel ABI mismatch | -| **Impact** | pip-installed torchvision crashes on NMS operator | -| **Solution** | Build torchvision from source | -| **Build Time** | ~15 minutes additional | - -### ARM64 Runtime Patches - -| Package | Issue | Workaround | -|---------|-------|------------| -| pycolmap | No ARM64 wheel | Runtime patch in api.py | -| evo | No ARM64 wheel | Runtime patch in api.py | +| Duration | 600.06 seconds | +| Status | **PASSED** | +| Throughput | 40.79 FPS (stable) | +| Latency (mean) | 24.73ms | +| Latency (min) | 24.25ms | +| Latency (max) | 27.88ms | +| Latency (p99) | 25.19ms | +| Thermal Throttling | None detected | ---- +### PyTorch Baseline (Pre-TensorRT) -## Desktop GPU (Reference) - -*To be measured on desktop GPU for comparison* - -### Expected Test Configuration - -| Component | Specification | -|-----------|---------------| -| GPU | NVIDIA RTX 3090 / 4090 | -| CUDA | 12.x | -| PyTorch | 2.x | -| TensorRT | 10.x (opset 18+ support) | - -### Expected Performance Targets - -| Model | Backend | Expected FPS | -|-------|---------|--------------| -| DA3-SMALL | PyTorch FP32 | ~30-40 | -| DA3-SMALL | TensorRT FP16 | ~60-80 | -| DA3-BASE | PyTorch FP32 | ~20-25 | -| DA3-BASE | TensorRT FP16 | ~40-50 | +| Model | Backend | Resolution | FPS | Inference Time | +|-------|---------|------------|-----|----------------| +| DA3-Small | PyTorch FP32 | 518x518 | ~5.2 | ~193ms | --- -## Comparison Targets - -### TensorRT Performance (When Available) - -Based on typical TensorRT speedups (2-4x over PyTorch), expected performance on Jetson Orin NX 16GB: +## Test Conditions -| Model | Resolution | Backend | Expected FPS | -|-------|------------|---------|--------------| -| DA3-SMALL | 518x518 | TensorRT FP16 | ~15-20 | -| DA3-SMALL | 308x308 | TensorRT FP16 | ~30-40 | -| DA3-BASE | 518x518 | TensorRT FP16 | ~10-15 | - -**Note**: These are projected values. Actual performance depends on opset compatibility resolution. +- **Power Mode**: MAXN (15W) +- **Cooling**: Active fan cooling +- **Benchmark Tool**: trtexec (TensorRT benchmark utility) +- **Iterations**: 100 inference passes +- **Warmup**: 2000ms --- -## Measurement Methodology - -### Standard Test Procedure - -1. **Start Container** - ```bash - docker compose up -d depth-anything-3-jetson - docker exec -it da3_ros2_jetson bash - ``` +## Key Technical Details -2. **Run Node with Logging** - ```bash - ros2 launch depth_anything_3_ros2 depth_anything_3.launch.py \ - image_topic:=/camera/image_raw \ - model_name:=depth-anything/DA3-SMALL \ - log_inference_time:=true - ``` +- **Architecture**: Host-Container Split (TRT on host, ROS2 in container) +- **Base Image**: dustynv/ros:humble-ros-base-l4t-r36.2.0 +- **Build Command**: `--memPoolSize=workspace:2048MiB` (TRT 10.x syntax) +- **ONNX Input Shape**: 5D `pixel_values:1x1x3xHxW` -3. **Measure Topic Rate** - ```bash - ros2 topic hz /depth_anything_3/depth - ``` - -4. **Monitor System** - ```bash - # GPU utilization - watch -n 1 nvidia-smi - - # Thermal (Jetson) - tegrastats --interval 1000 - ``` +--- -### Reporting Format +## Recommendations -When adding new baselines, include: -- Test date -- Hardware specification -- Software versions (JetPack, CUDA, PyTorch) -- Model and resolution -- Inference time and FPS -- Test conditions (power mode, cooling, etc.) +| Use Case | Configuration | FPS | Latency | +|----------|---------------|-----|---------| +| Real-time robotics | DA3-Small @ 308x308 | 93 | 11ms | +| Balanced | DA3-Small @ 400x400 | 64 | 16ms | +| High quality | DA3-Small @ 518x518 | 40 | 25ms | +| Quality-focused | DA3-Base @ 518x518 | 19 | 51ms | +| Offline processing | DA3-Large @ 518x518 | 7.5 | 132ms | --- -**Last Updated**: 2026-01-30 +**Last Updated**: 2026-02-02 diff --git a/docs/JETSON_BENCHMARKS.md b/docs/JETSON_BENCHMARKS.md new file mode 100644 index 0000000..c059e56 --- /dev/null +++ b/docs/JETSON_BENCHMARKS.md @@ -0,0 +1,216 @@ +# Depth Anything 3 - Jetson Orin NX Benchmarks + +Performance benchmarks for Depth Anything 3 (DA3) models running on NVIDIA Jetson Orin NX 16GB with TensorRT 10.3 optimization. + +**Test Date:** February 2, 2026 +**Hardware:** Jetson Orin NX 16GB (JetPack 6.2) +**TensorRT Version:** 10.3 +**Precision:** FP16 + +--- + +## Executive Summary + +| Configuration | FPS | Latency | Use Case | +|--------------|-----|---------|----------| +| DA3-Small @ 256x256 | **110 FPS** | 9.1ms | High-speed robotics | +| DA3-Small @ 308x308 | **93 FPS** | 10.9ms | Real-time robotics (recommended) | +| DA3-Small @ 400x400 | **64 FPS** | 15.8ms | Balanced quality/speed | +| DA3-Small @ 518x518 | **40 FPS** | 25.0ms | Higher quality | +| DA3-Base @ 518x518 | **19 FPS** | 51.4ms | Quality-focused | +| DA3-Large @ 518x518 | **7.5 FPS** | 132ms | Offline processing | + +**Recommendation:** For real-time robotics applications, use DA3-Small at 308x308 or 400x400 resolution for optimal speed/quality balance. + +--- + +## Resolution Benchmarks (DA3-Small) + +Testing DA3-Small model at different input resolutions to find the optimal speed/quality tradeoff. + +| Resolution | Throughput (FPS) | Latency (mean) | Latency (p99) | Engine Size | +|------------|------------------|----------------|---------------|-------------| +| 518x518 | 40.09 | 25.0ms | 25.6ms | 64MB | +| 400x400 | 63.58 | 15.8ms | 16.4ms | 60MB | +| 308x308 | 92.58 | 10.9ms | 11.1ms | 60MB | +| 256x256 | 110.20 | 9.1ms | 9.3ms | 56MB | + +### Resolution Scaling Analysis + +- **2x speedup:** Reducing from 518 to 400 yields 1.6x faster inference +- **2.3x speedup:** 308x308 provides 2.3x improvement over native resolution +- **2.7x speedup:** 256x256 achieves maximum throughput at 110 FPS + +### Quality Considerations + +Lower resolutions reduce depth map detail but maintain relative depth accuracy. For navigation and obstacle avoidance, 308x308 provides sufficient spatial resolution while enabling real-time performance. + +--- + +## Model Size Benchmarks (518x518) + +Comparing DA3 model variants at native 518x518 resolution. + +| Model | Parameters | Throughput (FPS) | Latency (mean) | Latency (p99) | Engine Size | +|-------|------------|------------------|----------------|---------------|-------------| +| DA3-Small | ~24M | 40.05 | 25.0ms | 25.6ms | 64MB | +| DA3-Base | ~97M | 19.24 | 51.4ms | 52.8ms | 211MB | +| DA3-Large | ~335M | 7.51 | 132.2ms | 134.2ms | 674MB | + +### Model Scaling Analysis + +- **Small to Base:** 4x more parameters, 2x slower (19 vs 40 FPS) +- **Base to Large:** 3.5x more parameters, 2.6x slower (7.5 vs 19 FPS) +- **Small to Large:** 14x more parameters, 5.3x slower (7.5 vs 40 FPS) + +### Quality vs Speed Tradeoff + +| Model | Relative Quality | Relative Speed | Best For | +|-------|-----------------|----------------|----------| +| Small | Good | Fastest | Real-time robotics, embedded | +| Base | Better | Moderate | Quality-sensitive applications | +| Large | Best | Slowest | Offline processing, benchmarking | + +--- + +## Memory Usage + +### GPU Memory (Estimated) + +| Model | Resolution | GPU Memory | +|-------|------------|------------| +| DA3-Small | 518x518 | ~1.2GB | +| DA3-Small | 308x308 | ~0.8GB | +| DA3-Base | 518x518 | ~2.5GB | +| DA3-Large | 518x518 | ~6GB | + +### Disk Space (TRT Engines) + +| Configuration | Engine Size | +|---------------|-------------| +| DA3-Small (all resolutions) | ~240MB total | +| DA3-Base (518x518) | 211MB | +| DA3-Large (518x518) | 674MB | + +--- + +## Deployment Recommendations + +### Real-Time Robotics (30+ FPS required) + +**Recommended:** DA3-Small @ 308x308 or 400x400 +- 308x308: 93 FPS with 11ms latency +- 400x400: 64 FPS with 16ms latency +- Both leave headroom for other processing + +### Quality-Focused Applications (15+ FPS acceptable) + +**Recommended:** DA3-Small @ 518x518 or DA3-Base @ 518x518 +- Small: 40 FPS, good quality +- Base: 19 FPS, better quality for detailed scenes + +### Offline/Batch Processing + +**Recommended:** DA3-Large @ 518x518 +- 7.5 FPS suitable for non-real-time applications +- Best depth quality for dataset generation + +--- + +## Test Methodology + +### Hardware Configuration + +- **Device:** NVIDIA Jetson Orin NX 16GB +- **JetPack:** 6.2 +- **Power Mode:** MAXN (15W) +- **Cooling:** Active fan cooling + +### Benchmark Parameters + +- **Tool:** trtexec (TensorRT benchmark utility) +- **Iterations:** 100 inference passes +- **Warmup:** 2000ms +- **Precision:** FP16 + +### Engine Build Settings + +```bash +trtexec \ + --onnx=model.onnx \ + --saveEngine=model.engine \ + --fp16 \ + --memPoolSize=workspace:2048MiB \ + --optShapes=pixel_values:1x1x3xHxW +``` + +--- + +## Reproducing Benchmarks + +### Prerequisites + +1. Jetson Orin NX with JetPack 6.2+ +2. TensorRT 10.3 +3. This repository deployed via `deploy_jetson.sh` + +### Running Resolution Benchmarks + +```bash +cd ~/depth_anything_3_ros2 +bash scripts/benchmark_resolutions.sh +``` + +### Running Model Size Benchmarks + +```bash +cd ~/depth_anything_3_ros2 +bash scripts/benchmark_models.sh +``` + +--- + +## Version History + +| Date | Changes | +|------|---------| +| 2026-02-02 | Initial benchmarks on Jetson Orin NX 16GB | +| 2026-02-02 | Added thermal/stability validation (10-min sustained load test) | + +--- + +## Thermal/Stability Validation + +### 10-Minute Sustained Load Test + +Test performed with DA3-Small @ 518x518 under continuous inference load. + +| Metric | Value | +|--------|-------| +| Duration | 600.06 seconds | +| Status | **PASSED** | +| Throughput | 40.79 FPS | +| Latency (mean) | 24.73ms | +| Latency (min) | 24.25ms | +| Latency (max) | 27.88ms | +| Latency (p99) | 25.19ms | + +### Stability Analysis + +- **FPS Stability:** Maintained consistent 40.79 FPS throughout 10-minute test +- **Latency Variance:** Only 3.63ms spread (min to max) indicating stable thermals +- **Thermal Throttling:** None detected (performance remained constant) +- **p99 Latency:** 25.19ms ensures predictable real-time behavior + +### Conclusions + +The Jetson Orin NX 16GB demonstrates excellent thermal stability for sustained DA3 inference workloads. No performance degradation was observed over the 10-minute test period, making it suitable for continuous robotics applications. + +--- + +## References + +- [Depth Anything V3 Paper](https://arxiv.org/abs/2401.10891) +- [ONNX Community Models](https://huggingface.co/onnx-community) +- [TensorRT Documentation](https://docs.nvidia.com/deeplearning/tensorrt/) +- [Jetson Orin NX Specs](https://developer.nvidia.com/embedded/jetson-orin-nx) diff --git a/docs/JETSON_DEPLOYMENT_GUIDE.md b/docs/JETSON_DEPLOYMENT_GUIDE.md index db8c6e2..f5999e4 100644 --- a/docs/JETSON_DEPLOYMENT_GUIDE.md +++ b/docs/JETSON_DEPLOYMENT_GUIDE.md @@ -5,10 +5,11 @@ | Component | Version | Notes | |-----------|---------|-------| | Platform | Jetson Orin NX 16GB | Seeed reComputer | -| JetPack | 6.2.1 (L4T R36.4.7) | Required for TRT 10.3 | +| JetPack | 6.2 (L4T R36.4) | Required for TRT 10.3 | | TensorRT | 10.3.0.30 | Host-side inference | -| CUDA | 12.6.11 | Host | -| Performance | 35.3 FPS @ 518x518 FP16 | 6.8x speedup over PyTorch | +| CUDA | 12.6 | Host | +| Performance | **40 FPS @ 518x518 FP16** | 7.7x speedup over PyTorch | +| Performance | **93 FPS @ 308x308 FP16** | 17.8x speedup over PyTorch | --- @@ -17,35 +18,35 @@ Due to broken TensorRT Python bindings in available Jetson containers ([Issue #714](https://github.com/dusty-nv/jetson-containers/issues/714)), we use a split architecture: ``` -┌─────────────────────────────────────────────────────────────────┐ -│ HOST (JetPack 6.2+) │ -│ ┌─────────────────────────────────────────────────────────┐ │ -│ │ TRT Inference Service (Python) │ │ -│ │ - Loads engine with host TensorRT 10.3 │ │ -│ │ - Watches /tmp/da3_shared/input.npy │ │ -│ │ - Writes /tmp/da3_shared/output.npy │ │ -│ └─────────────────────────────────────────────────────────┘ │ -│ ▲ │ -│ │ shared memory │ -│ ▼ │ -│ ┌─────────────────────────────────────────────────────────┐ │ -│ │ Docker Container (L4T r36.2.0) │ │ -│ │ ┌─────────────────────────────────────────────────┐ │ │ -│ │ │ ROS2 Depth Node │ │ │ -│ │ │ - Subscribes to /image_raw │ │ │ -│ │ │ - Writes input to shared memory │ │ │ -│ │ │ - Reads depth from shared memory │ │ │ -│ │ │ - Publishes to /depth │ │ │ -│ │ └─────────────────────────────────────────────────┘ │ │ -│ └─────────────────────────────────────────────────────────┘ │ -└─────────────────────────────────────────────────────────────────┘ ++---------------------------------------------------------------+ +| HOST (JetPack 6.2+) | +| +----------------------------------------------------------+ | +| | TRT Inference Service (Python) | | +| | - Loads engine with host TensorRT 10.3 | | +| | - Watches /tmp/da3_shared/input.npy | | +| | - Writes /tmp/da3_shared/output.npy | | +| +----------------------------------------------------------+ | +| ^ | +| | shared memory | +| v | +| +----------------------------------------------------------+ | +| | Docker Container (L4T r36.2.0) | | +| | +----------------------------------------------------+ | | +| | | ROS2 Depth Node | | | +| | | - Subscribes to /image_raw | | | +| | | - Writes input to shared memory | | | +| | | - Reads depth from shared memory | | | +| | | - Publishes to /depth | | | +| | +----------------------------------------------------+ | | +| +----------------------------------------------------------+ | ++---------------------------------------------------------------+ ``` **Why this approach:** - `dustynv/l4t-pytorch:r36.4.0` has broken TensorRT Python bindings - `dustynv/ros:humble-pytorch-l4t-r36.4.0` does not exist - Container TRT 8.6.2 cannot build DA3 engines (DINOv2 incompatibility) -- Host TRT 10.3 works perfectly (validated at 26.4ms latency) +- Host TRT 10.3 works perfectly (validated at 25ms latency) --- @@ -58,7 +59,7 @@ bash scripts/deploy_jetson.sh --host-trt This script: 1. Verifies TensorRT 10.3 on host -2. Downloads ONNX model if missing +2. Downloads ONNX model if missing (auto-installs huggingface_hub) 3. Builds TensorRT FP16 engine (~2 min) 4. Starts host inference service 5. Starts Docker container with shared memory mount @@ -87,7 +88,7 @@ mkdir -p models/tensorrt --optShapes=pixel_values:1x1x3x518x518 ``` -**Build time:** ~2 minutes | **Engine size:** ~58 MB +**Build time:** ~2 minutes | **Engine size:** ~64 MB ### Step 3: Start Host Inference Service @@ -116,21 +117,31 @@ ros2 launch depth_anything_3_ros2 depth_anything_3.launch.py use_shared_memory:= ## Performance Results -| Metric | Value | -|--------|-------| -| Throughput | 35.3 FPS | -| Latency (median) | 26.4 ms | -| Latency (p95) | 33.3 ms | -| GPU Temp | 44-45C | -| Speedup | 6.8x vs PyTorch | +See [JETSON_BENCHMARKS.md](JETSON_BENCHMARKS.md) for comprehensive benchmarks. -### Resolution Options +### Quick Reference -| Resolution | Expected FPS | Use Case | -|------------|--------------|----------| -| 518x518 | ~35 FPS | High quality | -| 400x400 | ~45 FPS | Balanced | -| 308x308 | ~55 FPS | High speed | +| Configuration | FPS | Latency | Use Case | +|--------------|-----|---------|----------| +| DA3-Small @ 518x518 | 40 | 25ms | High quality | +| DA3-Small @ 400x400 | 64 | 16ms | Balanced | +| DA3-Small @ 308x308 | 93 | 11ms | Real-time robotics | +| DA3-Small @ 256x256 | 110 | 9ms | High-speed | + +### Model Variants + +| Model | FPS (518x518) | Latency | Engine Size | +|-------|---------------|---------|-------------| +| DA3-Small | 40 | 25ms | 64MB | +| DA3-Base | 19 | 51ms | 211MB | +| DA3-Large | 7.5 | 132ms | 674MB | + +### Thermal Stability + +10-minute sustained load test **PASSED**: +- Throughput: 40.79 FPS (stable) +- Latency variance: < 5% +- No thermal throttling detected --- @@ -142,14 +153,8 @@ The host service and container communicate via memory-mapped files: |------|-----------|--------| | `/tmp/da3_shared/input.npy` | Container -> Host | float32 [1,1,3,518,518] | | `/tmp/da3_shared/output.npy` | Host -> Container | float32 [1,518,518] | -| `/tmp/da3_shared/ready.flag` | Host -> Container | Empty file (signal) | -| `/tmp/da3_shared/request.flag` | Container -> Host | Empty file (signal) | - -**Synchronization:** -1. Container writes `input.npy`, creates `request.flag` -2. Host detects `request.flag`, runs inference, writes `output.npy`, creates `ready.flag` -3. Container detects `ready.flag`, reads `output.npy` -4. Both delete flag files +| `/tmp/da3_shared/request` | Container -> Host | Timestamp signal | +| `/tmp/da3_shared/status` | Host -> Container | "ready", "complete:time", "error:msg" | --- @@ -196,11 +201,14 @@ TensorRT 10.x changed CLI syntax: |------|---------| | `scripts/deploy_jetson.sh` | Automated deployment | | `scripts/trt_inference_service.py` | Host-side TRT inference | +| `scripts/benchmark_resolutions.sh` | Resolution benchmark script | +| `scripts/benchmark_models.sh` | Model size benchmark script | +| `scripts/thermal_stability_test.sh` | Thermal validation script | | `depth_anything_3_ros2/da3_inference.py` | Inference wrapper (shared memory support) | -| `models/tensorrt/da3-small-fp16.engine` | TRT engine (58 MB) | +| `models/tensorrt/da3-small-fp16.engine` | TRT engine (64 MB) | | `docker-compose.yml` | Container config | --- -**Last Updated:** 2026-01-31 -**Validated On:** Jetson Orin NX 16GB, JetPack 6.2.1, TensorRT 10.3.0.30 +**Last Updated:** 2026-02-02 +**Validated On:** Jetson Orin NX 16GB, JetPack 6.2, TensorRT 10.3.0.30 diff --git a/docs/TENSORRT_DA3_PLAN.md b/docs/TENSORRT_DA3_PLAN.md deleted file mode 100644 index b3baa54..0000000 --- a/docs/TENSORRT_DA3_PLAN.md +++ /dev/null @@ -1,126 +0,0 @@ -# Depth Anything V3 on Jetson Orin NX: TensorRT 8.6 won't work - -**The definitive answer: DA3 + TensorRT 8.6 is not achievable without TensorRT upgrade.** The ika-rwth-aachen repository—the only working DA3+TRT implementation—requires TensorRT 10.9, not 8.6. Your "caskConvolutionV2Forward" error stems from TRT 8.6's fundamental inability to process DINOv2's Einsum-based attention patterns. The solution is upgrading to **JetPack 6.2** (TensorRT 10.3), which fully supports Jetson Orin NX and includes a 70% AI TOPS boost via "Super Mode." - ---- - -## Why TensorRT 8.6 cannot build DA3 engines - -The DINOv2 backbone at DA3's core uses `F.scaled_dot_product_attention()` which exports to ONNX as complex Einsum operations. TensorRT 8.6 has three critical limitations: - -- **Einsum restrictions**: TRT 8.6 does not support more than 2 inputs, ellipsis notation, or diagonal operations in Einsum layers. DINOv2's attention patterns like `"nlhd,nhdv,nlh->nlhv"` fail completely. -- **Missing ViT optimizations**: TRT 8.6 lacks Multi-Head Attention (MHA) fusion for Vision Transformers—attention operations remain as unoptimized separate MatMul/Softmax chains. -- **Format incompatibility**: The error `"caskConvolutionV2Forward could not find any supported formats"` indicates TRT 8.6 cannot find compatible CUDA kernels for DINOv2's specific tensor format/precision combinations. - -NVIDIA's TensorRT GitHub Issue #4537 confirms these DINOv2 compilation failures persist even into early TRT 10.x versions, with full fixes arriving in **TensorRT 10.8+**. - ---- - -## How ika-rwth-aachen solves this (they use TRT 10.9) - -The ika-rwth-aachen/ros2-depth-anything-v3-trt repository achieves DA3+TRT compatibility through one key requirement: **TensorRT 10.9**. They don't use custom plugins, graph surgery, or workarounds—they simply rely on TRT 10.x's improved transformer support. - -Their tested configurations use Docker images `nvcr.io/nvidia/tensorrt:25.08-py3` and `nvcr.io/nvidia/tensorrt:25.03-py3`, both containing TensorRT 10.9 with CUDA 12.8+. The only preprocessing step is modifying DA3's source before ONNX export: - -```python -# File: src/depth_anything_3/api.py -# Change bfloat16 to float16 (ONNX doesn't fully support bfloat16) -autocast_dtype = torch.float16 # NOT torch.bfloat16 -``` - -Export uses fixed input shape `[1, 3, 280, 504]` and produces two outputs: metric depth and sky classification. Pre-built ONNX models are available at **huggingface.co/TillBeemelmanns/Depth-Anything-V3-ONNX**. - ---- - -## The upgrade path: JetPack 6.2 with TensorRT 10.3 - -JetPack 6.2 is the production release for Jetson Orin NX with TensorRT 10.3—a massive upgrade from 8.6. This version includes critical ViT/transformer fixes and enables "Super Mode" which boosts your Orin NX 16GB to **up to 70% more AI TOPS**. - -| Component | JetPack 6.0 (Current) | JetPack 6.2 (Target) | -|-----------|----------------------|---------------------| -| TensorRT | **8.6** | **10.3** | -| CUDA | 12.2 | 12.6 | -| cuDNN | 8.9 | 9.3 | -| ViT Support | Limited | Enhanced MHA fusion | - -**Upgrade procedure:** -```bash -# On Jetson Orin NX, upgrade from JetPack 6.0/6.1 to 6.2 -sudo apt-add-repository universe -sudo apt-add-repository multiverse -sudo apt-get update -sudo apt-get install nvidia-jetpack -sudo reboot -``` - -Note: Firmware update may be required. Verify post-upgrade with `dpkg -l | grep nvidia`. JetPack 7 is **not available** for Orin NX—it's exclusive to the new Jetson Thor platform. - ---- - -## Alternative approaches if upgrade is impossible - -### Option 1: ONNX Runtime hybrid execution (partial acceleration) - -ONNX Runtime with TensorRT Execution Provider can accelerate TRT-compatible ops while falling back to CUDA EP for problematic layers: - -```python -import onnxruntime as ort - -sess = ort.InferenceSession('da3.onnx', providers=[ - ('TensorrtExecutionProvider', { - 'trt_op_types_to_exclude': 'Einsum,LayerNormalization', - 'trt_fp16_enable': True, - 'trt_engine_cache_enable': True - }), - 'CUDAExecutionProvider' -]) -``` - -This yields **20-40% slower performance** than full TRT but remains faster than pure CUDA inference. - -### Option 2: Use Depth Anything V2 instead - -The spacewalk01/depth-anything-tensorrt repository provides working TRT 8.6 implementations for **V1 and V2 only**. V2 uses a similar DINOv2 backbone but with different intermediate feature extraction that happens to work with older TRT versions. Quality is comparable to V3 for many use cases. - -```bash -# spacewalk01 export for V2 -python export_v2.py --encoder vitb --input-size 518 -``` - -Known issue: INT64 weights require casting to INT32 on some Jetson configurations. - -### Option 3: Modify DINOv2 attention implementation (complex) - -Replace `scaled_dot_product_attention` with explicit operations before ONNX export: - -```python -def explicit_attention(q, k, v, scale=None): - scale = scale or (1.0 / math.sqrt(q.size(-1))) - attn = (q @ k.transpose(-2, -1)) * scale - attn = F.softmax(attn, dim=-1) - return attn @ v -``` - -This requires forking DA3, modifying the DINOv2 backbone attention modules, and re-exporting to ONNX. Significant development effort with uncertain results on TRT 8.6. - ---- - -## Pre-built resources and working implementations - -| Resource | Description | URL | -|----------|-------------|-----| -| DA3 ONNX Models | Pre-exported ONNX (280×504 input) | huggingface.co/TillBeemelmanns/Depth-Anything-V3-ONNX | -| ROS2 DA3 TRT Node | Working TRT 10.9 implementation | github.com/ika-rwth-aachen/ros2-depth-anything-v3-trt | -| Jetson Orin Depth | V1/V2 for Jetson (small model only) | github.com/IRCVLab/Depth-Anything-for-Jetson-Orin | -| Seeed Studio Guide | DA3 deployment on Jetson AGX Orin | wiki.seeedstudio.com/deploy_depth_anything_v3_jetson_agx_orin | -| DA2 TensorRT | V1/V2 with TRT 8.6 support | github.com/spacewalk01/depth-anything-tensorrt | - -The IRCVLab repository works on Jetson Orin but recommends the **small (vits14) model only** due to memory constraints—larger models may exceed 16GB on Orin NX. - ---- - -## Conclusion - -**Upgrade to JetPack 6.2 is the definitive solution.** TensorRT 10.3 includes the transformer/ViT improvements necessary for DA3's DINOv2 backbone. No custom plugins, graph surgery, or architectural modifications exist that make DA3 work reliably on TRT 8.6—the Einsum limitations are fundamental. - -If upgrading is impossible, use ONNX Runtime's hybrid execution for partial TRT acceleration, or switch to Depth Anything V2 which has proven TRT 8.6 compatibility. The performance difference between V2 and V3 is marginal for most depth estimation tasks, making V2 a practical alternative when TRT version constraints cannot be changed. \ No newline at end of file diff --git a/docs/TENSORRT_OPTIMIZATION_PLAN.md b/docs/TENSORRT_OPTIMIZATION_PLAN.md deleted file mode 100644 index 7f24de1..0000000 --- a/docs/TENSORRT_OPTIMIZATION_PLAN.md +++ /dev/null @@ -1,119 +0,0 @@ -# TensorRT Optimization Plan for Jetson Deployment - -## Executive Summary - -The TensorRT optimization plan is **approved** with critical updates from the recent audit. - -**Audit Findings (Jan 30 2026):** -1. **Resolution Mismatch**: 1024x1024 resolution (used for Large models) is NOT divisible by 14 (ViT patch size), which serves as a hard constraint. -2. **Opset Compatibility**: Custom `ika-rwth-aachen` models (Opset 20) fail on TensorRT 8.6 due to missing Gelu plugin. Our plan uses `onnx-community` models which must be verified for compatibility. - -**Verdict: PROCEED with Plan + Resolution Fixes** - ---- - -## 1. Audit Analysis & Fixes - -### A. Resolution Constraint (Critical) -**Finding**: ViT models require input dimensions divisible by 14. -- `1024 / 14 = 73.14` (INVALID) -- `1022 / 14 = 73.0` (VALID) -- `1036 / 14 = 74.0` (VALID) - -**Fix**: Update `model_catalog.yaml` and `OPTIMIZATION_GUIDE.md` to use **1022** instead of 1024. - -### B. TensorRT Compatibility -**Finding**: Logs show Opset 20 `Gelu` operator failure on TensorRT 8.6. -**Mitigation**: -- Phase 3 uses `onnx-community` models. We must verify if they use Opset < 20 or if TRT 8.6 supports them. -- **Fallback**: If FP16 build fails, fallback to ONNX Runtime with TensorRT execution provider (Phase 4 extension). - ---- - -## 2. Implementation Plan - -### Phase 1: Fix Dependencies in Dockerfile - -**File:** `Dockerfile` - -**Changes:** -1. Add `python3-dev` to system dependencies. -2. Add `pycuda`, `huggingface_hub`, `onnxruntime-gpu` (fallback). - -```dockerfile -RUN if [ "$BUILD_TYPE" = "jetson-base" ]; then \ - pip3 install --no-cache-dir pycuda huggingface_hub onnxruntime-gpu; \ - fi -``` - -### Phase 2: Add TensorRT Build Arguments - -**File:** `Dockerfile` - -Add `BUILD_TENSORRT`, `TENSORRT_MODEL`, `TENSORRT_PRECISION`. - -### Phase 3: Implement Optional Engine Building - -**File:** `Dockerfile` - -Use `--auto` flag. **CRITICAL**: Update `scripts/build_tensorrt_engine.py` to target 1022 resolution for 'large' models instead of 1024. - -### Phase 4: Enhance Entrypoint Script - -**File:** `docker/ros_entrypoint.sh` - -Add runtime auto-build logic with fallback warning. - -### Phase 5: Fix docker-compose.yml - -Add `NVIDIA_DRIVER_CAPABILITIES` and volume mounts. - -### Phase 6: Update Configurations (Audit Fix) - -**File:** `config/model_catalog.yaml` - -Update all `1024` references to `1022`. - -```yaml -AGX_ORIN_64GB: - height: 1022 # Changed from 1024 (must be divisible by 14) - width: 1022 -``` - ---- - -## 3. Build Strategy Options - -### Option A: Build-Time (Baked into Image) -- **Pros:** Fastest startup -- **Cons:** Platform-specific - -### Option B: Runtime Auto-Build (Recommended) -- **Pros:** Flexible, one-time delay -- **Cons:** Startup delay on first run - ---- - -## 4. Verification Plan - -1. **Resolution Check**: Verify 1022x1022 works with `trtexec`. -2. **Opset Check**: Verify `onnx-community` model Opset version matches TRT 8.6 support. -3. **End-to-End**: Run `ros2 launch ...` with `log_inference_time:=true`. - ---- - -## 5. Risk Config - -| Risk | Mitigation | -|------|------------| -| Opset 20 not supported | Use older Opset export or ONNX Runtime fallback | -| Resolution mismatch | Enforce mod-14 resolutions (1022, 518, 308) | -| System Memory OOM | 1022x1022 requires ~4.5GB VRAM (Safe for AGX) | - ---- - -## Immediate Next Steps (Audit Response) - -1. **Update `model_catalog.yaml`**: Change 1024 -> 1022. -2. **Update Docs**: Reflect 1022 as Ultra resolution. -3. **Proceed with Docker fixes**: Implement dependencies and build args. diff --git a/scripts/benchmark_models.sh b/scripts/benchmark_models.sh new file mode 100644 index 0000000..963fd8a --- /dev/null +++ b/scripts/benchmark_models.sh @@ -0,0 +1,150 @@ +#!/bin/bash +# Benchmark different DA3 model sizes +# Tests: small, base, large at 518x518 +# +# Usage: bash scripts/benchmark_models.sh [small|base|large|all] +# Default: tests all models + +set -e + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_DIR="$(dirname "$SCRIPT_DIR")" +cd "$REPO_DIR" + +export PATH="$HOME/.local/bin:$PATH" + +ONNX_DIR="models/onnx" +TRT_DIR="models/tensorrt" +TRTEXEC="/usr/src/tensorrt/bin/trtexec" + +# Parse argument +MODEL_TO_TEST="${1:-all}" + +echo "========================================" +echo "DA3 Model Size Benchmark" +echo "========================================" +echo "" + +mkdir -p "$ONNX_DIR" "$TRT_DIR" + +# Function to download and build a model +benchmark_model() { + local MODEL_SIZE=$1 # small, base, or large + local HF_REPO="onnx-community/depth-anything-v3-${MODEL_SIZE}" + local ONNX_MODEL="$ONNX_DIR/da3-${MODEL_SIZE}-embedded.onnx" + local ENGINE="$TRT_DIR/da3-${MODEL_SIZE}-fp16-518.engine" + + echo "=== Testing DA3-${MODEL_SIZE^^} (518x518) ===" + + # Download ONNX model if needed + if [ ! -f "$ONNX_MODEL" ]; then + echo "Downloading DA3-${MODEL_SIZE^^} ONNX model..." + + # Auto-install dependencies if needed + python3 -c "import huggingface_hub" 2>/dev/null || { + echo " Installing huggingface_hub..." + pip3 install huggingface_hub 2>&1 | tail -1 + } + + python3 -c "import onnx" 2>/dev/null || { + echo " Installing onnx..." + pip3 install onnx 2>&1 | tail -1 + } + + python3 << PYEOF +import os +from huggingface_hub import snapshot_download +import onnx + +onnx_dir = "models/onnx" +hf_download_dir = os.path.join(onnx_dir, "hf-download-${MODEL_SIZE}") +output_model = "${ONNX_MODEL}" + +print(" Downloading from HuggingFace: ${HF_REPO}") +snapshot_download( + repo_id="${HF_REPO}", + local_dir=hf_download_dir, + allow_patterns=["*.onnx", "*.onnx_data"] +) + +print(" Embedding weights into single ONNX file...") +model_path = os.path.join(hf_download_dir, "onnx", "model.onnx") +model = onnx.load(model_path) +onnx.save(model, output_model, save_as_external_data=False) +print(f" Created: {output_model}") +PYEOF + + if [ $? -ne 0 ]; then + echo "ERROR: Failed to download DA3-${MODEL_SIZE^^} model" + return 1 + fi + fi + + ONNX_SIZE=$(du -h "$ONNX_MODEL" 2>/dev/null | cut -f1 || echo "N/A") + echo " ONNX model: $ONNX_MODEL ($ONNX_SIZE)" + + # Build TensorRT engine if needed + if [ ! -f "$ENGINE" ]; then + echo "" + echo "Building TensorRT engine for DA3-${MODEL_SIZE^^}..." + echo " (Large models may take 5-10 minutes)" + $TRTEXEC \ + --onnx="$ONNX_MODEL" \ + --saveEngine="$ENGINE" \ + --fp16 \ + --memPoolSize=workspace:2048MiB \ + --optShapes=pixel_values:1x1x3x518x518 \ + 2>&1 | tee /tmp/trtexec_${MODEL_SIZE}_build.log | grep -E "(Building|Serializing|SUCCESS|ERROR|Throughput)" + + if [ ! -f "$ENGINE" ]; then + echo "ERROR: Failed to build DA3-${MODEL_SIZE^^} engine" + echo "Check /tmp/trtexec_${MODEL_SIZE}_build.log for details" + return 1 + fi + fi + + ENGINE_SIZE=$(du -h "$ENGINE" 2>/dev/null | cut -f1 || echo "N/A") + echo " Engine: $ENGINE ($ENGINE_SIZE)" + + echo "" + echo "Benchmarking DA3-${MODEL_SIZE^^}..." + $TRTEXEC --loadEngine="$ENGINE" --iterations=100 --warmUp=2000 2>&1 | grep -E "(mean|median|Throughput)" + echo "" +} + +# Run benchmarks based on argument +case "$MODEL_TO_TEST" in + small) + benchmark_model "small" + ;; + base) + benchmark_model "base" + ;; + large) + benchmark_model "large" + ;; + all) + benchmark_model "small" + benchmark_model "base" + benchmark_model "large" + ;; + *) + echo "Usage: $0 [small|base|large|all]" + exit 1 + ;; +esac + +echo "========================================" +echo "Model Size Comparison Summary" +echo "========================================" + +# Show all engine sizes +echo "" +echo "Engine Files:" +ls -lh $TRT_DIR/da3-*-fp16-518.engine 2>/dev/null | awk '{print " "$9": "$5}' || echo " No engines found" + +echo "" +echo "Model Size Reference:" +echo " Small: ~24M params, fastest inference" +echo " Base: ~97M params, balanced" +echo " Large: ~335M params, highest quality" diff --git a/scripts/demo.sh b/scripts/demo.sh index 510e6bb..097b9ec 100644 --- a/scripts/demo.sh +++ b/scripts/demo.sh @@ -285,23 +285,58 @@ if [ "$USE_TRT" = true ]; then # Check for ONNX model if [ ! -f "$ONNX_MODEL" ]; then echo " Downloading ONNX model from HuggingFace..." - if command -v huggingface-cli &> /dev/null; then - huggingface-cli download onnx-community/depth-anything-v3-small \ - --local-dir "$ONNX_DIR/hf-download" \ - --include "*.onnx" "*.onnx_data" 2>/dev/null - - # Embed weights - python3 -c " -import onnx -model = onnx.load('$ONNX_DIR/hf-download/onnx/model.onnx') -onnx.save(model, '$ONNX_MODEL', save_as_external_data=False) -print(' ONNX model ready') -" - else - echo -e "${RED}ERROR: huggingface-cli not found${NC}" - echo " Install with: pip install huggingface_hub" - echo " Or use --no-trt to skip TensorRT" - exit 1 + + # Auto-install huggingface_hub if missing + if ! python3 -c "import huggingface_hub" 2>/dev/null; then + echo " Installing huggingface_hub..." + if ! pip3 install huggingface_hub 2>&1 | tail -2; then + echo -e "${YELLOW}WARNING: Failed to install huggingface_hub${NC}" + USE_TRT=false + fi + fi + + # Auto-install onnx if missing + if [ "$USE_TRT" = true ] && ! python3 -c "import onnx" 2>/dev/null; then + echo " Installing onnx..." + if ! pip3 install onnx 2>&1 | tail -2; then + echo -e "${YELLOW}WARNING: Failed to install onnx${NC}" + USE_TRT=false + fi + fi + + # Download and embed model + if [ "$USE_TRT" = true ]; then + python3 << 'PYEOF' +import os +import sys +try: + from huggingface_hub import snapshot_download + import onnx + + onnx_dir = "models/onnx" + hf_download_dir = os.path.join(onnx_dir, "hf-download") + output_model = os.path.join(onnx_dir, "da3-small-embedded.onnx") + + print(" Downloading ONNX model...") + snapshot_download( + repo_id="onnx-community/depth-anything-v3-small", + local_dir=hf_download_dir, + allow_patterns=["*.onnx", "*.onnx_data"] + ) + + print(" Embedding weights into single ONNX file...") + model_path = os.path.join(hf_download_dir, "onnx", "model.onnx") + model = onnx.load(model_path) + onnx.save(model, output_model, save_as_external_data=False) + print(f" Created: {output_model}") +except Exception as e: + print(f"ERROR: {e}", file=sys.stderr) + sys.exit(1) +PYEOF + if [ $? -ne 0 ]; then + echo -e "${YELLOW}WARNING: Failed to download ONNX model, falling back to PyTorch${NC}" + USE_TRT=false + fi fi fi @@ -342,11 +377,34 @@ if [ "$USE_TRT" = true ]; then chmod 777 "$SHARED_DIR" # Check Python dependencies - if ! python3 -c "import tensorrt; import pycuda.driver" 2>/dev/null; then - echo -e "${YELLOW}WARNING: Host Python missing tensorrt or pycuda${NC}" + # Check TensorRT + if ! python3 -c "import tensorrt" 2>/dev/null; then + echo -e "${YELLOW}WARNING: TensorRT Python bindings not found${NC}" echo " Falling back to PyTorch" USE_TRT=false else + # Auto-install numpy if missing + if ! python3 -c "import numpy" 2>/dev/null; then + echo " Installing numpy..." + if ! pip3 install numpy 2>&1 | tail -2; then + echo -e "${YELLOW}WARNING: Failed to install numpy, falling back to PyTorch${NC}" + USE_TRT=false + fi + fi + + # Auto-install pycuda if missing + if [ "$USE_TRT" = true ] && ! python3 -c "import pycuda.driver" 2>/dev/null; then + echo " Installing pycuda (required for TRT inference)..." + if pip3 install pycuda 2>&1 | tail -3; then + echo -e "${GREEN} pycuda installed successfully${NC}" + else + echo -e "${YELLOW}WARNING: Failed to install pycuda, falling back to PyTorch${NC}" + USE_TRT=false + fi + fi + fi + + if [ "$USE_TRT" = true ]; then # Start TRT service python3 "$SCRIPT_DIR/trt_inference_service.py" \ --engine "$TRT_ENGINE" \ @@ -423,7 +481,7 @@ echo "" echo -e "${BOLD}Demo Configuration:${NC}" echo " Camera: ${CAMERA_DEVICE:-"(topic: $IMAGE_TOPIC)"}" echo " Image Topic: $IMAGE_TOPIC" -echo " Backend: $([ "$USE_TRT" = true ] && echo "TensorRT FP16 (~35 FPS)" || echo "PyTorch (~5 FPS)")" +echo " Backend: $([ "$USE_TRT" = true ] && echo "TensorRT FP16 (~40 FPS)" || echo "PyTorch (~5 FPS)")" echo " RViz2: $([ "$LAUNCH_RVIZ" = true ] && echo "Yes" || echo "No")" echo " Monitor: $([ "$LAUNCH_MONITOR" = true ] && echo "Yes" || echo "No")" echo "" diff --git a/scripts/deploy_jetson.sh b/scripts/deploy_jetson.sh index 43f0a5b..3428723 100644 --- a/scripts/deploy_jetson.sh +++ b/scripts/deploy_jetson.sh @@ -17,10 +17,13 @@ # /image_raw (sub) TRT 10.3 engine # /depth (pub) ~30 FPS inference # -# Performance: 35.3 FPS @ 518x518 (6.8x speedup over PyTorch) +# Performance: 40 FPS @ 518x518 (7.7x speedup over PyTorch) set -e +# Ensure ~/.local/bin is in PATH (pip installs CLI tools there) +export PATH="$HOME/.local/bin:$PATH" + SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" REPO_DIR="$(dirname "$SCRIPT_DIR")" cd "$REPO_DIR" @@ -102,24 +105,50 @@ mkdir -p "$ONNX_DIR" "$TRT_DIR" if [ ! -f "$ONNX_MODEL" ]; then echo " Downloading from HuggingFace..." - # Check for huggingface-cli - if command -v huggingface-cli &> /dev/null; then - huggingface-cli download onnx-community/depth-anything-v3-small \ - --local-dir "$ONNX_DIR/hf-download" \ - --include "*.onnx" "*.onnx_data" - - # Embed external weights into single file - echo " Embedding weights into single ONNX file..." - python3 -c " -import onnx -model = onnx.load('$ONNX_DIR/hf-download/onnx/model.onnx') -onnx.save(model, '$ONNX_MODEL', save_as_external_data=False) -print(' Created: $ONNX_MODEL') -" - else - echo -e "${RED}ERROR: huggingface-cli not found${NC}" - echo " Install with: pip install huggingface_hub" - echo " Or manually download ONNX model to: $ONNX_MODEL" + # Auto-install dependencies if not available + python3 -c "import huggingface_hub" 2>/dev/null || { + echo " Installing huggingface_hub..." + pip3 install huggingface_hub 2>&1 | tail -1 + } + + python3 -c "import onnx" 2>/dev/null || { + echo " Installing onnx..." + pip3 install onnx 2>&1 | tail -1 + } + + # Download and embed model using Python API (more reliable than CLI) + python3 << 'PYEOF' +import os +import sys + +try: + from huggingface_hub import snapshot_download + import onnx + + onnx_dir = "models/onnx" + hf_download_dir = os.path.join(onnx_dir, "hf-download") + output_model = os.path.join(onnx_dir, "da3-small-embedded.onnx") + + print(" Downloading ONNX model...") + snapshot_download( + repo_id="onnx-community/depth-anything-v3-small", + local_dir=hf_download_dir, + allow_patterns=["*.onnx", "*.onnx_data"] + ) + + print(" Embedding weights into single ONNX file...") + model_path = os.path.join(hf_download_dir, "onnx", "model.onnx") + model = onnx.load(model_path) + onnx.save(model, output_model, save_as_external_data=False) + print(f" Created: {output_model}") + +except Exception as e: + print(f"ERROR: {e}", file=sys.stderr) + sys.exit(1) +PYEOF + + if [ $? -ne 0 ]; then + echo -e "${RED}ERROR: Failed to download ONNX model${NC}" exit 1 fi fi @@ -183,13 +212,48 @@ if [ "$HOST_TRT" = true ]; then chmod 777 "$SHARED_DIR" # Check for required Python packages - if ! python3 -c "import tensorrt; import pycuda.driver" 2>/dev/null; then - echo -e "${RED}ERROR: Host Python missing tensorrt or pycuda${NC}" - echo " Install with: pip3 install pycuda" - echo " TensorRT Python bindings should be available via JetPack" + echo " Checking Python dependencies..." + + # Check TensorRT (should be available via JetPack) + if ! python3 -c "import tensorrt" 2>/dev/null; then + echo -e "${RED}ERROR: TensorRT Python bindings not found${NC}" + echo " TensorRT should be available via JetPack 6.2+" + echo " Try: sudo apt install python3-libnvinfer python3-libnvinfer-dev" exit 1 fi + # Auto-install numpy if missing + if ! python3 -c "import numpy" 2>/dev/null; then + echo " Installing numpy..." + if pip3 install numpy 2>&1 | tail -2; then + echo -e "${GREEN} numpy installed successfully${NC}" + else + echo -e "${RED}ERROR: Failed to install numpy${NC}" + exit 1 + fi + fi + + # Auto-install pycuda if missing + if ! python3 -c "import pycuda.driver" 2>/dev/null; then + echo " Installing pycuda (required for TRT inference)..." + if pip3 install pycuda 2>&1 | tail -3; then + echo -e "${GREEN} pycuda installed successfully${NC}" + else + echo -e "${RED}ERROR: Failed to install pycuda${NC}" + echo " Try manually: pip3 install pycuda" + exit 1 + fi + + # Verify installation + if ! python3 -c "import pycuda.driver" 2>/dev/null; then + echo -e "${RED}ERROR: pycuda installed but import failed${NC}" + echo " This may indicate a CUDA configuration issue" + exit 1 + fi + fi + + echo -e "${GREEN} TensorRT, numpy, and pycuda ready${NC}" + # Start TRT inference service in background echo " Starting inference service..." python3 "$SCRIPT_DIR/trt_inference_service.py" \ @@ -237,7 +301,7 @@ echo "Resolution: 518x518 FP16" if [ "$HOST_TRT" = true ]; then echo -e "Mode: ${CYAN}Host-Container Split (TRT on host)${NC}" - echo "Expected: ~30-35 FPS" + echo "Expected: ~40 FPS @ 518x518 (93 FPS @ 308x308)" echo "" echo "TRT Service: PID $TRT_SERVICE_PID (log: /tmp/trt_service.log)" echo "Shared Dir: $SHARED_DIR" diff --git a/scripts/thermal_stability_test.sh b/scripts/thermal_stability_test.sh new file mode 100644 index 0000000..f87beab --- /dev/null +++ b/scripts/thermal_stability_test.sh @@ -0,0 +1,204 @@ +#!/bin/bash +# Phase 4: Thermal and Stability Validation +# Runs sustained TRT inference while monitoring GPU temperature and FPS stability +# +# Usage: bash scripts/thermal_stability_test.sh [duration_minutes] [resolution] +# Default: 10 minutes, 518x518 + +set -e + +DURATION_MIN=${1:-10} +RESOLUTION=${2:-518} +DURATION_SEC=$((DURATION_MIN * 60)) + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_DIR="$(dirname "$SCRIPT_DIR")" +cd "$REPO_DIR" + +TRTEXEC="/usr/src/tensorrt/bin/trtexec" +ENGINE="models/tensorrt/da3-small-fp16-${RESOLUTION}.engine" +LOG_DIR="logs/thermal_test_$(date +%Y%m%d_%H%M%S)" +TEMP_LOG="$LOG_DIR/temperature.csv" +FPS_LOG="$LOG_DIR/fps.csv" +SUMMARY="$LOG_DIR/summary.txt" + +mkdir -p "$LOG_DIR" + +echo "========================================" +echo "Phase 4: Thermal/Stability Validation" +echo "========================================" +echo "" +echo "Configuration:" +echo " Duration: ${DURATION_MIN} minutes" +echo " Resolution: ${RESOLUTION}x${RESOLUTION}" +echo " Engine: $ENGINE" +echo " Log directory: $LOG_DIR" +echo "" + +if [ ! -f "$ENGINE" ]; then + echo "ERROR: Engine not found: $ENGINE" + echo "Run deploy_jetson.sh or benchmark scripts first." + exit 1 +fi + +# Initialize CSV headers +echo "timestamp,elapsed_sec,gpu_temp_c,cpu_temp_c,power_mw" > "$TEMP_LOG" +echo "timestamp,elapsed_sec,throughput_fps,latency_mean_ms,latency_p99_ms" > "$FPS_LOG" + +# Function to get GPU temperature +get_gpu_temp() { + cat /sys/devices/gpu.0/hwmon/hwmon*/temp1_input 2>/dev/null | awk '{print $1/1000}' || echo "N/A" +} + +# Function to get CPU temperature +get_cpu_temp() { + cat /sys/devices/virtual/thermal/thermal_zone*/temp 2>/dev/null | head -1 | awk '{print $1/1000}' || echo "N/A" +} + +# Function to get power consumption +get_power() { + cat /sys/bus/i2c/drivers/ina3221/1-0040/hwmon/hwmon*/in*_input 2>/dev/null | head -1 || echo "N/A" +} + +echo "Starting temperature monitor in background..." +START_TIME=$(date +%s) + +# Background temperature monitoring (every 5 seconds) +( + while true; do + NOW=$(date +%s) + ELAPSED=$((NOW - START_TIME)) + if [ $ELAPSED -ge $DURATION_SEC ]; then + break + fi + + TIMESTAMP=$(date +%H:%M:%S) + GPU_TEMP=$(get_gpu_temp) + CPU_TEMP=$(get_cpu_temp) + POWER=$(get_power) + + echo "$TIMESTAMP,$ELAPSED,$GPU_TEMP,$CPU_TEMP,$POWER" >> "$TEMP_LOG" + sleep 5 + done +) & +TEMP_PID=$! + +echo "Starting sustained inference test..." +echo "" +echo "Progress:" + +# Run inference in batches and log FPS +BATCH_ITERATIONS=500 +BATCH_WARMUP=1000 +BATCHES=$((DURATION_SEC / 15)) # Roughly 15 sec per batch + +for i in $(seq 1 $BATCHES); do + NOW=$(date +%s) + ELAPSED=$((NOW - START_TIME)) + + if [ $ELAPSED -ge $DURATION_SEC ]; then + break + fi + + REMAINING=$((DURATION_SEC - ELAPSED)) + PROGRESS=$((ELAPSED * 100 / DURATION_SEC)) + + # Get current temperature + GPU_TEMP=$(get_gpu_temp) + + echo -ne "\r [$PROGRESS%] Elapsed: ${ELAPSED}s | Remaining: ${REMAINING}s | GPU: ${GPU_TEMP}C " + + # Run benchmark batch + RESULT=$($TRTEXEC --loadEngine="$ENGINE" --iterations=$BATCH_ITERATIONS --warmUp=$BATCH_WARMUP 2>&1) + + # Parse results + THROUGHPUT=$(echo "$RESULT" | grep "Throughput:" | awk '{print $2}') + LATENCY_MEAN=$(echo "$RESULT" | grep "Latency:" | grep -oP 'mean = \K[0-9.]+') + LATENCY_P99=$(echo "$RESULT" | grep "Latency:" | grep -oP 'percentile\(99%\) = \K[0-9.]+') + + TIMESTAMP=$(date +%H:%M:%S) + echo "$TIMESTAMP,$ELAPSED,$THROUGHPUT,$LATENCY_MEAN,$LATENCY_P99" >> "$FPS_LOG" +done + +echo "" +echo "" +echo "Stopping temperature monitor..." +kill $TEMP_PID 2>/dev/null || true +wait $TEMP_PID 2>/dev/null || true + +# Generate summary +echo "========================================" +echo "Generating Summary" +echo "========================================" + +{ + echo "Thermal/Stability Test Summary" + echo "==============================" + echo "" + echo "Test Configuration:" + echo " Duration: ${DURATION_MIN} minutes" + echo " Resolution: ${RESOLUTION}x${RESOLUTION}" + echo " Engine: $ENGINE" + echo " Date: $(date)" + echo "" + + # Temperature stats + echo "Temperature Statistics:" + if [ -f "$TEMP_LOG" ] && [ $(wc -l < "$TEMP_LOG") -gt 1 ]; then + GPU_TEMPS=$(tail -n +2 "$TEMP_LOG" | cut -d',' -f3 | grep -v "N/A") + if [ -n "$GPU_TEMPS" ]; then + GPU_MIN=$(echo "$GPU_TEMPS" | sort -n | head -1) + GPU_MAX=$(echo "$GPU_TEMPS" | sort -n | tail -1) + GPU_AVG=$(echo "$GPU_TEMPS" | awk '{sum+=$1; count++} END {printf "%.1f", sum/count}') + echo " GPU Temp: min=${GPU_MIN}C, max=${GPU_MAX}C, avg=${GPU_AVG}C" + + # Check if stayed under 80C + if (( $(echo "$GPU_MAX < 80" | bc -l) )); then + echo " Status: PASS (stayed under 80C target)" + else + echo " Status: WARNING (exceeded 80C target)" + fi + fi + fi + echo "" + + # FPS stats + echo "Performance Statistics:" + if [ -f "$FPS_LOG" ] && [ $(wc -l < "$FPS_LOG") -gt 1 ]; then + FPS_VALUES=$(tail -n +2 "$FPS_LOG" | cut -d',' -f3 | grep -v "^$") + if [ -n "$FPS_VALUES" ]; then + FPS_MIN=$(echo "$FPS_VALUES" | sort -n | head -1) + FPS_MAX=$(echo "$FPS_VALUES" | sort -n | tail -1) + FPS_AVG=$(echo "$FPS_VALUES" | awk '{sum+=$1; count++} END {printf "%.2f", sum/count}') + FPS_STDDEV=$(echo "$FPS_VALUES" | awk -v avg="$FPS_AVG" '{sum+=($1-avg)^2; count++} END {printf "%.2f", sqrt(sum/count)}') + + echo " Throughput: min=${FPS_MIN} FPS, max=${FPS_MAX} FPS, avg=${FPS_AVG} FPS" + echo " Stability (stddev): ${FPS_STDDEV} FPS" + + # Check stability (stddev < 5% of mean) + THRESHOLD=$(echo "$FPS_AVG * 0.05" | bc -l) + if (( $(echo "$FPS_STDDEV < $THRESHOLD" | bc -l) )); then + echo " Status: PASS (stable, stddev < 5% of mean)" + else + echo " Status: WARNING (variable performance)" + fi + fi + + LATENCY_VALUES=$(tail -n +2 "$FPS_LOG" | cut -d',' -f4 | grep -v "^$") + if [ -n "$LATENCY_VALUES" ]; then + LAT_AVG=$(echo "$LATENCY_VALUES" | awk '{sum+=$1; count++} END {printf "%.2f", sum/count}') + echo " Latency (mean): ${LAT_AVG} ms" + fi + fi + echo "" + + echo "Log Files:" + echo " Temperature: $TEMP_LOG" + echo " FPS: $FPS_LOG" + echo "" + echo "==============================" + +} | tee "$SUMMARY" + +echo "" +echo "Full results saved to: $LOG_DIR"