feat(profiling): integrate TraceLens trace analysis

.gitignore

@@ -128,9 +128,12 @@ venv/
 .venv/
 
 # model relatives
+# Anchored to the repo root: these are a MAD project's own model files, and the
+# directories madengine populates in the working directory during a run. Left
+# unanchored they also swallow packaged source such as src/madengine/scripts/.
 docker/
-scripts/
-*.json
+/scripts/
+/*.json
 .*_env/
 .vscode/
 

docs/cli-reference.md

@@ -442,6 +442,86 @@ madengine report to-email --directory ./results --verbose
 
 ---
 
+##### `report tracelens` - Analyze GPU Traces
+
+Generate [TraceLens](https://github.com/AMD-AGI/TraceLens) performance reports from the trace artifacts a run left behind (`torch_profiler_output/`, `rocprof_output/`, `slurm_results/`, `k8s_results/`).
+
+Requires TraceLens: `pip install 'madengine[tracelens]'`. `--discover-only` works without it.
+
+**Usage:**
+
+```bash
+madengine report tracelens [OPTIONS]
+```
+
+**Options:**
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `--root` | TEXT | `"."` | Directory searched recursively for traces |
+| `--output-dir` | TEXT | `tracelens_output` | Directory for generated reports |
+| `--mode` | TEXT | `auto` | `auto`, `pytorch`, `rocprof`, `pftrace`, or `collective` |
+| `--python` | TEXT | | Interpreter that has TraceLens installed |
+| `--gpu-arch` | TEXT | | GPU arch (e.g. `MI300X`) enabling roofline bound classification |
+| `--world-size` | INTEGER | trace count | Rank count for the collective report |
+| `--max-traces` | INTEGER | `0` | Cap traces analyzed per kind (`0` means no cap) |
+| `--discover-only` | FLAG | `False` | List discovered traces without running TraceLens |
+| `--verbose` | FLAG | `False` | Enable verbose logging |
+
+**Examples:**
+
+```bash
+# Analyze every trace found under the current directory
+madengine report tracelens
+
+# See what would be analyzed first
+madengine report tracelens --discover-only
+
+# Enable roofline bound classification
+madengine report tracelens --gpu-arch MI300X
+
+# Multi-rank collective analysis of a distributed run
+madengine report tracelens --root slurm_results --mode collective --world-size 8
+```
+
+**Output:** Per-trace reports in `--output-dir`, plus `tracelens_summary.csv` and `tracelens_summary.json`.
+
+See the [Profiling Guide](profiling.md#tracelens---tracelens-trace-analysis) for which trace formats map to which report.
+
+---
+
+##### `report tracelens-compare` - Diff TraceLens Reports
+
+Compare two or more TraceLens reports into a single diff workbook. The first report is the baseline; every metric gains `_diff` and `_pct` columns relative to it.
+
+**Usage:**
+
+```bash
+madengine report tracelens-compare REPORT... [OPTIONS]
+```
+
+**Options:**
+
+| Option | Short | Type | Default | Description |
+|--------|-------|------|---------|-------------|
+| `--output` | `-o` | TEXT | `tracelens_comparison.xlsx` | Output comparison workbook |
+| `--names` | | TEXT | | Display tag per report (repeat the flag) |
+| `--python` | | TEXT | | Interpreter that has TraceLens installed |
+| `--verbose` | `-v` | FLAG | `False` | Enable verbose logging |
+
+**Examples:**
+
+```bash
+# Compare a baseline against a candidate run
+madengine report tracelens-compare baseline.xlsx candidate.xlsx
+
+# Label each report and choose the output path
+madengine report tracelens-compare a.xlsx b.xlsx \
+  --names before --names after -o diff.xlsx
+```
+
+---
+
 ### `database` - Upload to MongoDB
 
 Upload CSV performance data to MongoDB database.

docs/profiling.md

@@ -342,6 +342,167 @@ madengine run --tags your_model \
   }'
 ```
 
+### torch_profiler_dynolog - On-Demand PyTorch Traces
+
+Capture `torch.profiler` (Kineto) traces from a running PyTorch workload without editing the model script. Instead of wrapping the command, this tool starts the [dynolog](https://github.com/facebookincubator/dynolog) daemon inside the container; PyTorch registers with it because the tool sets `KINETO_USE_DAEMON=1`, and a background script then requests a trace with `dyno gputrace`.
+
+```json
+{
+  "tools": [
+    {"name": "torch_profiler_dynolog"}
+  ]
+}
+```
+
+**Output:** `torch_profiler_output/libkineto_trace_<pid>.json` (one file per rank)
+
+**Requirements:**
+
+- The workload must be PyTorch >= 1.13. Nothing is captured from non-PyTorch models.
+- Iteration-based capture counts `optimizer.step()` calls. Workloads without an optimizer (pure inference) should set `TORCH_PROFILE_ITERATIONS` to `0` to fall back to duration-based capture.
+- The pre-script downloads the dynolog `.deb` from GitHub, so the container needs outbound HTTPS on the first run (x86_64 Debian/Ubuntu base image). Set `DYNOLOG_DEB_URL` to use a mirror, or bake `dynolog` and `dyno` into the image to skip the download entirely.
+
+**Environment Variables:**
+
+| Variable | Default | Purpose |
+|----------|---------|---------|
+| `TORCH_PROFILE_WARMUP_S` | `60` | Delay before the first trace request, so the workload reaches steady state |
+| `TORCH_PROFILE_ITERATIONS` | `5` | Iterations to capture. `0` switches to `TORCH_PROFILE_DURATION_MS` |
+| `TORCH_PROFILE_DURATION_MS` | `500` | Capture window when iteration counting is not used |
+| `TORCH_PROFILE_RETRY_INTERVAL_S` | `15` | Wait between attempts while no PyTorch process has registered |
+| `TORCH_PROFILE_MAX_ATTEMPTS` | `40` | Attempts before giving up |
+| `TORCH_PROFILE_PROCESS_LIMIT` | `64` | Ranks to trace. Upstream defaults to 3, which silently drops most ranks |
+| `TORCH_PROFILE_RECORD_SHAPES` | `1` | Record input shapes (needed for TraceLens per-operator analysis) |
+| `TORCH_PROFILE_WITH_STACKS` | `1` | Record CPU call stacks |
+| `TORCH_PROFILE_WITH_MODULES` | `1` | Record the `nn.Module` hierarchy |
+| `TORCH_PROFILE_WITH_FLOPS` | `0` | Estimate FLOPs per operator |
+| `TORCH_PROFILE_PROFILE_MEMORY` | `0` | Record allocator events |
+
+For a short-lived workload, shorten the warmup so the request lands while the model is still running:
+
+```json
+{
+  "tools": [
+    {
+      "name": "torch_profiler_dynolog",
+      "env_vars": {
+        "TORCH_PROFILE_WARMUP_S": "20",
+        "TORCH_PROFILE_MAX_ATTEMPTS": "10"
+      }
+    }
+  ]
+}
+```
+
+**No trace produced?** The teardown script reports how many traces it found. `no PyTorch process registered` means `dyno gputrace` never matched the workload: confirm the model is PyTorch, and raise `TORCH_PROFILE_WARMUP_S` and `TORCH_PROFILE_MAX_ATTEMPTS` for slow-starting jobs.
+
+### tracelens - TraceLens Trace Analysis
+
+[TraceLens](https://github.com/AMD-AGI/TraceLens) turns raw traces into operator, kernel, roofline, and collective reports. It analyzes traces the other profiling tools produce, so always pair it with a profiler; on its own it has nothing to read.
+
+Each trace format is routed to the matching TraceLens report:
+
+| Trace produced by | Format | TraceLens report |
+|-------------------|--------|------------------|
+| `torch_profiler_dynolog` | Kineto JSON | Per-operator, kernel summary, roofline, `nn.Module` breakdown |
+| `rocprofv3_lightweight` | rocprofv3 JSON | Kernel summary and details |
+| `rocprofv3_perfetto` | `.pftrace` | HIP activity, HIP API, and memory-copy reports |
+| Multiple per-rank Kineto traces | Kineto JSON | Multi-rank collective report |
+
+**Unreadable formats:** TraceLens cannot read rocprofv3's default SQLite (`*_results.db`) or RPD (`.rpd`) databases. Those are listed in the summary as `SKIPPED` with a pointer at a preset that works — use `rocprofv3_lightweight` for JSON or `rocprofv3_perfetto` for `.pftrace`. For `rpd`, point TraceLens at the `trace.json` its post-script writes alongside the database.
+
+#### Analyzing on the Host (Recommended)
+
+Running analysis on the host keeps TraceLens' pinned `protobuf` and `xprof` out of your workload image:
+
+```bash
+pip install 'madengine[tracelens]'
+
+# Analyze every trace a run left behind
+madengine report tracelens
+
+# See what would be analyzed, without running TraceLens
+madengine report tracelens --discover-only
+
+# Enable roofline bound classification
+madengine report tracelens --gpu-arch MI300X
+
+# Analyze a distributed run's collected artifacts
+madengine report tracelens --root slurm_results --mode collective --world-size 8
+```
+
+**Output:** `tracelens_output/` with per-trace reports plus `tracelens_summary.csv` and `tracelens_summary.json`
+
+Quantify the effect of a change by diffing two runs' reports. The first report is the baseline; every metric gains `_diff` and `_pct` columns relative to it:
+
+```bash
+madengine report tracelens-compare baseline.xlsx candidate.xlsx \
+  --names before --names after -o diff.xlsx
+```
+
+#### Analyzing in the Container
+
+Stack the `tracelens` tool **after** a profiler to get reports as part of the run itself:
+
+```json
+{
+  "tools": [
+    {"name": "rocprofv3_lightweight"},
+    {"name": "tracelens"}
+  ]
+}
+```
+
+**Output:** `tracelens_output/` copied to the working directory alongside the profiler's own output
+
+The pre-script installs TraceLens into an isolated virtualenv at `/opt/madengine-tracelens-venv` (no system site-packages), so its dependency pins cannot disturb the workload's Python environment. This needs outbound HTTPS to `github.com` on the first run. Analysis failures are reported as warnings and never fail a passing model run.
+
+Use a mode-specific variant to restrict analysis to one trace kind:
+
+| Tool | Analyzes |
+|------|----------|
+| `tracelens` | Every supported trace found (default) |
+| `tracelens_pytorch` | Kineto traces only |
+| `tracelens_rocprof` | rocprofv3 JSON only |
+| `tracelens_pftrace` | Perfetto traces only |
+| `tracelens_collective` | Multi-rank collective report only |
+
+**Full PyTorch pipeline** — capture and analyze in one run:
+
+```bash
+madengine run --tags your_model \
+  --additional-context '{
+    "gpu_vendor": "AMD",
+    "guest_os": "UBUNTU",
+    "tools": [
+      {"name": "torch_profiler_dynolog"},
+      {"name": "tracelens"}
+    ]
+  }'
+```
+
+**Environment Variables:**
+
+| Variable | Default | Purpose |
+|----------|---------|---------|
+| `TRACELENS_VENV` | `/opt/madengine-tracelens-venv` | Virtualenv holding TraceLens |
+| `TRACELENS_OUTPUT_DIR` | `tracelens_output` | Directory for generated reports |
+| `TRACELENS_MODE` | `auto` | `auto`, `pytorch`, `rocprof`, `pftrace`, or `collective` |
+| `TRACELENS_GPU_ARCH` | unset | GPU arch (e.g. `MI300X`) enabling roofline bound classification |
+| `TRACELENS_WORLD_SIZE` | trace count | Rank count for the collective report |
+| `TRACELENS_MAX_TRACES` | unset | Cap traces analyzed per kind |
+| `TRACELENS_GIT_REF` | pinned commit | TraceLens revision to install |
+| `TRACELENS_PIP_SPEC` | git URL at `TRACELENS_GIT_REF` | Full pip spec, for private mirrors |
+
+#### Distributed Runs
+
+SLURM and Kubernetes runs collect `torch_profiler_output/` and `tracelens_output/` alongside the other profiling directories, per node and per rank. Analyze the collected tree on the host by pointing `--root` at it:
+
+```bash
+madengine report tracelens --root slurm_results
+madengine report tracelens --root k8s_results --mode collective --world-size 16
+```
+
 ### rocblas_trace - rocBLAS Library Tracing
 
 Trace rocBLAS API calls and configurations:
@@ -681,6 +842,8 @@ madengine run --tags model \
 |------|---------------|---------|
 | `rocprof` | `rocprof_output/*` | GPU kernel traces, HIP API calls |
 | `rpd` | Various RPD files | ROCm profiler data |
+| `torch_profiler_dynolog` | `torch_profiler_output/*.json` | Kineto traces, one per rank |
+| `tracelens` | `tracelens_output/*` | TraceLens reports plus `tracelens_summary.csv` |
 | `rocblas_trace` | `library_trace.csv`, logs | rocBLAS API calls |
 | `miopen_trace` | `library_trace.csv`, logs | MIOpen API calls |
 | `tensile_trace` | `library_trace.csv`, logs | Tensile operations |

examples/profiling-configs/README.md

@@ -110,7 +110,29 @@ madengine run --tags your_model \
   --additional-context-file examples/profiling-configs/rocm_trace_lite_default.json
 ```
 
-### 7. Multi-Node Distributed (`rocprofv3_multi_node.json`)
+### 7. TraceLens Analysis (`torch_profiler_tracelens.json`, `tracelens_rocprofv3.json`)
+
+**Use Case**: Turn raw traces into operator, kernel, roofline, and collective reports with [TraceLens](https://github.com/AMD-AGI/TraceLens). TraceLens only analyzes traces, so it is always stacked after a profiler.
+
+- **`torch_profiler_tracelens.json`** — captures `torch.profiler` (Kineto) traces on demand via dynolog, then generates the PyTorch operator and roofline reports. The workload must be PyTorch, and iteration-based capture needs an `optimizer.step()` loop.
+- **`tracelens_rocprofv3.json`** — profiles with `rocprofv3_lightweight` (JSON) and generates the kernel summary reports.
+
+**Requirements / notes:**
+
+- The trace pre-script installs TraceLens into an isolated virtualenv at `/opt/madengine-tracelens-venv`, so its pinned `protobuf` and `xprof` cannot disturb the workload's Python environment. The container needs **HTTPS access to GitHub** on the first run. Override the revision with `TRACELENS_GIT_REF`, or the whole spec with `TRACELENS_PIP_SPEC`.
+- TraceLens cannot read rocprofv3's default SQLite (`*_results.db`) or RPD databases. Use `rocprofv3_lightweight` (JSON) or `rocprofv3_perfetto` (`.pftrace`); unreadable artifacts are listed in `tracelens_output/tracelens_summary.csv` as `SKIPPED` with guidance.
+- To keep TraceLens out of the workload image entirely, skip the in-container tool and analyze on the host instead: `pip install 'madengine[tracelens]'` then `madengine report tracelens`. See the [Profiling Guide](../../docs/profiling.md#tracelens---tracelens-trace-analysis).
+
+**Usage**:
+```bash
+madengine run --tags your_model \
+  --additional-context-file examples/profiling-configs/torch_profiler_tracelens.json
+
+madengine run --tags your_model \
+  --additional-context-file examples/profiling-configs/tracelens_rocprofv3.json
+```
+
+### 8. Multi-Node Distributed (`rocprofv3_multi_node.json`)
 
 **Use Case**: Large-scale distributed training on SLURM clusters
 
@@ -217,6 +239,8 @@ The wrapper script auto-detects which profiler is available and formats the comm
 
 **Other:** `rocm_trace_lite` (RTL **lite** mode) and `rocm_trace_lite_default` (RTL **default** mode) — kernel dispatch SQLite trace via [rocm-trace-lite](https://sunway513.github.io/rocm-trace-lite/index.html), installed from **GitHub Release wheels** by the trace pre-script (not PyPI; see [Profiling Guide](../../docs/profiling.md)). Not a rocprofv3 preset; do not combine with `rocprof` / `rocprofv3_*` on the same run.
 
+**PyTorch tracing and analysis:** `torch_profiler_dynolog` captures Kineto traces on demand, and `tracelens` (or the mode-specific `tracelens_pytorch`, `tracelens_rocprof`, `tracelens_pftrace`, `tracelens_collective`) generates TraceLens reports from whatever traces a co-selected profiler produced. Neither is a rocprofv3 preset; `tracelens` is analysis only and must be stacked after a profiler.
+
 ## Counter Definition Files
 
 Counter files are located at `src/madengine/scripts/common/tools/counters/`:
@@ -247,6 +271,9 @@ gpu_info_vram_profiler_output.csv   # VRAM usage over time
 library_trace.csv                    # Library API calls (if library tracing enabled)
 
 rocm_trace_lite_output/trace.db       # rocm-trace-lite (also trace.json.gz / trace_summary.txt as emitted by RTL)
+
+torch_profiler_output/*.json          # Kineto traces, one per rank (torch_profiler_dynolog)
+tracelens_output/                     # TraceLens reports plus tracelens_summary.csv
 ```
 
 ## Visualization

examples/profiling-configs/torch_profiler_tracelens.json

@@ -0,0 +1,19 @@
+{
+  "gpu_vendor": "AMD",
+  "guest_os": "UBUNTU",
+  "tools": [
+    {
+      "name": "torch_profiler_dynolog",
+      "env_vars": {
+        "TORCH_PROFILE_ITERATIONS": "5",
+        "TORCH_PROFILE_WARMUP_S": "60"
+      }
+    },
+    {
+      "name": "tracelens",
+      "env_vars": {
+        "TRACELENS_MODE": "pytorch"
+      }
+    }
+  ]
+}

examples/profiling-configs/tracelens_rocprofv3.json

@@ -0,0 +1,15 @@
+{
+  "gpu_vendor": "AMD",
+  "guest_os": "UBUNTU",
+  "tools": [
+    {
+      "name": "rocprofv3_lightweight"
+    },
+    {
+      "name": "tracelens",
+      "env_vars": {
+        "TRACELENS_MODE": "rocprof"
+      }
+    }
+  ]
+}

pyproject.toml

@@ -45,6 +45,14 @@ classifiers = [
   "Operating System :: OS Independent",
 ]
 
+[project.optional-dependencies]
+# Host-side trace analysis for `madengine report tracelens`. Kept optional
+# because TraceLens pins protobuf>=6.31 and xprof, which can conflict with other
+# packages in a shared environment.
+tracelens = [
+  "TraceLens @ git+https://github.com/AMD-AGI/TraceLens.git@6f9bcdbf6cc9911eb650de57b345917ea4d31a17",
+]
+
 [project.scripts]
 madengine = "madengine.cli.app:cli_main"