Skip to content

[release/2.12] Pin torchaudio to ROCm/audio release/2.11.0.1 - #3382

Open
ethanwee1 wants to merge 1 commit into
release/2.12from
ew/rocm-24803-pin-torchaudio-2.11
Open

[release/2.12] Pin torchaudio to ROCm/audio release/2.11.0.1#3382
ethanwee1 wants to merge 1 commit into
release/2.12from
ew/rocm-24803-pin-torchaudio-2.11

Conversation

@ethanwee1

@ethanwee1 ethanwee1 commented Jun 30, 2026

Copy link
Copy Markdown

https://amd-hub.atlassian.net/browse/ROCM-24803

Summary

Rebased onto current release/2.12. Pins related_commits to the source-only ROCm GPU port for torchaudio 2.11.0.1 (ROCm/audio#17 @ a0da46b3).

The Windows CUDAExtension cross-drive HIPIFY fix is split out to ROCm/pytorch#3562 per review feedback.

related_commits

Uses ethanwee1/audio fork URL until #17 merges; then flip origin to https://github.com/ROCm/audio.

Validation

TheRock Windows gfx110X-all build dispatched on combined branch ew/rocm-24803-validation (this pin + #3562).

Companion PRs

@ethanwee1

Copy link
Copy Markdown
Author

TheRock test build (Windows, gfx110X-all) for this torchaudio pin:
https://github.com/ROCm/TheRock/actions/runs/28466063021

Inputs:

  • workflow: Multi-Arch Build Windows PyTorch Wheels
  • pytorch_git_ref: ew/rocm-24803-pin-torchaudio-2.11 (this branch on ROCm/pytorch)
  • amdgpu_families: gfx110X-all
  • python_version: 3.12
  • rocm_version: 7.14.0.dev0+331251018497de4049c7d8447c7b2d1dcdfd012e

Verifies torchaudio resolves from ROCm/audio release/2.11.0.1 (commit b83203e8, version 2.11.0.1) and builds/sanity-checks a wheel.

@rocm-repo-management-api

rocm-repo-management-api Bot commented Jun 30, 2026

Copy link
Copy Markdown

Jenkins build for 397caab617f3b3beb91836468259cc663e1210b9 commit finished as NOT_BUILT
Links: Pipeline Overview / Build artifacts / Test Results

@ethanwee1

Copy link
Copy Markdown
Author

This PR now contains two commits:

  1. Pin torchaudio to ROCm/audio release/2.11.0.1 (validated: builds version 2.11.0.1+devrocm...).
  2. Fix Windows cross-drive HIPIFY failure in torch/utils/cpp_extension.py (CUDAExtension), which blocked the torchaudio wheel build on runners where the checkout is on a subst'd B: drive aliasing C: (ValueError: path is on mount 'C:', start on mount 'B:').

New TheRock test build (Windows, gfx110X-all) with both changes:
https://github.com/ROCm/TheRock/actions/runs/28474577213

Previous build (pin only, failed on the cross-drive bug now fixed): https://github.com/ROCm/TheRock/actions/runs/28466063021

@rocm-repo-management-api

rocm-repo-management-api Bot commented Jun 30, 2026

Copy link
Copy Markdown

Jenkins build for 212d846a9952acce94eefcb7b3bcf9a69225e841 commit finished as NOT_BUILT
Links: Pipeline Overview / Build artifacts / Test Results

@ethanwee1

Copy link
Copy Markdown
Author

Build status update

TheRock test build (Windows, gfx110X-all): https://github.com/ROCm/TheRock/actions/runs/28474577213FAILED (in the torchaudio compile), but it got materially further than the prior run.

Changes in this PR (3 commits) and what each achieved:

  1. Pin torchaudio to ROCm/audio release/2.11.0.1 (commit b83203e8) in related_commits — resolves + builds version 2.11.0.1+devrocm... (final release, not the pip-skipped 2.11.0a0 from main). ✅ validated by the build.
  2. Fix Windows cross-drive HIPIFY failure in torch/utils/cpp_extension.py (CUDAExtension) — realpath-normalize the build dir and hipified source before os.path.relpath, so a subst'd B: drive that aliases C: no longer throws ValueError: path is on mount 'C:', start on mount 'B:'. ✅ validated: the new build had zero cross-drive errors and torchaudio actually started compiling (the log even reveals the real path C:\<GUID>\src\pytorch_audio, confirming B: was a subst alias).

Why the build still fails (out of scope for this PR):
torchaudio's rnnt/forced_align GPU extensions on release/2.11.0.1 are CUDA-only and not ported to ROCm. First error:

src/libtorchaudio/cuda_utils.h:3:10: fatal error: 'cuda_runtime_api.h' file not found

This is not a one-line header fix — behind it, rnnt/gpu/gpu_transducer.h and the GPU kernel headers (math.cuh, gpu_kernel_utils.cuh, gpu_kernels.cuh) are entirely wrapped in #ifdef USE_CUDA (empty on ROCm), and macros.h/options.h gate their CUDA includes/device macros on USE_CUDA rather than USE_ROCM, while extension.py still adds the .cu GPU sources for USE_ROCM. Making torchaudio's GPU ops build on ROCm/Windows is a HIP porting effort across many kernel files (and needs GPU testing).

Disposition: the GPU port is being handed to the torchaudio/audio team. This PR stands on its own as (1) the correct torchaudio version pin and (2) a general Windows cross-drive HIPIFY fix, both build-validated up to the torchaudio ROCm GPU-port gap.

@rocm-repo-management-api

rocm-repo-management-api Bot commented Jul 1, 2026

Copy link
Copy Markdown

Jenkins build for 212d846a9952acce94eefcb7b3bcf9a69225e841 commit finished as FAILURE
Links: Pipeline Overview / Build artifacts / Test Results

@rocm-repo-management-api

rocm-repo-management-api Bot commented Jul 1, 2026

Copy link
Copy Markdown

Jenkins build for 74bce3687d4eacf122f181b438885d907a49c1cb commit finished as NOT_BUILT
Links: Pipeline Overview / Build artifacts / Test Results

jithunnair-amd pushed a commit that referenced this pull request Jul 1, 2026
https://amd-hub.atlassian.net/browse/ROCM-24803?focusedCommentId=500051

Interim unblock: pins torchaudio to the stable release/2.11 (2.11.0),
which already builds and publishes on Windows, replacing the floating
pytorch/audio main (2.11.0a0) pin.

The torchaudio 2.11.0.1 ROCm GPU port is tracked separately in
#3382 and ROCm/audio#16.
@ethanwee1

Copy link
Copy Markdown
Author

Reworked to the HIPIFY-based torchaudio port (ROCm/audio#17, mirrors torchvision) instead of the manual compat shim. New build: https://github.com/ROCm/TheRock/actions/runs/28538325024

@rocm-repo-management-api

rocm-repo-management-api Bot commented Jul 1, 2026

Copy link
Copy Markdown

Jenkins build for 74bce3687d4eacf122f181b438885d907a49c1cb commit finished as FAILURE
Links: Pipeline Overview / Build artifacts / Test Results

@ethanwee1

Copy link
Copy Markdown
Author

Repointed related_commits back to the validated compat-shim port (ROCm/audio#16 @ a890448a on ew/rocm-24803-hip-port-gpu-ext). Closed ROCm/audio#17 (HIPIFY setup.py experiment).

Where the HIPIFY fixes live:

Prior passing TheRock run with this pin combo: https://github.com/ROCm/TheRock/actions/runs/28529085509

ethanwee1 added a commit to ethanwee1/audio that referenced this pull request Jul 6, 2026
…yle)

Mirror torchvision's setup.py approach: run hipify_python.hipify() over
libtorchaudio CUDA sources and headers, then compile the generated .hip
outputs.

Fixes vs the first HIPIFY attempt (TheRock 28538325024):

1. Do not widen `#ifdef USE_CUDA` to `USE_ROCM` in headers — that pulled
   `cuda_fp16.h` into CPU `.cpp` compiles because PyTorch defines USE_ROCM
   globally on ROCm extension builds. Keep upstream USE_CUDA guards; pass
   `-DUSE_CUDA` on hipcc/nvcc only for GPU translation units.
2. Hipify with `os.path.realpath()` paths and depth-aware include globs so
   headers like `cuda_utils.h` are rewritten on the same inode Windows runners
   use when B: is a subst alias of the C: checkout (otherwise the compiler
   reads un-hipified headers from the underlying path).
3. Include `cuctc/` in the hipify scope.

Reference: https://github.com/ROCm/vision/blob/release/0.27/setup.py

Paired with ROCm/pytorch#3382 (torchaudio pin + Windows cross-drive HIPIFY fix
in torch/utils/cpp_extension.py — that fix is in PyTorch, not here).
@ethanwee1

Copy link
Copy Markdown
Author

Pin updated to audio HIPIFY v2 0c7509ef. Windows cross-drive fix remains in this PR's cpp_extension.py; torchaudio GPU port is HIPIFY-only in ROCm/audio#17.

@rocm-repo-management-api

rocm-repo-management-api Bot commented Jul 6, 2026

Copy link
Copy Markdown

Jenkins build for 20ed9fc17d7e46e02019b8f652ca8037fcac073e commit finished as FAILURE
Links: Pipeline Overview / Build artifacts / Test Results

@rocm-repo-management-api

rocm-repo-management-api Bot commented Jul 8, 2026

Copy link
Copy Markdown

Jenkins build for 7b07f69b1ff1b8b60426138073ec3163051217e9 commit finished as FAILURE
Links: Pipeline Overview / Build artifacts / Test Results

@ethanwee1

Copy link
Copy Markdown
Author

Pin now on HIPIFY v3 (ROCm/audio#17 133b6687). Windows gfx110X-all build passed: https://github.com/ROCm/TheRock/actions/runs/28953516514 — produced torchaudio-2.11.0.1+devrocm...-win_amd64.whl.

@ethanwee1
ethanwee1 requested a review from jithunnair-amd July 16, 2026 15:26
ethanwee1 added a commit to ethanwee1/audio that referenced this pull request Jul 17, 2026
…header + guards)

Addresses review feedback (jithunnair-amd) to not invoke hipify manually in
the torchaudio setup file. Instead this ports libtorchaudio to ROCm via
fork source updates only, relying on the built-in CUDAExtension hipify for
the .cu source bodies:

- cuda_compat.h maps the CUDA runtime symbols used in headers to their HIP
  equivalents under USE_ROCM.
- Widen the rnnt/forced_align GPU #ifdef USE_CUDA guards, macros.h, and
  Options::stream_ to also cover USE_ROCM.
- Route cub through hipcub under USE_ROCM.

The Windows B:/C: subst cross-drive HIPIFY fix lives in ROCm/pytorch#3382
(torch/utils/cpp_extension.py).
@rocm-repo-management-api

rocm-repo-management-api Bot commented Jul 17, 2026

Copy link
Copy Markdown

Jenkins build for 19ca05d9a4931735adf4fddbbea53de643810edf commit finished as FAILURE
Links: Pipeline Overview / Build artifacts / Test Results

ethanwee1 added a commit to ethanwee1/audio that referenced this pull request Aug 13, 2026
…header + guards)

Addresses review feedback (jithunnair-amd) to not invoke hipify manually in
the torchaudio setup file. Instead this ports libtorchaudio to ROCm via
fork source updates only, relying on the built-in CUDAExtension hipify for
the .cu source bodies:

- cuda_compat.h maps the CUDA runtime symbols used in headers to their HIP
  equivalents under USE_ROCM.
- Widen the rnnt/forced_align GPU #ifdef USE_CUDA guards, macros.h, and
  Options::stream_ to also cover USE_ROCM.
- Route cub through hipcub under USE_ROCM.

The Windows B:/C: subst cross-drive HIPIFY fix lives in ROCm/pytorch#3382
(torch/utils/cpp_extension.py).
Point related_commits at the source-only HIP port (ROCm/audio#17) on
release/2.11.0.1. Uses ethanwee1/audio fork until #17 merges; then flip
origin to https://github.com/ROCm/audio.

Companion: ROCm/audio#17, ROCm/pytorch cpp_extension fix (separate PR).

ROCM-24803
@ethanwee1

Copy link
Copy Markdown
Author

Rebased stack ready for review. TheRock Windows gfx110X-all validation build (pin + cpp_extension fix on ew/rocm-24803-validation):

Run: https://github.com/ROCm/TheRock/actions/runs/31709842398

Input Value
pytorch_git_ref ew/rocm-24803-validation
amdgpu_families gfx110X-all
test_amdgpu_families gfx110X-all
python_version 3.12
rocm_version 7.14.0a20260624
rocm_package_index_url https://rocm.nightlies.amd.com/whl-multi-arch/

PR split (ROCM-24803):

  • This PR — related_commits pin only (rebased onto release/2.12)
  • ROCm/pytorch#3562 — Windows B:/C: CUDAExtension realpath fix
  • ROCm/audio#17 — source-only HIP port @ a0da46b3 on release/2.11.0.1

cc @jithunnair-amd @pablo-garay for review

@ethanwee1

Copy link
Copy Markdown
Author

Build failure analysis for run 31709842398:

Not a PyTorch/torchaudio code issue. The build died in Install requirements before checkout/build started:

choco install --no-progress -y ninja --version 1.13.1
A newer version of ninja (v1.13.2) is already installed.
Chocolatey installed 0/1 packages. 1 packages failed.

The windows-2022 runner image was updated (20260802.262) and now ships ninja 1.13.2, while TheRock main still pins 1.13.1 via chocolatey (which refuses downgrade).

Redispatch using Pablo's ninja-tolerant workflow branch (same fix as users/garayp/win-ninja-tolerant) + the ROCm version from the July passing build:

New run: https://github.com/ROCm/TheRock/actions/runs/31711078734

Input Value
repository / ref ROCm/TheRock @ users/garayp/win-ninja-tolerant
rocm_version 7.14.0.dev0+331251018497de4049c7d8447c7b2d1dcdfd012e (same as 28953516514)
pytorch_git_ref ew/rocm-24803-validation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant