Skip to content

Adpat mori to RoCM 714 container - #541

Open
QizhouZhang97 wants to merge 14 commits into
mainfrom
dev/adpat_rocm714
Open

Adpat mori to RoCM 714 container#541
QizhouZhang97 wants to merge 14 commits into
mainfrom
dev/adpat_rocm714

Conversation

@QizhouZhang97

@QizhouZhang97 QizhouZhang97 commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Moves CI to rocm/pytorch:rocm7.14_ubuntu24.04_py3.12_pytorch_release_2.12.0, dropping 7.2.4.

The 7.14 image has no /opt/rocm — ROCm comes from the rocm-sdk-core wheel, which lacks CMake configs, and its hipconfig shim shadows the real one and ignores HIP_PATH.

  • CMakeLists.txt: resolve ROCm before project(... HIP) without hipconfig; pin CMAKE_HIP_COMPILER; RPATH the ROCm libdir so no LD_LIBRARY_PATH is needed
  • Dockerfile.dev: apt ROCm 7.14 dev stack, /usr/lib64/libc.so symlink
  • Dockerfile.cco: pin flydsl<0.3
  • nightly: py3.10 dropped (no 7.14 image)
  • EPv2 gather hangs on 7.14 — a ROCm regression (passes on 7.2.4); gated off, scatter still covered.

Tested on 8×MI355X: CCO, SDMA, EPv1 (335 passed), mori check all green.

Comment thread docker/Dockerfile.cco Outdated
Comment thread docker/Dockerfile.cco Outdated
Comment on lines +68 to +56
RUN pip install --no-cache-dir flydsl
# Pinned <0.3: 0.3.x dropped `flydsl.expr.buffer_ops`, still imported directly
# by the cco examples (04/05/07/08). Unpin once they move to the 0.3.x API.
RUN pip install --no-cache-dir "flydsl<0.3"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

现在应该已经支持flydsl0.3.0了,可以对应升级一下

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0.3.0貌似并没有bufferops?

Comment thread .claude/skills/deploy-mori/SKILL.md Outdated
Comment thread docker/Dockerfile.dev

@jhchouuu jhchouuu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two things in docker/Dockerfile.dev worth fixing before this lands. Both hit every base image, including the 7.2.4 py3.10 nightly lane that now builds from this same file.

1. ENV LD_LIBRARY_PATH=/opt/rocm/lib overwrites instead of appending (line 55). The Dockerfile.cco it replaces had :${LD_LIBRARY_PATH}, so whatever the base image set is now dropped.

ENV LD_LIBRARY_PATH=/opt/rocm/lib:${LD_LIBRARY_PATH}

2. ENV LD_PRELOAD=... is unconditional, but the install that provides the file is not (line 61). When the base already ships a ROCm dev stack the apt step is skipped, yet the preload still points at /opt/rocm/lib/libhsa-runtime64.so.1. If that path doesn't exist, every command in the image prints ld.so: object ... cannot be preloaded ...: ignored. Suggest gating it the same way as the install (ARG, or set it from the else branch).

Unrelated: the PR body still says "EPv2 gather hangs on 7.14 — ... gated off, scatter still covered", but e59ff2b removed the gating and later commits cleaned up the comments. Worth updating so the squashed commit message doesn't carry it, and it'd be useful to note whether gather turned out to be fixed on 7.14 or the original diagnosis was off.

@QizhouZhang97

Copy link
Copy Markdown
Contributor Author

Two things in docker/Dockerfile.dev worth fixing before this lands. Both hit every base image, including the 7.2.4 py3.10 nightly lane that now builds from this same file.

1. ENV LD_LIBRARY_PATH=/opt/rocm/lib overwrites instead of appending (line 55). The Dockerfile.cco it replaces had :${LD_LIBRARY_PATH}, so whatever the base image set is now dropped.

ENV LD_LIBRARY_PATH=/opt/rocm/lib:${LD_LIBRARY_PATH}

2. ENV LD_PRELOAD=... is unconditional, but the install that provides the file is not (line 61). When the base already ships a ROCm dev stack the apt step is skipped, yet the preload still points at /opt/rocm/lib/libhsa-runtime64.so.1. If that path doesn't exist, every command in the image prints ld.so: object ... cannot be preloaded ...: ignored. Suggest gating it the same way as the install (ARG, or set it from the else branch).

Unrelated: the PR body still says "EPv2 gather hangs on 7.14 — ... gated off, scatter still covered", but e59ff2b removed the gating and later commits cleaned up the comments. Worth updating so the squashed commit message doesn't carry it, and it'd be useful to note whether gather turned out to be fixed on 7.14 or the original diagnosis was off.

Two things in docker/Dockerfile.dev worth fixing before this lands. Both hit every base image, including the 7.2.4 py3.10 nightly lane that now builds from this same file.

1. ENV LD_LIBRARY_PATH=/opt/rocm/lib overwrites instead of appending (line 55). The Dockerfile.cco it replaces had :${LD_LIBRARY_PATH}, so whatever the base image set is now dropped.

ENV LD_LIBRARY_PATH=/opt/rocm/lib:${LD_LIBRARY_PATH}

2. ENV LD_PRELOAD=... is unconditional, but the install that provides the file is not (line 61). When the base already ships a ROCm dev stack the apt step is skipped, yet the preload still points at /opt/rocm/lib/libhsa-runtime64.so.1. If that path doesn't exist, every command in the image prints ld.so: object ... cannot be preloaded ...: ignored. Suggest gating it the same way as the install (ARG, or set it from the else branch).

Unrelated: the PR body still says "EPv2 gather hangs on 7.14 — ... gated off, scatter still covered", but e59ff2b removed the gating and later commits cleaned up the comments. Worth updating so the squashed commit message doesn't carry it, and it'd be useful to note whether gather turned out to be fixed on 7.14 or the original diagnosis was off.

移除容器中对ld_library_path, ld_preload的依赖

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants