Skip to content

(feat) Enable MLX5 + MI300X CI - #560

Open
QizhouZhang97 wants to merge 2 commits into
mainfrom
dev/mlx_cq_create
Open

(feat) Enable MLX5 + MI300X CI#560
QizhouZhang97 wants to merge 2 commits into
mainfrom
dev/mlx_cq_create

Conversation

@QizhouZhang97

@QizhouZhang97 QizhouZhang97 commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Motivation

On mlx5 hosts without a working peer-memory bridge, DEVX registration of GPU
memory fails with EFAULT and MORI aborts at assert(cqUmem). Peer-memory is
missing from inbox ib_uverbs, and OFED cannot supply it on kernels >= 6.3:
ib_register_peer_memory_client is exported non-GPL, so symbol_get() rejects
it and amdgpu's peer-direct bridge never registers.

Technical Details

Register the GPU CQ/QP/DBR umems and the atomic ibuf through dmabuf
(mlx5dv_devx_umem_reg_ex, ibv_reg_dmabuf_mr), which pins via
ib_umem_dmabuf_get_pinned and needs no peer-memory client. Both helpers fall
back to by-address registration, so hosts that already work are unaffected. The
symbol is resolved optionally and its descriptor vendored, keeping the build
green on rdma-core v39. Also tolerates hipErrorHostMemoryAlreadyRegistered
when sibling QPs share a UAR page.

CI gains an MI300X_MLX platform (2x MI300X, 8x CX-7 NDR InfiniBand). SL/TC are
omitted -- mlx5 applies them only on the RoCE path.

Test Result

Two-node EP dispatch/combine (v1, v1_ll, async_ll), bit-exact hierarchical
AllGather, and MORI-IO all pass on the new runner.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant