Skip to content

ch4/ipc: fix host (cma ipc) -> device (gpu/skip ipc) copies for oversized data types - #7924

Open
nmnobre wants to merge 2 commits into
pmodels:mainfrom
nmnobre:bcast
Open

ch4/ipc: fix host (cma ipc) -> device (gpu/skip ipc) copies for oversized data types#7924
nmnobre wants to merge 2 commits into
pmodels:mainfrom
nmnobre:bcast

Conversation

@nmnobre

@nmnobre nmnobre commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Pull Request Description

The aim is to fix the bcast tests when MPIR_CVAR_CH4_CMA_ENABLE=1, which is the default on my machine:

../../installed/bin/mpirun -n 2 ./coll/bcast -seed=106 -testsize=2 -type=MPI_INT -count=65530 -evenmemtype=device -oddmemtype=host
Error class 15 (Other MPI error, error stack:
internal_Bcast(128)......................: MPI_Bcast(buffer=0x7fa3c4e00000, count=2, dtype=USER<hvector>, 1, MPI_COMM_WORLD) failed
MPID_Bcast(290)..........................: 
MPIDI_Bcast_allcomm_composition_json(236): 
MPIDI_Bcast_intra_composition_alpha(255).: 
MPIC_Recv(210)...........................: 
MPIC_Wait(99)............................: 
MPIR_Wait(745)...........................: 
MPIDI_IPC_rndv_cb(302)...................: 
MPIDI_CMA_copy_data(67).................)
expected 0 at 0x7fa3c523ffe0, but found -1
DTP_obj_buf_check failed.
coll [nesting levels: 3
0: hvector [numblks 1, blklen 1, stride 10484648]
1: resized [lb 2621160, extent 2621160]
2: blkindx [numblks 32765, blklen 1, displs (65528,65526,65524,65522,65520,65518,65516,65514,65512,65510,...,18,16,14,12,10,8,6,4,2,0)]
base: MPI_INT
]

I now realise that the tests were crashing on my machine for different reasons they are crashing in the Jenkins...
I suppose the CI machines have CMA disabled?

Also, can I ask how I can control the correct execution of alltoall, allgather_gpu, allgatherv_gpu at the bottom of testlist.gpu given we are setting MPIR_CVAR_ALLGATHERV_POSIX_INTRA_ALGORITHM=ipc_read (only valid on ZE)? How's Jenkins not crashing for those?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant