[FEA] Multi-node Out of Core Streaming KMeans API#2066
Conversation
…nto combine-batch
…nto combine-batch
…nto combine-batch
|
/ok to test defb785 |
| - nccl ${{ nccl_version }} | ||
| # ucx/ucxx are linked by the multi-GPU (MG) tests | ||
| - libucxx ${{ ucxx_version }} | ||
| - ucx |
There was a problem hiding this comment.
These are needed at build time, right? We also need to list these under the cached host env, under line 74.
There was a problem hiding this comment.
We also need to list these under the cached host env, under line 74.
@divyegala this is the place from where I removed it, right? Should I add it again?
|
/ok to test 8dd17f3 |
bdice
left a comment
bdice
left a comment
There was a problem hiding this comment.
Ah, this is an entirely new dependency on ucxx. We have several other tasks to handle.
https://github.com/NVIDIA/cuvs/blob/main/ci/release/update-version.sh needs changes like raft has.
Also reference how RAFT uses UCXX_VERSION and UCXX_BRANCH throughout the repo, we'll need to replicate all that infrastructure.
|
/ok to tesst ccd5e43 |
|
/ok to test ccd5e43 |
…into mnmg-streaming
|
/ok to test 21864d2 |
|
/ok to test 4da8077 |
|
/ok to test a0edd32 |
|
/ok to test c8db7ff |
| FORK ${RAFT_FORK} | ||
| PINNED_TAG ${RAFT_PINNED_TAG} | ||
| ENABLE_MNMG_DEPENDENCIES OFF | ||
| ENABLE_MNMG_DEPENDENCIES ${BUILD_MG_ALGOS} |
There was a problem hiding this comment.
| ENABLE_MNMG_DEPENDENCIES ${BUILD_MG_ALGOS} | |
| ENABLE_MNMG_DEPENDENCIES ${BUILD_MG_ALGOS} |
|
/ok to test fe98618 |
| if(Python_FOUND) | ||
| include(${rapids-cmake-dir}/cython-core/find_prefix_paths.cmake) | ||
| rapids_cython_find_prefix_paths("${Python_EXECUTABLE}" _cuvs_wheel_prefix_paths) | ||
| set_property(GLOBAL PROPERTY FIND_LIBRARY_USE_LIB64_PATHS TRUE) |
There was a problem hiding this comment.
I don’t fully understand what this code path is aiming to do, but conda doesn’t use lib64 paths iirc. Normally I would assume this is used in wheels, but we don’t build C++ tests for wheels. What’s going on here?
There was a problem hiding this comment.
I had done that to ensure that if those dependencies are coming from pip, they are found. Because find_package was failing if I get pylibraft from pip. I agree that we no longer do this in our conda builds, so it is not needed
| NAME CLUSTER_KMEANS_MG_TEST | ||
| PATH cluster/kmeans_mg.cu |
There was a problem hiding this comment.
Please mark all multi-GPU tests as SERIAL. You probably don’t want to run two 2-GPU tests concurrently on a 4-GPU system. It just gets messy. We recently discussed this topic for cudf and that was the right choice. rapidsai/cudf#23188
…into mnmg-streaming
|
/ok to test 07b5a39 |
8 participants
Merge after #2015 and #2017
Allows a stream of input matrices per worker, that are further batched using the
streaming_batch_sizeparameter. Reasoning: We should be able to supply dask partitions (on host) directly without having to concatenate them into one consolidated matrix.As a part of this PR, we also unify the multi-GPU implementations into one (earlier the out of core implementation was separate).
Tests: We get rid of the separate out of core test file. The single MG testing unit is taking care of both out of core and on device matrices.