update main - #19
Closed
zchuango wants to merge 301 commits into
Closed
Conversation
* Add minimal HCA peer affinity primitives * Add RDMA slice affinity logging * Precompute resolved HCA peer affinity by local NIC * Format HCA peer affinity changes * Remove GPU HCA affinity override Drop MC_GPU_HCA_AFFINITY in favor of MC_CUSTOM_TOPO_JSON and keep HCA peer affinity precomputation limited to cuda topology entries. * Resolve topology merge conflict * Avoid config log merge conflict * Log RDMA slice affinity at verbose level
) UbContext::doConstruct() assigns endpoint_store_ only after the virtual construct() succeeds, returning early otherwise. UrmaContext::construct() returns ERR_CONTEXT on the normal device-init failures (openDevice, urma_create_jfce/jfc/jfr), leaving endpoint_store_ as a null shared_ptr. The destructor then calls endpoint_store_->destroy() unconditionally, dereferencing null when device initialization failed. Guard the call like the sibling members in the same destructor, which are already null-safe (worker_pool_.reset() and the urma_context_ check).
* [TE] add HPE Slingshot (cxi) backend * [TE] fix bugs in cxi and add some tests * [TE] cxi, fix format & typos, remove EFA references * fix typos.toml to include HPE
…2505) These three transports snapshot the output `status` BEFORE polling the hardware (cudaEventQuery / hixl GetTransferStatus). When the poll transitions the task to COMPLETED/FAILED/TIMEOUT, only the task is updated; `status` keeps the pre-poll PENDING value, so the caller sees the completion one poll cycle late. Move the `status = TransferStatus{...}` read to after the poll on every return path. Same class of bug already fixed for io_uring/gds. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Lancer <maruixiang6688@gmail.com>
--------- Co-authored-by: Ke Yang <yangke@approaching.ai>
…2577) Co-authored-by: JieTang <tangjie66@huawei.com>
* [TE] Name AscendDirectTransport worker threads Set each AscendThreadPool worker thread name to "ascend-wk-<i>" via pthread_setname_np on Linux so they are identifiable in top -H / ps -L / perf instead of inheriting the process name. No-op on non-Linux. * [TE] Simplify AscendThreadPool thread naming Drop SetAscendThreadName helper and __linux__ guards; call pthread_setname_np directly like hixl. Thread names ascend-wk-<i> stay within the 15-byte pthread limit (pool size <= 16). --------- Co-authored-by: lbjyx <youxiao@huawei.com>
…details to other places (#2652) --------- Co-authored-by: Ke Yang <yangke@approaching.ai> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
…Init (#2651) BucketStorageBackend::Init() parses each scanned filename's stem with std::stoll at two sites. The whole method body is wrapped in one try/catch that returns INTERNAL_ERROR, so a single stray file with a non-numeric stem (a temp/backup/renamed file still carrying the bucket extension) makes std::stoll throw and aborts the entire store init. Guard each std::stoll and continue past the offending entry, mirroring the existing skip-and-continue handling already used in both loops and the env-parse fix in #2629. Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
* Align TENT RDMA lifecycle and rail affinity * Reformat * Add benchmarks * Reformat * Fix minor issues * Restore testcases
* Add MCCL all-to-all fallback for MACA EP * Stabilize MACA EP P2P path * Stabilize MACA EP P2P path
Co-authored-by: Aionw <aionw@users.noreply.github.com>
Co-authored-by: ruanzhao <ruanzhao@kingsoft.com> Co-authored-by: Claude Opus 4.7 <noreply@owtffssent.com>
…2637) Co-authored-by: ruanzhao <ruanzhao@kingsoft.com> Co-authored-by: Claude Opus 4.7 <noreply@owtffssent.com>
…egation (#2682) * [TE] Support rdma+hip multi-protocol segments for single-node disaggregation On a single node, the transfer engine registers every buffer into all installed transports and publishes them under one shared local segment. When both RDMA and the HIP intra-node transport are installed (the AMD prefill/decode disaggregation case), three things broke: 1. HipTransport::registerLocalMemory rejected host (non-device) memory with "Unsupported memory type" and returned -1. Callers that register a batch across all transports (e.g. the PD metadata/aux buffers that live in host memory) thus rolled back the whole batch and tore down every session. Now non-device memory is skipped gracefully (return 0) and left to RDMA/TCP; any sticky HIP error from the failed query is cleared so it does not poison subsequent HIP calls made by the caller. 2. HipTransport::install overwrote the local segment another transport had already published, collapsing "rdma" to "hip". It now composes with the existing local segment so the node advertises "rdma,hip". 3. The multi-protocol segment encode/decode allow-list was hardcoded to CXL+TCP / CXL+RDMA (max 2 protocols). It is generalized to accept any combination of protocols that have a per-buffer emit/parse branch (cxl/tcp/rdma/hip), and a hip per-buffer branch (addr + shm_name) is added. Buffers are tagged with their owning protocol so a mixed segment round-trips. MultiTransport::selectTransport now routes per request for a comma-protocol segment: each request goes to the transport that owns the buffer covering the target address, preferring hip when an address is registered under multiple protocols (device KV over intra-node XGMI). The single-protocol hip decoder also skips buffers with an empty shm_name defensively. Validated single-node 1P1D (prefill GPU0 / decode GPU1) on MI355X over the real rdma+hip path: without the change KV transfer fails ("Corrupted segment descriptor ... protocol hip", "Failed to get kvcache from prefill instance"); with it the completion returns normally and the logs show no transfer errors. * [TE] Address review: priority-based multi-protocol routing and COW segment install - selectTransport: choose the target transport for a comma-protocol segment by a fixed priority (hip > cxl > rdma > tcp) instead of buffer registration order, and locate CXL buffers via offset + cxl_base_addr rather than the unused addr field. - HipTransport::install: build the local segment on a copy of the existing descriptor and publish it via addLocalSegment instead of mutating the shared descriptor in place, and avoid appending "hip" to the protocol twice. * [TE] Apply clang-format-20 to multi-protocol changes
#2667) Co-authored-by: Aionw <aionw@users.noreply.github.com>
* [Bugfix] Separate post and poll in RDMA worker pool Dedicate thread 0 to polling all CQs and remap the remaining workers to cover all posting shards. * [Bugfix] Fix worker pool fallback and redispatch logic --------- Co-authored-by: cheengguo <cheengguo@tencent.com>
Co-authored-by: Aionw <aionw@users.noreply.github.com>
Co-authored-by: Yuchen Kou <kouyuchen@approaching.ai>
* Apply per Device stream to avoid GPU 0 kmd traffic * Event synchronization via cudaEventSynchronize for caller thread and tranfer stream * Align code style with main branch * Update per-device event pool --------- Co-authored-by: 百麒 <yaozhong.lyz@alibaba-inc.com>
Signed-off-by: Lancer <maruixiang6688@gmail.com> Signed-off-by: Lancer <maruixiang@sunrise-ai.com>
…re (#1789) Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.59.0 to 1.79.3. - [Release notes](https://github.com/grpc/grpc-go/releases) - [Commits](grpc/grpc-go@v1.59.0...v1.79.3) --- updated-dependencies: - dependency-name: google.golang.org/grpc dependency-version: 1.79.3 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
… getaddrinfo (#2919) * fix: bind wildcard listen address (0.0.0.0/::) without going through getaddrinfo * test(wildcard-listen): cover IPv6 :: wildcard and guard free-port Address review feedback on #2919: - Bind the HTTP server test to the IPv6 wildcard :: (0.0.0.0 is already covered by the RPC test) so the numeric-literal parse path is exercised for both address families. - Assert getFreeTcpPort() > 0 before casting to unsigned short, so a -1 failure does not silently become port 65535. - Return EAI_FAIL instead of EAI_SYSTEM from the interposer fallback; EAI_SYSTEM asks callers to inspect errno, which is not set here. * test: drop wildcard_listen_test, ylt test_acceptor already covers it
Co-authored-by: jipengtian <jipengtian@xiaohongshu.com>
Co-authored-by: Yuchen Kou <kouyuchen@approaching.ai>
* Store: support tensor APIs with dummy client * Store: free dummy local SHM on teardown * Store: use real-owned buffers for dummy tensors * Store: format dummy tensor buffer allocation * Store: serialize dummy SHM writes * Store: remove dummy write serialization * docs: update dummy tensor staging lifetime note
…tric underflow (#3080) During snapshot restore (standby/master), LoadLatestSnapshot builds a throwaway SegmentManager and deserializes allocators via Serializer<OffsetBufferAllocator>::deserialize(). That path assigns cur_size_ straight from the persisted snapshot without going through the live allocate()/adoptImportedBuffer() paths, so no inc_allocated_mem_size() is paired with it. The (often throwaway) allocator is destroyed right after, and its destructor still calls dec_allocated_mem_size(cur_size_) to undo its contribution to the global gauge; with no matching inc the signed gauge goes negative (e.g. -12288 in the repro), and get_summary_string() then passes that negative int64 to byte_size_to_string(uint64_t), wrapping it to ~16777216 TB. Fix by pairing inc_allocated_mem_size(segment, cur_size_) where cur_size_ is restored, so inc/dec stay symmetric and the gauge ends at 0 after the temporary allocator is destroyed. The live restore path (RestoreOffsetBufferAllocator) goes through allocate() which already inc()s, so this does not double-count. Replaces the previous display-only clamp in get_summary_string(), which only masked the symptom.
…#3160) EraseMetadata cleaned up the tenant-side offloading_task but not the mirror entry the master had also pushed into the client's LocalDiskSegment::offloading_objects. On BatchRemove or BatchEvict the next OffloadObjectHeartbeat drained that task-less key back to the client, which happily wrote a bucket file and called NotifyOffloadSuccess — resurrecting the key as a LOCAL_DISK-only metadata entry, or in the pathological case leaving an orphan bucket on disk with no master metadata to reference it. Symptoms in production (mooncake-store SSD offload with BatchEvict-heavy workload): - SSD metric drops but bucket files stay on disk - LOCAL_DISK-only replicas appear for keys the caller already removed - Later Puts for the same key fail with OBJECT_ALREADY_EXISTS - -707 LEASE_EXPIRED / -200 NO_AVAILABLE_HANDLE storms as pinned memory refuses to free Fix: erase the storage key from every client's offloading_objects map when EraseMetadata cleans up offloading_tasks. Iteration is over mounted local-disk segments (typically 1..64 clients) and each erase is a hash-map no-op on segments that don't hold the key, so the cost stays bounded. Regression: BatchRemoveDropsOffloadingObjectsMirror in offload_on_evict_test. Without the fix the test observes 3 stale entries drained after BatchRemove; with the fix, zero. Co-authored-by: huniu20 <huniu20@users.noreply.github.com>
…ors (#3198) * [TE] Add tests for RDMA MR straddling and submit failure Add test coverage for RDMA transfer slice handling and error paths: - Cross-MR boundary transfer straddling on source and target buffers - Prevention of slice double-free on submit error - Batch and task state completion on partial submit failure Signed-off-by: Ruiqing Feng <ruiqing.feng@broadcom.com> * [Bugfix][TE] Split RDMA slices at MR boundaries and handle submit errors - Split transfer slices at MR boundaries to avoid multi-MR WRs - Mark unposted slices and unstarted tasks as FAILED on submit errors Signed-off-by: Ruiqing Feng <ruiqing.feng@broadcom.com> --------- Signed-off-by: Ruiqing Feng <ruiqing.feng@broadcom.com>
Without relaxed ordering, bursty RDMA traffic can saturate switch buffers and trigger PFC pause frames, which back-propagate congestion across the fabric and increase network operational overhead. Enabling RO by default avoids this and improves out-of-the-box usability: users unaware of this option would otherwise miss the benefit entirely. The option remains fully overridable via MC_IB_PCI_RELAXED_ORDERING=0. Update env-var descriptions and documentation accordingly.
…devices (#3209) A peer segment descriptor carries devices[], buffers[].rkey[] and priority_matrix as three independent fields, and decodeSegmentDesc() never checks them against each other. The rdma, efa and cxi transports then pick a device with Topology::selectDevice(), whose HCA list is built from the distinct HCA names in that priority_matrix, and use the resulting device_id to index both buffers[buffer_id].rkey and devices[]. Its range is therefore decoupled from devices.size(). A descriptor with a short devices array, a longer rkey vector and many distinct HCAs drives device_id past the end of devices[]. The rkey bound (in WorkerPool::selectPeerDevice for rdma, inside selectDevice itself for efa and cxi) still passes, and the subsequent devices[device_id].name accesses read a DeviceDesc out of bounds and copy a wild std::string. In the RDMA transport those are worker_pool.cpp:216 and :256, running in a transfer worker thread with no enclosing try/catch, so the resulting SIGSEGV is an uncaught process crash. A publisher emits exactly one key per device (one per RdmaContext in context_list_), so require rkey.size() == devices.size() for a non-empty rkey vector in decodeSegmentDesc(), on both the single-protocol and multi-protocol paths. This extends the existing rkey/lkey consistency check, rejects the malformed peer once instead of per-slice, and covers all three transports. barex is excluded because it hands the whole rkey vector to the peer rather than indexing it by device_id. Buffers with an empty rkey (registered local-only) stay valid. Keep a device_id < devices.size() bound as defense in depth on the value actually used to index the array, in all three submit paths: WorkerPool::selectPeerDevice() for rdma, and the per-slice resolve loops in efa_context.cpp and cxi_context.cpp. Add two tests over the real decode path: a peer publishing 64 keys for 1 device is rejected, and a well-formed 4-key/4-device descriptor still resolves. Signed-off-by: echarris <echarris@smcm.edu>
* [PG] Fix initialization with world size 1 * [PG] Quick fix for ylt. * fix(pg): clean up shared transfer engine before exit --------- Co-authored-by: Xun Sun <UNIDY2002@outlook.com>
github-actions
Bot
added
documentation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Module
mooncake-transfer-engine)mooncake-store)mooncake-ep)mooncake-pg)mooncake-integration)mooncake-p2p-store)mooncake-wheel)mooncake-common)mooncake-rl)Type of Change
How Has This Been Tested?
Test commands:
# Example: bash scripts/run_ci_test.shTest results:
Checklist
./scripts/code_format.shpre-commit run --all-filesand all hooks passAI Assistance Disclosure