merge main - #22
Closed
zchuango wants to merge 366 commits into
Closed
Conversation
…#2519 step 2) (#2763) * [TENT] Opt-in earliest-deadline-first dispatch in admission queue (RFC #2519 step 2) Step 1 (#2618) added Request.deadline_ns + post-hoc MLU observability. This adds the first policy step alogfans green-lit: an opt-in EDF ordering in LocalTransferAdmissionQueue::pickForDispatch. - QueueLimits gains `deadline_aware` (default false). - When false: pickForDispatch keeps strict FIFO — behavior unchanged. - When true: the dispatch queue is stably reordered earliest-deadline-first before selection; owners without a deadline (deadline_ns == 0) keep FIFO order behind all deadlined owners. - Selection still respects the existing owner/byte capacity limits; this only reorders *which* queued owner is picked next, it does not admit or reject. No codec / local-decode / bandwidth-prediction here — this is the ordering layer only, strictly additive and gated behind the opt-in flag. Motivation from measurement (H20 / CX-7 RoCEv2, TENT backend): sweeping the deadline shows a transition band (~200us on this HW) where feasible and missed transfers coexist (mean MLU 1.31, ~13% feasible) — exactly where EDF ordering has leverage. Below/above that band ordering buys nothing. Misses there are driven by concurrency queueing, which is what this reordering targets. Adds 3 unit tests (EDF order, undeadlined-last, and FIFO-unchanged default); full admission_queue_test suite passes (16/16). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * [TENT] EDF: order fifo_ at admission instead of re-sorting every dispatch Addresses review on #2763: pickForDispatch used to std::stable_sort the whole fifo_ (up to max_outstanding_owners, default 1024) on every call, and it is called on every complete/poll/submit — re-sorting an already-ordered queue repeatedly, even when nothing new was admitted, and even when a byte limit lets it consume only a few entries. Instead keep fifo_ EDF-ordered as owners arrive: when deadline_aware, tryAdmit inserts each owner at its earliest-deadline-first position (upper_bound, so same-deadline owners keep FIFO tie-break — identical ordering to the old stable sort). pickForDispatch then just consumes from the front, dropping the sort entirely. Hot dispatch path goes from O(N log N) to O(picked); the cost moves to one O(N) ordered insert per admit. Default (deadline_aware == false) is unchanged plain FIFO push_back. Adds a test admitting out-of-order deadlines across separate tryAdmit calls to cover the ordered-insert path; full admission_queue_test passes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: 彦纾 <wangyanshu.wys@alibaba-inc.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…#2780) * [Bench] Enable replay speedup and multi-threading * Correct multithreading * Update benchmarks/storage_benchmark_v1/benchmark.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Change multi-thread benchmarking * Add notes --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
…#2764) * [TENT] Deadline-infeasible drop + degradation hook (RFC #2519 step 3) Builds on step 2 (#2763 EDF ordering) and step 1 (#2618 deadline_ns + MLU observability). Adds the degradation layer: owners predicted to miss their deadline are dropped from dispatch instead of sent, and a local-decode signal is raised so the caller can recompute locally. - QueueLimits.mlu_local_threshold (θ_local, default 0 = disabled). - setDegradationPolicy(bandwidth_provider, hooks, now_provider): dependency- injected so the admission queue stays decoupled from the device-selection layer and remains unit-testable. now_provider defaults to steady_clock. - pickForDispatch gains an optional `dropped_owner_ids` out-param. When drop is enabled (θ_local > 0, deadline_aware, bandwidth provider set), an owner whose predicted MLU (= length/bw / (deadline - now)) reaches θ_local — or whose deadline is already past — is charged out of the outstanding accounting, marked terminal (CANCELED), reported in dropped_owner_ids, and triggers on_local_decode_suggested. Everything else dispatches as before. Interface only: no codec / local-decode body (those live in vLLM/SGLang; TENT only raises the signal), as scoped in the RFC. Strictly additive and fully opt-in — θ_local = 0 (default) means zero behavior change from step 2. 5 new unit tests (drop infeasible / keep feasible / expired-deadline drop / disabled-when-threshold-zero / no-drop-without-bandwidth-provider, incl. hook invocation and outstanding-accounting checks). Full admission_queue_test suite passes 21/21. Motivation (H20 / CX-7 RoCEv2, TENT backend): the MLU sweep in #2519 shows the 100us deadline tier sits at mean MLU 2.63 — well past a θ_local of ~1.5 — so under contention these transfers are provably infeasible and are exactly what step 3 would drop to local-decode rather than waste bandwidth on. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * ci: re-trigger (tent-ci apt/sccache network flake, not a code failure) * ci: re-trigger build-flags (runner killed mid-build, resource flake #2611) --------- Co-authored-by: 彦纾 <wangyanshu.wys@alibaba-inc.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…2214) * feat(store): add optional RFC #1527 KV events publisher on master Implement an opt-in ZMQ publisher on mooncake_master that emits standardized KV cache events for Dynamo global KV indexer integration. - Add KvEventPublisher with async bounded queue and background worker - Publish stored/removed events on PutEnd, Remove, and eviction paths - Wire RFC #1527 envelope fields plus optional vLLM/SGLang compat aliases - Expose gflags/config toggles (enable_kv_events, bind endpoint, backend_id) - Add GET /kv_events/status on the master metrics HTTP server - Document Mooncake master publisher usage in indexer API design Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com> * docs(conductor): add KV event field provenance matrix (SGLang vs master vs register) Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com> * fix(store): address KV events PR review feedback - Fix msgpack map sizes with ComputeEventMapSize helper - Close ZMQ message parts on partial send failure - Emit per-medium removed events when memory replicas are evicted - Remove spurious removed event on PutRevoke before PutEnd - Use condition_variable::wait, drain full queue on shutdown, htobe64 - Drop bounded queue/drop-on-full to avoid indexer consistency gaps - Make libzmq optional via ENABLE_KV_EVENTS (stub when disabled) - Add missing condition_variable include and key_util for hash parsing Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com> * style(store): apply clang-format-20 to KV events changes Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com> * fix(store): fix KV events CI link failure when libzmq is absent Propagate ENABLE_KV_EVENTS=OFF to the parent CMake scope when libzmq is not found so kv_event_publisher_test uses the header stub consistently with mooncake_store. Add libzmq3-dev to dependencies.sh for full builds. Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com> * fix(store): link libzmq in Rust bindings when KV events enabled mooncake_store pulls in ZMQ symbols when libzmq is present; propagate the optional -lzmq link in build.rs using the same has_library pattern as uring/etcd. Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com> * fix(store): link libzmq in Go CGO flags for KV events Go integration tests link libmooncake_store.a statically and need -lzmq when the KV events publisher is compiled in. Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com> * ci: re-trigger build after runner disk-space failure Previous build (3.12) failed during Codecov upload with 'No space left on device' after all tests passed. No code changes. Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com> * fix(store): disable KV events compile flag when libzmq is absent When libzmq was not found, ENABLE_KV_EVENTS was only cleared in the parent CMake scope while the local value stayed ON. That defined MOONCAKE_ENABLE_KV_EVENTS=1 without linking kv_event_publisher.cpp, breaking mooncake_master on platforms like the Ascend CI image. Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com> * fix(store): address review comments — msgpack map_size, ZMQ leak, PutRevoke event, cmake guard - PutRevoke: emit PublishKvRemoved before erasing an invalidated object so that downstream indexers are notified when a revoked key is removed - cmake: upgrade ZMQ-not-found diagnostic from WARNING to FATAL_ERROR when ENABLE_KV_EVENTS is explicitly ON, since the user opted in - Verified: msgpack map_size values (18/15/15/13) are already correct after ComputeEventMapSize refactor in ac17fbc - Verified: ZMQ zmq_msg_close is already called on every send-failure path after the fix in ac17fbc Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(cmake): default ENABLE_KV_EVENTS to OFF KV events require libzmq which is not available on all CI runners (e.g. Ascend). Default to OFF so builds succeed without it; users who want the feature opt in with -DENABLE_KV_EVENTS=ON. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(store): align KV events publisher with RFC #1527 spec - Set stored base_block_idx=0 so at least one placement field is present - Emit per-object tenant_id from master metadata, not only global config - Stop spurious removed events on PutRevoke (abort before PutEnd) - Publish removed on disk/NOF eviction paths with medium=disk - Pass explicit medium to eviction removed helper Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com> * fix(store): address PR #2214 review and CI issues - Fix clang-format on kv_event_publisher.h (CI Check code format) - Add optional object_key field (kv_events_emit_object_key, default on) for Dynamo matching on Mooncake store keys without decimal/0x seq_hash encoding - Publish events with empty seq_hashes when object_key is emitted but hash cannot be parsed; update indexer API docs and field matrix - Wire emit_object_key through master config, gflags, and master.json Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com> * style: clang-format master_admin_service.cpp Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com> * fix(store): emit per-block KV events without global semantic fields Align Mooncake master publisher with Dynamo KV events model: each event describes a pooled block (seq_hash/object_key, medium, tenant_id), not process-wide model/block_size/lora/dp_rank invariants. Omit unknown envelope fields (nil) and supply stream dimensions via indexer POST /register. Deprecate master flags that previously stamped every event. Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com> * feat(store): include group ID in KV events Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com> --------- Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Ishan Dhanani <ishandhanani@gmail.com>
…ng (#2758/#2467) (#2790) updateBestMapping() maps a local NIC to a remote NIC per (local_numa, remote_numa) pair. For the same-NUMA case it uses direct_rails_, which loadDefault() builds via same-name matching (mlx5_5 -> mlx5_5). But for the cross-NUMA case it used positional assignment, remote_devices[remote_numa][i % remote_cnt], ignoring device names. On a multi-bond dual-NUMA RoCEv2 fabric where the two nodes disagree on which NUMA a same-named NIC sits in (e.g. an overlay NIC), this maps a local NIC to an unrelated remote NIC on a different physical/overlay network. The QP then never reaches RTR -> 'transport retry counter exceeded' (#2467) / cross-node modify-to-RTR EINVAL(22) (#2758). Fix: in the cross-NUMA branch, prefer a same-name remote device (mirroring loadDefault()'s Priority-1 matching) before falling back to positional assignment. Same-NUMA behavior is unchanged; positional fallback still applies when no same-name remote device exists in the target NUMA domain. Add a cross-NUMA same-name unit test (asymmetric NUMA layout as in #2467). Verified: the changed translation unit compiles cleanly in-tree; the full tent_rail_monitor_test binary could not be linked in my environment due to an unrelated GDS/cuFile build dependency (cufile.h absent), so the gtest was not run locally. Co-authored-by: 彦纾 <wangyanshu.wys@alibaba-inc.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
) Bumps [golang.org/x/net](https://github.com/golang/net) from 0.48.0 to 0.55.0. - [Commits](golang/net@v0.48.0...v0.55.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-version: 0.55.0 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
--------- Co-authored-by: chatgpt-codex-connector[bot] <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.51.0 to 0.52.0. - [Commits](golang/crypto@v0.51.0...v0.52.0) --- updated-dependencies: - dependency-name: golang.org/x/crypto dependency-version: 0.52.0 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…gine data plane port (#2784) Previously, passing --host=ip:port would cause "IPC server stopped" because the port-containing string was forwarded to coro_rpc_server without stripping the port. Now getHostNameWithoutPort() is used to extract the bare IP for coro_rpc_server binding, while the port flows through to TransferEngine as intended. Signed-off-by: tan changzhi <544463199@qq.com>
* [TENT] Opt-in per-entry priority promotion (#2528) Workers::promoteTimedOutRequests drains a whole priority queue but decides promotion from the HEAD entry only, then promotes every entry when the head has timed out. This over-promotes freshly enqueued, non-starving entries and, conversely, ignores timed-out entries behind a fresh head. It also promotes at most one level per tick. Factor the promotion decision into promotion_policy.h (DecidePromotionHeadOnly = today's behavior, DecidePromotionPerEntry = promote exactly the timed-out entries) so it is unit-testable without the RDMA stack, and wire an opt-in config flag: * transports/rdma/priority_promotion_per_entry (default false) keeps the historical head-only 'flush the tier' policy byte-for-byte; * when true, each pass promotes only the entries that have themselves timed out, and both MEDIUM->HIGH and LOW->MEDIUM are considered each tick so a starving LOW entry is not stalled behind an unrelated MEDIUM promotion. promotion_policy_test adds 5 deterministic cases reproducing both defects and verifying the two policies agree on the all-timed-out / empty / no-timestamp cases the head-only design targets. See issue #2528. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * [TENT] per-entry promotion: add config docs + guard unsigned underflow Addresses review feedback on #2528/#2788: * docs/design/tent/qos.md: document transports/rdma/priority_promotion_per_entry (default false = head-only, true = per-entry) and the existing priority_promotion_timeout_us (per @alogfans). * promotion_policy.h: guard current_ts >= ts before the unsigned subtraction so a non-monotonic clock / race (current_ts < enqueue_ts) cannot underflow and spuriously mark an entry timed out. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: guard promotion decision indices * perf: keep default promotion path allocation-light --------- Co-authored-by: 彦纾 <wangyanshu.wys@alibaba-inc.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Yanshu <237344440@qq.com>
…2808) * [TENT] Expose Request.deadline_ns and policy_name to Python bindings The C++ Request struct already carries deadline_ns (RFC #2519, #2618) and policy_name (#2640/#2759), but the Python bindings only exposed priority and transport_hint. This left deadline / policy as internal-only fields, unreachable from the Python API that upper layers (Store, SGLang, vLLM) use. Expose both as optional constructor args (defaulting to existing behavior: deadline_ns=0, policy_name=None) and as read/write properties. Fully backward compatible: callers that omit the new args are unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: include optional for pybind request fields --------- Co-authored-by: 彦纾 <wangyanshu.wys@alibaba-inc.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Yanshu <237344440@qq.com>
…an one staging chunk (#2815) Caught while testing the TPU staging path on a real TPU VM (v5p-8, libtpu 0.0.32, PJRT C API 0.83) for the first time. Until now the feature had only ever run against the mock adapter. ProxyManager stages a transfer in chunk_size (4 MiB) pieces and passes `token + chunk_offset` to the platform for every chunk after the first (proxy_manager.cpp:76). The adapter ABI only ever specified base-address classification, so `isDevicePtr(token + off)` returned false, TENT classified TPU HBM as host memory, and TpuPlatform::copy fell through to CpuPlatform::copy -- a plain memcpy of the device token. On real PJRT that memcpy does not crash: PJRT_Buffer_UnsafePointer returns an address that is host-readable but does not hold the buffer's data. The staging copy therefore produced garbage and reported COMPLETED. Chunk 0 was correct and every subsequent chunk was silently wrong, so any transfer over 4 MiB was corrupted without an error anywhere. The mock could not catch this because its "device" pointers were ordinary host memory, so the accidental memcpy happened to produce the right bytes. Fixes: - tpu_pjrt_abi.h / tpu_pjrt_shim.h: state that classification and copy entrypoints must resolve interior addresses to the registered buffer whose range contains them, that a copy may not run past a buffer's end, and that the token must never be dereferenced. - tpu_transport.cpp: require exactly one TPU-device side per staging hop and fail loudly otherwise, so a non-conforming or absent adapter can no longer degrade into a silent memcpy. - mock_tpu_pjrt_adapter.cpp: model the two properties that matter -- tokens are opaque (a poisoned read-only mapping, data lives in shadow storage) and addresses may be interior. A copy that bypasses the adapter now yields 0xDD instead of accidentally passing. - common.cmake: -DUSE_TPU=ON silently compiled zero TPU code, because all TPU sources live under tent/ which is gated on USE_TENT. This is why the bug was invisible to every build. Now a hard error. Tests: two new regression tests in tent_tpu_pjrt_shim_test and a new tent_tpu_transport_test (6 cases) that drives the staging hop the way ProxyManager does. Verified they fail against the pre-fix code (LocalStageCopiesFromInteriorDeviceOffset reports COMPLETED while delivering 0xDD) and pass after. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* clear legacy benchmark results * Change vllm performance benchmark docs menu * rename vllm performance file name * Change Mooncake performance docs paths * Change SGLang performance docs paths * update the sglang and vllm performance description --------- Co-authored-by: Ke Yang <yangke@approaching.ai>
…#2803) Signed-off-by: Csrayz <33659823+Csrayz@users.noreply.github.com>
…anager (#2805) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* [TENT] Add IntentType enum to Request for Transfer Intent API Define standard intent categories (FOREGROUND_GET, BACKGROUND_PREFETCH, MIGRATION, CHECKPOINT, WEIGHT_LOADING, STAGING_INTERNAL) so TENT can identify a request's business semantics before scheduling. Changes: - types.h: add IntentType enum class + Request::intent_type field (default INTENT_UNSPEC, behavior byte-identical to today) - pybind.cpp: export IntentType enum, add intent_type/policy_name/ deadline_ns to Request constructor and as readwrite attributes - intent_type_test.cpp: 6 gtest cases covering defaults, assignment, integer values, field independence, copy, and batch usage Relates to: TENT roadmap "Transfer Intent API" * ci: retrigger CI * fix: keep intent type binding scoped * test: wire intent type coverage into cmake --------- Co-authored-by: 彦纾 <wangyanshu.wys@alibaba-inc.com> Co-authored-by: Yanshu <237344440@qq.com>
) * [TENT] admission queue: add deadline proximity promotion When a queued owner's remaining slack (deadline_ns - now) falls below the configurable promotion_slack_ns threshold, it is promoted to the front of the dispatch queue via stable_partition, ahead of owners with comfortable slack or no deadline. This complements the existing step-2 EDF ordering by dynamically boosting urgency as deadlines approach, ensuring near-deadline transfers get dispatched preferentially even if they were admitted after requests with later deadlines. Key design choices: - Opt-in: promotion_slack_ns defaults to 0 (disabled). - Requires deadline_aware = true (like all deadline features). - stable_partition preserves EDF order within promoted/non-promoted groups. - Composes with step-3 drop: promotion happens first, then infeasible owners are still dropped from the (now-reordered) front. - NowProvider reused from step-3 (falls back to steady_clock). Ref: RFC #2519 step 4 * fix: wire deadline promotion runtime config * fix: avoid deque stable partition in dispatch * perf: reuse deadline promotion scratch buffers * bench: add deadline promotion hot-path benchmark * perf: retain faster local partition buffers * perf: retain benchmarked stable partition --------- Co-authored-by: 彦纾 <wangyanshu.wys@alibaba-inc.com> Co-authored-by: Yanshu <237344440@qq.com>
* [transfer-engine] Add show-link diagnostic tool for NIC topology inspection Add a connectivity diagnostic utility that displays local RDMA NIC information and topology selection matrix. This helps operators understand which NICs are discovered, their NUMA affinity, link speed, and how the topology-aware device selection maps storage types to NICs. Components: - show_links.h/cpp: Core logic (collectLocalNics, buildShowLinksReadable/Json) - show_link.cpp: CLI binary with --json/--discover_only flags - C API: showLinks() exposed via transfer_engine_c.h - rdma_context: Add activeWidth() accessor for bandwidth calculation - show_links_test.cpp: Basic gtest coverage Tested on 4-node H20 cluster (5 NICs per node, mlx5_0 + mlx5_bond_0..3): - NIC discovery: 5/5 devices found on both nodes - NUMA topology: Correctly distinguishes preferred vs available NICs - Cross-node probe: Verified at 0.5ms latency (3 consecutive runs) - Speed calculation: 200Gbps (HDR 50Gbps x 4 lanes) matches ibstat * style: clang-format show_links.cpp and transfer_engine.cpp * fix: respect show-link json output * fix: handle show-links before engine init * fix: make show-link discover-only discover topology * fix: show topology nics without rdma transport * fix: harden show-link C API and speed reporting * style: format show-links JSON assertion * fix: include integer types in show-links header --------- Co-authored-by: 彦纾 <wangyanshu.wys@alibaba-inc.com> Co-authored-by: Yanshu <237344440@qq.com>
* [TENT] admission queue: add deadline proximity promotion When a queued owner's remaining slack (deadline_ns - now) falls below the configurable promotion_slack_ns threshold, it is promoted to the front of the dispatch queue via stable_partition, ahead of owners with comfortable slack or no deadline. This complements the existing step-2 EDF ordering by dynamically boosting urgency as deadlines approach, ensuring near-deadline transfers get dispatched preferentially even if they were admitted after requests with later deadlines. Key design choices: - Opt-in: promotion_slack_ns defaults to 0 (disabled). - Requires deadline_aware = true (like all deadline features). - stable_partition preserves EDF order within promoted/non-promoted groups. - Composes with step-3 drop: promotion happens first, then infeasible owners are still dropped from the (now-reordered) front. - NowProvider reused from step-3 (falls back to steady_clock). Ref: RFC #2519 step 4 * [transfer-engine] Add graceful shutdown for SIGTERM/SIGINT/SIGABRT When a Transfer Engine process is killed by SIGTERM/SIGINT/SIGABRT, RDMA resources (QPs, MRs) are left dangling. The peer side's QP still references the now-invalid MR, causing RDMA READ to silently return undefined data instead of an error. This adds an opt-in enableGracefulShutdown() API that: - Registers atexit() to call freeEngine() on all active engines - Installs signal handlers that call exit(128+signo) to trigger atexit - Ensures the existing deconstruct() path (QP destroy + MR dereg) runs The API is opt-in to avoid interfering with applications that manage their own signal handlers (e.g., gRPC, Python interpreters). * fix: make graceful shutdown signal path safe * fix: cover tent shutdown and preserve abort semantics * fix: preserve graceful shutdown across moves * fix: avoid post-fork shutdown handler hang --------- Co-authored-by: 彦纾 <wangyanshu.wys@alibaba-inc.com> Co-authored-by: Yanshu <237344440@qq.com>
…ers (#2842) * [TE] Make ThreadLocalStorage per-instance and reclaim per-thread holders Fixes #2717. The previous implementation kept its thread-local slot as a 'thread_local static' member of the class template — one slot per template instantiation, shared by every ThreadLocalStorage<T> instance in the process — so two instances (e.g. two engines' SegmentManager remote-desc caches) aliased each other's per-thread state. The slot was also a raw pointer to a heap holder that nothing ever deleted, so the deregistration logic in the holder destructor was unreachable and every thread leaked one holder per instantiation. Rework the storage around a per-thread map keyed by a process-monotonic instance id (never reused, so a recycled allocation cannot alias a stale entry), with a one-entry cache in front so the common get() stays one thread_local access plus a compare. Per-thread values are owned by the thread and destroyed at thread exit. A control block jointly owned by the storage and every thread node makes both teardown orders safe: a thread exiting first deregisters from the registry; an owner destroyed first marks the block dead and exiting threads skip the registry. forEach() runs under the registry mutex and visits exactly the live registered values. Validation (96-core box, GCC 13): - New thread_local_storage_test: 8/8. Against the previous implementation the same binary fails 5/7 (aliasing, deregistration, resurrection, forEach, churn) and LeakSanitizer reports 288 bytes in 18 allocations from get(). - ASAN/UBSAN and TSAN clean for 50 repeated runs each (both teardown orders exercised). - Full TENT suite 23/23 (SegmentManager's tl_remote_cache_ exercises the storage end-to-end). - get() hot path at -O3: 0.33-0.41 ns before, 0.66-0.69 ns after (+~0.3 ns; the caller's next step is a clock_gettime and map lookup). * review: null-check control in sweepDeadNodes Unreachable today (the sweep runs before try_emplace on the same thread and control is assigned immediately after emplace by a nothrow shared_ptr copy), but consistent with ~ThreadNode's defensive check and robust to future reordering.
Co-authored-by: KMSorSMS <yzwliam@126.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
…lm paths, add NCCL env & offline model cache (#2811) * fix(tone_tests): inject NCCL_GIN_TYPE=0 env var into test container * fix(ci): select cu130 wheel artifact for integration test to match CUDA 13 image * fix(tone_tests): correct sglang test path and create log dir in run-single * fix(tone_tests): correctly clean stale wheels in get_whl to avoid installing wrong CUDA wheel * fix(tone_tests): fix vllm test OOM (gpu-mem-util 0.85) and correct mooncake proxy path * fix(tone_tests): drain GPU memory between cases in run-all to avoid cross-test OOM * fix(tone_tests): run vllm test on GPU 6,7 to align with sglang tests * feat(tone_tests): use local HF cache offline when model already downloaded * fix(tone_tests): force-kill host GPU-holding PIDs in drain to fully clear residual memory * fix(tone_tests): restart container between run-all cases to reset GPU/ERDMA state (keeps deps)
… destination (#2850) * [TransferEngine] Acknowledged TCP framing: COMPLETED means applied at destination Fixes the data-integrity half of #2086. The TCP data plane reported a WRITE as COMPLETED when its final chunk entered the initiator's kernel socket buffer: destination memory could still be mutating megabytes later (measured 41% of 2.4 MB writes torn after COMPLETED on loopback), and a server-side rejection was invisible (a single-chunk WRITE to an unregistered address reported success). Errored connections were also returned to the pool on is_open() alone, letting a protocol-desynced socket corrupt subsequent requests, and session_mutex_ was locked and unlocked on different threads (UB) while synchronizing nothing. Protocol v2 (negotiated and wire-compatible): - Servers advertise tcp_proto_version=2 in the segment descriptor; old readers ignore the field and descriptors without it default to v1. Unflagged requests remain byte-identical for old initiators. - WRITE: the server sends an 8-byte status frame only after the final chunk has been applied to destination memory. The client reads that frame concurrently with the body, so a rejection or a legacy peer's bogus payload can abort a large write instead of deadlocking on two full socket directions. - An early negative or malformed acknowledgment closes the socket immediately, but FAILED is not published until the outstanding async_write handler has released the caller-owned source buffer. - READ: the server prefixes the payload with a status frame (back-to-back, no extra RTT), so rejections are signaled rather than inferred from a dropped connection. - Status frames carry a 32-bit magic so a stale v2 descriptor that reaches a legacy server fails quickly instead of silently misinterpreting the byte stream; MC_TCP_PROTO=1 remains an operational rollback hatch. - Connections are re-pooled only after a cleanly terminated exchange; anything else is closed and dropped. session_mutex_ is removed because shared_from_this already owns each sequential handler chain. Validation (96-core dev4new, GCC 13, CUDA 12.9 where enabled): - tcp_write_visibility_test 6/6 x 5; the two large early-abort/source- quiescence cells passed 20/20 each. The visibility reproducer went from 166/400 torn reads on v1 to 0/400 on v2 with three noise writers. - ASAN/UBSAN/LSAN: 6/6 clean. CPU TSan: the two new quiescence cells are clean after suppressing three pre-existing engine-wide races in the custom RWSpinlock, polling counters, and slice-cache reuse. - transfer_engine_bench TCP loopback A/B (equal build flags, 2 runs, write, 4 threads x batch 32): 16 KB 0.25/0.29 vs 0.24/0.30 GB/s; 64 KB 0.88/0.84 vs 0.88/0.97; 256 KB 1.28/1.15 vs 1.34/1.30; 1 MB 1.54/1.53 vs 1.62/1.46 - flat within noise. * [TransferEngine] Run TCP visibility tests with HTTP metadata * [TransferEngine] Bound the v2 status-frame wait so stale descriptors fail instead of hanging An adversarial re-review found a hole in the stale-descriptor story: the fail-fast path relies on the legacy peer's bytes not parsing as a status frame, but a request SHORTER than a frame (1-7 bytes) never yields 8 bytes at all. The v1 server streams size bytes of 'READ payload' and then keeps the connection open awaiting the next header (for WRITE it is symmetrically stuck parsing our body as a partial header), TCP slices have no timeout worker, so both sides waited forever. Give the two status-frame reads a deadline. It covers only the frame, never payload streaming: a READ status precedes any payload, and the WRITE deadline is armed only once the body is done, when a well-behaved server owes at most one chunk's apply plus 8 bytes. Expiry just closes the socket; the pending read's handler owns the failure path, including source-buffer quiescence for WRITE. Handlers all run on the transport's single io thread, so no new races. Default 30s, MC_TCP_STATUS_TIMEOUT_SEC overrides (tests use 2s). New test drives both directions of a 4-byte request against a persistent fake v1 peer and asserts failure arrives after the deadline (not before - an early failure would mean the wrong path failed) and well before the old forever-hang. Full suite passes 3x in both P2PHANDSHAKE and HTTP metadata modes.
Co-authored-by: maxlisongsong <maxlisongsong@didiglobal.com>
#3259) * [TENT] Fix double-free in HttpMetaStore under concurrent workers: a shared CURL* is not thread-safe * [TENT] http: use request handle for curl_easy_escape instead of nullptr * [TENT] http: fix clang-format * ci: retrigger checks (flaky graceful_shutdown_test) --------- Co-authored-by: jiayuzailiu <jiayuzailiu@tencent.com>
* [CI/Build] Simplify wheel workflows * [CI/Build] Preserve host build parallelism
…3308) * [PG] Build the device worker with C++17 for CMake 3.22 compatibility * fix(musa): build decoupled PG device runtime --------- Co-authored-by: Xun Sun <UNIDY2002@outlook.com>
TransferEngineImpl::unregisterLocalMemory returned on the first transport whose unregister failed, leaving the region registered on the remaining transports and skipping the local bookkeeping erase. Collect the first error but attempt every transport, mirroring the batch path made best-effort in #2869. Refs #2869 Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
* [TransferEngine] Skip the CUDA pointer probe on GPU-less hosts getMemoryLocation() probes cudaPointerGetAttributes for every registered buffer in a CUDA-enabled build. On a GPU-less host (e.g. an RDMA-only real-client sidecar) the call fails with no device and logs an ERROR per buffer, drowning the real signal, before falling back to the host path. Detect device presence once via cudaGetDeviceCount and skip the probe entirely when no device exists, logging a single WARNING. Hosts with a GPU are unaffected. Refs #2937 Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com> * Extract the CUDA device presence probe into a named helper Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com> * clang-format: join the device-count condition onto one line --------- Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
* fix(ci): resolve nightly build failures * ci: preserve nightly build environment on install * fix(efa): correct configuration comment
Co-authored-by: Le1zyCatt <2490162471@qq.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: stmatengss <11641725+stmatengss@users.noreply.github.com>
…ket (#3289) Co-authored-by: maxlisongsong <maxlisongsong@didiglobal.com>
--------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: stmatengss <11641725+stmatengss@users.noreply.github.com>
) MasterMetricManager uses yalantinglibs dynamic_gauge_1t for per-segment metrics (mem_allocated_size_per_segment_, mem_total_capacity_per_segment_, nof_allocated_size_per_segment_, nof_total_capacity_per_segment_). When a segment is unmounted via CommitUnmountSegment, dec_total_mem_capacity() and dec_allocated_nof_size() only decrement the gauge value to 0 but do not remove the label entry from the gauge's internal map. This causes stale 0-value entries to persist indefinitely in Prometheus output. The problem is especially visible after a master restart with snapshot restore: the restored segments carry old client IDs, the reaper eventually expires those clients and calls CommitUnmountSegment, which decrements capacity to 0 but leaves the label behind. After clients remount with new IDs, the old segment names linger as capacity=0 entries. Fix: add remove_segment_metrics() and remove_nof_segment_metrics() that call remove_label_value() on the per-segment gauges, and invoke them in: - ScopedSegmentAccess::CommitUnmountSegment (memory segments) - ScopedNoFSegmentAccess::CommitUnmountSegment (NoF segments) - SegmentManager::releaseCapacityMetrics() (HA teardown) - ~MasterService() standby allocated-size cleanup (HA teardown) Signed-off-by: leonzzhu <leonzzhu@tencent.com>
github-actions
Bot
added
documentation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Module
mooncake-transfer-engine)mooncake-store)mooncake-ep)mooncake-pg)mooncake-integration)mooncake-p2p-store)mooncake-wheel)mooncake-common)mooncake-rl)Type of Change
How Has This Been Tested?
Test commands:
# Example: bash scripts/run_ci_test.shTest results:
Checklist
./scripts/code_format.shpre-commit run --all-filesand all hooks passAI Assistance Disclosure