merge main

.claude/skills/mooncake-api/SKILL.md

@@ -19,6 +19,14 @@ Use this skill when users ask about:
 - Mooncake EP (Expert Parallelism) and Mooncake Backend
 - Troubleshooting Mooncake Python API issues
 
+## Routing Guidance
+
+- For most vLLM and SGLang users, start from `docs/source/getting_started/quick-start.md`.
+- For PD disaggregation, direct users to the SGLang/vLLM integration guides listed in Quick Start. Those guides own the serving-framework configuration.
+- For Mooncake Store integrations, direct users to the SGLang/vLLM Store setup guides listed in Quick Start. Do not duplicate `mooncake_master` startup commands in general API answers unless the user is working outside those frameworks.
+- For direct low-level Transfer Engine usage, use `docs/source/design/transfer-engine/index.md#using-transfer-engine-in-your-projects`.
+- For API signatures and method details, use the Python API references under `docs/source/python-api-reference/`.
+
 ## Core Components
 
 ### 1. Mooncake Store (Distributed KV Cache)
@@ -307,6 +315,7 @@ mooncake_master --default_kv_lease_ttl=5000
 - `MC_STORE_MEMCPY`: Enable local memcpy optimization (set to "1")
 - `MC_STORE_CLIENT_METRIC`: Enable client metrics (enabled by default)
 - `MC_YLT_LOG_LEVEL`: Log level (trace/debug/info/warn/error/critical)
+- `MOONCAKE_STORE_CHECKSUM`: Enable diagnostic object-level CRC-64 checks (set to "1" before creating any writer or reader client)
 
 ## Common Patterns
 
@@ -465,6 +474,17 @@ if result != 0:
     raise RuntimeError(f"Failed to register buffer: {result}")
 ```
 
+### Corrupted Data or Garbled Output
+```python
+# Set this before importing Mooncake or creating any Store client.
+import os
+os.environ["MOONCAKE_STORE_CHECKSUM"] = "1"
+
+from mooncake.store import MooncakeDistributedStore
+```
+
+Enable the switch on every writer and reader client process, then reproduce with full-object `put`/`upsert` and `get` operations. Treat `CHECKSUM_MISMATCH` (-801) as a failed read and do not use the destination buffer. Objects without checksum metadata and range reads are not verified. This mode scans object data, stages GPU buffers to host memory, and disables the local hot cache, so use it only for diagnosis.
+
 ### Service Connectivity
 ```bash
 # Check master is running
@@ -510,6 +530,8 @@ curl http://localhost:8080/metadata
 ## Documentation Links
 
 - Full API Reference: https://kvcache-ai.github.io/Mooncake/
+- Quick Start: docs/source/getting_started/quick-start.md
 - Mooncake Store: docs/source/python-api-reference/mooncake-store.md
 - Transfer Engine: docs/source/python-api-reference/transfer-engine.md
+- Transfer Engine direct usage: docs/source/design/transfer-engine/index.md#using-transfer-engine-in-your-projects
 - EP Backend: docs/source/python-api-reference/ep-backend.md

.claude/skills/mooncake-troubleshoot/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: mooncake-troubleshoot
-description: Automatically diagnose Mooncake deployment and runtime issues. Checks services (mooncake_master, metadata server), RDMA devices, environment variables, connectivity, memory limits, and analyzes logs for common error patterns. Use when Mooncake deployment fails, services won't start, connections fail, or you encounter runtime errors like "Error from etcd client", "No matched device found", "Failed to register memory", "NO_AVAILABLE_HANDLE", or any RDMA/networking issues. Also use when user asks to troubleshoot, debug, diagnose, or fix Mooncake problems.
+description: Automatically diagnose Mooncake deployment and runtime issues. Checks services (mooncake_master, metadata server), RDMA devices, environment variables, connectivity, memory limits, object integrity, and analyzes logs for common error patterns. Use when Mooncake deployment fails, services won't start, connections fail, data is corrupted or garbled, or you encounter runtime errors like "Error from etcd client", "No matched device found", "Failed to register memory", "NO_AVAILABLE_HANDLE", "CHECKSUM_MISMATCH", or any RDMA/networking issues. Also use when user asks to troubleshoot, debug, diagnose, or fix Mooncake problems.
 ---
 
 # Mooncake Deployment Troubleshooting
@@ -62,8 +62,8 @@ echo "https_proxy: $https_proxy"
 Verify critical environment variables are set correctly:
 
 ```bash
-# Display all MC_* variables
-env | grep ^MC_
+# Display Mooncake variables
+env | grep -E '^(MC_|MOONCAKE_STORE_CHECKSUM=)'
 
 # Key variables to check:
 echo "MC_METADATA_SERVER: $MC_METADATA_SERVER"
@@ -76,6 +76,7 @@ echo "MC_GID_INDEX: $MC_GID_INDEX"
 echo "MC_MTU: $MC_MTU"
 echo "MC_IB_PORT: $MC_IB_PORT"
 echo "MC_ENABLE_DEST_DEVICE_AFFINITY: $MC_ENABLE_DEST_DEVICE_AFFINITY"
+echo "MOONCAKE_STORE_CHECKSUM: $MOONCAKE_STORE_CHECKSUM"
 ```
 
 **Key variables:**
@@ -88,6 +89,7 @@ echo "MC_ENABLE_DEST_DEVICE_AFFINITY: $MC_ENABLE_DEST_DEVICE_AFFINITY"
 - `MC_GID_INDEX` - RDMA GID index (set if GID is all zeros)
 - `MC_MTU` - RDMA MTU size
 - `MC_ENABLE_DEST_DEVICE_AFFINITY=1` - Reduce QP creation (fix "Failed to create QP")
+- `MOONCAKE_STORE_CHECKSUM=1` - Enable diagnostic object-level CRC-64 checks; set before starting every writer and reader client process
 
 ### 4. RDMA Device Check
 
@@ -216,6 +218,8 @@ Search logs for common error patterns and their meanings:
   - **Fix:** Increase `default_kv_lease_ttl` in master startup
 - `OBJECT_NOT_FOUND` (-704) → Object doesn't exist
 - `SEGMENT_NOT_FOUND` (-101) → No available segments
+- `CHECKSUM_MISMATCH` (-801) → Full-object read differs from the checksum captured before the write
+  - **Action:** Treat the read as failed and do not use the destination buffer
 - `Failed to get description of XXX` → Segment name mismatch
   - **Fix:** Ensure segment name matches `local_hostname` from peer
 
@@ -244,6 +248,18 @@ env | grep MC_FORCE_TCP
 - RDMA device names must exist on the machine
 - Ports must not be in use by other services
 
+### 9. Object Integrity Diagnostics
+
+When the user reports corrupted data or garbled output that may originate from Mooncake Store:
+
+1. Verify that the Store clients, primary master, and standby master use checksum-capable binaries from the same version.
+2. Set `MOONCAKE_STORE_CHECKSUM=1` before starting every writer and reader client process, then restart or recreate existing clients.
+3. Reproduce with full-object `put`/`upsert` and `get` operations. Range reads, including `get_into_ranges`, are not covered.
+4. If `CHECKSUM_MISMATCH` (-801) is returned, the covered read differs from the checksum captured over the writer's source object. Treat the read as failed and do not consume the destination buffer.
+5. If `object_checksum_absent` appears at VLOG(1), metadata has no checksum and verification was skipped. This can occur for existing objects or objects written by clients without the switch.
+6. If verification succeeds, continue investigating corruption outside the covered Store path. A checksum mismatch identifies a difference but does not by itself distinguish transfer corruption from storage corruption.
+7. Warn that this diagnostic mode scans all object data, stages GPU buffers to host memory, and disables the local hot cache. Disable it after diagnosis.
+
 ## Error Code Quick Reference
 
 ### Transfer Engine Error Codes
@@ -266,6 +282,7 @@ env | grep MC_FORCE_TCP
 | -707 | LEASE_EXPIRED | Lease expired | Increase lease TTL |
 | -704 | OBJECT_NOT_FOUND | Object doesn't exist | Check object key |
 | -101 | SEGMENT_NOT_FOUND | No available segments | Check segment registration |
+| -801 | CHECKSUM_MISMATCH | Full-object data differs from its stored checksum | Reject the read buffer and investigate the Store data path |
 | -900 | RPC_FAIL | RPC failed | Check network/master |
 | -1000 | ETCD_OPERATION_ERROR | etcd operation failed | Check etcd status |
 

.devcontainer/Dockerfile

@@ -8,7 +8,6 @@ RUN apt-get update -y \
 RUN apt-get install -y libibverbs-dev \
     libunwind-dev \
     libgoogle-glog-dev \
-    libgtest-dev \
     libjsoncpp-dev \
     libnuma-dev \
     libpython3-dev \

.github/CODEOWNERS

@@ -6,19 +6,19 @@
 # EP: @UNIDY2002 UNIDY2002@outlook.com
 # PG: @UNIDY2002 UNIDY2002@outlook.com
 
-.github @stmatengss @ykwd @Ann-1024 @luketong777
-/docs @ShangmingCai @stmatengss @ykwd
+.github @stmatengss @ykwd @Ann-1024 @luketong777 @Aionw
+/docs @ShangmingCai @stmatengss @ykwd @UNIDY2002 @alogfans @staryxchen
 /mooncake-ep @UNIDY2002 @ympcMark @yuechen-sys
 /mooncake-integration/transfer_engine @ShangmingCai @alogfans
-/mooncake-integration/store @ykwd @stmatengss
+/mooncake-integration/store @ykwd @stmatengss @zxpdemonio
 /mooncake-pg @UNIDY2002 @ympcMark @yuechen-sys
 /mooncake-store @ykwd @stmatengss @XucSh @YiXR
-/mooncake-store/*/ha/ @Libotry @YiXR @00fish0
-/mooncake-transfer-engine @alogfans @doujiang24 @chestnut-Q
-/mooncake-transfer-engine/tent @alogfans @doujiang24 @chestnut-Q @staryxchen @00fish0 @dtcccc 
+/mooncake-store/*/ha/ @Libotry @YiXR @00fish0 @Icedcoco
+/mooncake-transfer-engine @alogfans @doujiang24 @chestnut-Q @staryxchen
+/mooncake-transfer-engine/tent @alogfans @doujiang24 @chestnut-Q @staryxchen @00fish0 @dtcccc
 /mooncake-transfer-engine/*/transport/hip_transport/ @alogfans @amd-arozanov
 /mooncake-transfer-engine/*/transport/ascend_transport/ @alogfans @ascend-direct-dev
 /mooncake-transfer-engine/*/transport/efa_transport/ @alogfans @whn09
-/mooncake-wheel @ShangmingCai @stmatengss
+/mooncake-wheel @ShangmingCai @stmatengss @zxpdemonio
 /scripts/tone_tests @luketong777
 /scripts/ascend/ @ascend-direct-dev @VNightMare @MingYang119

.github/actions/free-disk-space/action.yml

@@ -0,0 +1,15 @@
+name: Free runner disk space
+description: Remove preinstalled toolchains that Mooncake CI does not use
+
+runs:
+  using: composite
+  steps:
+    - name: Remove unused runner tools
+      shell: bash
+      run: |
+        sudo rm -rf -- \
+          /usr/share/dotnet \
+          /opt/ghc \
+          /opt/hostedtoolcache/CodeQL \
+          /usr/local/lib/android
+        df -h

.github/workflows/_build-efa-wheel.yaml

@@ -0,0 +1,223 @@
+name: _build-efa-wheel
+
+# Shared AWS EFA wheel build used by pull-request CI and release workflows.
+# EFA hardware is not required to build: the distro libfabric is used for
+# compilation, then auditwheel leaves libfabric/libefa to the system runtime.
+
+on:
+  workflow_call:
+    inputs:
+      variant:
+        # cuda | cuda13 | non-cuda
+        type: string
+        required: true
+      use-cuda:
+        type: boolean
+        required: true
+      python-versions:
+        # JSON array consumed by the called workflow's Python matrix.
+        type: string
+        required: true
+      build-profile:
+        # ci uses Ninja and CI diagnostics; release preserves the release make build.
+        type: string
+        required: true
+      cmake-args:
+        type: string
+        required: true
+      variant-flag:
+        # build_wheel.sh package variant, e.g. EFA_BUILD.
+        type: string
+        required: true
+      cuda-version:
+        type: string
+        default: '12.8.1'
+      torch-cuda-arch-list:
+        type: string
+        default: ''
+      artifact-prefix:
+        type: string
+        required: true
+
+env:
+  SCCACHE_GHA_ENABLED: "true"
+
+jobs:
+  build:
+    runs-on: ubuntu-22.04
+    strategy:
+      matrix:
+        python-version: ${{ fromJSON(inputs.python-versions) }}
+    env:
+      BUILD_PROFILE: ${{ inputs.build-profile }}
+      CMAKE_ARGS: ${{ inputs.cmake-args }}
+      EFA_VARIANT: ${{ inputs.variant }}
+      USE_CUDA: ${{ inputs.use-cuda }}
+      VARIANT_FLAG: ${{ inputs.variant-flag }}
+      CUDA_VERSION: ${{ inputs.cuda-version }}
+
+    steps:
+      - name: Validate build inputs
+        run: |
+          case "$BUILD_PROFILE" in
+            ci|release) ;;
+            *) echo "::error::Unknown EFA build profile: $BUILD_PROFILE"; exit 1 ;;
+          esac
+
+          case "$EFA_VARIANT:$USE_CUDA:$VARIANT_FLAG" in
+            cuda:true:EFA_BUILD|cuda13:true:EFA_CU13_BUILD|non-cuda:false:EFA_NON_CUDA_BUILD) ;;
+            *) echo "::error::Inconsistent EFA variant inputs"; exit 1 ;;
+          esac
+        shell: bash
+
+      - name: Configure CUDA architecture list
+        if: ${{ inputs.torch-cuda-arch-list != '' }}
+        env:
+          ARCH_LIST: ${{ inputs.torch-cuda-arch-list }}
+        run: echo "TORCH_CUDA_ARCH_LIST=$ARCH_LIST" >> "$GITHUB_ENV"
+        shell: bash
+
+      - name: Checkout source
+        uses: actions/checkout@v4
+        with:
+          persist-credentials: ${{ inputs.build-profile == 'release' }}
+
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+
+      - name: Free up disk space
+        uses: ./.github/actions/free-disk-space
+
+      - name: Install CUDA Toolkit
+        if: ${{ inputs.use-cuda }}
+        uses: Jimver/cuda-toolkit@v0.2.29
+        with:
+          cuda: ${{ inputs.cuda-version }}
+          method: 'network'
+          sub-packages: '["nvcc", "nvrtc-dev"]'
+          non-cuda-sub-packages: '["libcusparse-dev", "libcublas-dev", "libcusolver-dev"]'
+
+      - name: Run sccache-cache
+        uses: mozilla-actions/sccache-action@v0.0.9
+
+      - name: Configure sccache
+        uses: actions/github-script@v7
+        with:
+          script: |
+            core.exportVariable('ACTIONS_RESULTS_URL', process.env.ACTIONS_RESULTS_URL || '');
+            core.exportVariable('ACTIONS_RUNTIME_TOKEN', process.env.ACTIONS_RUNTIME_TOKEN || '');
+
+      - name: Install dependencies
+        run: |
+          sudo apt update -y
+          packages=(libfabric-dev libfabric1)
+          if [ "$BUILD_PROFILE" = ci ]; then
+            packages=(ninja-build "${packages[@]}")
+          fi
+          sudo apt install -y "${packages[@]}"
+          sudo bash -x dependencies.sh -y
+          if [ "$BUILD_PROFILE" = ci ]; then
+            df -h
+          fi
+        shell: bash
+
+      - name: Configure project
+        run: |
+          generator=()
+          if [ "$BUILD_PROFILE" = ci ]; then
+            generator=(-G Ninja)
+          fi
+
+          cuda_args=(-DUSE_CUDA=OFF)
+          if [ "$USE_CUDA" = true ]; then
+            cuda_args=(
+              -DUSE_CUDA=ON
+              -DCMAKE_EXE_LINKER_FLAGS=-L/usr/local/cuda/lib64/stubs
+            )
+          fi
+
+          read -r -a profile_args <<< "$CMAKE_ARGS"
+          mkdir build
+          cd build
+          cmake "${generator[@]}" .. \
+            -DUSE_EFA=ON \
+            -DLIBFABRIC_INCLUDE_DIR=/usr/include \
+            -DLIBFABRIC_LIBRARY=/usr/lib/x86_64-linux-gnu/libfabric.so \
+            -DENABLE_SCCACHE=ON \
+            -DCMAKE_BUILD_TYPE=Release \
+            "${profile_args[@]}" \
+            "${cuda_args[@]}"
+        shell: bash
+
+      - name: Build project
+        run: |
+          if [ "$USE_CUDA" = true ]; then
+            export LD_LIBRARY_PATH=/usr/local/cuda/lib64/stubs:$LD_LIBRARY_PATH
+            export LIBRARY_PATH=/usr/local/cuda/lib64/stubs:$LIBRARY_PATH
+          fi
+
+          cd build
+          if [ "$BUILD_PROFILE" = release ]; then
+            make -j3
+            sudo make install
+          else
+            cmake --build .
+            sudo cmake --install .
+            df -h
+          fi
+        shell: bash
+
+      - name: Run sccache stat for check
+        if: ${{ env.SCCACHE_PATH != '' }}
+        shell: bash
+        run: ${SCCACHE_PATH} --show-stats
+
+      - name: Generate Python version tag
+        id: python-tag
+        run: |
+          echo "python_version_tag=$(echo ${{ matrix.python-version }} | tr -d '.')" >> "$GITHUB_OUTPUT"
+        shell: bash
+
+      - name: Build Python wheel
+        run: |
+          export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib
+          export "$VARIANT_FLAG=1"
+          PYTHON_VERSION=${{ matrix.python-version }} \
+            OUTPUT_DIR=dist-py${{ steps.python-tag.outputs.python_version_tag }} \
+            ./scripts/build_wheel.sh
+        shell: bash
+
+      - name: Verify libfabric is excluded from the wheel
+        run: |
+          WHL=$(ls mooncake-wheel/dist-py${{ steps.python-tag.outputs.python_version_tag }}/*.whl | head -1)
+          echo "Inspecting $WHL"
+          if unzip -l "$WHL" | grep -iE 'libfabric|libefa'; then
+            echo "::error::libfabric/libefa must NOT be bundled in the EFA wheel"
+            exit 1
+          fi
+          echo "OK: libfabric/libefa correctly excluded (resolve to system EFA at runtime)"
+        shell: bash
+
+      - name: Verify CUDA runtime dependency
+        if: ${{ inputs.use-cuda }}
+        run: |
+          WHL=$(ls mooncake-wheel/dist-py${{ steps.python-tag.outputs.python_version_tag }}/*.whl | head -1)
+          inspect_dir=$(mktemp -d)
+          unzip -q "$WHL" mooncake/engine.so -d "$inspect_dir"
+          cuda_major=${CUDA_VERSION%%.*}
+          if ! readelf -d "$inspect_dir/mooncake/engine.so" | \
+              grep -Fq "Shared library: [libcudart.so.${cuda_major}]"; then
+            echo "::error::EFA wheel does not depend on libcudart.so.${cuda_major}"
+            readelf -d "$inspect_dir/mooncake/engine.so" | grep NEEDED
+            exit 1
+          fi
+          echo "OK: EFA wheel depends on libcudart.so.${cuda_major}"
+        shell: bash
+
+      - name: Upload Python wheel artifact
+        uses: actions/upload-artifact@v4
+        with:
+          name: ${{ inputs.artifact-prefix }}-py${{ steps.python-tag.outputs.python_version_tag }}
+          path: mooncake-wheel/dist-py${{ steps.python-tag.outputs.python_version_tag }}/*.whl