qec/realtime: Relay BP (nv-qldpc) gpu_roce profile for the HSB decoding-server test#670
Open
cketcham2333 wants to merge 6 commits into
Open
qec/realtime: Relay BP (nv-qldpc) gpu_roce profile for the HSB decoding-server test#670cketcham2333 wants to merge 6 commits into
cketcham2333 wants to merge 6 commits into
Conversation
…ng-server test Extend hsb_fpga_decoding_server_test.sh to drive the nv-qldpc Relay BP decoder through the decoding server's gpu_roce transport -- the self-relaunching device-graph scheduler (GpuRoceTransceiver): enqueue_syndromes / get_corrections / reset_decoder execute as DEVICE_CALLs on the GPU and the captured RelayBP decode graph fires device-side (fire-and-forget + tail-self-relaunch) when a window completes. This replaces hololink_qldpc_graph_decoder_bridge for the decoding-server flow; the old bridge test is untouched. hsb_fpga_decoding_server_test.sh: - --transport cpu_roce|gpu_roce, defaulted from the decoder profile (pymatching -> cpu_roce HOST_CALL path, nv-qldpc-decoder -> gpu_roce), plus --gpu, --proprietary-archive, --nv-qldpc-plugin. - Data generation (surface_code-4-yaml, fresh each run) adds --use-relay-bp for the nv-qldpc profile and injects `transport: gpu_roce` into the generated YAML: DecoderServer selects its transceiver from the per-decoder transport key, the generator omits non-default optionals, and the default (cpu_roce) resolves to the not-yet-implemented CpuRoceTransceiver stub. - gpu_roce server launch: Hololink parameters go via HOLOLINK_* env (the server's gpu_roce mode ignores the cpu_roce CLI flags), remote QP converted to decimal ($((qp)) -- GpuRoceConfig::from_env parses base-10 only), CUDA_MODULE_LOADING=EAGER as with the old bridge launcher, and readiness keyed on "QEC_DECODING_SERVER_READY gpu_roce" (the transceiver prints the QP/RKey/Buffer handshake before that line, without the bridge banner). - Build phase: builds gpu_roce_transceiver (HSB) and cudaq-realtime-bridge-hololink (cuda-quantum), wires CUDAQ_QEC_REALTIME_CUDEVICE_PROPRIETARY_ARCHIVE / GPU_ROCE_TRANSCEIVER_LIB / CUDAQ_REALTIME_BRIDGE_HOLOLINK_LIBRARY into the cudaqx configure, clears the CMake cache (find_library NOTFOUND staleness), and symlinks the nv-qldpc plugin into build/lib/decoder-plugins. All gated on the proprietary archive's presence so pymatching-only rigs build unchanged. The plugin symlink is also created opportunistically on --build-less runs. decoding-server-cqr/CMakeLists.txt (bug fix, separable): - CUDAQ_GPU_ROCE_AVAILABLE was set as CACHE INTERNAL on success, but the plain `set(... FALSE)` at the top of the file shadows the cache under CMP0126 (NEW), so the compile-definition gate for cudaq-qec-decoding-server never fired: the library was silently built WITHOUT gpu_roce (make_transport threw "CUDAQ_GPU_ROCE_AVAILABLE is not set" at runtime) while the sibling decoding_server target -- which reads the cache -- reported "gpu_roce transport enabled". Set the normal variable TRUE alongside the cache entry. Signed-off-by: Chuck Ketcham <cketcham@nvidia.com>
With CUDAQ_GPU_ROCE_AVAILABLE now actually reaching the library target (the CMP0126 normal-vs-cache shadow fix), cudaq-qec-decoding-server links libdoca_gpunetio.so and libcudaq-realtime-bridge-hololink.so -- both of which reference the CUDA Driver API (cu*). Consumers of the static lib (test_decoding_server_core, test_decoders_yaml, decoding_server) inherit those dependencies, and on driverless build machines (CI containers: CUDA toolkit present, no GPU driver, no libcuda.so.1) their links fail with dozens of `undefined reference to cuStreamCreate` etc. -- the failure seen on all four QEC Build-and-test CI jobs. Link CUDA::cuda_driver alongside CUDA::cudart in the gpu_roce block: the CMake target resolves to the real driver where one is installed and to the toolkit's stubs (lib/stubs/libcuda.so) in driverless environments. This matches the existing precedent in unittests/utils (playback and the old bridge already link CUDA::cuda_driver). Signed-off-by: Chuck Ketcham <cketcham@nvidia.com>
With CUDAQ_GPU_ROCE_AVAILABLE actually reaching the library target (the CMP0126 fix), every consumer of cudaq-qec-decoding-server (test_decoding_server_core, test_decoders_yaml, the CQR plugin) inherited runtime dependencies on libdoca_gpunetio.so and libcudaq-realtime-bridge-hololink.so, which require libcuda.so.1 at LOAD time. On driverless CI machines (CUDA toolkit + HSB + DOCA present, no GPU driver) all four QEC Build-and-test jobs failed: gtest_discover_tests executes the test binaries at build time and the loader cannot satisfy libcuda.so.1 -- link-time driver stubs satisfy ld but not the runtime loader. Restructure so the core library carries no GPU RoCE dependencies while keeping full CI build coverage of the HSB path: - New optional component cudaq-qec-decoding-server-gpuroce holds GpuRoceTransceiver.cpp plus GpuRoceFactory.cpp, the strong definition of the factory (cudaqx_qec_make_gpu_roce_transceiver) that DecodingServer.cpp now references weakly. All DOCA / hololink / CUDA-driver link deps move to this component. - make_transport(gpu_roce) calls the factory when it is linked in and throws a clear "GPU RoCE support is not linked into this binary" error otherwise; the ifdef and the direct GpuRoceTransceiver construction are gone from the core. - The scheduler hookup no longer dynamic_casts to the concrete type: ITransceiver grows a launch_device_scheduler(void*) virtual (default: no device scheduler), overridden by GpuRoceTransceiver to forward to launch_scheduler(). The config-driven constructor keys the hookup on the transport enum instead. - decoding_server links the component WHOLE_ARCHIVE in its proprietary-archive-gated gpu_roce block -- a weak reference does not pull archive members. - A link canary (cudaq-qec-decoding-server-gpuroce-linkcheck; never executed, not installed, not a test) preserves CI link validation of the component: a static archive alone never resolves symbols, and the one real consumer (the daemon's gpu_roce block) is gated on the proprietary cudevice archive, which CI does not provision. The canary forces full link resolution against the hololink / DOCA libraries with the driver stubs -- the same check that used to happen incidentally via the test binaries. - decoding_server's QEC_HAVE_GPU_ROCE_TRANSPORT gate is split from the proprietary-archive gate: the gpu_roce CLI branch references only the (now dependency-free) core DecodingServer API, so it compiles and links whenever HSB/DOCA are present -- CI now builds that branch too. The component + cudevice whole-archive link stays behind the archive gate, so the CI daemon remains loadable on driverless runners. Branch [2a] wraps server startup in try/catch, turning what used to be std::terminate on any gpu_roce bring-up failure into an "ERROR: gpu_roce startup failed: ..." message with exit 1. Net CI coverage: GpuRoceTransceiver.cpp compiles against the HSB/DOCA headers (component builds in `all`), its link resolves (canary), and the daemon's gpu_roce CLI branch compiles and links (split gate). No binary CI executes carries a libcuda.so.1 dependency. On dev rigs the daemon keeps full gpu_roce. As a bonus, the CQR plugin .so is loadable on driverless machines again (it too had inherited the deps). Signed-off-by: Chuck Ketcham <cketcham@nvidia.com>
bmhowe23
reviewed
Jul 10, 2026
bmhowe23
left a comment
bmhowe23
left a comment
Collaborator
There was a problem hiding this comment.
Thanks, Chuck. Sharing some comments now ... I'm not sure to what extent we want to handle my comments in this PR vs a follow-up PR. We can discuss offline.
…est prebuilt Ben Howe's PR 670 review flagged that the test script implies source trees + a build toolchain in the decoding-server Docker image (it built cuda-quantum, HSB, and the cudaqx server from source). Rework `do_build` so the script builds ONLY the `surface_code-4` generator -- the one artifact not shipped in the image -- and consumes everything else PREBUILT: the decoder server (with gpu_roce linked in), the playback tool, the HSB / cudaq-realtime shared libs, and the decoder plugins. On a dev rig those come from the /workspaces/*/build trees (resolve_paths already points there); in a productized container they come from their installed locations. - `do_build` no longer builds cuda-quantum, HSB, or the decoder server. It checks the cudaqx build dir is configured and runs `cmake --build --target surface_code-4-yaml` (ninja regenerates from the cached configure if CMakeLists changed and recompiles only what is stale). A clean, unconfigured rig can no longer bootstrap from this script -- that is the intended trade. - Remove `PROPRIETARY_ARCHIVE` from the script entirely. It was only ever a build-time input to the gpu_roce daemon, which the script no longer builds; the prebuilt daemon already carries it. `NV_QLDPC_PLUGIN` stays -- it is a runtime dlopen input (generator + server), supplied via `--nv-qldpc-plugin` or `CUDAQ_QEC_NV_QLDPC_PLUGIN` (eventual home: an all_libs_release.yml artifact). Signed-off-by: Chuck Ketcham <cketcham@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds a Relay BP (nv-qldpc) profile to the HSB decoding-server test, driving the decoder
through the server's
gpu_rocetransport — the self-relaunching device-graph scheduler(
GpuRoceTransceiver):enqueue_syndromes/get_corrections/reset_decoderexecute asDEVICE_CALLs on the GPU, and the captured RelayBP decode graph fires device-side(fire-and-forget + tail self-relaunch) when a window completes. This replaces
hololink_qldpc_graph_decoder_bridgefor the decoding-server flow; the old bridge test isuntouched. The pymatching/cpu_roce profile is unchanged and re-verified.
Changes
hsb_fpga_decoding_server_test.sh--transport cpu_roce|gpu_roce, defaulted from the decoder profile (pymatching→cpu_roce,nv-qldpc-decoder→gpu_roce); new--gpu,--proprietary-archive,--nv-qldpc-pluginflags.surface_code-4-yaml) adds--use-relay-bpfor thenv-qldpc profile and injects
transport: gpu_roceinto the generated YAML —DecoderServerselects its transceiver from the per-decodertransport:key, the generatoromits non-default optionals, and the default (
cpu_roce) resolves to the not-yet-implementedCpuRoceTransceiverstub.HOLOLINK_*env (the server's gpu_roce modeignores the cpu_roce CLI flags); remote QP converted to decimal
(
GpuRoceConfig::from_envparses base-10 only);CUDA_MODULE_LOADING=EAGERas with the oldbridge launcher; readiness keyed on
QEC_DECODING_SERVER_READY gpu_roce(the transceiverprints the QP/RKey/Buffer handshake before that line, without the bridge banner).
gpu_roce_transceiver+ cuda-quantumcudaq-realtime-bridge-hololink,wires
CUDAQ_QEC_REALTIME_CUDEVICE_PROPRIETARY_ARCHIVE/GPU_ROCE_TRANSCEIVER_LIB/CUDAQ_REALTIME_BRIDGE_HOLOLINK_LIBRARYinto the cudaqx configure, clears the CMake cache(find_library NOTFOUND staleness), and symlinks the nv-qldpc plugin into
build/lib/decoder-plugins(also opportunistically on--build-less runs). All gated on theproprietary archive's presence, so pymatching-only rigs build exactly as before.
decoding-server-cqr/CMakeLists.txt— bug fix (separable)CUDAQ_GPU_ROCE_AVAILABLEwas set asCACHE INTERNALon success, but the plainset(... FALSE)at the top of the file shadows the cache under CMP0126 (NEW), so thecompile-definition gate for
cudaq-qec-decoding-servernever fired: the library silently builtwithout gpu_roce (
make_transportthrew "CUDAQ_GPU_ROCE_AVAILABLE is not set" at runtime)while the sibling
decoding_servertarget — which reads the cache — reported"gpu_roce transport enabled". Fixed by setting the normal variable alongside the cache entry.