Make trt throw if fails to do inference#680
Merged
melody-ren merged 6 commits intoJul 15, 2026
Merged
Conversation
converged is a statement about the solution of a completed decode; it must never stand in for decoder health or inference success. Previously the TensorRT decoder conflated all three: - a failed initialization was swallowed, leaving a decoder object that returned converged=false zero-filled results on every decode - a failed inference was caught and reported the same way - a failed enqueueV3 was ignored entirely, which could report stale device buffers as a successful converged decode Initialization and inference failures now throw. The decoding server already maps worker exceptions to error responses, and the direct path propagates them to the caller, so an inference error and an unconverged decode are now distinct observables end to end. The in-source test counter this decoder once carried is not restored; execution evidence now comes from the decoding server, which keeps a per-session decode counter and, when QEC_DECODING_SERVER_STATS is set, prints one per-decoder counter line at shutdown. The surface-code test harness asserts every decoder completed one decode per shot from those lines (REQUIRE_SERVER_DECODE_COUNTS), and the trt test gate now warns loudly when python-onnx is missing instead of silently registering no trt tests. Unit tests asserting the old swallow-and-fabricate behavior are rewritten to expect the throws. Signed-off-by: Melody Ren <melodyr@nvidia.com>
From an independent review pass: - remove a dangling doc-comment fragment left by the failure_result_size removal - initialize Impl::stream and guard the -1 binding indices in ~Impl: both are live on the throwing-constructor path that is now the contract (the destructor runs against a partially constructed Impl) - enforce one-result-per-input at the decode_batch boundary so a misbehaving external global decoder returning short cannot turn the positional indexing in decode() into out-of-bounds access - warn per failed gate when the trt surface-code tests do not register: missing plugin target and missing python-onnx are different problems with different remedies, and the target-missing case was still a silent skip - correct the stats-print ordering comment in the decoding server tool and a stale python test comment describing the old swallow-and-warn constructor behavior Signed-off-by: Melody Ren <melodyr@nvidia.com>
…ed-does-not-mean-decoder-ready Signed-off-by: Melody Ren <melodyr@nvidia.com>
Signed-off-by: Melody Ren <melodyr@nvidia.com>
melody-ren
marked this pull request as ready for review
July 14, 2026 23:06
bmhowe23
approved these changes
Jul 14, 2026
melody-ren
enabled auto-merge (squash)
July 15, 2026 00:00
bmhowe23
added a commit
to bmhowe23/cudaqx
that referenced
this pull request
Jul 15, 2026
…ridge branch Main landed the squashed PR 670 plus NVIDIA#656/NVIDIA#673/NVIDIA#674/NVIDIA#678/NVIDIA#680/NVIDIA#683/NVIDIA#688/ NVIDIA#690/NVIDIA#691/NVIDIA#692. Resolution keeps this branch's design and ports main's content into it: - Naming: main's GpuRoce{Transceiver,Factory,LinkCheck} map 1:1 onto this branch's DeviceGraph* files (the de-transport-ing follow-up requested in the PR 670 review). Main's post-review fixes are ported into the renamed files: ring-size overflow + host-page-size alignment validation, gpu_id resolved from the decoder's cuda_device_id instead of an env var, the factory taking pinned_cuda_device, and the linkcheck exercising the new factory signature. - Schema: per-decoder 'dispatch: host|device_graph' plus the top-level transport section stay; main's per-decoder 'transport:' key does not return. cuda_device_id is adopted as a per-decoder placement knob (config struct + YAML trait), and the hsb script now injects 'dispatch: device_graph' + cuda_device_id and drives the server with QEC_DEVICE_GRAPH_* env and the device_graph READY sentinel. - Features adopted from main: decoder CUDA-device pinning (worker-thread pin via promise, graph-capture pin, hardware_guards.h), per-session decode counters + print_session_stats + QEC_DECODING_SERVER_STATS server-side stats printing, DecodingServer ctor exception safety, virtualized realtime decoder API (NVIDIA#674), CMP0126 fix, CUDA::cudart on the core lib, and the cudevice-archive dedup in the server link. - DecodingSession combines main's set_graph_capture_device with this branch's reserved-SMs decode-graph capture (QEC_DEVICE_GRAPH_RESERVED_SMS). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
anjbur
added a commit
to anjbur/cudaqx
that referenced
this pull request
Jul 16, 2026
This reverts commit efbb6f9. Signed-off-by: Angela Burton <angelab@nvidia.com>
anjbur
added a commit
to anjbur/cudaqx
that referenced
this pull request
Jul 16, 2026
…integration Signed-off-by: Angela Burton <angelab@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Trt decoder was using
convergedflag to indicate if the decoder is ready. Previously the TensorRT decoder had the following behaviour:Initialization and inference failures now throw. The decoding server already maps worker exceptions to error responses, and the direct path propagates them to the caller, so an inference error and an unconverged decode are now distinct observables end to end.