TRT 11.1 strongly-typed FP32 engine silently produces wrong results for a DETR-style detection model (correct in ONNXRuntime and TRT 10.13)

[ArgoHA]

Description

Building a strongly-typed engine from an FP32 ONNX of a D-FINE (DETR-family, deformable-attention decoder) detection model on TensorRT 11.1.0.106 produces silently wrong outputs: detection scores diverge from ONNXRuntime by up to 0.15 (absolute, on sigmoid outputs) and output boxes by up to 597 px, where FP32-vs-FP32 agreement should be ~1e-4. The build completes with no warnings or errors.

The same ONNX file is exact in ONNXRuntime, and the same model executes correctly on TensorRT 10.13.3.9. On a real dataset (VisDrone test set) the mis-execution collapses recall 0.49 -> 0.34 and F1 0.587 -> 0.458.

Repo where issue was encountered - D-FINE-seg

Environment

TensorRT Version: 11.1.0.106 (tensorrt / tensorrt-cu13 wheels from PyPI)

NVIDIA GPU: GeForce RTX 5070 Ti, 16 GB (sm_120)

NVIDIA Driver Version: 580.159.03

CUDA Version: 13.0

CUDNN Version: 9.20.0.48 (not used by the repro; TRT wheels vendor their own deps)

Operating System: Ubuntu 22.04.5 LTS, kernel 6.8.0-124-generic

Python Version (if applicable): 3.13.12

Tensorflow Version (if applicable): —

PyTorch Version (if applicable): 2.12.1+cu130 (used only for CUDA buffers in the repro script; model is loaded from ONNX)

Baremetal or Container (if so, version): baremetal, pip wheels (pip install tensorrt==11.1.0.106 onnxruntime==1.21.1 onnx==1.21.0)

Relevant Files

Model link: https://drive.google.com/file/d/1Fs6nh2Edv1-1xFbqnIN5zTvE4nzfe7Ta/view?usp=share_link

• model.onnx — D-FINE-S detection model (HGNetv2 backbone + hybrid encoder + 3-layer deformable-attention decoder + fused postprocessor), FP32, static [1,3,640,640] input, outputs labels [1,300] int64, boxes [1,300,4], scores [1,300]. Exported with torch.onnx.export (dynamo, opset 19), simplified with onnxsim.
• The bug also reproduces with a TorchScript-exporter (dynamo=False, opset 17) export of the same weights — it is not specific to one exporter's graph flavor.
• repro.py — self-contained comparison script (below), deterministic synthetic input, no dataset needed.

Steps To Reproduce

Commands or scripts:

pip install tensorrt==11.1.0.106 onnxruntime==1.21.1 onnx==1.21.0 numpy torch
python repro.py model.onnx

repro.py:

import sys
import numpy as np
import onnxruntime as ort
import tensorrt as trt
import torch

onnx_path = sys.argv[1]

# deterministic structured input (gradient + rectangles, image-like, in [0,1])
rng = np.random.default_rng(0)
x = np.tile(np.linspace(0.2, 0.8, 640, dtype=np.float32)[None, :], (640, 1))
img = np.stack([x, x.T, 0.5 * (x + x.T)], 0)
for _ in range(40):
    x0, y0 = rng.integers(0, 560, 2)
    w, h = rng.integers(20, 80, 2)
    img[:, y0 : y0 + h, x0 : x0 + w] = rng.random(3, dtype=np.float32)[:, None, None]
inp = img[None]  # [1,3,640,640]

# ONNXRuntime reference
sess = ort.InferenceSession(onnx_path, providers=["CPUExecutionProvider"])
ort_out = dict(zip([o.name for o in sess.get_outputs()], sess.run(None, {"input": inp})))

# TensorRT: strongly-typed engine from the same FP32 ONNX
logger = trt.Logger(trt.Logger.WARNING)
builder = trt.Builder(logger)
network = builder.create_network()  # strongly typed (TRT >= 11 default)
parser = trt.OnnxParser(network, logger)
assert parser.parse(open(onnx_path, "rb").read())
config = builder.create_builder_config()
config.set_memory_pool_limit(trt.MemoryPoolType.WORKSPACE, 2 << 30)
engine = trt.Runtime(logger).deserialize_cuda_engine(
    builder.build_serialized_network(network, config)
)
ctx = engine.create_execution_context()

bufs = {}
for i in range(engine.num_io_tensors):
    name = engine.get_tensor_name(i)
    shape = tuple(engine.get_tensor_shape(name))
    dt = {"FLOAT": torch.float32, "INT64": torch.int64, "HALF": torch.float16}[
        engine.get_tensor_dtype(name).name
    ]
    bufs[name] = torch.empty(shape, dtype=dt, device="cuda")
    ctx.set_tensor_address(name, bufs[name].data_ptr())
bufs["input"].copy_(torch.from_numpy(inp))
ctx.execute_async_v3(torch.cuda.current_stream().cuda_stream)
torch.cuda.synchronize()

for name in ("scores", "boxes"):
    ref, out = ort_out[name].ravel(), bufs[name].float().cpu().numpy().ravel()
    print(f"{name:7s} max_abs_diff={np.abs(ref - out).max():.4f}")
print("scores top5 ORT:", np.round(np.sort(ort_out["scores"].ravel())[::-1][:5], 4))
print("scores top5 TRT:", np.round(np.sort(bufs["scores"].float().cpu().numpy().ravel())[::-1][:5], 4))

Observed output (RTX 5070 Ti, driver 580.159.03, TRT 11.1.0.106):

scores  max_abs_diff=0.1495
boxes   max_abs_diff=597.0000
scores top5 ORT: [0.1603 0.1174 0.0951 0.0739 0.0698]
scores top5 TRT: [0.2487 0.2349 0.2262 0.2203 0.2194]

Expected: max_abs_diff ~1e-4 or less (FP32 vs FP32). The engine builds without any warning; the wrong results are silent.

Have you tried the latest release?: Yes — 11.1.0.106 is the newest TensorRT on PyPI at the time of filing. The model executes correctly on TensorRT 10.13.3.9 (weak-typed build; FP16 engine matches the PyTorch F1 on our full benchmark).

Can this model run on other frameworks? For example run ONNX model with ONNXRuntime (polygraphy run <model.onnx> --onnxrt): Yes — ONNXRuntime (CPU EP) executes the file exactly (it is the reference in the repro). PyTorch source model agrees with ONNXRuntime.

Additional context

• The mis-execution appears tied to FP32 compilation of the transformer decoder region. Related observations on the same model, all silent:
  • A mixed engine authored as FP16 backbone/encoder + FP32 decoder (explicit Cast boundaries, strongly typed) collapses the same way (dataset F1 0.587 -> 0.445), while an all-FP16 engine (only Softmax kept FP32) is essentially correct (F1 0.584–0.586).
  • Inserting FP32 islands (~100 nodes of box-decode arithmetic) into the otherwise-FP16 graph measurably degrades accuracy instead of improving it.
  • With FP16 engines, accuracy differs between two ONNX graph flavors of the same weights (torch dynamo exporter 0.584 vs TorchScript exporter 0.586 dataset F1) — consistent with a graph-compilation sensitivity rather than kernel precision.
• All converted/authored ONNX variants used in these experiments execute identically to the FP32 reference in ONNXRuntime, so the divergence is introduced at TensorRT build/execution time.

[whateverforever]

do you have cumsum (for example in your pos enc?) #4551

[ArgoHA]

Nope

[JESUSROYETH]

Hi @ArgoHA 👋

I could reproduce this on a different GPU — an RTX 4060 Laptop (Ada, sm_89), same TRT 11.1.0.106 / CUDA 13, with your model.onnx and repro script. The strongly-typed FP32 engine diverges from ONNXRuntime by the same range you reported:

scores  max|Δ| = 0.196
boxes   max|Δ| = 605 px
labels  differ by up to 9

So this is not specific to sm_120 / Blackwell — it shows up on Ada too, which points at a TRT 11.1 regression rather than something about your new card. polygraphy run model.onnx --trt --onnxrt flags the same three outputs as failing, and it uses its own random inputs, so it does not depend on the exact input either.

I then ran a few probes in order to narrow it down:

1. Weakly-typed FP32 network (the legacy default) also diverges — so turning off strong typing is not a workaround, and the strongly-typed path is not really the cause.
2. config.set_tactic_sources(0) (TRT-native kernels only, no cuBLAS/cuDNN) still diverges — so the bad kernel is a TRT-native tactic, not an external library.
3. A single GridSample node with your decoder's attributes (bilinear, zeros, align_corners=0, feat [8,32,80,80], grid [8,300,3,2]) matches ORT to ~1e-5 — so the deformable-sampling op on its own is fine, the error is in the fused graph, not that op.
4. The magnitude changes a bit from run to run, which fits a tactic-selection problem: a specific fused kernel returns wrong values and gets picked during timing.

I could not pin the exact layer yet. Marking all outputs to bisect breaks the fusions (each tensor becomes an output), and on 8 GB that either masks the bug or runs out of memory, so finding the offending tactic needs an IAlgorithmSelector dump or a look from the TRT side.

For now the only verified-correct path is staying on 10.13.3.9, as you already found. I'm happy to lock and dump the winning tactics with IAlgorithmSelector on the Ada repro and share which kernel flips the output, since it reproduces here.