Environment
Platform: xilinx_u55c_gen3x16_xdma_3_202210_1 (Alveo U55C)
XRT: 2.19.194 (2025.1), hash 7d8151e6ee73c6ec2e99501a58c9c2eca6cc68ce
Driver: xrt (sw/runtime/xrt, C API path — CPP_API not defined for this build)
Bitstream built with v++ 2023.1, targeting xilinx_u55c_gen3x16_xdma_3_202210_1, kernel freq 250MHz
Host: Ubuntu 24.04.4, kernel 6.8.0-134-generic, xocl/xclmgmt 2.19.194
Reproduced via ./ci/blackbox.sh --driver=xrt --app=hip/vecadd --args="-n 1048576"
Symptom
Running the same command repeatedly alternates cleanly between failing and passing:
run 1: Bus error (core dumped), make: *** Error 135
run 2: passes
run 3: Bus error (core dumped)
run 4: passes
... (repeats indefinitely)
Root cause (isolated)
Ruled out first (confirmed not the cause):
SC firmware/xsabin version mismatch (was present, fixed independently, did not change this behavior)
mmap/BAR "not ready yet" race (tested with retry-on-signal guard around register accesses — no effect)
Fixed startup delay before first register access (tested with a blind 6s sleep before first CU register write — no effect)
Confirmed via instrumented register reads/writes in sw/runtime/xrt/vortex.cpp, init():
ap_ctrl status before reset write: 0x4 // ap_idle=1, CU healthy/idle
[write MMIO_CTL_ADDR (0x00) = CTL_AP_RESET (1<<4)]
ap_ctrl status after reset write: 0xdeadfa11 // XRT's AXI-Lite read-failure sentinel
The CU's s_axi_ctrl interface is healthy and idle immediately before this write. The write of CTL_AP_RESET (1 << 4) to offset 0x00 (MMIO_CTL_ADDR) is what breaks it — every register access afterward (including all CP_BASE-offset command-processor reads/writes in cp_init()/cp_submit_cl_) returns 0xdeadfa11 or triggers SIGBUS.
dmesg confirms an AXI firewall trip immediately following:
xocl_mailbox_srv: Card is in a BAD state, please issue xbutil reset
xocl_hot_reset: resetting device...
$ xbmgmt examine --device --report firewall
Level 3 CTRL_USER: 0x80004 (RECS_CONTINUOUS_RTRANSFERS_MAX_WAIT|RECS_WRITE_TO_BVALID_MAX_WAIT)
i.e. writes/reads issued to the CU's s_axi_ctrl AXI-Lite slave never receive a BVALID/RVALID handshake after the reset-bit write — consistent with that write leaving the interface's decode logic held in a non-responsive state, with no automatic self-clear/follow-up write observed in this code path.
XRT auto-recovers the card via hot reset after the firewall trip, which explains the alternating pattern: each fresh process re-triggers the bad write → firewall trip → auto-reset; the next process then starts from a freshly-reset, healthy card and succeeds — until it repeats the same faulty sequence itself.
Relevant code (sw/runtime/xrt/vortex.cpp):
cpp#define MMIO_CTL_ADDR 0x00
#define CTL_AP_RESET (1 << 4)
...
CHECK_ERR(this->write_register(MMIO_CTL_ADDR, CTL_AP_RESET), {
return err;
});
The xclbin's declared s_axi_ctrl is a standard Vitis/HLS-style control interface (ap_ctrl_hs convention), where bit 4 conventionally corresponds to auto_restart, not a self-clearing reset — no ap_start (bit 0) is asserted anywhere in this path, and no follow-up write clears bit 4. Whether this is:
a missing ap_start assertion,
a missing de-assert/clear write after reset,
or CTL_AP_RESET's bit meaning something CU-specific that no longer matches this platform/build,
Flagging here for maintainer input.
Suggested next debugging step for whoever picks this up: temporarily skip the CTL_AP_RESET write entirely and confirm the CU stays responsive / the firewall doesn't trip, to conclusively isolate this single write as sufficient-and-necessary for the fault (I've confirmed it's associated but haven't yet tested removing it in isolation).
Happy to provide the full gdb bt, dmesg captures, and xclbinutil output from my debugging session if useful.
Environment
Platform: xilinx_u55c_gen3x16_xdma_3_202210_1 (Alveo U55C)
XRT: 2.19.194 (2025.1), hash 7d8151e6ee73c6ec2e99501a58c9c2eca6cc68ce
Driver: xrt (sw/runtime/xrt, C API path — CPP_API not defined for this build)
Bitstream built with v++ 2023.1, targeting xilinx_u55c_gen3x16_xdma_3_202210_1, kernel freq 250MHz
Host: Ubuntu 24.04.4, kernel 6.8.0-134-generic, xocl/xclmgmt 2.19.194
Reproduced via ./ci/blackbox.sh --driver=xrt --app=hip/vecadd --args="-n 1048576"
Symptom
Running the same command repeatedly alternates cleanly between failing and passing:
run 1: Bus error (core dumped), make: *** Error 135
run 2: passes
run 3: Bus error (core dumped)
run 4: passes
... (repeats indefinitely)
Root cause (isolated)
Ruled out first (confirmed not the cause):
SC firmware/xsabin version mismatch (was present, fixed independently, did not change this behavior)
mmap/BAR "not ready yet" race (tested with retry-on-signal guard around register accesses — no effect)
Fixed startup delay before first register access (tested with a blind 6s sleep before first CU register write — no effect)
Confirmed via instrumented register reads/writes in sw/runtime/xrt/vortex.cpp, init():
ap_ctrl status before reset write: 0x4 // ap_idle=1, CU healthy/idle
[write MMIO_CTL_ADDR (0x00) = CTL_AP_RESET (1<<4)]
ap_ctrl status after reset write: 0xdeadfa11 // XRT's AXI-Lite read-failure sentinel
The CU's s_axi_ctrl interface is healthy and idle immediately before this write. The write of CTL_AP_RESET (1 << 4) to offset 0x00 (MMIO_CTL_ADDR) is what breaks it — every register access afterward (including all CP_BASE-offset command-processor reads/writes in cp_init()/cp_submit_cl_) returns 0xdeadfa11 or triggers SIGBUS.
dmesg confirms an AXI firewall trip immediately following:
xocl_mailbox_srv: Card is in a BAD state, please issue xbutil reset
xocl_hot_reset: resetting device...
$ xbmgmt examine --device --report firewall
Level 3 CTRL_USER: 0x80004 (RECS_CONTINUOUS_RTRANSFERS_MAX_WAIT|RECS_WRITE_TO_BVALID_MAX_WAIT)
i.e. writes/reads issued to the CU's s_axi_ctrl AXI-Lite slave never receive a BVALID/RVALID handshake after the reset-bit write — consistent with that write leaving the interface's decode logic held in a non-responsive state, with no automatic self-clear/follow-up write observed in this code path.
XRT auto-recovers the card via hot reset after the firewall trip, which explains the alternating pattern: each fresh process re-triggers the bad write → firewall trip → auto-reset; the next process then starts from a freshly-reset, healthy card and succeeds — until it repeats the same faulty sequence itself.
Relevant code (sw/runtime/xrt/vortex.cpp):
cpp#define MMIO_CTL_ADDR 0x00
#define CTL_AP_RESET (1 << 4)
...
CHECK_ERR(this->write_register(MMIO_CTL_ADDR, CTL_AP_RESET), {
return err;
});
The xclbin's declared s_axi_ctrl is a standard Vitis/HLS-style control interface (ap_ctrl_hs convention), where bit 4 conventionally corresponds to auto_restart, not a self-clearing reset — no ap_start (bit 0) is asserted anywhere in this path, and no follow-up write clears bit 4. Whether this is:
a missing ap_start assertion,
a missing de-assert/clear write after reset,
or CTL_AP_RESET's bit meaning something CU-specific that no longer matches this platform/build,
Flagging here for maintainer input.
Suggested next debugging step for whoever picks this up: temporarily skip the CTL_AP_RESET write entirely and confirm the CU stays responsive / the firewall doesn't trip, to conclusively isolate this single write as sufficient-and-necessary for the fault (I've confirmed it's associated but haven't yet tested removing it in isolation).
Happy to provide the full gdb bt, dmesg captures, and xclbinutil output from my debugging session if useful.