diff --git a/docs/dfx/global-backpressure-design.md b/docs/dfx/global-backpressure-design.md new file mode 100644 index 0000000000..1886375302 --- /dev/null +++ b/docs/dfx/global-backpressure-design.md @@ -0,0 +1,174 @@ +# Global DFX Backpressure — block-on-contention + dual-signal freeze + +The shared design for how every DFX profiling subsystem (L2Swimlane, PMU, +DepGen, ArgsDump, ScopeStats, tensor_dump) reacts when the host collector +cannot keep the device-side buffer pool refilled. It replaces the older +"bounded wait, then drop the record" model with a **resident block-on-contention +freeze**: an AICPU writer that runs out of ready-queue space or free buffers +parks at its buffer-switch gate until the host clears the contention, rather +than dropping profiling data. + +Read alongside the code it documents: + +- `src/common/platform/include/common/dfx_backpressure_device.h` — the shared + `DfxBackpressureHeader` layout and the AICPU-side primitives. +- `src/common/platform/include/aicpu/profiler_device_engine.h` — the device + producer control flow that calls those primitives (`wait_for_ready_queue_space`, + `wait_for_free_queue_entry`, `enqueue_ready`). +- `src/common/platform/include/host/profiler_base.h` — + `ProfilerAlgorithms::update_backpressure_freeze`, the single host-side freeze + state machine. +- Capacity sizing that keeps this path cold in normal runs: + [dfx-buffer-capacity-audit.md](dfx-buffer-capacity-audit.md). + +## Why block instead of drop + +The device writers sit on the AICPU scheduler's critical path. The previous +model gave them a short bounded wait for a ready-queue slot or a free buffer and, +on expiry, counted a dropped record and moved on. That keeps the workload moving +but silently loses profiling data exactly when the system is most interesting +(contended). For DFX correctness we prefer to **stall the producer and lose no +records**, provided the stall cannot deadlock and cannot hang forever on a host +crash. The mechanism below delivers that: resident blocking, bounded only by a +30-second host-crash backstop, with no opt-out. + +## The two gate classes (issue #997) + +An AICPU writer only ever blocks at one of two buffer-switch gates, so the +coordination is split into two independent freezes — one per gate class: + +| Gate | Blocks when | Device signal | Host freeze | +| ---- | ----------- | ------------- | ----------- | +| **push** (`rq_*`) | the per-thread ready queue is full (`next_tail == head`) | `rq_contended` | `rq_freeze_active` | +| **pop** (`fq_*`) | the free queue is empty (`head == tail`) | `fq_contended` | `fq_freeze_active` | + +`*_freeze_active` are host→device; `*_contended` are device→host leader signals +that the host consumes and clears. All four are `volatile uint32_t` 0/1 flags in +the `DfxBackpressureHeader` that every subsystem's `DataHeader` embeds once +(physically per-subsystem: each subsystem has its own shared-memory region, so +swimlane's freeze is independent of PMU's). + +Splitting by gate class is what makes the freeze **common-mode**: every lane +that reaches a given gate parks at the same gate, producing one lane-aligned gap +in the swimlane instead of one lane sparsifying and skewing a bottleneck read. + +## Device side + +Three primitives in `dfx_backpressure_device.h`, driven from +`profiler_device_engine.h`: + +1. **Leader signal** — `mark_rq_contended` / `mark_fq_contended` set the + `*_contended` flag once per wait (guarded by a local `signalled` bool). The + host observes the sticky flag on its next mgmt tick and opens the matching + freeze. +2. **Peer freeze barrier** — `push_freeze_barrier` / `pop_freeze_barrier` spin + while the host holds the freeze open (`*_freeze_active != 0`), relaxing via + the repo-wide `SPIN_WAIT_HINT()` (a no-op on onboard silicon, `sched_yield()` + in sim). Each carries an acquire `rmb()` after the gate so the host's + pre-release queue writes are visible to the reads that follow. +3. **Leader park** — `wait_for_release` is used only by a leader whose own + reclaim depends on the host completing a full open→drain→release cycle on the + free-queue side (only tensor_dump's arena barrier today). It spins on the + **disjunction** `fq_contended || fq_freeze_active`. + +The push and pop loops in the engine bridge the host round-trip: raising +`*_contended` does not itself park the lane — until the host opens the freeze the +barrier sees `*_freeze_active == 0` and returns at once, so the loop re-checks +the queue and picks up a slot the instant one appears. Once the host has opened +the freeze, the barrier holds the lane until release. + +### Timeout backstop + +Every barrier and contention spin is bounded by +`Module::kBackpressureWaitCycles`, which every subsystem points at the single +`PLATFORM_DFX_BACKPRESSURE_TIMEOUT_CYCLES` constant +(`PLATFORM_PROF_SYS_CNT_FREQ * 30`, i.e. a 30-second wall-clock ceiling that +scales per arch — a2a3 50 MHz, a5 1000 MHz). The budget exists **only** to break +an infinite spin on host crash or hardware failure; it is not a normal-path +wait. When it expires the writer breaks to its single failure exit +(`return false`), and the caller (`switch_buffer`) accounts the dropped records. +A 30-second expiry therefore means "the host is gone", not "the buffer was +briefly full". + +## Host side — the freeze state machine + +`ProfilerAlgorithms::update_backpressure_freeze` runs once per tick of the +single `mgmt_replenish_loop`. It must be single-writer for `*_freeze_active`, +which is why it lives on the one replenish thread, not the (possibly sharded) +drain threads. It is idle at zero cost until a lane raises `*_contended`. + +**Open (per gate, independent).** If a gate's freeze is closed and its +`*_contended` reads non-zero, set `*_freeze_active = 1`, publish it, then clear +`*_contended`. For the pop gate the freeze is opened *before* `fq_contended` is +consumed so the disjunction `wait_for_release` spins on stays continuously true +— see the escape-window note below. + +**Release (conjunction, both gates together).** While either freeze is open, +release **both** only once: + +1. the ready queue is fully drained (`queue_heads[q] == queue_tails[q]` for every + thread), **and** +2. the free queues are refilled to their attainable initial upper limit + (`replenish_free_queues` returns `pushed == 0`), **and** +3. the per-subsystem `backpressure_release_ready()` predicate holds. + +Order matters: RQ-empty is checked first, because once the ready queue is empty +the drain threads are idle and it is safe for the replenish thread to be the sole +free-queue writer (drain-driven `top_up_free_queue` is suppressed while either +freeze is open). Resuming into RQ-empty + FQ-at-limit is one clean common-mode +gap with no immediate re-stall. + +### Per-subsystem release predicate + +`backpressure_release_ready()` is a CRTP hook, default `true`. A subsystem whose +collector owns a separate, independently-overwritten region must override it to +hold the release until that region is drained — otherwise the device resumes and +overwrites not-yet-pulled data. Only **tensor_dump** does this today: RQ-empty +alone does not imply its payload arena has been pulled. The hook is evaluated on +the replenish thread inside the freeze loop, so overrides must be cheap, +non-blocking, and read only atomics. + +## Two correctness arguments + +**No (0,0) escape window (pop gate).** `wait_for_release` spins on the +disjunction `fq_contended || fq_freeze_active`. If the host cleared +`fq_contended` before opening `fq_freeze_active`, a leader could observe both +zero mid-transition and escape the park prematurely. Opening the freeze *before* +consuming the contention keeps the predicate continuously true from +`mark_fq_contended()` through release. + +**Deadlock-free at any pool size.** The conjunction release cannot wedge +regardless of how small the pool is: + +- RQ-drain is host-driven and independent of any buffer a lane holds at a push + gate — the host can always empty the ready queue. +- "FQ refilled" is defined as `pushed == 0` against the *attainable* initial + limit `min(kSlotCount, BUFFERS)`, not an unreachable exact-`kSlotCount` fill. + +So even a one-buffer-per-pool stress shape drains and releases. The AICPU writer +also publishes its full buffer to the ready queue *before* trying to acquire a +replacement, so the host can always observe the full buffer and return a +recycled one. + +## Memory ordering across the PCIe boundary + +`*_freeze_active` (host→device) writes go through `write_range_to_device` and +`*_contended` / `queue_tails` (device→host) reads through `read_range_from_device` +so **a5 (non-SVM)** sees them; **a2a3 (SVM)** short-circuits both to no-ops +because the header already lives in shared device memory. Device-side barriers +(`rmb()` after a gate, `wmb()` before publishing a freeze/tail) pair with the +host's `wmb()` before each `write_range_to_device` so a released lane never reads +a stale queue. + +## What this observably does under stress + +When the pool is deliberately shrunk until the host cannot keep up, the freeze +fires repeatedly and the AICPU scheduler parks at the pop gate. Because the +scheduler is parked, in-flight tasks whose `FIN` it has not yet polled have their +recorded `finish_time` pushed out to the freeze-release instant — i.e. the freeze +**inflates the very `finish_time` it is collecting**. Task *dispatch→finish* +durations in the swimlane are then a measurement artifact of the freeze, not real +kernel time; the unaffected AICore-side `start`/`end` timestamps remain the +ground truth for kernel duration. This is expected: the mechanism trades producer +stall (and skewed AICPU-observed timing) for zero record loss, and normal-sized +pools keep the path cold so it never triggers in production runs. diff --git a/docs/profiling-framework.md b/docs/profiling-framework.md index 5ccf09e46f..c6725fdfbd 100644 --- a/docs/profiling-framework.md +++ b/docs/profiling-framework.md @@ -49,8 +49,11 @@ Each profiling subsystem needs the same plumbing on the host: losing late entries. The AICPU producer side has a matching repeated shape: wait for ready-queue -space, publish a full buffer, wait for a free replacement, install it as the -current buffer, and account dropped records if bounded backpressure expires. +space, publish a full buffer, wait for a free replacement, and install it as the +current buffer. Both waits are resident block-on-contention gates — the writer +parks until the host clears the freeze — not bounded drops; records are only +lost if the 30-second host-crash backstop trips. See +[dfx/global-backpressure-design.md](dfx/global-backpressure-design.md). Before unification this was near-identical control flow repeated across collectors. The framework collapses the host side to one implementation @@ -162,8 +165,8 @@ Provides: start. Split drain threads do not bulk-mirror the whole shared-memory region; they refresh only their queue indices / entries before advancing `queue_heads`. On an empty scan, split drain does a short - busy-poll window before falling back to the 10 us sleep, so micro-bursts - are less likely to miss AICPU's bounded wait window. + busy-poll window before falling back to the 10 us sleep, so a lane parked at + a buffer-switch gate is released promptly instead of riding out the freeze. - Optional collector sharding (`Module::kMaxCollectorThreads` caps the shard arrays; the live shard count is the runtime `min(aicpu_thread_num, kMaxCollectorThreads)`) — each collector drains one host ready shard and @@ -272,10 +275,13 @@ schema, flush/finalize behavior, and small layout hooks locally. The engine currently provides: -- `wait_for_ready_queue_space` — bounded wait for a per-thread ready queue - slot. -- `wait_for_free_queue_entry` — bounded wait for a replacement buffer in a - free queue, with acquire ordering before reading `buffer_ptrs[]`. +- `wait_for_ready_queue_space` — block-on-contention park for a per-thread + ready-queue slot (push gate): raises `rq_contended` and holds until the host + clears `rq_freeze_active`, bounded only by the 30 s host-crash backstop. +- `wait_for_free_queue_entry` — block-on-contention park for a replacement + buffer in a free queue (pop gate): raises `fq_contended` and holds until the + host clears `fq_freeze_active`, with acquire ordering before reading + `buffer_ptrs[]`. - `enqueue_ready` — write the ready entry, `wmb()`, then advance `queue_tails[q]`. - `pop_free` — pop a free buffer, clear its count, install @@ -291,7 +297,7 @@ Each AICPU writer defines a local `XxxDeviceModule` trait with: | ------ | ------- | | `Context` | Per-call context such as header pointer, ready-queue thread, core/pool id, or local cache pointer | | `using DataHeader / State / FreeQueue / Buffer` | Device shared-memory layout types | -| `kReadyQueueSize`, `kSlotCount`, `kBackpressureWaitCycles` | Queue geometry and bounded wait budget | +| `kReadyQueueSize`, `kSlotCount`, `kBackpressureWaitCycles` | Queue geometry and the 30 s host-crash backstop for the block-on-contention gates | | `header(ctx)`, `ready_thread(ctx)`, `free_queue(state)` | Locate the shared header and queues | | `current_ptr/set_current_ptr`, `current_seq/set_current_seq` | Access the active device buffer state | | `count/set_count` | Access the active buffer's record count | @@ -409,10 +415,14 @@ Two things follow: SPSC by shard. Cross-shard return routing does not change ownership: replenish remains the only recycled-lane producer, and each drain shard remains its lane's only consumer. -- Device-side queue backpressure is bounded for the profiling writers that - use this protocol. If the host does not make ready-queue space or - free-queue entries visible within the short wait budget, AICPU records a - drop and keeps the workload moving instead of spinning indefinitely. +- Device-side queue backpressure is resident block-on-contention for the + profiling writers that use this protocol: on a full ready queue or empty free + queue the AICPU writer parks at its buffer-switch gate and loses no records + until the host clears the freeze. There is no opt-out and no short-wait drop; + the only bound is the 30-second host-crash backstop + (`PLATFORM_DFX_BACKPRESSURE_TIMEOUT_CYCLES`), and a record is dropped only if + that trips — i.e. the host is gone. Full state machine and its correctness + arguments: [dfx/global-backpressure-design.md](dfx/global-backpressure-design.md). - The AICPU writer publishes a full buffer to the ready queue before acquiring its replacement buffer. If no replacement is visible yet, the current pointer is cleared and later records first try to recover from diff --git a/src/a2a3/platform/include/common/platform_config.h b/src/a2a3/platform/include/common/platform_config.h index 4474f1cfcd..6fd20ab116 100644 --- a/src/a2a3/platform/include/common/platform_config.h +++ b/src/a2a3/platform/include/common/platform_config.h @@ -182,6 +182,14 @@ constexpr int PLATFORM_PROF_READYQUEUE_SIZE = */ constexpr uint64_t PLATFORM_PROF_SYS_CNT_FREQ = 50000000; // 50 MHz +/** + * Unified spin-wait timeout for DFX subsystem backpressure gates (system-counter + * cycles). Every DFX collector's park loop aborts after this budget on host + * crash / hardware failure. Expressed against PLATFORM_PROF_SYS_CNT_FREQ so it + * scales per arch and stays a 30 s wall-clock ceiling. + */ +constexpr uint64_t PLATFORM_DFX_BACKPRESSURE_TIMEOUT_CYCLES = PLATFORM_PROF_SYS_CNT_FREQ * 30; + /** * Timeout duration for performance data collection (seconds) */ diff --git a/src/a2a3/platform/include/common/pmu_profiling.h b/src/a2a3/platform/include/common/pmu_profiling.h index a5e2baaa32..79511b071e 100644 --- a/src/a2a3/platform/include/common/pmu_profiling.h +++ b/src/a2a3/platform/include/common/pmu_profiling.h @@ -31,6 +31,7 @@ #include #include "common/core_type.h" +#include "common/dfx_backpressure_device.h" #include "common/platform_config.h" // DAV_2201 hardware counter count. @@ -228,7 +229,8 @@ struct PmuDataHeader { volatile uint32_t queue_tails[PLATFORM_MAX_AICPU_THREADS]; // AICPU writes (producer) uint32_t num_cores; uint32_t event_type; // PmuEventType value, written by host at init - uint32_t pad[2]; + // DFX backpressure coordination (unified across all DFX subsystems). + DfxBackpressureHeader backpressure; } __attribute__((aligned(64))); // ============================================================================= diff --git a/src/a2a3/platform/shared/aicpu/pmu_collector_aicpu.cpp b/src/a2a3/platform/shared/aicpu/pmu_collector_aicpu.cpp index d60553b3ff..e4c47a1a39 100644 --- a/src/a2a3/platform/shared/aicpu/pmu_collector_aicpu.cpp +++ b/src/a2a3/platform/shared/aicpu/pmu_collector_aicpu.cpp @@ -49,7 +49,7 @@ static PmuDataHeader *s_pmu_header = nullptr; // Populated by pmu_aicpu_init(); 0 means "no PMU for this core" (sim). static uint64_t s_pmu_reg_addrs[PLATFORM_MAX_CORES] = {0}; -static constexpr uint64_t kPmuQueueBackpressureWaitCycles = PLATFORM_PROF_SYS_CNT_FREQ / 50000; // 20 us +static constexpr uint64_t kPmuQueueBackpressureWaitCycles = PLATFORM_DFX_BACKPRESSURE_TIMEOUT_CYCLES; extern "C" void set_platform_pmu_base(uint64_t pmu_data_base) { g_platform_pmu_base = pmu_data_base; } diff --git a/src/a5/platform/include/common/platform_config.h b/src/a5/platform/include/common/platform_config.h index 6783075a89..e89430e93d 100644 --- a/src/a5/platform/include/common/platform_config.h +++ b/src/a5/platform/include/common/platform_config.h @@ -209,6 +209,14 @@ constexpr int PLATFORM_PROF_READYQUEUE_SIZE = */ constexpr uint64_t PLATFORM_PROF_SYS_CNT_FREQ = 1000000000; // 1000 MHz +/** + * Unified spin-wait timeout for DFX subsystem backpressure gates (system-counter + * cycles). Every DFX collector's park loop aborts after this budget on host + * crash / hardware failure. Expressed against PLATFORM_PROF_SYS_CNT_FREQ so it + * scales per arch and stays a 30 s wall-clock ceiling. + */ +constexpr uint64_t PLATFORM_DFX_BACKPRESSURE_TIMEOUT_CYCLES = PLATFORM_PROF_SYS_CNT_FREQ * 30; + /** * Timeout duration for performance data collection (seconds) */ diff --git a/src/a5/platform/include/common/pmu_profiling.h b/src/a5/platform/include/common/pmu_profiling.h index e13cbccac5..5020a277bb 100644 --- a/src/a5/platform/include/common/pmu_profiling.h +++ b/src/a5/platform/include/common/pmu_profiling.h @@ -35,6 +35,7 @@ #include #include "common/core_type.h" +#include "common/dfx_backpressure_device.h" #include "common/platform_config.h" /** @@ -267,6 +268,8 @@ struct PmuDataHeader { volatile uint32_t queue_tails[PLATFORM_MAX_AICPU_THREADS]; // AICPU writes (producer) uint32_t num_cores; uint32_t event_type; // PmuEventType value, written by host at init + // DFX backpressure coordination (unified across all DFX subsystems). + DfxBackpressureHeader backpressure; uint32_t pad[2]; } __attribute__((aligned(64))); diff --git a/src/a5/platform/onboard/host/device_runner.cpp b/src/a5/platform/onboard/host/device_runner.cpp index 66c1e5b0d3..1b0a53d0d7 100644 --- a/src/a5/platform/onboard/host/device_runner.cpp +++ b/src/a5/platform/onboard/host/device_runner.cpp @@ -770,8 +770,8 @@ void DeviceRunner::finalize_collectors() { int DeviceRunner::init_l2_swimlane(int num_aicore, int aicpu_thread_num, int device_id) { int rc = l2_swimlane_collector_.initialize( - num_aicore, aicpu_thread_num, device_id, l2_swimlane_level_, prof_alloc_cb, /*register_cb=*/nullptr, - prof_free_cb, output_prefix_ + num_aicore, aicpu_thread_num, device_id, l2_swimlane_level_, prof_alloc_cb, + /*register_cb=*/nullptr, prof_free_cb, output_prefix_ ); if (rc == 0) { kernel_args_.args.l2_swimlane_data_base = diff --git a/src/a5/platform/shared/aicpu/pmu_collector_aicpu.cpp b/src/a5/platform/shared/aicpu/pmu_collector_aicpu.cpp index 101bae1cff..7250d48831 100644 --- a/src/a5/platform/shared/aicpu/pmu_collector_aicpu.cpp +++ b/src/a5/platform/shared/aicpu/pmu_collector_aicpu.cpp @@ -60,7 +60,7 @@ static PmuAicoreRing *s_pmu_aicore_rings[PLATFORM_MAX_CORES]; // Populated by pmu_aicpu_init(); 0 means "no PMU for this core" (sim). static uint64_t s_pmu_reg_addrs[PLATFORM_MAX_CORES] = {0}; -static constexpr uint64_t kPmuQueueBackpressureWaitCycles = PLATFORM_PROF_SYS_CNT_FREQ / 50000; // 20 us +static constexpr uint64_t kPmuQueueBackpressureWaitCycles = PLATFORM_DFX_BACKPRESSURE_TIMEOUT_CYCLES; extern "C" void set_platform_pmu_base(uint64_t pmu_data_base) { g_platform_pmu_base = pmu_data_base; } diff --git a/src/a5/platform/sim/host/device_runner.cpp b/src/a5/platform/sim/host/device_runner.cpp index ffd2152efb..1b22b0e16f 100644 --- a/src/a5/platform/sim/host/device_runner.cpp +++ b/src/a5/platform/sim/host/device_runner.cpp @@ -700,8 +700,8 @@ void DeviceRunner::finalize_collectors() { int DeviceRunner::init_l2_swimlane(int num_aicore, int aicpu_thread_num, int device_id) { int rc = l2_swimlane_collector_.initialize( - num_aicore, aicpu_thread_num, device_id, l2_swimlane_level_, prof_alloc_cb, /*register_cb=*/nullptr, - prof_free_cb, output_prefix_ + num_aicore, aicpu_thread_num, device_id, l2_swimlane_level_, prof_alloc_cb, + /*register_cb=*/nullptr, prof_free_cb, output_prefix_ ); if (rc == 0) { kernel_args_.l2_swimlane_data_base = diff --git a/src/common/platform/include/aicpu/profiler_device_engine.h b/src/common/platform/include/aicpu/profiler_device_engine.h index d761e01869..d6034e895e 100644 --- a/src/common/platform/include/aicpu/profiler_device_engine.h +++ b/src/common/platform/include/aicpu/profiler_device_engine.h @@ -14,6 +14,7 @@ #include #include "aicpu/device_time.h" +#include "common/dfx_backpressure_device.h" #include "common/memory_barrier.h" #include "common/platform_config.h" @@ -24,6 +25,14 @@ namespace profiling_device { // Module supplies the concrete shared-memory layout and subsystem-specific // ready-entry/drop hooks; this engine owns the queue handoff and buffer-switch // control flow. +// +// The push/pop gates here implement the device half of the block-on-contention +// backpressure protocol: on a full ready queue or empty free queue the writer +// parks at its buffer-switch gate (via dfx_backpressure_device.h) until the host +// clears the freeze, and only breaks to the single failure exit when the +// 30-second host-crash backstop (Module::kBackpressureWaitCycles) trips. Full +// design — dual-signal freeze, conjunction release, (0,0) escape-window and +// deadlock-freedom arguments — in docs/dfx/global-backpressure-design.md. template struct DeviceProfilerEngine { using Context = typename Module::Context; @@ -37,34 +46,63 @@ struct DeviceProfilerEngine { return false; } + // Push gate with incremental release support: + // 1. Check RQ slot availability (unified check, no duplication) + // 2. If slot exists, allow use even during freeze (incremental release) + // 3. Gate control happens in enqueue_ready() after push operation + // 4. No slot: mark contention and spin (triggers backpressure mechanism) + // Timeout-protected to prevent infinite spin on host crash or hardware failure + bool contended_signalled = false; const uint64_t start = get_sys_cnt_aicpu(); do { + // Unified RQ slot check (check once, use result for both freeze and non-freeze) uint32_t current_tail = header->queue_tails[thread_idx]; uint32_t current_head = header->queue_heads[thread_idx]; uint32_t next_tail = (current_tail + 1) % Module::kReadyQueueSize; + if (next_tail != current_head) { + // Slot available - return for both freeze and non-freeze cases + // enqueue_ready will handle the push and then wait for gate if needed *tail_out = current_tail; *head_out = current_head; return true; } - if (Module::kBackpressureWaitCycles == 0) { - break; - } + + // No slot available - mark contention and spin + // This triggers backpressure: freeze may open, RQ may drain + dfx_backpressure::mark_rq_contended(header, &contended_signalled); + + // Timeout protection if (get_sys_cnt_aicpu() - start >= Module::kBackpressureWaitCycles) { - break; + break; // timeout — fall through to the single failure exit below } + SPIN_WAIT_HINT(); } while (true); return false; } - static bool wait_for_free_queue_entry(FreeQueue *free_queue, uint32_t *head_out, uint32_t *tail_out) { + static bool + wait_for_free_queue_entry(DataHeader *header, FreeQueue *free_queue, uint32_t *head_out, uint32_t *tail_out) { if (free_queue == nullptr) { return false; } - const uint64_t start = get_sys_cnt_aicpu(); + // Pop gate. The loop, not any single call, is what holds a starved lane: + // 1. FQ slot available -> take it and return. + // 2. FQ empty -> raise fq_contended (leader signal, once per wait). + // 3. pop_freeze_barrier parks only while the host holds fq_freeze open. + // Raising fq_contended does not itself park this lane: until the host + // observes the signal and opens the freeze, the barrier sees + // fq_freeze_active==0 and returns at once, so this loop re-checks and + // bridges the host round-trip. Once frozen, the barrier spins here + // until release; its timeout arms only then and is the sole give-up + // (host dead/hung mid-freeze). + // 4. Freeze released -> loop re-checks and picks up the refilled slot. + bool contended_signalled = false; + do { + // Step 1: Check FQ slot availability uint32_t head = free_queue->head; uint32_t tail = free_queue->tail; if (head != tail) { @@ -73,12 +111,18 @@ struct DeviceProfilerEngine { rmb(); // acquire: order the tail read above before the caller's buffer_ptrs read return true; } - if (Module::kBackpressureWaitCycles == 0) { - break; - } - if (get_sys_cnt_aicpu() - start >= Module::kBackpressureWaitCycles) { - break; + + // Step 2: No slot available - mark contention + dfx_backpressure::mark_fq_contended(header, &contended_signalled); + + // Step 3: Park while the host holds the freeze open; returns at once + // when it is not open. Timeout fires only while frozen. + if (!dfx_backpressure::pop_freeze_barrier(header, Module::kBackpressureWaitCycles)) { + break; // gate timeout — fall through to the single failure exit below } + + // Step 4: Gate not held (never opened yet, or released) — re-check the + // free queue; an open→drain→refill cycle leaves a slot to pick up here. } while (true); return false; @@ -97,6 +141,17 @@ struct DeviceProfilerEngine { Module::write_ready_entry(ctx, current_tail, buffer_ptr, buffer_seq); wmb(); // publish: entry fields visible before the tail advance header->queue_tails[q] = next_tail; + + // Push-gate global-sync park: the held buffer is now flushed (nothing in + // hand), so blocking here is a clean, hostage-free stop. Every lane that + // reaches its push gate during rq_freeze converges here → one aligned + // common-mode gap; production resumes only after the host clears the + // freeze (conjunction release). + if (!dfx_backpressure::push_freeze_barrier(header, Module::kBackpressureWaitCycles)) { + // Gate timeout: the entry is already published to the ready queue, + // so the caller (switch_buffer) accounts the dropped records. + return -1; + } return 0; } @@ -108,7 +163,7 @@ struct DeviceProfilerEngine { FreeQueue *free_queue = Module::free_queue(state); uint32_t head = 0; uint32_t tail = 0; - if (!wait_for_free_queue_entry(free_queue, &head, &tail)) { + if (!wait_for_free_queue_entry(Module::header(ctx), free_queue, &head, &tail)) { return nullptr; } diff --git a/src/common/platform/include/common/args_dump.h b/src/common/platform/include/common/args_dump.h index 129909e2da..7af0cec9f3 100644 --- a/src/common/platform/include/common/args_dump.h +++ b/src/common/platform/include/common/args_dump.h @@ -43,6 +43,7 @@ #include #include +#include "common/dfx_backpressure_device.h" #include "common/platform_config.h" // ============================================================================= @@ -259,6 +260,8 @@ struct DumpDataHeader { uint64_t arena_size_per_thread; uint32_t magic; uint32_t dump_args_level; // DumpArgsLevel: 0=off, 1=partial, 2=full, 3=full_json_only + // DFX backpressure coordination (unified across all DFX subsystems). + DfxBackpressureHeader backpressure; } __attribute__((aligned(64))); // ============================================================================= diff --git a/src/common/platform/include/common/dep_gen.h b/src/common/platform/include/common/dep_gen.h index 60a81fc3c6..62cb64bbdd 100644 --- a/src/common/platform/include/common/dep_gen.h +++ b/src/common/platform/include/common/dep_gen.h @@ -45,6 +45,7 @@ #include #include "arg_direction.h" // CORE_MAX_TENSOR_ARGS +#include "common/dfx_backpressure_device.h" #include "common/platform_config.h" // ============================================================================= @@ -301,7 +302,8 @@ struct DepGenDataHeader { volatile uint32_t queue_heads[PLATFORM_MAX_AICPU_THREADS]; // Host reads (consumer) volatile uint32_t queue_tails[PLATFORM_MAX_AICPU_THREADS]; // AICPU writes (producer) uint32_t num_instances; // Always 1 for now - uint32_t _pad[3]; + // DFX backpressure coordination (unified across all DFX subsystems). + DfxBackpressureHeader backpressure; } __attribute__((aligned(64))); // ============================================================================= diff --git a/src/common/platform/include/common/dfx_backpressure_device.h b/src/common/platform/include/common/dfx_backpressure_device.h new file mode 100644 index 0000000000..d9aaa24177 --- /dev/null +++ b/src/common/platform/include/common/dfx_backpressure_device.h @@ -0,0 +1,164 @@ +/* + * Copyright (c) PyPTO Contributors. + * This program is free software, you can redistribute it and/or modify it under the terms and conditions of + * CANN Open Software License Agreement Version 2.0 (the "License"). + * Please refer to the License for details. You may not use this file except in compliance with the License. + * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, + * INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. + * See LICENSE in the root of the software repository for the full text of the License. + * ----------------------------------------------------------------------------------------------------------- + */ + +/** + * @file dfx_backpressure_device.h + * @brief Unified DFX backpressure (block-on-contention + global-sync freeze) + * coordination — shared layout + AICPU-side primitives. + * + * `DfxBackpressureHeader` is the per-subsystem coordination block. Every DFX + * subsystem's DataHeader embeds ONE `DfxBackpressureHeader backpressure;` member + * instead of re-declaring the three fields. It is physically per-subsystem (each + * subsystem has its own shared-memory region) — that is the per-subsystem peer + * group: swimlane's freeze is independent of pmu's. The struct *definition* is + * shared here; the *instances* are necessarily separate. + * + * The two device-side reactions (park at the gate / raise the leader signal) + * live here too. The host side of the mechanism (the freeze open/release state + * machine) lives once in `ProfilerAlgorithms::update_backpressure_freeze` + * (host/profiler_base.h). Every DFX subsystem carries the header block and runs + * the machinery; block-on-contention is the only behavior — it is idle at zero + * cost until a lane raises `contended`. + */ + +#ifndef SRC_COMMON_PLATFORM_INCLUDE_COMMON_DFX_BACKPRESSURE_DEVICE_H_ +#define SRC_COMMON_PLATFORM_INCLUDE_COMMON_DFX_BACKPRESSURE_DEVICE_H_ + +#include + +#include "aicpu/device_time.h" +#include "common/memory_barrier.h" + +// SPIN_WAIT_HINT() is the repo-wide AICPU spin-wait relax used by every +// resource-wait spin (ring/dep-pool full, lock_fanout ticket locks, scheduler +// dispatch). It is platform-tiered: onboard silicon expands to a no-op (AICPU +// owns its A55 core), sim adds sched_yield() so the oversubscribed host cores +// don't starve the AICore threads running real kernels. +// +// This header must NEVER (re)define SPIN_WAIT_HINT once a translation unit +// already has it. It is pulled — via each DFX DataHeader — into always-on +// scheduler / orchestrator / aicore TUs that already got the platform's +// sched_yield() definition (from inner_kernel.h / pto_runtime2_types.h). An +// unguarded `#define SPIN_WAIT_HINT() ((void)0)` here would clobber that on +// include order, turning those TUs' yielding spins into busy spins and starving +// the AICore threads (sim scheduler no-progress timeout). So: reuse whatever the +// platform already provides; only supply the fallback — for pure host, +// struct-only builds where spin_hint.h is off the path and the spin templates +// are never instantiated — when nobody else has defined it. +#ifndef SPIN_WAIT_HINT +#if __has_include("spin_hint.h") +#include "spin_hint.h" +#else +#define SPIN_WAIT_HINT() ((void)0) +#endif +#endif + +// Per-subsystem backpressure coordination block, embedded once in every DFX +// DataHeader. Two independent global freezes, one per gate — together they +// deliver #997's "block until RQ fully drained AND FQ fully refilled": +// rq_* : ready-queue-full (push gate). Host opens rq_freeze on rq_contended; +// every lane parks at its push gate; host releases once RQ is drained. +// fq_* : free-queue-empty (pop gate). Host opens fq_freeze on fq_contended; +// every lane parks at its pop gate; host releases once FQ is refilled to +// its initial upper limit min(kSlotCount, BUFFERS). +// freeze_active are host→device; contended are device→host (leader signal, host +// consumes + clears). Block-on-contention is the only behavior — no opt-out gate. +struct DfxBackpressureHeader { + volatile uint32_t rq_freeze_active; // host → device (push gate) + volatile uint32_t rq_contended; // device → host (ready-queue-full leader) + volatile uint32_t fq_freeze_active; // host → device (pop gate) + volatile uint32_t fq_contended; // device → host (free-queue-empty leader) +}; + +namespace dfx_backpressure { + +// Global-sync peer freeze at the push gate: while the host has opened rq_freeze, +// park here so every lane stops at its push gate (common-mode, lane-aligned gap) +// instead of one lane sparsifying and misleading bottleneck reading. Relaxes via +// SPIN_WAIT_HINT() (platform-tiered). Null-safe. Timeout-protected to prevent infinite spin on +// host crash or hardware failure. Returns false on timeout, true when gate opens normally. +template +inline bool push_freeze_barrier(const Header *header, uint64_t timeout_cycles) { + if (header == nullptr) { + return true; + } + const uint64_t start = get_sys_cnt_aicpu(); + while (header->backpressure.rq_freeze_active != 0) { + if (get_sys_cnt_aicpu() - start >= timeout_cycles) { + return false; // timeout: gate failed to open + } + SPIN_WAIT_HINT(); + } + rmb(); // acquire: order host's pre-release queue writes before the reads that follow the gate + return true; +} + +// Global-sync peer freeze at the pop gate: while the host has opened fq_freeze, +// park here so every lane stops at its pop gate. Symmetric to +// push_freeze_barrier for the free-queue side. Null-safe. Timeout-protected to prevent +// infinite spin on host crash or hardware failure. Returns false on timeout, true when +// gate opens normally. +template +inline bool pop_freeze_barrier(const Header *header, uint64_t timeout_cycles) { + if (header == nullptr) { + return true; + } + const uint64_t start = get_sys_cnt_aicpu(); + while (header->backpressure.fq_freeze_active != 0) { + if (get_sys_cnt_aicpu() - start >= timeout_cycles) { + return false; // timeout: gate failed to open + } + SPIN_WAIT_HINT(); + } + rmb(); // acquire: order host's pre-release queue writes before the reads that follow the gate + return true; +} + +// Leader signals (once per wait, via the `signalled` guard) when a lane hits +// real contention, so the host opens the matching global freeze that parks the +// peer lanes too. Idempotent sticky flags; the host consumes + clears them. +template +inline void mark_rq_contended(Header *header, bool *signalled) { + if (!*signalled && header != nullptr) { + header->backpressure.rq_contended = 1; + *signalled = true; + } +} +template +inline void mark_fq_contended(Header *header, bool *signalled) { + if (!*signalled && header != nullptr) { + header->backpressure.fq_contended = 1; + *signalled = true; + } +} + +// Leader park for a lane whose own reclaim depends on the host completing an +// open→drain→release cycle on the FREE-queue (pop) side — only tensor_dump's +// arena barrier today (engine-gate leaders instead spin on a real free slot). +// Blocks until the host has both opened fq_freeze covering this contention AND +// released it. Uses the DISJUNCTION (fq_contended || fq_freeze_active): the host +// opens fq_freeze before consuming fq_contended, so the predicate stays +// continuously true from mark_fq_contended() to release — no (0,0) escape +// window. Relaxes via SPIN_WAIT_HINT() (platform-tiered). Null-safe. +template +inline void wait_for_release(const Header *header) { + if (header == nullptr) { + return; + } + while (header->backpressure.fq_contended != 0 || header->backpressure.fq_freeze_active != 0) { + SPIN_WAIT_HINT(); + } + rmb(); // acquire: order host's pre-release writes before the leader's post-release reads +} + +} // namespace dfx_backpressure + +#endif // SRC_COMMON_PLATFORM_INCLUDE_COMMON_DFX_BACKPRESSURE_DEVICE_H_ diff --git a/src/common/platform/include/common/l2_swimlane_profiling.h b/src/common/platform/include/common/l2_swimlane_profiling.h index 4c0c10332e..d41488ab72 100644 --- a/src/common/platform/include/common/l2_swimlane_profiling.h +++ b/src/common/platform/include/common/l2_swimlane_profiling.h @@ -60,6 +60,7 @@ #include #include "common/core_type.h" +#include "common/dfx_backpressure_device.h" #include "common/platform_config.h" // ============================================================================= @@ -434,6 +435,9 @@ struct L2SwimlaneDataHeader { uint32_t num_orch_phase_threads; // Number of orch-phase pools the AICPU initialized uint32_t num_phase_cores; // Number of valid entries in core_to_thread (0 = unset) int8_t core_to_thread[PLATFORM_MAX_CORES]; // core_id → scheduler thread index (-1 = unassigned) + + // DFX backpressure coordination (unified across all DFX subsystems). + DfxBackpressureHeader backpressure; } __attribute__((aligned(64))); // ABI lock for the merged header. The phase metadata fields and the diff --git a/src/common/platform/include/common/scope_stats.h b/src/common/platform/include/common/scope_stats.h index 64fcfd7c18..48ef191672 100644 --- a/src/common/platform/include/common/scope_stats.h +++ b/src/common/platform/include/common/scope_stats.h @@ -47,6 +47,7 @@ #include #include +#include "common/dfx_backpressure_device.h" #include "common/platform_config.h" #define PTO2_SCOPE_STATS_MAX_RING_DEPTH 4 @@ -153,6 +154,8 @@ struct ScopeStatsDataHeader { volatile uint32_t queue_heads[PLATFORM_MAX_AICPU_THREADS]; // Host reads (consumer) volatile uint32_t queue_tails[PLATFORM_MAX_AICPU_THREADS]; // AICPU writes (producer) uint32_t num_instances; // Always 1 for now + // DFX backpressure coordination (unified across all DFX subsystems). + DfxBackpressureHeader backpressure; // Per-ring static capacities — written once by AICPU at orchestrator init // (scope_stats_set_ring_capacity / scope_stats_set_tensormap_capacity). diff --git a/src/common/platform/include/host/profiler_base.h b/src/common/platform/include/host/profiler_base.h index bc202c8772..464dd9beeb 100644 --- a/src/common/platform/include/host/profiler_base.h +++ b/src/common/platform/include/host/profiler_base.h @@ -435,7 +435,21 @@ struct ProfilerAlgorithms { ); return; } - (void)top_up_free_queue(mgr, site.kind, *site.free_queue, site.buffer_size, q); + + // Drain-driven free_queue top-up, suppressed while EITHER freeze is open so + // the mgmt replenish thread (conjunction release) is the sole free_queue + // writer. This drain read is not synchronized against the mgmt write, and + // try_pop advances queue_heads before it — so mgmt can observe RQ-empty and + // start its own top_up while a drain thread is here. The two writers stay + // apart by timing, not structure: the freeze_active=1 store precedes the + // whole RQ-drain window, so it is visible here well before the last drained + // entry. The fields stay plain volatile (they are also host→device shared, + // so cannot become std::atomic); a stale read only defers this top_up, or + // in the vanishing open-edge overlap duplicates a free_queue slot — + // bounding the worst case to DFX diagnostic loss, never compute data. + if (header->backpressure.rq_freeze_active == 0 && header->backpressure.fq_freeze_active == 0) { + (void)top_up_free_queue(mgr, site.kind, *site.free_queue, site.buffer_size, q); + } if (!mgr.push_to_ready(site.info, q)) { (void)mgr.retire_unqueued_buffer(site.kind, site.info.dev_buffer_ptr, q); @@ -495,6 +509,120 @@ struct ProfilerAlgorithms { return pushed; } + // DFX backpressure global-sync freeze state machine (host-owned). Runs each + // mgmt tick for every DFX subsystem; idle at zero cost until a lane raises + // contention. Two per-gate freezes, one per gate class (#997 "a thread blocks + // only at its buffer-switch gate"): + // rq_freeze (push gate): opened on rq_contended (ready-queue-full); every + // lane parks at its push gate. + // fq_freeze (pop gate): opened on fq_contended (free-queue-empty); every + // lane parks at its pop gate. + // Each freeze is opened independently by its leader, but BOTH release together + // on the conjunction #997 "block until RQ fully drained AND FQ fully refilled" + // — resuming into RQ-empty + FQ-at-limit is one clean common-mode gap. + // Deadlock-free at any pool size: RQ-drain is host-driven and independent of + // any held buffer, and FQ "refilled" is the attainable initial limit (pushed + // ==0), not the unreachable exact-kSlotCount fill. + // + // *_freeze_active are host→device, *_contended / queue_tails are device→host. + // Host writes go through write_range_to_device and device→host reads through + // read_range_from_device so a5 (non-SVM) sees them; a2a3 (SVM) short-circuits + // both to no-ops since the header already lives in shared device memory. + // extra_release_ready is the per-subsystem release predicate + // (Derived::backpressure_release_ready(), default true). The caller + // (mgmt_replenish_loop) evaluates it because this static has no Derived. + // tensor_dump uses it to hold the release until its collector has pulled all + // arena bytes (RQ-empty alone does not imply that — see buffer_pool_manager). + template + static void update_backpressure_freeze(Mgr &mgr, DataHeader *header, bool extra_release_ready = true) { + // --- Open each per-gate freeze on its own contention --- + // A thread blocks only at its own buffer-switch gate (#997 "same gate + // class"): push lanes park on rq_freeze (ready-queue-full), pop lanes on + // fq_freeze (free-queue-empty). Each is opened independently by its leader. + if (header->backpressure.rq_freeze_active == 0 && + mgr.read_range_from_device(&header->backpressure.rq_contended, sizeof(header->backpressure.rq_contended)) == + 0 && + header->backpressure.rq_contended != 0) { + header->backpressure.rq_freeze_active = 1; + wmb(); + mgr.write_range_to_device( + &header->backpressure.rq_freeze_active, sizeof(header->backpressure.rq_freeze_active) + ); + header->backpressure.rq_contended = 0; + mgr.write_range_to_device(&header->backpressure.rq_contended, sizeof(header->backpressure.rq_contended)); + LOG_WARN( + "%s DFX backpressure TRIGGERED: ready-queue-full, push-gate freeze OPENED — all AICPU lanes parked at " + "their push gate", + Module::kSubsystemName + ); + } + if (header->backpressure.fq_freeze_active == 0 && + mgr.read_range_from_device(&header->backpressure.fq_contended, sizeof(header->backpressure.fq_contended)) == + 0 && + header->backpressure.fq_contended != 0) { + // Open fq_freeze BEFORE consuming fq_contended so the disjunction + // (fq_contended || fq_freeze_active) that wait_for_release() spins on + // stays continuously true — no (0,0) escape window. + header->backpressure.fq_freeze_active = 1; + wmb(); + mgr.write_range_to_device( + &header->backpressure.fq_freeze_active, sizeof(header->backpressure.fq_freeze_active) + ); + header->backpressure.fq_contended = 0; + mgr.write_range_to_device(&header->backpressure.fq_contended, sizeof(header->backpressure.fq_contended)); + LOG_WARN( + "%s DFX backpressure TRIGGERED: free-queue-empty, pop-gate freeze OPENED — all AICPU lanes parked at " + "their pop gate", + Module::kSubsystemName + ); + } + + // --- Conjunction release (#997 "block until RQ fully drained AND FQ fully + // refilled") --- + // While either freeze is open, release BOTH only once RQ is drained AND + // the free queues are refilled to their initial upper limit. Resuming into + // RQ-empty + FQ-at-limit is one clean common-mode gap (no immediate + // re-stall). Deadlock-free regardless of pool size: RQ-drain is + // host-driven and independent of any buffer held at a push gate, and FQ + // "refilled" is pushed==0 (the attainable initial limit min(kSlotCount, + // BUFFERS)), not the unreachable exact-kSlotCount fill. Order matters: wait + // RQ-empty FIRST — once RQ is empty the drain threads are idle, so it is + // safe for THIS thread to top up the free queues (drain-driven top_up is + // suppressed while either freeze is open). + if (header->backpressure.rq_freeze_active == 0 && header->backpressure.fq_freeze_active == 0) { + return; + } + if (mgr.read_range_from_device(&header->queue_tails[0], sizeof(header->queue_tails)) != 0) { + return; + } + for (int q = 0; q < PLATFORM_MAX_AICPU_THREADS; q++) { + if (header->queue_heads[q] != header->queue_tails[q]) return; // RQ not drained + } + uint64_t pushed = replenish_free_queues(mgr, header); + if (pushed != 0 || !extra_release_ready) { + return; // FQ not yet refilled to its attainable initial limit + } + // Both conjuncts met → release both freezes together. + if (header->backpressure.rq_freeze_active != 0) { + header->backpressure.rq_freeze_active = 0; + wmb(); + mgr.write_range_to_device( + &header->backpressure.rq_freeze_active, sizeof(header->backpressure.rq_freeze_active) + ); + } + if (header->backpressure.fq_freeze_active != 0) { + header->backpressure.fq_freeze_active = 0; + wmb(); + mgr.write_range_to_device( + &header->backpressure.fq_freeze_active, sizeof(header->backpressure.fq_freeze_active) + ); + } + LOG_WARN( + "%s DFX backpressure RELEASED: ready-queue drained AND free-queues refilled — AICPU lanes resume", + Module::kSubsystemName + ); + } + private: static int recycled_warm_target(int kind, int shard_count) { return profiler_module_recycled_warm_target(kind, shard_count); @@ -622,6 +750,16 @@ class ProfilerBase { ProfilerBase(const ProfilerBase &) = delete; ProfilerBase &operator=(const ProfilerBase &) = delete; + // DFX backpressure per-subsystem release predicate (CRTP hook). Default: + // the freeze may release as soon as the framework's RQ-empty + FQ-full hold. + // A subsystem whose collector owns a separate, independently-overwritten + // region (only tensor_dump today: the payload arena) MUST override this to + // return false until its collector has drained that region, else the device + // resumes and overwrites not-yet-pulled data. Called on the mgmt_replenish + // thread inside the freeze loop — overrides must be cheap, non-blocking, and + // read only atomics. + bool backpressure_release_ready() const { return true; } + private: friend Derived; ProfilerBase() = default; @@ -1020,6 +1158,17 @@ class ProfilerBase { size_t drained = manager_.drain_done_into_recycled(); uint64_t replenished = Alg::replenish_recycled_pools(manager_, header); + // DFX backpressure global-sync freeze state machine. Runs on this + // single replenish thread — the state machine must be single-writer + // for freeze_active. Always active (block-on-contention is the only + // behavior); idle at zero cost until a lane raises `contended`. + // Per-subsystem release predicate (CRTP): default true, only + // tensor_dump overrides it (gate release on collector-quiesce); + // evaluated here because update_backpressure_freeze is a static with + // no Derived, and must be cheap + non-blocking (freeze hot loop). + const bool extra_release_ready = static_cast(this)->backpressure_release_ready(); + Alg::update_backpressure_freeze(manager_, header, extra_release_ready); + if (drained == 0 && replenished == 0) { std::this_thread::sleep_for(std::chrono::microseconds(10)); } diff --git a/src/common/platform/shared/aicpu/args_dump_aicpu.cpp b/src/common/platform/shared/aicpu/args_dump_aicpu.cpp index 8d87c064cf..df0a0ddf09 100644 --- a/src/common/platform/shared/aicpu/args_dump_aicpu.cpp +++ b/src/common/platform/shared/aicpu/args_dump_aicpu.cpp @@ -50,7 +50,7 @@ static inline void account_dropped_records(DumpBufferState *state, uint32_t drop state->dropped_record_count = (next < prev) ? UINT32_MAX : next; } -static constexpr uint64_t kDumpQueueBackpressureWaitCycles = PLATFORM_PROF_SYS_CNT_FREQ / 50000; // 20 us +static constexpr uint64_t kDumpQueueBackpressureWaitCycles = PLATFORM_DFX_BACKPRESSURE_TIMEOUT_CYCLES; static bool g_enable_dump_args = false; // Dump level latched from the header in dump_args_init(). The selective diff --git a/src/common/platform/shared/aicpu/dep_gen_collector_aicpu.cpp b/src/common/platform/shared/aicpu/dep_gen_collector_aicpu.cpp index bf879eb8e8..72191b8506 100644 --- a/src/common/platform/shared/aicpu/dep_gen_collector_aicpu.cpp +++ b/src/common/platform/shared/aicpu/dep_gen_collector_aicpu.cpp @@ -43,7 +43,7 @@ static DepGenDataHeader *s_dep_gen_header = nullptr; static DepGenBufferState *s_dep_gen_state = nullptr; static int s_orch_thread_idx = -1; // set via dep_gen_aicpu_set_orch_thread_idx -static constexpr uint64_t kDepGenQueueBackpressureWaitCycles = PLATFORM_PROF_SYS_CNT_FREQ / 50000; // 20 us +static constexpr uint64_t kDepGenQueueBackpressureWaitCycles = PLATFORM_DFX_BACKPRESSURE_TIMEOUT_CYCLES; extern "C" void set_platform_dep_gen_base(uint64_t dep_gen_data_base) { g_platform_dep_gen_base = dep_gen_data_base; } diff --git a/src/common/platform/shared/aicpu/l2_swimlane_collector_aicpu.cpp b/src/common/platform/shared/aicpu/l2_swimlane_collector_aicpu.cpp index 301a9fd1f5..fa65d22555 100644 --- a/src/common/platform/shared/aicpu/l2_swimlane_collector_aicpu.cpp +++ b/src/common/platform/shared/aicpu/l2_swimlane_collector_aicpu.cpp @@ -126,7 +126,7 @@ extern "C" uint64_t get_platform_l2_swimlane_aicore_rotation_table() { } L2SwimlaneLevel get_l2_swimlane_level() { return g_l2_swimlane_level; } -static constexpr uint64_t kL2SwimlaneQueueBackpressureWaitCycles = PLATFORM_PROF_SYS_CNT_FREQ / 50000; // 20 us +static constexpr uint64_t kL2SwimlaneQueueBackpressureWaitCycles = PLATFORM_DFX_BACKPRESSURE_TIMEOUT_CYCLES; struct L2SwimlaneTaskDeviceModule { struct Context { @@ -204,7 +204,7 @@ static L2SwimlaneTaskDeviceModule::Context l2_task_context( } static bool wait_for_free_queue_entry(L2SwimlaneFreeQueue *free_queue, uint32_t *head_out, uint32_t *tail_out) { - return L2SwimlaneTaskEngine::wait_for_free_queue_entry(free_queue, head_out, tail_out); + return L2SwimlaneTaskEngine::wait_for_free_queue_entry(s_l2_swimlane_header, free_queue, head_out, tail_out); } /** diff --git a/src/common/platform/shared/aicpu/scope_stats_collector_aicpu.cpp b/src/common/platform/shared/aicpu/scope_stats_collector_aicpu.cpp index 148e2e4285..2682dab701 100644 --- a/src/common/platform/shared/aicpu/scope_stats_collector_aicpu.cpp +++ b/src/common/platform/shared/aicpu/scope_stats_collector_aicpu.cpp @@ -45,7 +45,7 @@ static int s_orch_thread_idx = -1; // set via scope_stats_aicpu_set_orch_thread // unroll_heap_offset). Reset in set_platform_scope_stats_base. static uint64_t s_heap_wraps[PTO2_SCOPE_STATS_MAX_RING_DEPTH][2] = {}; -static constexpr uint64_t kScopeStatsQueueBackpressureWaitCycles = PLATFORM_PROF_SYS_CNT_FREQ / 50000; // 20 us +static constexpr uint64_t kScopeStatsQueueBackpressureWaitCycles = PLATFORM_DFX_BACKPRESSURE_TIMEOUT_CYCLES; namespace { diff --git a/tests/ut/cpp/common/test_buffer_pool_manager.cpp b/tests/ut/cpp/common/test_buffer_pool_manager.cpp index dad64292a7..87c400de5b 100644 --- a/tests/ut/cpp/common/test_buffer_pool_manager.cpp +++ b/tests/ut/cpp/common/test_buffer_pool_manager.cpp @@ -9,6 +9,7 @@ * ----------------------------------------------------------------------------------------------------------- */ +#include "common/dfx_backpressure_device.h" #include "host/buffer_pool_manager.h" #include "host/profiler_base.h" @@ -75,6 +76,7 @@ struct AlgorithmFreeQueue { struct AlgorithmHeader { AlgorithmFreeQueue free_queue; + DfxBackpressureHeader backpressure; }; struct AlgorithmReadyEntry { diff --git a/tests/ut/cpp/common/test_profiler_base.cpp b/tests/ut/cpp/common/test_profiler_base.cpp index b26313024c..fe9f3cff5a 100644 --- a/tests/ut/cpp/common/test_profiler_base.cpp +++ b/tests/ut/cpp/common/test_profiler_base.cpp @@ -20,6 +20,7 @@ * only one shard exists. Both shapes are covered here. */ +#include "common/dfx_backpressure_device.h" #include "host/profiler_base.h" #include @@ -56,6 +57,7 @@ struct TestHeader { volatile uint32_t queue_heads[PLATFORM_MAX_AICPU_THREADS]; volatile uint32_t queue_tails[PLATFORM_MAX_AICPU_THREADS]; TestFreeQueue free_queue; + DfxBackpressureHeader backpressure; }; struct TestReadyBufferInfo {