milnor_gpu: segmented no-copy resident master/basis (kill the realloc-doubling spike) - #18
Draft
JoeyBF wants to merge 120 commits into
Draft
milnor_gpu: segmented no-copy resident master/basis (kill the realloc-doubling spike)#18JoeyBF wants to merge 120 commits into
JoeyBF wants to merge 120 commits into
Conversation
…hanism" This reverts commit 71a21b6.
The relaxed wavefront keeps many bidegrees in flight at once, so at any instant it is likely that some job is inside a linear-algebra critical section (ParallelGuard). The scheduler re-spawned a bounced job immediately, which just re-checked is_in_parallel, found it still busy, and bounced again — spawning a whole rayon job per re-check and pegging every core on a retry storm that does no useful work. Instead the receiver checks the flag itself (a cheap atomic load) and parks a bidegree only when the section is genuinely busy. A job acquires and releases its guards many times and spends most of its time outside them, so the section frees far more often than jobs complete; parked bidegrees are therefore re-checked via a short recv_timeout while anything is parked, and re-spawned as soon as the section frees. Incoming messages are still handled the instant they arrive. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MCUtWj6P6suSZqvATCdg6d
is_in_parallel was a global count of active par_iter critical sections, so a step_resolution job bounced whenever *any* thread was in one. Under the relaxed wavefront many bidegrees are in flight, so that flag is almost always set and nearly every job bounced, producing the retry churn the parking mitigation only softened. The priority inversion the guard exists to prevent is narrower: a worker that initiated a par_iter blocks in the join and work-steals, and if it steals another (heavy, nested-parallel) resolution step, that step stalls the section the worker is blocked on. A stolen job runs on the stealer's own OS thread, so a thread-local depth counter reports exactly whether *this* worker is a blocked guard holder. Jobs picked up by a free worker read zero and run, letting independent bidegrees resolve concurrently instead of serializing behind any single critical section. The scheduler thread never holds a guard, so it can no longer read the flag to sense saturation; park bounced bidegrees and retry them on each completion or a short recv_timeout tick. Bounces are now rare (only a genuine steal-onto-a-blocked-holder), so the parking path barely engages. The classical scheduler shares the guard and benefits the same way, so its immediate-respawn no longer storms. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MCUtWj6P6suSZqvATCdg6d
Pins the invariant the previous commit relies on — a ParallelGuard held on one thread reads as absent on another — so a future change that reverts to a shared counter fails loudly instead of silently reintroducing the retry storm. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MCUtWj6P6suSZqvATCdg6d
…-graph-d69k0s' into hpc
…ead-local
The batched multiply serialized every launch behind one RESIDENT mutex held
across the whole marshal+upload+kernel+readback section, and additionally
pinned all work to CUDA stream 0. With the relaxed dependency graph exposing
~max_s-wide bidegree parallelism, that lock collapsed a ~12-core CPU wavefront
to ~2.6 busy cores and left the GPU idle 80% of the time — making NASSAU_GPU=1
a net 1.4x slowdown over CPU-only at stem 130 (193s vs 142s).
cubecl 0.10 does not need the lock: a per-device runner thread already
serializes all server access (concurrent client calls are memory-safe), and
memory pools are per-stream. So:
- RESIDENT becomes a thread_local RefCell: each rayon worker keeps its own
admissible cache and cs/mk device handles, created and consumed only on the
thread (and thus the default per-thread CUDA stream) that owns them, so no
handle ever crosses threads and no cross-stream event sync fires.
- The GPU_STREAM{value:0}.executes pin is removed; each worker launches on its
own default stream, so independent bidegrees marshal and execute
concurrently. memory_cleanup now trims only the calling worker's pool.
Stem 130 (S_2, s<=152, 16-core H200 box): 193s/2.6 cores (old mutex GPU) and
142s/10 cores (CPU-only) -> 44-49s/5.6 cores. Verified bit-identical to the
CPU path with NASSAU_GPU_VERIFY=1 at stem 80 (MIN_WORK=0, every build) and
stem 130 (default gate, all offloaded/chunked launches, concurrent workers).
Note: concurrency raises peak host memory (concurrent marshal buffers across
workers); a 16-worker VERIFY run at stem 130 exceeded a ~48GB cgroup, while
normal runs fit comfortably. Bound RAYON_NUM_THREADS if memory-constrained.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… resident master)
The mutex-removal commit let many workers run device sections concurrently, which
exposed three unbounded memory consumers at record stems (>100GB host AND device
by stem 150, measured):
1. Unbounded launch transients: the all-rows reuse build allocated its full output
in one shot, per in-flight worker. Fixed by splitting builds into row blocks
bounded by NASSAU_GPU_BLOCK_MB (default 512MB) of output AND GPU_PAIR_CHUNK
kernel threads — one launch per block, subsuming the former pair-chunk loop
(rows are independent, so blocks concatenate exactly).
2. Unbounded stream count: every worker thread got its own CUDA stream, and each
stream's pool retains freed slabs indefinitely. Fixed by NASSAU_GPU_CONCURRENCY
(default 8) permits that double as stream slots: at most 8 device sections run
at once, on 8 fixed streams. A permit must never be held across a rayon parallel
section (par_iter chunks execute on guard-free threads that can steal a bidegree
job which then parks on acquire — observed deadlock); it is acquired only for
the strictly sequential layout+device section. Do NOT raise to 16: measured
catastrophic (>30x) slowdown from cross-stream sync churn.
3. Per-thread resident duplication: the thread-local resident store copied the
admissible-matrix master (~8.5GB at stem 150, growing with degree) once per
worker, on host and device. Fixed by re-sharing it: host master behind an
RwLock (enumeration outside the write lock), one device mirror behind a small
mutex, handles shared across threads/slots (cubecl event-syncs cross-stream
reuse). Re-uploads are needs-based — only when a launch dereferences past the
uploaded prefix — since re-uploading on mere growth serialized multi-GB copies
on nearly every frontier launch (measured 1.5x wall regression).
Stem 150 (S_2, s<=152, 16-core H200 box), verified bit-identical to CPU at
stem 80 (every build, forced multi-block) and stem 130 (all offloaded launches):
wall cores host RSS device
before this commit 682s 3.3 137 GB 140 GB (full card)
after (32 workers) 721s 3.8 65 GB 37 GB
CPU-only reference 771s 10.3 4.4 GB —
Verdict: at record stems the GPU path now merely ties CPU-only while using far
more memory — the CPU path (no row-reuse matrix, per-signature builds) is both
frugal and wavefront-parallel. Recommend CPU-only for the stem-300 production
run; the GPU path remains correct, memory-bounded, and a real win at mid stems
(3x at stem 130).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replaces the count-based launch cap (NASSAU_GPU_CONCURRENCY=8 exclusive sections) with two decoupled controls: - NASSAU_GPU_MEM_BUDGET_MB (default 4096): admission weighted by a launch's output bytes, so dozens of small low-stem launches run concurrently again (the count cap throttled exactly the region that never had a memory problem) while the frontier stays bounded to ~budget/block-size in flight. - NASSAU_GPU_STREAMS (default 8): fixed CUDA stream slots, round-robin and SHARED (small launches serialize on a stream rather than demanding an exclusive one), so stream/pool count is bounded independently of concurrency. Master device uploads are now prefix-only with doubling: a launch ships max(need, 2*uploaded) entries, not the whole master, so frontier launches (which append new high-degree R each t) no longer re-ship gigabytes of untouched tail. Stem 130 improved 187s -> 150s; stem 150 memory 65/37 -> 68/30 GB, verified bit-identical (stem 80 all-builds, stem 130 all offloaded). But a slots x budget sweep is FLAT (8/4G=150s, 16/8G=170s, 32/16G=160s, 64/32G=201s): concurrency knobs are not the ceiling. The ceiling is Amdahl — the GPU accelerates only the Milnor multiply (~17% of frontier wall time; row_reduce/signature_matrix/readback dominate and are CPU/serial through cubecl's single runner thread), so the end-to-end GPU:CPU ratio is flat ~1.13x across the 130-150 heavy bands, not widening. Widening it would require offloading row_reduce (PR SpectralSequences#274's RREF). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…tiply path
The zero-signature image matrix (d_s applied to the zero-sig source basis,
column-masked to the zero-sig target) was the last per-bidegree Milnor multiply
still on the CPU — a serial per-row apply_to_basis_element_restricted, ~17% of
frontier wall time in the perf profile. But it is the *same* restricted multiply
as the QI-source `full_matrix` already built via restricted_partial_matrix_maybe_gpu,
just on d_s = differentials[b.s()] instead of d_{s-1}, and its target
mask/dimension are exactly the `target_mask`/`target_dim` already computed for the
bidegree (d_s and d_{s-1} share the target module modules[b.s()-1]). So route it
through the same GPU-offloaded, work-gated, already-verified path and apply the
column mask on CPU; drop the serial `signature_matrix` method. (Reinstates the
"signature_matrix offload" win from the original nassau_gpu branch, lost in the
SpectralSequences#272 relaxed-graph merge.) row_reduce stays on CPU — the signature-masked matrices
are very flat (~100 x 100000), a poor RREF target for the GPU.
Correctness: GPU Ext chart byte-identical to CPU-only through (100,152);
NASSAU_GPU_VERIFY passes at stem 130.
This shrinks the serial tail that Amdahl-capped the GPU:CPU ratio, so the
arithmetic-intensity advantage finally shows through and the gap WIDENS with stem
(S_2, s<=152, 16-core H200 box, w=32):
band GPU CPU ratio
130->140 159s 206s 1.30x
140->150 278s 423s 1.52x
cum 0->150 596s 771s 1.29x (was 723s, a near-tie)
Memory stays bounded by the same byte-budget/block machinery (this path reuses
multiply_batch_on_gpu). Next lever: the full-reuse-matrix readback.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… a host zero buffer Per-thread stack sampling at stem 145 showed the wavefront's serial stalls were a rayon worker pegged in __memcpy_ssse3 inside create_from_slice — host-side upload marshaling, NOT readback (cubecl 0.10 already does async D2H off pinned memory with the event wait on the worker thread, so the runner is free during the copy). The dominant offender: the batched multiply allocated + zeroed a host `vec![0u32; out_len]` (hundreds of MB at the frontier) and memcpy'd it up as the kernel's XOR accumulator, every launch/block. Allocate out_h uninitialized (client.empty) and zero it with a trivial on-device kernel (zero_u32), same stream as the multiply so it is ordered before it. Removes the host memset, the non-pinned host->device copy, and the transfer itself; on-device zeroing is memory-bound (microseconds on an H200). Verified: GPU Ext chart byte-identical to CPU through (100,152); NASSAU_GPU_VERIFY passes at stem 130. Bands (S_2, s<=152, 16-core H200 box, w=32), vs the prior signature-offload binary: 0->130 159 -> 141s (ties CPU-only 142; was a 0.89x loss) 0->140 318 -> 245s 130->140 marginal 104s vs CPU 206s = 1.98x (was 1.30x) peak RSS 46GB, GPU 28GB (both down). Next serial upload to check: term_pparts / the per-product record arrays. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The shared admissible master reaches ~3GB by stem 138 (cs 1.2GB + mk 1.9GB), and every launch took the RESIDENT_DEV mutex to read its device handles — with a growth-triggering launch doing a multi-GB create_from_slice re-upload *while holding that mutex*. Per-thread stack sampling showed the frontier collapsing to a single thread memcpy-ing gigabytes while every other bidegree blocked on the lock (upload byte-size instrumentation under NASSAU_GPU_DEBUG confirmed the master, not term data or the seqno table, as the giant upload). Make handle reads lock-free (RESIDENT_DEV: Mutex -> RwLock) and move the upload memcpy outside that lock, serialized only among uploaders by a separate RESIDENT_UPLOAD mutex with a re-check that coalesces a burst of growth-needing launches into one upload. A launch whose R's are already resident proceeds without ever blocking on someone else's upload. Verified GPU chart byte-identical to CPU through (100,152); NASSAU_GPU_VERIFY passes at stem 130. Removes the lock-held-across-copy stall, but occupancy only rose ~3.5->4.5 cores: the dominant limiter is upstream (thin GPU bidegrees + wavefront width), not this lock. Kept because it is correct and matters more at stem 300 where the master is larger. Also adds a per-buffer upload-size line to NASSAU_GPU_DEBUG. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…er-product alloc storm A frontier launch has ~1e5-1e6 products, and the marshal built term data as `Vec<(Vec<u16>, Vec<u32>)>` — two heap allocations per product (~1e6 tiny allocs per launch) — then extend-copied them into the flat upload buffers. Per-thread profiling of the GPU path showed this as a dominant chunk of the per-bidegree CPU "envelope" (~16% _int_malloc/_int_free plus the marshal copy) that wraps each (fast) kernel and, because the wavefront is only ~10-15 bidegrees wide, cannot be hidden — so the GPU sits idle between brief spikes. Precompute the term-count prefix sum (`term_off`), size the flat `term_pparts`/ `term_lens` once, and parallel-fill each product's disjoint slice in place (unsafe but sound: prefix-sum ranges never alias). The later layout loop just reads `term_off[pi]` for `prod_term_start` — no per-product allocation, no concat copy. Verified GPU chart byte-identical to CPU through (100,152). Same GPU results, far less allocation and marshal work per launch. (The remaining per-product `GpuProduct.term_indices: Vec<usize>` built in extract is the next alloc to flatten.) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…er-splitting The row-block splitter caps a launch at GPU_PAIR_CHUNK thread-pairs (the kernel indexes threads by u32 ABSOLUTE_POS, ceiling 2^32). It was set to 1<<30 (~1.07e9), ~4x below the real ceiling — so every billion-pair giant was chopped into ~4 launches, each a separate upload + kernel + BLOCKING readback round-trip, even though its output is only ~350 MB (well under gpu_block_bytes). Debug confirmed the giants pegged at 1.07e9 pairs; this, not the byte budget, was the binding split, which is why a NASSAU_GPU_BLOCK_MB sweep was flat. Raise it to 3.9e9 (leaves ~0.39e9 headroom under 2^32; the splitter always takes >=1 row and a lone row past 2^32 still trips the per-block u32::try_from assert; grid stays ~1.5e7 cubes, far under 2^31). Giants now run as a single launch (max total_pairs observed 3.90e9), collapsing 4 round-trips to 1. GPU 0->140 (w=100): ~245-288s -> 216s. Chart byte-identical to CPU through (100,152). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…dices The batched multiply re-gathered and re-uploaded every term's zero-padded p-part (`term_pparts`, width*2 bytes/term) on every launch — the dominant per-launch H2D transfer, plus a large parallel host gather. Make the basis itself resident on the device instead: build it once (grown incrementally as higher degrees appear, mirroring the admissible master) and upload only `term_gei[slot]`, the term's global basis-element index `global_base[s_degree] + ti` (4 bytes/term). The kernel reads the p-part from `basis_pparts[gei*width..]` with length `basis_lens[gei]`. At stem 140 the per-launch term transfer drops from ~width/2x larger to term_gei=95 MB, the basis is a one-time few-MB upload, and the per-term p-part gather is gone. Kernel change is minimal: params `term_pparts, term_lens` -> `basis_pparts, basis_lens, term_gei` (net +1 array arg), launch-arg order preserved 1:1 with the signature. `multiply_pair` is unchanged. An A/B toggle (`NASSAU_GPU_BASIS_PASSTHROUGH=1`) binds the per-launch term buffers as the "basis" with an identity index map, reproducing the old behaviour through the new kernel — so a single binary can isolate a kernel-signature bug from a resident host/upload bug. Both paths verified GPU==CPU per launch at stem 80 (MIN_WORK=0), and the resident path chart-matches CPU at (100,152). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
`NASSAU_GPU=1` could not use multiple CUDA streams: the shared resident admissible master + Milnor basis were re-`create_from_slice`d into a NEW device handle on every growth, and that handle churn broke cubecl's per-handle cross-stream synchronization, so a launch on one stream could read the master while another was mid-upload -> wrong multiply (`dx != 0`), "Memory page" panic, or hang. Fix: make the resident buffers STABLE and grow them IN PLACE — the read-only shared global (model-weights) pattern cubecl supports across streams. A small `copy_into_*` kernel writes the new tail (uploaded to scratch via `create_from_slice`) at the buffer's append offset; the handle changes only on a rare capacity doubling, which is barrier-protected (`RESIDENT_REALLOC`: device sections hold the read lock across the multiply, a realloc takes the write lock and quiesces them). Each worker gets a stable per-thread stream id (`thread_stream_id`); default `NASSAU_GPU_STREAMS = 8`. Key gotcha (a stem-150 `dx != 0`): cubecl's `ArrayArg` length is u32, so a buffer of exactly 2^32 elements truncates to length 0 and the copy writes nothing (buffer reads all zeros). The doubling `cap` jumped 2^31 -> 2^32 right at stem 150's `masks` size. Capacity is clamped to `RESIDENT_MAX_CAP = 2^32 - 1`; a single resident buffer cannot exceed that (a larger master needs splitting — the old create_from_slice path had the same limit). Copies are chunked under the u32 ABSOLUTE_POS thread limit. Validated: VERIFY (GPU==CPU per launch) at 8 streams, chart-match to CPU, 0 dx-crashes over many stem-150 reps at 1 and 8 streams. Perf note: multi-stream is correct but ~neutral vs single-stream at stem 140/150 (the per-device runner serializes kernel submission); it may help at higher stems with a wider heavy-bidegree wavefront. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
Instrument Matrix::row_reduce to emit a `fp::rr` tracing event for every p=2 reduction with min(rows,cols) >= 1024, recording rows/cols/min and whether the device RREF was taken (path="gpu") or it fell back to CPU M4RI (path="cpu"). The event inherits the active nassau span so each line carries its bidegree/signature context. Confirms on a stem-150 run that every reduction with min >= 8192 (up to 25091x30275, incl. the heavy zero-signature base solves) dispatches to the GPU with no fallbacks; everything below the 8192 FP_CUDA_RR_THRESHOLD stays on CPU. tracing is added as a gpu-gated optional dep. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
The intermittent stem-150 hang (flat sm=100%, ~1/6 of runs, always on the zero-signature base solve) was a deadlock in the fp-cuda cooperative row-reduce kernel `panel_factor_coop`. Its grid-wide spin barrier (launched via `cuLaunchCooperativeKernel`) requires ALL its CTAs co-resident, but the algebra Milnor multiply runs on a *separate* CUDA runtime (cubecl) whose kernels concurrently occupy SMs. When a cooperative row-reduce launched while cubecl multiply kernels were resident, its CTAs could not all co-reside; the missing ones never reached the barrier and the resident ones spun forever. Fix: a cross-runtime `fp::GPU_EXCLUSIVE` RwLock. The cooperative row-reduce takes the write lock (drains in-flight cubecl multiplies, then runs with the GPU to itself) around its launch+download; every cubecl multiply takes the read lock across its whole device section (launch through readback, so releasing means the kernel has actually completed). Readers run concurrently; a pending row-reduce briefly excludes them. No lock cycle (cubecl never takes the fp-cuda ctx lock) and no same-thread read->write (a solve's multiply releases before its row-reduce). Also adds a `gpu_row_reduce` tracing span around the GPU reduce -- the diagnostic that localized the wedge (an unclosed span names a stuck reduce, distinguishing an RREF hang from a multiply hang). Validated on H200: stem-150 x16 with 0 wedges (baseline ~1/6), wall time unchanged (261-359s), GPU-vs-CPU chart match preserved. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
The device RREF launched three cooperative kernels — panel_factor_coop, promote_coop, block_reduce_coop — synchronized by a hand-rolled grid-wide spin barrier. cuLaunchCooperativeKernel requires all the grid's CTAs to be co-resident, which only holds when this process owns the whole GPU. When another CUDA runtime shares the device (cubecl's Milnor multiply in the nassau GPU resolution), its kernels occupy SMs, the reduce's CTAs can't all co-reside, the missing ones never reach the barrier, and the resident ones spin forever — the intermittent stem-150 wedge (flat sm=100%). Add an FP_CUDA_RR_COOP switch (rr_coop()). Default off: the forward pass runs the single-CTA panel_factor one limb at a time, promotion uses the grid- strided promote_pivots, and back-substitution uses the single-CTA block_reduce_rref — none launched cooperatively, so the reduce composes with concurrent GPU work. Set FP_CUDA_RR_COOP=1 to opt into the cooperative path on a dedicated GPU (measured 2-3x faster at Nassau strides, up to ~10x on large dense half-rank matrices). Both paths validated bit-exact vs CPU row_reduce (row_reduce_demo) and vs the CPU BLAS3 oracle at 2^16/2^17 (reduce_pow2_half). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
The composable default from the previous commit fell back to single-CTA kernels
(panel_factor, block_reduce_rref) that use one SM of ~132 — 8-10x slower than the
cooperative path on large dense matrices, since the cooperative kernels' real win
is spreading each sequential bit-step across the whole grid.
Recover that all-SM parallelism WITHOUT a cooperative launch by replacing the
in-grid grid_sync with kernel-boundary (stream-ordered) synchronization, the way
cuSOLVER/cuBLAS build grid-wide multi-step algorithms:
- panel_factor: pf_find -> pf_swap -> pf_xor per bit-step (panel_factor_streamed)
- block_reduce: br_cond -> br_xor per pivot (block_reduce_elem streamed arm)
All per-step state stays on the device (pivot count, find-first result, pivword,
clear-conditions), so the host issues every launch without a readback and the
latency hides behind GPU work; only the final (pr, pivcols) is copied back. No
cuLaunchCooperativeKernel anywhere on the default path, so it can't deadlock
against a concurrent cubecl kernel. The streamed back-substitution also adopts the
cooperative path's wide-block TRSM (bp=1024 + X.U GEMM) since the GEMM composes.
Perf (H200 half-rank square, device-only) vs cooperative:
2^13 s128 0.080 vs 0.066 (1.21x) 2^15 s512 0.642 vs 0.577 (1.11x)
2^14 s256 0.195 vs 0.166 (1.17x) 2^16 s1024 2.27 vs 1.17 (1.95x)
2^17 s2048 8.63 vs 4.52 (1.91x)
Within ~1.1-1.2x of cooperative through stride 512 (the Nassau regime; the wedge
matrix was stride 215) and 3-6x faster than the single-CTA fallback at large
sizes. The residual past stride 1024 is where cooperative additionally fuses
promote/block-reduce; closable with CUDA graphs if those sizes ever matter.
Both paths validated bit-exact vs CPU row_reduce (row_reduce_demo) and the CPU
BLAS3 oracle at 2^16 (reduce_pow2_half).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
Merge pf_find and pf_swap into pf_find_swap using a threadfence "last-CTA finalize" (a grid-wide reduction, not a barrier, so no co-residency needed), cutting the streamed forward pass from 3 to 2 launches per bit-step and removing the serial 1-thread swap kernel. Cap the streamed grid at FP_CUDA_PF_CTAS (128) like the cooperative kernel. Add FP_CUDA_RR_TIMING to split forward/back timing. Diagnostic result: the streamed forward pass is work-bound (~cols^2), not launch-bound — merging, capping, and grid size leave it unchanged — so it is the per-step relaunch efficiency vs the persistent cooperative grid, ~2.7x at 2^17 but shrinking with size (3.0x at 2^16). Still bit-exact vs CPU row_reduce and the BLAS3 oracle at 2^16. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
pf_step does column cc-1's clear and column cc's find+swap in one launch, since the forward sweep alternates xor(col j) / find(col j+1) over the same below-row range — so each thread reads its row once and sees its own XOR before scanning. Halves the forward pass's launches (one per column instead of two). Marginal on its own (~5%), confirming the forward-pass gap vs cooperative is the per-step GPU relaunch cost of non-persistent kernels, not launch count — it resists launch reduction and only closes with scale (the O(cols) term vanishing against O(cols^2) work): ~2x total at 2^17, ~1.4x at 2^18. Bit-exact vs the CPU BLAS3 oracle at 2^16. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
The cross-runtime fp::GPU_EXCLUSIVE RwLock existed only to keep the cooperative fp-cuda row-reduce (cuLaunchCooperativeKernel grid barrier) from co-scheduling against a concurrent cubecl Milnor multiply, which could prevent CTA co-residency and deadlock the grid barrier (the stem-150 wedge). The default row-reduce is now the streamed kernel-boundary path (no cooperative launch anywhere), so it composes with concurrent GPU work by construction — the lock is unnecessary. Remove the RwLock, its fp re-export, and the read-side guard in the algebra Milnor-multiply device section. The gpu_row_reduce tracing span is kept (a useful wedge/hang diagnostic). FP_CUDA_RR_COOP=1 still selects the cooperative kernels for a dedicated GPU; do not combine that with the concurrent nassau multiply. Validated: stem-150 resolves in 242s with 30042 GPU row-reduce events and no wedge (full hunt in progress). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
The batched Milnor multiply silently corrupted results once the shared `masks` admissible-master crossed 2^32 u16 elements (~8.6 GB, around S_2 stem ~160-180): the buffer length and the per-R offsets were u32, so cubecl's default U32 `address_type` truncated them (`len as u32`), giving out-of-range reads and `dx != 0` (d^2 != 0) panics at e.g. (180,92). (160,82) stayed just under the ceiling and was clean. Fix uses cubecl 0.10's first-class 64-bit addressing rather than hand-rolled buffer splitting: - `multiply_batch_kernel` -> `#[cube(launch_unchecked, address_type = "u64")]`. Static u64 (not "dynamic"): dynamic picks u32 `usize` for small blocks and then narrows the u64 offset arrays on read (`usize::cast_from(u64)` under a u32 address type), corrupting results. `launch_unchecked` because checked mode emits `min(u64, u64)`, which NVRTC rejects as an ambiguous overload; every access is in-bounds by construction (uploaded `need_*` prefix + per-column `j` guards). - `RInfo.cs_off/mk_off` u32 -> u64; `r_cs_offset`/`r_mk_offset` bound as `Array<u64>`. - Resident grow copies (`copy_into_u16`/`_u32`) -> `#[cube(launch_unchecked, address_type = "dynamic")]` (scalar usize offsets adapt without narrowing; dynamic keeps small copies on u32). - Drop `RESIDENT_MAX_CAP` and its clamps (the u32 ceiling is gone); assert `out_len <= u32::MAX` loudly (the row-block splitter guarantees it). Validated on H200: (160,82) rc=0 clean, 517s vs 486s baseline (+6.4% for the static-u64 multiply, partly offset by unchecked dropping the bounds clamp); (180,92) ran 1500s with zero dx panics (previously a deterministic panic ~130-180s), i.e. correct well past the 2^32 masks boundary. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
…he resident host master `resolve_through_stem S_2 "" 180 92` was OOM-killed at the 500 GB cgroup limit (~280 GB anon + ~240 GB shmem). Measurement (env `NASSAU_MEM_REPORT`, plus a pure-CPU control at ~5-7 GB) showed the resolution's own data — differentials and module tables — is only ~1.5 GB; the ~500 GB was entirely GPU-runtime host memory. Two fixes cut it to ~90 GB at stem 180 (dx-clean): 1. Default `NASSAU_GPU_STREAMS` 8 -> 1. cubecl gives each CUDA stream its own page-locked (pinned) host pool that `memory_cleanup` never trims; ≥2 streams ballooned pinned host memory to 140-240 GB, single-stream holds it at ~4-6 GB. The payoff of multi-stream is ~nil here: cubecl's server is single-threaded, so extra streams buy no CPU concurrency, only GPU kernel overlap the big saturating multiplies barely use. Override for a dedicated large-RAM node. 2. Stop retaining the resident admissible master host-side. `RESIDENT_HOST` kept the full `col_sums`/`masks` (~54 GB at stem 180) forever, duplicating the device copy, even though after upload it is never read again (offsets come from `index`; growth uploads only the new tail; a capacity realloc copies the old *device* buffer). Now it keeps only the not-yet-uploaded tail (`*_pending`, ~sub-GB) plus a logical length, freeing each `R`'s data the moment it reaches the GPU — invariant `dev.uploaded == len - pending.len()`. Also: bound the pinned staging on resident growth to `STAGE_CHUNK` chunks, and a gated `NASSAU_MEM_REPORT` (differentials/modules/resident heap breakdown) for ongoing memory work. Validated on H200: stem 180 peak RSS ~90 GB (was 500 GB OOM), dx-clean through the >2^32 masks region and the heavy solves. Remaining growth for higher stems is real GPU working memory (concurrent dense output matrices) + GPU device memory, not retained host duplicates. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
…unded Even with the resident host-master freed, host RSS still grew ~linearly toward the cgroup limit at high stems — not a retained duplicate but the GPU path materializing whole dense output matrices. The CPU walks signatures one at a time (small working set); the GPU offload built the entire matrix at once and held its dense readback (`num_rows × num_limbs` u32, ~12 GB regions at stem 180) alongside the assembled matrix. Two caps bound it: 1. `reuse_full_matrix` now only builds the all-rows matrix when `rows × cols <= NASSAU_GPU_REUSE_MAX_WORK` (default 1e10). Above that it falls back to per-signature builds (each a bounded row subset, like the CPU), so the peak scales with the largest single signature, not the whole bidegree. 2. `get_partial_matrix_restricted` processes rows in batches sized so the dense readback stays under `NASSAU_GPU_MAX_READBACK_MB` (default 1024). Products are built in row order, so each batch is a contiguous slice with its `row` remapped batch-local; results XOR back into the global rows. The readback is freed between batches. Both are correctness-neutral (rows are independent): verified bit-for-bit vs the CPU under `NASSAU_GPU_VERIFY` with a 1 MB cap (many batches/build), 0 mismatches. Effect at stem 180 (streams=1 + resident de-dup + this): host anon goes from growing past 85 GB to a bounded ~18-24 GB, with no throughput loss — cross- bidegree rayon concurrency keeps the GPU fed, and the default cap only splits the few giant high-t builds into a handful of chunks. Peak RSS ~30 GB (was 500 GB OOM). This bounds the host working set so it scales toward higher stems; a D-deep async launch/collect pipeline (cubecl launches are async; only read_one blocks) can hide chunk latency if a much smaller cap is ever needed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
The resident admissible master (col_sums/masks) grows with degree and is the stem-300 device-memory wall: ~34 GB at stem 180, extrapolating past the H200's 143 GB before stem 300. This adds an opt-in eviction that bounds it. Policy (from the NASSAU_R_STATS probe): a *degree* threshold, not LRU. Low-degree R's are the stable hot core (reference span 0.99 of the run); high-degree R's are scattered-recurring (0.81) and also the biggest matrices, so a degree cap evicts the most bytes for the fewest references and is stem-independent (the kept set saturates). Env NASSAU_GPU_RESIDENT_MAX_DEGREE (default i32::MAX = keep all). Design — no kernel change. Every output row's products share one operation R (extract_restricted), so hot (deg<=cap) and cold rows are DISJOINT. multiply_batch_on_gpu partitions products, compacts each group to its own dense row range (so total readback stays num_rows, not 2x), runs the resident pass on hot and a Transient pass on cold, and scatters results back. Transient builds a per-block master with create_from_slice, freed with the launch, so the device copy never persists. Cold admissible data is cached host-side (COLD_HOST) so the expensive recompute is one-shot, like the resident path. Fast path (cap==MAX or all-hot) is byte-identical to before — zero regression by default. GpuProduct gains Clone for the partition. Validated bit-exact: NASSAU_GPU_VERIFY full S_2 stem-110 resolution at theta=10 (transient path heavily stressed) and theta=100: mismatches=0 dx=0 panics=0. Memory vs throughput (stem 180, ExclusivePages, streams=1): control (no eviction) 51 GB / 1349 s; theta=125 11 GB; theta=100 12 GB (master 34 -> ~1 GB), dx=0. But eviction costs 2-4x throughput even at a high theta (stem 150: control 282 s, theta=140 613 s) because the evicted high-degree matrices are the biggest and are re-uploaded every launch. So this is a fit-in-memory lever for stems where the master won't fit at all (a slow completion beats an OOM), tuned to the highest theta that fits; it is NOT a speedup. Next optimization to cut the re-upload: a bounded LRU device-side cold cache (upload once, reuse across a bidegree's launches). Also retained: the R-access probe (NASSAU_R_STATS/dump_r_stats) and the device/host [MEM] report (NASSAU_MEM_REPORT) used to characterize this — both env-gated no-ops. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
Foundational step toward generating admissible matrices (col_sums/masks) ON the GPU into transient scratch instead of storing/uploading the resident master -- the direction that eliminates both the stem-300 device-memory wall and the eviction re-upload cost (a given launch would enumerate its cold R's on-device, never uploading them). enumerate_admissible_ref reimplements AdmissibleMatrix::next using ONLY flag-guarded control flow -- no break, continue, or early return -- because that is the subset the cubecl DSL compiles cleanly (cf. multiply_pair, which tracks a `rejected` flag rather than breaking). `found` replaces the odometer's `return true`, `handled` replaces its `continue 'mid`. This validates the tricky restructuring on the CPU, where it is fast to debug, before the hard-to-debug #[cube] port -- which then becomes a mechanical transcription onto per-thread local Arrays (state is tiny: rows = |p_part|, cols <= 32). Test admissible_enum_ref_matches asserts bit-exact equivalence with admissible_matrices over every real R up to degree 60 (4155 R's, all match). Not yet ported to cubecl / not wired into the multiply -- next step. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
Transcribe enumerate_admissible_ref into a cubecl #[cube] kernel that generates each R's col_sums/masks for every admissible matrix directly into device scratch -- the on-GPU replacement for the resident/uploaded master. One thread per distinct R, all state in fixed-size per-thread local Arrays; flag-based control flow (while ... && !found / handled) since the DSL has no break/continue/return. Backend-agnostic host driver (generic over Runtime) so the identical kernel can run on CUDA and the cpu backend. New test admissible_enum_gpu_matches runs it on the H200 and asserts bit-exact output (values + per-R counts) vs the CPU reference: 1055 R's, 30385 matrices, all matching. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
Wire enumerate_admissible_kernel into the Transient (evicted) cold path of multiply_batch_block. A cold R's col_sums/masks are now GENERATED on the GPU into transient scratch (one enumeration launch, ordered before the multiply on the same stream) rather than built on the host and uploaded via create_from_slice. Only the small p-parts + per-R dimensions upload; the scratch is freed with the launch. This kills the per-launch H2D master re-upload that made eviction a 2-4x slowdown (bench 2026-07-27), the whole point of the in-kernel-enumeration direction. Also replace the COLD_HOST full-array cache with COLD_COUNT, a 12-byte-per-R (cs_len, mk_len, num_mats) shape cache. The evicted tail of the master now lives neither on the device nor the host -- only its sizes, needed up front to lay out the scratch offsets and the pair-count prefix sum. num_mats is counted once per distinct R (admissible_matrices, arrays dropped) and memoized. De-gate enumerate_admissible_kernel + ENUM_* caps + the MAX_XI_TAU import for production. Validated bit-exact: the isolation test (1055 R's / 30385 matrices) still passes, and NASSAU_GPU_VERIFY full S_2 stem-110 resolutions at theta=10 (nearly all R's cold -> enumeration path stressed) and theta=100 both report mismatches=0 dx=0 panics=0 rc=0. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
…refuted) Investigation of the eviction crash + a throughput bench that together show in-kernel enumeration cannot beat upload-based eviction: - enumerate_admissible_kernel -> u64 addressing (was default u32); did NOT fix the big-block CUDA_ERROR_LAUNCH_FAILED, so the fault is elsewhere in the Transient wiring, not the kernel. - admissible_enum_gpu_matches extended to degree 145, chunked per-degree: proves the kernel bit-exact vs the CPU reference (144903 R's, 185M matrices) across the full range the eviction path exercises -- so the kernel is correct. - bench_admissible_cpu_vs_gpu (ignored): CPU admissible_matrices 1.61s vs GPU kernel-only 2.02s (0.8x, SLOWER) for degrees 1-130. GPU enumeration is ~3x slower than just transferring the same arrays (0.68s readback) it replaces: the odometer is sequential per R with matrices-per-R spanning 1..millions, so the launch bottlenecks on its few longest threads at GPU scalar speed. The "trade GPU integer work for PCIe bandwidth" premise is refuted -- enumeration worsens the eviction re-upload cost instead of curing it. The enum wiring in multiply_batch_block's Transient path still faults at scale and should be reverted to upload-based eviction; kept here as validated, dormant code documenting the dead end. Default (theta=inf) path is unaffected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
The persistent helpers measured 2499s against 2412s for the thread::scope fan-out they replaced — 87s, and real rather than scatter: two near-identical serial configs came in at 3136s and 3138s, so run-to-run variance here is about +/-2s. The suspect is wake latency. A freshly spawned thread starts running immediately on a warm core; a parked helper must be woken from a blocking recv, which is a futex wake plus whatever placement the scheduler picks, and 384 mostly-idle helpers pay that on every block. std's mpsc parks almost at once; crossbeam-channel spins briefly first, and it is already a gpu-feature dependency. Applied to both directions — job dispatch and the result hand-back — since the caller blocks on the latter for every shard it farms out. 75/75 tests pass on 4 devices. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
This reverts 38bbdd7. Measured worse, on verified complete stem-200s (max_t=310, closed=28013, pairs=5.87e13, zero crashes): thread::scope spawn per block 2412s persistent helpers, std mpsc 2499s persistent helpers, crossbeam 2612s The theory was that the 87s the persistent helpers cost against thread::scope was futex wake latency, and that crossbeam's spin-before-park would recover it. It cost a further 113s instead. With 384 mostly-idle helpers, spinning appears to take CPU the marshalling threads need rather than saving a wake. Back to std mpsc at 2499s, which is the configuration to keep: +87s against thread::scope, in exchange for ~900k fewer spawn/join cycles per run and a thread count that derives from the rayon pool. 75/75 tests pass on 4 devices. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
Frontier GPU load was 50.5% on device 0 against ~15% on the other three at stem 210 — a 3.3x spread, up from 1.18x at stem 200, so it worsens with scale and three of four devices sat ~85% idle. Cause is in the policy's own justification. `dev_load` balances accumulated `num_mats`, which its doc defends by assuming `R`s are "used at broadly similar rates". The NASSAU_R_STATS probe at (150,110) says otherwise: of 173930 distinct `R`s, the top 1% carry 31% of references and the top 10% carry 78%. Equal `num_mats` therefore says little about equal work. Two things compounded it: assignment is permanent, and `min_by_key` broke ties toward device 0, which is where the early low-degree `R`s landed while every device still had near-zero load. Hashing balances neither count nor bytes on purpose — it draws each device an independent sample of the joint (size, reference-rate) distribution, and with ~174k `R`s and no single one above a fraction of a percent of references, both concentrate. It is also stateless: no accumulator under the write lock, no first-sight ordering, no tie-break. The mixing is load-bearing. PPart packs entry i at a fixed bit offset, so the low bits are r_1, which tracks internal degree, which tracks work (the probe's hot decile averages degree 70 vs the cold decile's 14). A bare bits() % 4 would partition by r_1. Measured over 168781 real `R`s: worst deviation 0.65% mixed vs 4.55% unmixed. The new test asserts both the overall balance and balance WITHIN degree bands — a hash uniform overall but skewed per band would still starve devices for long stretches, since the frontier walks degree upward. 76/76 tests pass on 4 devices. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
`MasterMode::Transient` — the `NASSAU_GPU_RESIDENT_MAX_DEGREE` eviction path that is the intended stem-300 memory lever — had no end-to-end test. `admissible_enum_gpu_matches` validates the enumeration kernel through a test-only harness, which is a different launch path: it says nothing about whether the scratch that kernel writes is laid out the way `multiply_batch_kernel` indexes it. Every existing batch test took the default `cap == i32::MAX` fast path and never entered the eviction code at all. `multiply_batch_gpu_inner` now takes the cap as an argument rather than reading the process-wide `LazyLock`, so one process can exercise all three regimes, and `multiply_batch_matches_reference_under_eviction` runs the same batch through all-resident, all-transient and the mixed two-launch split against one CPU golden, at both low and high degree. It passes in every regime (7684 products at degree 72), so the split logic and the on-device enumeration feeding the multiply are sound. Separately, `gpu_count()` counted `/proc/driver/nvidia/gpus` — the devices physically in the node, not the ones visible to the process. CUDA renumbers the visible subset to `0..n`, so under `CUDA_VISIBLE_DEVICES=1,2,3` the fourth shard opened device 3 and panicked with `CUDA_ERROR_INVALID_DEVICE`, taking the GPU workers down with it. The standard way to partition GPUs on a shared node silently broke the run. Also records that the enum kernel is bit-exact to degree 240 (1,593,460 R's, 16,949,543,206 matrices), so a high-stem LAUNCH_FAILED is not evidence against it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
…caps Three changes to `enumerate_admissible_kernel`, of which only one is a measured win. Sort the enumeration's `R`s by matrix count before launch (+8%). The kernel walks an odometer, so a thread's cost tracks its `R`'s matrix count, and a warp retires only when its slowest lane does. Matrix counts are heavily skewed — the R_STATS Lorenz curve has the hottest 1% of `R`s carrying 31% of references — so in basis order one huge `R` idles 31 lanes. Only the enumeration's own inputs are permuted: it writes each `R` at the absolute offset it is handed, so the scratch comes out byte-identical and the multiply's `r_cs_offset`/`prod_r_index` keep basis order untouched. `enum_warp_utilisation` (new, CPU-only) puts the modelled headroom at 2.16x — 30.9% of lane slots useful in basis order against 66.8% sorted. It cashed out as 1.08x measured, because an SM keeps many warps resident and hides most of the modelled stall. The docstring says so, so nobody sizes a decision on the model again. `ENUM_COL_CAP` was `WORKING_CAP` (32), conflating this cap with the multiply kernel's assembled-p-part array. `cols` is the widest bit-length of a p-part entry, bounded by the `r_1` field width — 11. That made `matrix`, the hottest per-thread local array, 320 u32 instead of 110. Also zero only the region each `R` reaches rather than the full cap. Both are strictly better on local-memory footprint and NEITHER is measurable end-to-end (1.884s -> 1.872s, inside noise) — kept for correctness of the derivation, not for speed. `enum_col_cap_bounds_real_rs` checks the derived caps against every real `R` to degree 400 (actual maxima: 8 rows, 9 cols). For context, in-kernel enumeration already costs 0.81x what merely uploading the same arrays costs, so there is no large multiple waiting here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
The eviction path bound its per-block enumeration scratch as segment 0 alone, so a block was capped at one segment — 4 GiB of u16. A stem-200 block wants 2.17e9 u16 of `masks`, just over the 2^31 line, so `NASSAU_GPU_RESIDENT_MAX_DEGREE=125` died on a hard assert partway through the run. That cap is why theta=125 could not complete. The fix is not a bigger segment. `master_seg_elems()` is deliberately under `u32::MAX` so a segment's element count cannot overflow cubecl's 32-bit array-length metadata — the same truncation class that caused the earlier silent corruption — and raising it would buy 2x and reopen that door. The resident path already spans MASTER_MAX_SEG segments through the same `seg_read`, so the transient path just had to use them: the ceiling goes 4 GiB -> 64 GiB with no new truncation surface. Layout places each `R` wholly inside one segment, with its `col_sums` and `masks` in the same-numbered segment, advancing both cursors to the next boundary together when either run would straddle. That keeps the enumeration as one launch per segment against the existing single-buffer kernel signature, so no segmented-write kernel is needed and the validated kernel is untouched. Offsets handed to each launch are segment-local; the multiply keeps reading global offsets through `seg_read` exactly as before. Verified by running the eviction test with NASSAU_GPU_MASTER_SEG_ELEMS shrunk to 1048576 and 262144, which forces many segments at degree 72 — all regimes still match the CPU golden, as does the full suite at both sizes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
…and log it `gpu_lock` arbitration was gated on `FP_CUDA_DEVICE == multiply_device()`, where `multiply_device()` read `NASSAU_GPU_DEVICE` — a variable nothing in `algebra` reads any more, left behind when the multiply became multi-GPU. It answered "device 0" however many GPUs the multiply was actually saturating, so `FP_CUDA_DEVICE=2` on a 4-GPU node would conclude "separate devices, no arbitration needed" while the multiply hammered device 2 too. The multiply shards across every visible device, so the test is `FP_CUDA_DEVICE < multiply_devices()`. This is a latent bug, NOT the cause of the theta=125 LAUNCH_FAILED: with default settings both sides resolved to 0, so arbitration was already enabled. Verified by the log line this adds, which exists because `[batch-stats] lock=` cannot distinguish "arbitration off" from "arbitration on but uncontended" — and the answer turned out to be a third thing. What the instrumentation actually exposed: `lock=0.0s` in every run *with arbitration enabled*, because the multiply takes `gpu_lock::shared()` inside the SUBMISSION closure and drops it when submission returns. Submission only enqueues; the kernels run long after. So the multiply releases the guard while its saturating kernels are still executing, the reduction then takes `exclusive()` believing the device is idle, and its thousands of tiny sequential launches interleave with them — the exact overlap the lock exists to prevent. Fixing that means holding the guard through the fence rather than the submit (the same scope error as the earlier GpuPermit bug); left for its own change since it alters the hot path's concurrency and invalidates the ~5 s/stem-200 cost estimate in the lock's docs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
…e multiply `gpu_lock::exclusive()` early-returned whenever `arbitration_needed()` was false, which conflated two independent questions: - does the MULTIPLY have to yield? (only when it shares the reduction's device) - do REDUCTIONS have to serialize? (always) The second is unconditional. The row reduction's GEMM is a persistent whole-device grid — `num_ctas = occupancy x SMs`, cluster-aligned, with cluster sync and DSMEM multicast — so two concurrent reductions each demand the entire GPU and neither can be placed. On Hopper that surfaces as a bare CUDA_ERROR_LAUNCH_FAILED, which compute-sanitizer does not attribute (0 invalid accesses across a whole run: it was never a memory bug). This is why putting the row reduction on its own GPU did not help on its own: `FP_CUDA_DEVICE=3` with the multiply on 0..2 turned arbitration off wholesale, so reductions stopped serializing against each other and the run still failed 63 times in 300 s. Isolating the device removes multiply contention and leaves reduction-vs-reduction contention untouched. With the split, an isolated reduction GPU runs clean at the FULL GEMM grid — no CTA cap, no throughput sacrificed. Measured on the theta=125 stem-200 repro that faulted at ~105 s in every other configuration: 400 s, 0 launch failures, 0 panics. The alternative was capping the persistent grid (`FP_CUDA_GEMM_CTAS`). The sweep on bench_kernel_only (16384^3, idle H200) shows why that is the wrong trade: the largest cap that survives the workload is 32, and 32 CTAs is 2107 TOPS against 8674 at full grid — 24% of peak. 64 CTAs reaches 49% but still fails. Throughput is linear in CTA count to ~128 (97% of peak), so the kernel only needs the device it is not being given. Not fixed here: the multiply takes `shared()` inside the SUBMISSION closure and drops it when submission returns, while its kernels are still resident — so on a SHARED device the yield is ineffective (`lock=0.0s` in every run). Correct fix is a dedicated fp-cuda driver thread that fences between reductions and overlaps transfers (copy engines do not consume SMs, so uploads can pipeline against compute). Isolating the reduction GPU sidesteps it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
Both fp-cuda entry points launch persistent whole-device grids (`num_ctas = occupancy x
SMs`, cluster-aligned): the row reduction's trailing GEMM and the standalone `try_mul`.
Two cannot be placed at once, and on Hopper the loser does not queue — it fails with a
bare CUDA_ERROR_LAUNCH_FAILED that compute-sanitizer cannot attribute.
`gpu_lock::exclusive()` covered `row_reduce` only; `try_mul` was deliberately lock-free
("concurrent callers do not interfere"), so even a dedicated reduction GPU had two
independent whole-device consumers and was not actually owned by anything. That is why
enabling the cooperative reduction path on an isolated device still wedged after a single
reduction: the cooperative kernel spun at its grid-wide barrier for CTAs a concurrent
`try_mul` was holding.
Routing both through one thread makes single-ownership structural instead of a discipline
each new call site must remember. Jobs run to completion there, and both end in a
device-to-host download, so serialization is on COMPLETION, not submission — the
distinction that matters, and the one `gpu_lock::shared()` still gets wrong on the multiply
side (taken inside the submit closure, dropped when submission returns, which is why
`[batch-stats] lock=` reads 0.0s everywhere).
Measured on the theta=125 stem-200 repro, reduction isolated to device 3, FULL GEMM grid:
200 s, 120 GPU reductions, 0 launch failures, both with and without FP_CUDA_RR_COOP.
Default shared-device config unchanged: 147 reductions, 0 failures, max_t=245 in 200 s.
Cooperative mode is now SAFE but not a win at this workload (closed=20406 vs 20306 in
200 s — noise). The reduction path is too small a share of the resolution for its ~2x to
show. It stays off by default.
Two things this does NOT do. The driver takes no `gpu_lock` guard: serialization among
fp-cuda jobs is structural, and taking the guard there deadlocked the run (it waits for
the multiply's readers while workers block on the driver), so yielding to the multiply on
a SHARED device still needs arranging without a guard held across a blocking job.
Transfers are serialized with compute, though copy engines do not consume SMs, so a later
change can pipeline the next job's upload against the current job's kernels.
`contended_acquisition_terminates_and_writers_are_exclusive` now measures reader overlap
in its own uncontended phase: with `exclusive` unconditional, the contended phase keeps a
writer queued almost always and writer preference correctly holds readers off, so
asserting overlap there measured the scheduler rather than the lock.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
The row reduction's GEMM sized its persistent grid to full occupancy (`occupancy x SMs`, ~264 CTAs on an H200). That grid is only placeable on a GPU this process owns outright. When anything else holds SMs the launch is not queued — it fails, as a bare CUDA_ERROR_LAUNCH_FAILED that compute-sanitizer cannot attribute (0 invalid accesses across a whole run: it was never a memory bug). Arbitrating with the other runtime would make fp-cuda and algebra depend on each other's scheduling, and would still not help a co-tenant in another process. Asking for a share needs no such knowledge: the persistent loop already handles any multiple of CLUSTER (fewer CTAs do more tile-iterations each), so grid size was never a correctness parameter. Default 1/16. The safe size is NOT a sharp threshold: on the theta=125 stem-200 workload 1/16 ran a clean 240 s while 1/8 failed 74 times, and a 32-CTA cap — essentially the same grid as 1/8 — had passed cleanly in an earlier run under the same nominal config. Where the launch stops being placeable depends on what the other tenant is doing at that moment. A share sharply reduces collision probability; it does not prove it to zero. Cost, from the idle-device sweep (bench_kernel_only, 16384^3): 16 CTAs = 1062 binary TOPS, 32 = 2107, full grid = 8674. Set FP_CUDA_GEMM_DEVICE_FRAC=1 on a dedicated linear-algebra GPU to restore full throughput. Left unresolved, and recorded in the source: why an oversubscribed grid fails rather than queueing. Ordinary launches schedule in waves; something here — thread-block clusters, the dynamic shared-memory request, or both — makes placement a hard launch-time requirement. Until that is understood no share can be called correct, only likelier to fit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
Replaces the "UNRESOLVED: why does an oversubscribed grid fail rather than queue" note with the answer, found in the GEMM's own history. Phase 9 (264b411, thread-block clusters + TMA multicast of B) introduced `__cluster_dims__`, B multicast with an all-ranks mask, and a cluster-wide empty barrier reached via `mapa`. A cluster's CTAs are co-resident by construction — a hard placement constraint an ordinary launch does not carry, and the reason the launch fails instead of scheduling in waves. Phase 8 (60b05cd) is not implicated, which is the useful half: its persistent loop strides (`tile += gridDim.x`) and was already work-capped (`sms.min(total_tiles)`), so the grouped rasterization that gives the kernel its L2 behaviour composes at any grid size. Only Phase 9's cluster layer has to go. That makes the fix the same one the row reduction already took (see `rr_coop`): a cluster-free variant, composable by construction, needing no knowledge of any other tenant — in this process or another. Cost is Phase 9's 2x cut in B HBM traffic, on top of the 8x GROUP_M rasterization already delivers, against the ~8x this share currently sacrifices (1062 TOPS at 1/16 vs 8674 at full grid). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
The GEMM's `__cluster_dims__(CLUSTER,1,1)` makes a cluster's CTAs co-resident by
construction -- rank 0 multicasts B straight into its mates' shared memory and
every consumer arrives on their empty barriers through `mapa`. So the launch does
not queue for SMs the way an ordinary grid does; when another runtime holds them
it fails outright with CUDA_ERROR_LAUNCH_FAILED. That is the fault compute-
sanitizer could never attribute (0 invalid accesses across a whole run: it was
never a memory bug), and the reason `FP_CUDA_GEMM_DEVICE_FRAC` had to give away
~8x of the throughput to make the resolution survive a shared GPU.
Template the kernel body on the cluster width and emit two entry points. Every
cluster-dependent construct has an exact single-CTA counterpart -- rank is 0,
cluster_sync drops (the preceding __syncthreads already orders it), arrive_cluster
becomes a local arrive, multicast becomes a plain TMA load, the empty barrier
counts 1 -- so the two kernels run identical arithmetic on an identical tile
schedule and differ only in B's HBM traffic and in whether the launch demands
co-resident CTAs.
The cost of dropping multicast turns out to be nearly nil. bench_kernel_only on
an idle H200, cluster-free vs cluster, correct=true idempotent=true throughout:
4096^3 4091 vs 4071 binary TOPS
8192^3 6687 vs 6778
16384^3 8501 vs 8632
32768^3 9608 vs 9664
Within 1.5% everywhere. GROUP_M rasterization was already keeping B resident in
L2, so the second-order saving multicast adds does not show up at these shapes --
and it was never worth a hard placement constraint.
So the cluster-free kernel takes the whole machine by default (frac 1) instead of
a 1/16 share, which is a ~8x throughput gain over the stopgap on a shared GPU.
`FP_CUDA_GEMM_COOP=1` opts back into the cluster kernel for a dedicated device,
mirroring `FP_CUDA_RR_COOP` one layer down.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
… it loses Both odometers -- `AdmissibleMatrix::next` and `enumerate_admissible_kernel` -- recompute an anti-diagonal bitsum `d` for every visited (row, col), and the same loop already maintains `masks[row+col] = d | new_entry`. That looks like a precomputation waiting to be exploited: collapse an O(cols) scan in the hottest triple-nested loop down to one load, in both implementations at once. Measured over 46,995,344 anti-diagonal computations (degree <= 120): d == masks 13,322,067 28.348% d subset-of masks 46,995,344 100.000% masks==0 (sound skip) 2,736,685 5.823% d-scan iterations 55,846,521 -> 1.19 per check The containment is exact and universal, and it does yield a sound skip (`masks[row+col] == 0` implies `d == 0`). But the scan it would skip averages 1.19 iterations: `(row+col+1).saturating_sub(rows)..col` is nearly always empty or a single step for real R's, because they are short and wide relative to the anti-diagonal. The skip fires on 5.8% of checks and avoids 6.9% of iterations of a loop that barely runs. There is nothing here. Keep the probe as an ignored diagnostic with the verdict in its docstring, so the shortcut is not re-proposed. Its assert is armed: 0 containment violations. With the earlier null result from cutting per-thread local state 3x (1.884s -> 1.872s), the odometer's arithmetic and its local state are both cleared. What is left proportional to the work is the emit: ~20 scattered 2-byte global stores per matrix, ~680 GB at degree 240. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
…HBM bandwidth Two null results had already cleared the enumeration arithmetic: shrinking the per-thread local state 3x was noise (1.884s -> 1.872s), and the anti-diagonal bitsum averages 1.19 iterations per check (ec3ed45). That left the emit as the only candidate proportional to the work -- `cs_len + mk_len` (~20) scattered 2-byte global stores per matrix, with adjacent lanes writing to unrelated `r_cs_out[ri]` offsets. Add a comptime `emit` flag so the stores compile OUT rather than branching around them, and time the identical odometer with and without. `mat` still increments, so the enumeration and its trip count are unchanged; only the writes differ. degree <= 130: 89,392 R's, 75,987,575 matrices, 1.03e9 u16 stores (2.07 GB) emit=true 0.0858s emit=false 0.0491s stores: 42.8% of kernel time, 1.75x if free Large, but NOT dominant -- the odometer is still 57%. The recoverable part is what the bandwidth says: 2.07 GB in 0.0367s is ~56 GB/s against ~4.8 TB/s of HBM, about 1% of what the device can do. That is the signature of scattered 2-byte transactions, not of a store volume we are stuck with. Packing u16 pairs into u32 stores, or staging a matrix through shared memory for coalesced warp writes, should recover most of the 42.8% without touching the format the multiply reads. Production launches pass `emit = true`; only the diagnostic passes false. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
Replacing the four `Vec`s in `AdmissibleMatrix` with fixed arrays sized to `PPart`'s structural caps (MAX_LEN = 10 rows, width(0) = 11 cols) removes four allocations per R, of which there are ~1.6e6 at degree 240. It looked like a free win. Measured on benches/nassau_milnor it is a net loss: op24xel32 6.7% slower op40xel1 2.2% slower op8xel24, op20xel24, op32xel8, op8xel8 0.3-0.6% slower op24xel24, op16xel32, op46xel1 0.1-0.3% faster rest no significant change The allocations were never the cost; the zeroing is. A typical R is ~4 rows x ~6 cols = 24 entries against a 110-entry cap, so `vec![0; rows * cols]` clears about a third of what `[0; 110]` does, and the saved mallocs do not pay for the extra stores. An end-to-end stem-110 CPU resolution agreed: 9.59s vs 9.71s, noise. Same effect as shrinking the GPU kernel's ENUM_COL_CAP 32 -> 11, from the other direction: over-sized fixed state costs more than the allocation it avoids. Noted on the struct so the change is not re-proposed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
f3391b7 measured the emit at 42.8% of the enum kernel and read its ~56 GB/s as ~1% of HBM -- implying a bandwidth pathology from scattered 2-byte writes, to be fixed by relayout. Two more probe modes say otherwise. emit=1 production 0.0858s emit=0 no stores 0.0490s emit=2 lane-adjacent (coalesced) 0.0829s 92.1% of the cost REMAINS emit=3 every other entry 0.0685s 53.1% remains Mode 2 writes the same bytes and the same number of stores with the warp's lanes hitting adjacent addresses -- perfect coalescing, garbage layout, timing probe only. If transactions or bandwidth were the constraint that would have collapsed the cost; it moved 8%. Mode 3 halves the store count and takes almost exactly half the cost. The emit is bound by store-instruction issue. So relayout is dead and packing is the fix, with width tracking the win directly since cost is linear in instruction count: 2 x u16 -> u32 half the stores ~21% of kernel time ~1.27x 4 x u16 -> u64 quarter the stores ~32% ~1.47x Same bytes moved either way; what is bought is issue slots. Requires padding each R's cs/mk stride to the packing width so a matrix's run starts word-aligned, and teaching the multiply kernel's reader to unpack. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
…head is steep `NASSAU_R_STATS` ranked Rs by reference count, and the eviction policy was tuned on that plus bytes. But enumeration cost is proportional to num_mats, so what a rebuild actually costs is `count * num_mats`, and a small hot R can cost the same as a big cold one. Record num_mats per R and report the cost-weighted concentration, plus the transient set on its own. S_2 (150,75), theta=125: total matrices enumerated (if every ref rebuilt) 385.5e9 transient (deg>125) 98,551 Rs (57%) 43.5M refs (3%) 22% of cost within transient, cost coverage top1%=21% top5%=51% top10%=68% top25%=88% Two things follow. Transient is 22% of total enumeration work but ~99% of GPU KERNEL time (nsys, transient-heavy config: enumerate_admissible_kernel 98.9%, multiply_batch_kernel 1.1%). No contradiction: resident Rs are enumerated once on the CPU at first sight and then live in the master, so they never cost GPU time again. Rebuild work and GPU time are different denominators and were being conflated. And the cost inside the transient set is NOT flat -- 5% of its Rs carry half of it. A partial cache has a head to exploit. Better, since bytes are ~num_mats and cost is count*num_mats, cost-per-byte is EXACTLY count: num_mats cancels, so admission ranked on plain reference count is optimal for any byte budget. The degree cutoff remains the right first filter (degree proxies bytes) but discards hot and cold alike among what it excludes; a small count-ranked cache layered on top of theta is the missing piece, with memory bounded by what it is given rather than by theta. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
…s already optimal
The previous commit reported that transient cost is concentrated (top 5% of Rs
carry 51%) and proposed a count-ranked pinned cache. Add a per-R CSV dump and
simulate that policy against the reference stream. It does not work, and the
reason retires the idea.
transient at (150,75): 98,551 Rs, 43.5M refs, 83.76e9 matrices, 8.2 GB total
budget oracle k=1 k=2 k=4 k=8 k=16
1% 2.7% 2.4% 2.5% 2.5% 2.5% 2.6%
5% 11.8% 11.7% 11.7% 11.7% 11.7% 11.6%
10% 21.6% 21.4% 21.3% 21.3% 21.2% 21.2%
25% 44.0% 43.8% 43.8% 43.7% 43.5% 43.0%
Savings are LINEAR in bytes cached, and every admission rule lands within 0.3pp
of an oracle that ranks by cost-per-byte with full hindsight. When the oracle
cannot beat arrival order there is no structure to exploit.
The "steep head" was an artifact of measuring concentration per R rather than per
BYTE. cost/byte is exactly `count`, and the high-cost Rs are the big ones, so 5%
of Rs is nowhere near 5% of bytes. Normalised by the memory it costs -- the only
thing a cache budget cares about -- the distribution is flat.
So a pinned cache is the same linear memory-for-time dial theta already is, and
adds nothing. What the numbers do show is that theta is not a parameter to tune
but a fallback: the whole transient set is 8.2 GB at (150,75), and giving it up
costs ~99% of GPU kernel time (2412s uncapped vs 18566s at theta=125 for stem
200). Theta should be as high as device memory allows, and the levers worth
having are the ones that make it unnecessary -- a smaller master (values are
<= 2^11 but stored as u16; bit-packing is ~31% fewer bytes) and faster
enumeration for whatever still misses.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
…gh as memory allows `resident_degree_cap`'s docstring justified the threshold on a byte metric: at theta<=125, 43% of distinct Rs stay resident and only 4% of references miss. That is true and it is the wrong axis. A miss is not a fixed cost -- it re-enumerates on the GPU in EVERY block touching that R (~442 times over a run at (150,75)) -- and enumerate_admissible_kernel is ~99% of GPU kernel time whenever the transient path is live, against multiply_batch_kernel's 1.1% (nsys). Stem 200 to max_t=310, all complete, 0 crashes: theta=inf 2412s theta=200 2865s 50.2 GB peak theta=125 18566s 44.9 GB peak theta=125 gives up 6.5x in wall time for 5.3 GB. The knee is sharp and sits above 125; the memory curve is much flatter than the time curve. exec fell 16x (26269s -> 1619s) and fence 32x (245398s -> 7596s) on identical work (pairs 5.88e13 both) -- the enumeration simply stops happening. So the cap is a fallback for exhausting device memory, not a knob to tune down, and the docstring now says so with the numbers. Also records the negative result on smarter eviction: a pinned count-ranked cache saves linearly in bytes cached, within 0.3pp of a full-hindsight oracle, because cost/byte is exactly `count` and the per-byte distribution is flat. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
… as an OOM
Picking theta was a blind guess: master bytes are dominated by the top degrees
(stem 150: 2.2 GB at theta<=125, 9.9 GB at theta<=150), so a lower-stem run
cannot predict a higher theta -- the degrees carrying the mass are absent from
it. The only way to learn theta was to set it too high and crash hours in.
Track master bytes per R degree at admission (one add per first-sight R). The
cumulative sum IS "how big the master would be at theta = d", so any run reports
the whole curve, and a budget picks theta off it directly. Printed with the
periodic batch-stats as well as at the end, so a run that dies on an allocation
still leaves the answer behind.
stem 140: full=4.1GB (1.0GB/GPU over 4 devices)
theta<=100 0.3GB, <=125 1.5GB, <=141 4.1GB
Which also corrects the premise this cap was reasoned about under. The master is
a SMALL share of device memory -- 1.0 GB/GPU at stem 140, 2.6 GB/GPU at stem 150,
against a ~50 GB peak at stem 200. theta caps the master, so it is not the knob
that governs peak memory; the concurrent dense output matrices are. A theta that
fits the master is necessary, not sufficient, and lowering theta buys far less
memory than its 6.5x wall-time cost (e7c8d2f) suggests it should.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
…n is empty step1 computes `desired_image` as the kernel of the augmentation `target_module -> cc_module` in degree t, by building a `target_dim x target_dim` augmented identity and row-reducing it. When the target complex is empty in degree t that map has a zero-dimensional codomain, so the kernel is the whole space and no computation can discover otherwise. Take it directly. `target_dim` is dim(A_t), so at high t this allocated and reduced a matrix large enough to reach the GPU RREF path purely to rediscover the entire space. The guard is on the codomain being empty, NOT on resolving the sphere: every finite target module is concentrated in finitely many degrees, so past its top cell this holds for all t -- almost the whole resolution. The sphere is only the extreme of it, firing from t = 1. Verified bit-identical output on S_2, C2 and C2_eta (stem 40, max_s 20) -- i.e. on non-sphere modules, where the guard is false at low t and true above the top cell. NOT verified faster. At stem 120 the difference is noise (4.85s vs 4.91s): the waste is O(dim(A_t)^2) so it only bites at high t, and the high-t A/B was killed before finishing. Landed on correctness and on there being no reason to do the work, not on a measured win. Also gate the `dump_master_by_degree` call behind `feature = "gpu"` with its neighbour -- it broke the non-GPU build. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
…rd thread seg_grow's MASTER_MAX_SEG check was an assert, and it runs on a shard's dedicated per-launch thread. A panic there kills only that launch: the next spawns a fresh thread and the resolution continues, having silently dropped the failed block's products. That is not hypothetical. On an uncapped stem-300 run all four shard threads hit it at b=(174,26) and the run carried on for another ~900k batches (batch-stats calls 828k -> 1740k) with GPU_DISABLED never set, because no fallback engaged -- nothing had failed from the resolution's point of view. The output would have been wrong, not slow, and nothing in the run's own reporting would have said so. There is no recovery from exhausting the segment table mid-master, so abort the process. The message now points at the actual fix -- lower theta, sized off `dump_master_by_degree` -- rather than only at raising the caps. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
…ate does not fix it The emit ablations (f3391b7, 1a7e0f9) said stores are 42.8% of this kernel and that store COUNT matters while coalescing does not. Both true, and both misleading about the mechanism. ncu on an idle H200: Compute (SM) 5.16% Avg Active Threads/Warp 5.88 / 32 Memory 13.92% Active Warps/Scheduler 2.09 DRAM 0.65% No Eligible 76.57% L2 Hit 98.83% Warp Cycles/Issued Instr 8.90 Nothing is saturated. Stores hit cache (L2 98.8%) and never reach DRAM, which is why making them lane-adjacent bought 8%: there was no transaction pressure to relieve. The kernel is latency-bound, and ncu attributes 45.6% of its 8.9 stall cycles to L1TEX waits on the odometer's own state -- `matrix`/`totals`/ `col_sums`/`masks` are indexed by runtime values, so they live in local memory. Tried the obvious fix: state in `Shared<[u32]>`, strided `elem * BLOCK + tid` so a warp hits consecutive banks. Bit-exact (admissible_enum_gpu_matches passes), and the mechanism is confirmed -- warp cycles per issued instruction 8.90 -> 5.92, a 33% cut. But 152 u32/thread is 38.9 KB per block, which forces the block from 256 to 64 threads, and active warps per scheduler fall 2.09 -> 1.61. Net 109.63 ms -> 112.72 ms, a 2.8% LOSS. Reverted. Two things worth carrying forward. The 33% is real and only paywalled behind the shared footprint, so a smaller state (`matrix` as u16 -- values are <= 2^11) may still collect it. And divergence is untouched by any of this: 5.88 of 32 lanes active, ~82% of the machine idle, because one thread owns one R and warps run at their longest member. That is now the largest single number in the profile. Also: ncu's "Est. Local Speedup: 76.57%" is not a forecast. It is the prize if the stall vanishes for free; here it cost occupancy and the trade was negative. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
…y are not the constraint Follow-up to the latency profile. ncu Occupancy/LaunchStats on the production kernel (idle H200, degree <= 130): Registers Per Thread 40 Block Limit Registers 24 blocks Theoretical Occupancy 75% Waves Per SM 0.44 Achieved Occupancy 8.16% Grid Size 1397 x 64 threads Registers allow 24 blocks/SM and a 75% theoretical ceiling, so shrinking them buys nothing -- the achieved figure is 8.16%, an order of magnitude below a ceiling that is already generous. Waves Per SM = 0.44 is the fact that matters. The H200 has 132 SMs x 24 block slots = 3168; the grid supplies 1397. One thread per R over 89392 Rs is ~89k threads against ~270k of device capacity, so the kernel cannot fill half the machine however well it runs. Achieved occupancy then falls to 8.16% because those few blocks drain raggedly -- the same divergence that leaves 5.88 of 32 lanes active. Production is likely worse still: this benchmark enumerates every R to degree 130 in one launch, while a production launch covers only one block's transient Rs. So the granularity is the bug: thread-per-R gives too FEW threads and wildly UNEQUAL ones, and stores / local memory / shared memory / registers are all adjustments to a kernel that has no work to hide latency with. That explains why every attempt today measured between -3% and +8%. The changes that would matter are structural: batch many more Rs per launch so the grid fills the device, or replace thread-per-R with lane cooperation over dynamically pulled work. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
…% of the grid `enumerate_admissible_kernel` is ~99% of GPU kernel time, and profiling it standalone gave Waves Per SM = 0.44: it could not fill half the device. But that benchmark enumerates every R to degree 130 in ONE launch, whereas production launches cover a single block-segment's transient Rs. Count what production actually submits. S_2 stem 150, max_s 60, theta=125: enum launches 10260 Rs/launch mean 1293 (max 34157) blocks/launch mean 6 waves/SM 0.002 Six blocks against an H200's 3168 slots. Production is 220x smaller than the benchmark and uses 0.2% of the machine per launch, ~1293 threads out of ~270k of capacity, 10260 times over. That settles batching vs lane cooperation, which was the open question. Lane cooperation (warp per R instead of thread per R) fixes divergence -- 5.88 of 32 lanes active -- and multiplies threads by 32, but the odometer is sequential so it is Amdahl-capped near 1.6x, and the grids would still be a fraction of a percent of the device. Batching Rs across launches attacks a factor of hundreds. It also explains why this kernel dominates GPU time despite being simple integer work: it is not slow, it is run 10260 times on an almost empty GPU, so the cost is launch latency and drain. And it closes the micro-optimisation thread -- stores, local memory, shared memory, registers and coalescing all tune a kernel that is idle by construction. The 42.8% emit share was real, but it is 42.8% of a kernel at 0.2% grid utilisation. The counters ride with `[batch-stats]`, so any run reports its own geometry. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
…erial
Production submits ~6 blocks per enum launch (waves/SM 0.002), so the obvious
move was to spread the same work over more SMs with smaller blocks. Measured on
S_2 stem 150 / max_s 60 / theta=125:
ENUM_THREADS blocks/launch waves/SM wall
256 6 0.002 561 s
64 21 0.007 569 s
32 41 0.013 542 s
6.5x the blocks for 3.4% — noise. Reverted.
The reason is the useful part. A launch's duration is set by its LONGEST SINGLE R
-- one thread, sequential odometer -- not by how many blocks it occupies, so
extra SMs idle beside the one thread still grinding. Rs/launch averages 1293 and
peaks at 34157, so the intra-launch spread is enormous.
And 10260 launches across a ~550 s run is ~50 ms apiece, which matches
`gpu_thread` running every device section on one thread and one stream: they are
strictly serial by design (that serialisation fixed a 370 s starvation bug).
This is what makes batching worth a factor of hundreds rather than a few percent:
merging launches converts a SUM into a MAX. Ten serialised 50 ms launches cost
500 ms; the same Rs in one launch cost ~50 ms, because the short Rs run beside
the long one instead of queueing behind it. Neither block size nor any per-thread
micro-optimisation can reach that, which is why every attempt has landed between
-3% and +8%.
Two candidates, recorded on the kernel: stream concurrency (cheap -- the launches
are already independent and FIFO dispatch could round-robin across N streams and
stay fair, but streams were pinned to 1 to fix a host-memory blowup from
per-stream pinned pools), or aggregating Rs across calls (needs enumeration
decoupled from the multiply that consumes its output).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
…bound `NASSAU_GPU_STREAMS` (default 1, i.e. unchanged) lets N worker threads drain one device's queue, each on its own `StreamId`, so independent device sections overlap instead of running strictly one after another. This was the cheap half of the "batch the enum kernel" plan: `enumerate_admissible_kernel` is ~99% of GPU kernel time and runs at `Waves Per SM = 0.002`, and merging serialised launches turns a SUM into a MAX. Measured (S_2 stem 150, max_s 60, theta=125), and it refutes the plan: | streams | wall | peak host RSS | peak GPU | |---------|-------|---------------|----------| | 1 | 528 s | 39.1 GB | 31.5 GB | | 2 | 527 s | 53.3 GB | 43.9 GB | | 4 | 598 s | 75.9 GB | 68.8 GB | Wall is flat at 2 and 13% worse at 4 while both memories roughly double. The memory growth is the load-bearing part of the result: it proves the sections really did overlap, so this is not "the streams did not engage" — concurrency happened and bought nothing. The conclusion is therefore about the workload, not about streams: at this configuration the resolution is not GPU-throughput-bound. Kernel time is 99% enum, but kernel time is not the critical path. Measured alongside: ~918% CPU on a 128-core node (~7% of the machine) with a wavefront only ~5 bidegrees wide, i.e. the limiter is a serial dependency chain on the host. Kept rather than reverted: a few lines, defaults to exactly the old behaviour, and it is the control that makes the "not GPU-bound" claim falsifiable elsewhere. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
… a time Both sides are little-endian packed F_2 bitvectors with bit i = column i, so four of the kernel's u32 limbs ARE one of `fp`'s u64 limbs, byte for byte. The readback can therefore be a truncating byte copy into a scratch `FpVector` plus one limb-wise `add`, instead of walking the set bits and calling `add_basis_element` on each. At the logged ~26% density that is cols/64 word XORs per row in place of ~0.26*cols bounds-checked entry writes. `update_from_bytes` refills the scratch in place, so the whole call allocates two buffers rather than one per row. Applies to both `get_partial_matrix` and the restricted variant; the restricted one also drops bits past `target_dim`, which is exactly what its old `col < target_dim` guard did — whole limbs by truncation, the partial final limb by mask. Verified with NASSAU_GPU_VERIFY over a full S_2 stem-40 max_s-30 resolution, which builds every matrix both ways and compares nonzero column sets: 0 mismatches. Honest timing result: no measurable win (528 s, identical to baseline at stem 150 / theta=125). That is the same story as the streams experiment — the run is bound by a serial dependency chain at ~7% CPU, so making CPU work cheaper does not move wall time. Kept because it is verified-equivalent and strictly less work, and it will matter once the serial chain is opened up. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
…, both negative
Ran the `NASSAU_PROBE_SIG_INDEP` probe that had been sitting unused: 19.2% of the
values a signature reads (27 028 of 140 614) were written by an earlier signature,
so the loop is a genuine forward substitution and the lift must stay ordered.
That only rules out reordering the lift. Everything above it reads read-only state,
so a windowed prepare stage feeding an ordered lift is legal, and it was built and
measured. It is worthless, for a reason worth recording so nobody rebuilds it:
sum of per-bidegree TOTAL signature time 4064.7 s
sum of per-bidegree MAX signature time 3895.7 s -> ceiling 1.04x
One signature is ~96% of its bidegree, so the ideal speedup is 4%. End to end a
window of 4 measured 537 s against a 528 s baseline while raising mean CPU from
918% to 1306% — 42% more CPU to lose 1.7%, exactly as the ceiling predicts. The
refactor is reverted; only the measurement is kept.
Context for what this closes off: the run uses ~7% of a 128-core node with a
wavefront ~5 bidegrees wide, and GPU stream concurrency was independently measured
to buy nothing. Three attempts to add concurrency have now failed, which locates
the limiter as latency on a serial chain rather than any throughput shortfall.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
… spin-wait
The zero signature is ~96% of its bidegree (it is unconstrained, so its mask is the
widest and its matrix the largest — reproducible in 100% of bidegrees across four
runs), and almost all of its work sat outside every span. Adds `cpu_restricted`,
`zs_masks`, `zs_select`, `zs_source_mask` and `zs_dx_init` to close that gap.
They are all small, which is itself the result — none of the CPU regions I suspected
is the cost:
extract_restricted 1062.7 s cpu_restricted 97.0 s
gpu_readback 319.6 s pair_prepass 90.1 s
marshal_terms 54.3 s zs_select 1.8 s
gpu_submit 0.3 s zs_dx_init 0.3 s
against a zero-signature `step` total of 4844.9 s, leaving ~66% unattributed. Span
accounting could not find it because `time.busy` is inclusive AND the thing being
waited on is not inside any span: `[batch-stats]` reports `fence=1024.2 s` (caller
blocked on the GPU completion fence) against a worker-side `exec=77.1 s`.
A sampling profile settles it. `perf record` on a steady-state stem-150 run, by
shared object:
43.84% libcuda.so 29.18% the binary 17.40% libc
The whole libcuda share is one tight address range (0x461860-0x461999): the driver's
spin-wait. The process burns ~44% of its CPU busy-waiting on the device.
That retro-explains every negative result in this branch of work. Extra CUDA streams,
a parallel signature-prepare window, and a cheaper matrix readback all failed for the
same reason — they add or cheapen CPU work on a path that is waiting for the GPU, and
the extra threads spin too. It also puts `enumerate_admissible_kernel` (99% of GPU
kernel time) back on the critical path as a LATENCY problem, which is a different
target from the throughput framing that streams were testing.
Top CPU symbols for whoever picks this up: `get_partial_matrix_restricted` 11.3%,
allocator (`_int_free`/`malloc_consolidate`/`malloc`/`_int_malloc`) 8.6%,
`__memcpy_ssse3` 5.9%, `__vdso_clock_gettime` 4.7%.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Replaces the doubling-realloc-with-copy growth of every resident device buffer (admissible master
col_sums/masksand the resident basispparts/lens) with a single segmented, append-only, no-copy growth mechanism.Why
The stem-140+
dx != 0(d²≠0) failures were traced to cubecl silently handing back the wrong device buffer under memory pressure (ManagedMemoryDescriptor id mismatch), driven by the master's realloc-doubling transient: at a growth point the old (cap) and new (2×cap) buffers are both live plus an on-device copy between them — a ~2–3× spike that tips cubecl into its corruption regime. Verified it is not the RREF (checked to 2^18 square; failing matrices are only ~10^4 square) and not the multiply (CPU-RREF of GPU output is clean).How
SegBuf+seg_grow!replaceGrowBuf,resident_dev_handle!,basis_dev_handles!,stage_upload!,RESIDENT_REALLOC, andRESIDENT_INIT_CAP(all deleted — no two coexisting growth paths). Growth allocates only the new fixed-size segment(s) and stage-writes the tail; existing segments are never reallocated or copied, so device peak islive + one_segment.multiply_batch_kernelbinds each store asMASTER_MAX_SEG(=16) segmentArrays and gathers a thread's matrixcs/mk+ its term p-part into smallWORKING_CAPlocals viaseg_read_u16/seg_read_u32(correct at any offset — no layout padding), then calls the unchanged puremultiply_pairwith base 0.master_seg_elems()(envNASSAU_GPU_MASTER_SEG_ELEMS, default1<<31,< u32::MAX) => 64 GiB/buffer over 16 segments. Over-cap is a cleanassert!, not corruption.Validation (bit-exact on H200)
seg_read_matches_contiguous— the 16-arg segment-read primitive.multiply_batch_matches_referenceacrossNASSAU_GPU_MASTER_SEG_ELEMS768–4096 — multi-segment single-launch gather.multiply_batch_incremental_growthatseg_elems=8192— cross-launch append into the partially-filled last segment.algebraGPU suite green. (3 remaining suite failures —test_ppart_multiplier_3,basis_element_from_string_total_milnor,test_evaluate_3— are pre-existing odd-prime/CPU tests outside this diff.)Next
Fire the dx-clean stem-200 run and capture the flat
[MEM] master=growth curve, then decide host-paging vs 4-GPU sharding for stem 300.🤖 Generated with Claude Code
https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf