perf(gqh): GQH matvec for gfx1201 — greedy 28.7 tok/s, MTP n-max=2 50.3 tok/s

[DeanoC]

What

Two commits on the GQH matvec (ggml/src/ggml-cuda/gqh.cu, gqh.cuh), measured on an AMD AI PRO R9700 (gfx1201, 32 GB) against Qwen38-27B GQH (298 GQH4 / 10 GQH3 tensors, 14.0 GiB):

1. 0317ed7 — port the batch-1 optimizations from geo-lucebox#32: software-pipelined wire loads (DEPTH=3), ncols launcher specialization, occupancy-fit rows-per-warp arm, one LDS level table per rung. Greedy tg64 20.8 → 28.6 tok/s.
2. f8dd7c3 — optimize the multi-column arm that MTP verify batches run. Greedy path untouched; MTP n-max=2 33.4 → 50.3 tok/s.

The multi-column defect

The generic <RUNG,8,1> instantiation re-read activations per (row, superblock, column) — 1024 B per warp per column against 137 B of wire. That is ~100 GB of activation reads per column per forward, at 12 TB/s effective, i.e. saturating L0. Weights stream once regardless of N, so N=3 should cost about N=1; instead each extra column cost **+9 ms**.

Fixes: exact-width instantiations with a shared activation read; wave-uniform addressing; a shortened per-superblock dependency chain; a fifth column for the exact-width arm (removing a 31 ms pp5 cliff); per-column activation addressing corrected beyond two columns.

Benchmarks

Steady-state per-forward (llama-bench -p 1,3,5 -b 8 -ub 8 -r 12):

	base	this PR
pp1	36.1 ms	36.1 ms (unchanged)
pp3	64.0 ms	38.6 ms
pp5	80.1 ms	44.5 ms
greedy tg64	20.8 tok/s (stock) / 28.6 (after 0317ed7)	28.7 tok/s

End-to-end MTP decode (llama-server --spec-type draft-mtp, temp 0, prose / code prompt):

n-max	base	this PR	acceptance
2	33.4	50.3 / 48.5 tok/s	63% / 59%
3	—	46.0 / 49.5 tok/s	44% / 51%
4	25.9	41.1 / 45.2 tok/s	34% / 41%

n-max=2 is the config to ship. n-max=4 is no longer a regression, but deeper drafts are now limited by drafter acceptance (63% → 34%), not verify cost — further depth gains are a drafter problem, not a matvec one.

Correctness

Two frozen sha gates on every landed pass — plain-greedy output (N=1 path) and greedy-MTP output (multi-column verify path) — both byte-equal to their references, accumulation term order untouched throughout. test-gqh-backend 54/54 bit-identical to the f32 reference over 3 rungs × 3 shapes × nvec 1..6, with a non-vacuity control that fails at nvec 5 so the new width is provably covered. Instantiations outside the exact-width cells are ISA-identical to base.

Refutations (recorded, nothing landed)

Three of ten evolution passes deliberately landed nothing:

• Byte-keyed level table — bit-exact and perfectly confined, but 7% slower. Isolated to LDS bank conflicts with a probe holding the instruction stream byte-identical while making addresses conflict-free.
• Occupancy pressure — a gated LDS-burn probe moved the arm from 16 waves/SIMD down to 3; pp3 did not move (collapses only at 1 wave). Occupancy is free here.
• Disabling CUDA graphs — reads +8.3% on a -r 3 rig but −3.4% in steady state (-r 24): ~23% of a -r 3 measurement is one-time graph-capture cost that amortizes. A benchmark-only gain, refused.

Two separate changes that cut 15% and 10% of hot-loop instructions both measured slower — on this kernel the currency is memory behavior, not instruction count.

Provenance

Evolved by the geo-evo kernel loop (10 iterations each phase). Full per-iteration record, traces, and measurement caveats: handoffs/progress-lucebox_gqh_matvec.md and handoffs/progress-gqh_mtp_multicol.md in geo-evo.

Measured on gfx1201 only. The width and rows thresholds are runtime-derived but validated on this device alone — a second-device run would be worth having before treating them as general.