[FA2] Use kernels-community/aiter-flash-attn-ck Hub kernel for ROCm FlashAttention (drop aiter dependency) - #14436
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
| AttentionBackendName.AITER_FA2_HUB, | ||
| # The `kernels-community/aiter-flash-attn-ck` CK kernel only ships bf16 `mha_fwd` instances; | ||
| # fp16 raises a cryptic "invalid argument for fmha_fwd" from CK, so reject it up front. | ||
| constraints=[_check_device_cuda, _check_qkv_dtype_bf16, _check_shape], |
There was a problem hiding this comment.
Shouldn't _check_qkv_dtype_bf16_or_fp16 already tackle it?
There was a problem hiding this comment.
Not for this case; that helper permits fp16 (it only rejects dtypes outside {bf16, fp16}), so it would let fp16 tensors through to the kernel, which is exactly the input that fails.
The build only ships bf16 (for now) mha_fwd instances. we can try to add fp16 and the backward afterwards
| _parallel_config: "ParallelConfig" | None = None, | ||
| ) -> torch.Tensor: | ||
| if attn_mask is not None: | ||
| raise ValueError("`attn_mask` is not supported for aiter attention") |
There was a problem hiding this comment.
Shouldn't it be aiter attention?
There was a problem hiding this comment.
This is inside _native_flash_attention, so it should say "native flash attention"; the original "aiter attention" here was a copy-paste bug, so I fixed it while touching this file.
What
Replaces the
aiterpip-package attention backend with the Hub kernelkernels-community/aiter-flash-attn-ck, loaded viakernels(same pattern asflash_hub/sage_hub).aiter->aiter_fa2_hub; no moreaiterpip dependency.-ckCK build is bf16-only, so the dtype constraint is bf16 (fp16 is rejected).Validation (AMD MI300, ROCm 7.2, torch 2.11)
Numerical parity vs PyTorch SDPA (random q/k/v,
B=2, S=1024, H=8, D=64):End-to-end: FLUX.1-schnell, same seed, 4 steps, bf16,
native(SDPA) vsaiter_fa2_hub- perceptually identical (mean pixel diff ~= 1.3%).Prompt: "A photorealistic close-up of a red fox sitting in a snowy pine forest at golden hour, soft rim light, shallow depth of field, highly detailed fur"