Skip to content

[FA2] Use kernels-community/aiter-flash-attn-ck Hub kernel for ROCm FlashAttention (drop aiter dependency) - #14436

Open
Abdennacer-Badaoui wants to merge 1 commit into
huggingface:mainfrom
Abdennacer-Badaoui:use-aiter-fa-from-hub
Open

[FA2] Use kernels-community/aiter-flash-attn-ck Hub kernel for ROCm FlashAttention (drop aiter dependency)#14436
Abdennacer-Badaoui wants to merge 1 commit into
huggingface:mainfrom
Abdennacer-Badaoui:use-aiter-fa-from-hub

Conversation

@Abdennacer-Badaoui

Copy link
Copy Markdown
Member

What

Replaces the aiter pip-package attention backend with the Hub kernel kernels-community/aiter-flash-attn-ck, loaded via kernels (same pattern as flash_hub / sage_hub).

  • Backend aiter -> aiter_fa2_hub; no more aiter pip dependency.
  • The -ck CK build is bf16-only, so the dtype constraint is bf16 (fp16 is rejected).

Validation (AMD MI300, ROCm 7.2, torch 2.11)

Numerical parity vs PyTorch SDPA (random q/k/v, B=2, S=1024, H=8, D=64):

dtype causal max_abs mean_rel
bf16 no 3.9e-03 1.3e-02
bf16 yes 3.1e-02 1.4e-02

End-to-end: FLUX.1-schnell, same seed, 4 steps, bf16, native (SDPA) vs aiter_fa2_hub - perceptually identical (mean pixel diff ~= 1.3%).

Prompt: "A photorealistic close-up of a red fox sitting in a snowy pine forest at golden hour, soft rim light, shallow depth of field, highly detailed fur"

flux_compare

@github-actions github-actions Bot added the size/M PR with diff < 200 LOC label Aug 10, 2026
@github-actions github-actions Bot added documentation Improvements or additions to documentation models utils labels Aug 10, 2026
@HuggingFaceDocBuilderDev

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

AttentionBackendName.AITER_FA2_HUB,
# The `kernels-community/aiter-flash-attn-ck` CK kernel only ships bf16 `mha_fwd` instances;
# fp16 raises a cryptic "invalid argument for fmha_fwd" from CK, so reject it up front.
constraints=[_check_device_cuda, _check_qkv_dtype_bf16, _check_shape],

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't _check_qkv_dtype_bf16_or_fp16 already tackle it?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not for this case; that helper permits fp16 (it only rejects dtypes outside {bf16, fp16}), so it would let fp16 tensors through to the kernel, which is exactly the input that fails.

The build only ships bf16 (for now) mha_fwd instances. we can try to add fp16 and the backward afterwards

_parallel_config: "ParallelConfig" | None = None,
) -> torch.Tensor:
if attn_mask is not None:
raise ValueError("`attn_mask` is not supported for aiter attention")

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't it be aiter attention?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is inside _native_flash_attention, so it should say "native flash attention"; the original "aiter attention" here was a copy-paste bug, so I fixed it while touching this file.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation models size/M PR with diff < 200 LOC utils

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants