Fold DyT alpha scalar into tanh LUT - #21952
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/21952
Note: Links to docs will display an error until the docs builds have been completed. ❌ 2 New Failures, 3 Unrelated FailuresAs of commit 54b33a0 with merge base 1b2838b ( NEW FAILURES - The following jobs have failed:
FLAKY - The following jobs failed but were likely due to flakiness present on trunk:
BROKEN TRUNK - The following job failed but were present on the merge base:👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
|
@cgreenberg has exported this pull request. If you are a Meta employee, you can view the originating Diff in D116560649. |
|
9501798 to
5414367
Compare
|
|
| @@ -0,0 +1,341 @@ | |||
| # Copyright 2025-2026 Arm Limited and/or its affiliates. | |||
There was a problem hiding this comment.
update copyright to Meta
Summary: Dynamic Tanh (DyT) normalization computes `tanh(alpha * x)` with a learned scalar alpha. Quantized, that lowers to a full-tensor integer Mul followed by a tanh TABLE. On Ethos-U the TABLE is free but the Mul is a real per-element cost, so the Mul is pure overhead. This adds `FoldDyTAlphaIntoLUTPass` in `backends/arm/_passes/`, which folds the alpha multiply into the tanh lookup table and deletes the Mul and its surrounding rescales. Folding alpha in floating point before quantization would change rounding, so the pass does it in the integer domain instead: it replays the exact TOSA SINGLE_ROUND RESCALE and Mul arithmetic over all 256 int8 input codes, feeds the results through the existing tanh quantization mapping, and materializes the result as one 256-entry TABLE. The rewrite is therefore byte-exact rather than approximate. It fails closed on anything it cannot prove: non-scalar alpha, activation-side rank views, and narrowed tanh ranges are all handled explicitly. Also adds `register_pass_factories_before()` to the Arm pass manager, a small hook for inserting a pass that needs access to the `ExportedProgram` ahead of a named target pass. `FoldDyTAlphaIntoLUTPass` needs it to read constant tensors. Note on one import: the pass imports `register_pass_factories_before` inside `register_fold_dyt_alpha_into_lut_pass()` rather than at module scope. `_passes/__init__.py` imports `arm_pass_manager` last and `arm_pass_manager` imports back from the package, so a module-scope import from a pass module that `__init__.py` re-exports is circular. Only the register helper needs it. The pass is inert until a model registers it, so this diff changes no behaviour on its own. Differential Revision: D116560649
5414367 to
54b33a0
Compare
This PR needs a
|
Summary:
Dynamic Tanh (DyT) normalization computes
tanh(alpha * x)with a learned scalaralpha. Quantized, that lowers to a full-tensor integer Mul followed by a tanh
TABLE. On Ethos-U the TABLE is free but the Mul is a real per-element cost, so
the Mul is pure overhead.
This adds
FoldDyTAlphaIntoLUTPassinbackends/arm/_passes/, which folds thealpha multiply into the tanh lookup table and deletes the Mul and its
surrounding rescales.
Folding alpha in floating point before quantization would change rounding, so the
pass does it in the integer domain instead: it replays the exact TOSA
SINGLE_ROUND RESCALE and Mul arithmetic over all 256 int8 input codes, feeds the
results through the existing tanh quantization mapping, and materializes the
result as one 256-entry TABLE. The rewrite is therefore byte-exact rather than
approximate. It fails closed on anything it cannot prove: non-scalar alpha,
activation-side rank views, and narrowed tanh ranges are all handled explicitly.
Also adds
register_pass_factories_before()to the Arm pass manager, a smallhook for inserting a pass that needs access to the
ExportedProgramahead of anamed target pass.
FoldDyTAlphaIntoLUTPassneeds it to read constant tensors.Note on one import: the pass imports
register_pass_factories_beforeinsideregister_fold_dyt_alpha_into_lut_pass()rather than at module scope._passes/__init__.pyimportsarm_pass_managerlast andarm_pass_managerimports back from the package, so a module-scope import from a pass module that
__init__.pyre-exports is circular. Only the register helper needs it.The pass is inert until a model registers it, so this diff changes no behaviour
on its own.
Differential Revision: D116560649
cc @digantdesai @freddan80 @per @zingo @oscarandersson8218 @mansnils @Sebastian-Larsson @robell @rascani