Skip to content

Core ML: stop the gather opt-out from failing tied embeddings - #21969

Open
john-rocky wants to merge 1 commit into
pytorch:mainfrom
john-rocky:coreml-tied-embedding-quantize
Open

Core ML: stop the gather opt-out from failing tied embeddings#21969
john-rocky wants to merge 1 commit into
pytorch:mainfrom
john-rocky:coreml-tied-embedding-quantize

Conversation

@john-rocky

Copy link
Copy Markdown
Contributor

Fixes #21856.

What happens today

op_linear_quantizer_config is applied with op_type_configs={"gather": None}, so that
embedding tables are not compressed by the linear quantizer. When a model ties its embedding
table to its output projection — one weight, two consumers — that constant reaches the
compressor configured two ways at once: quantized as a linear weight, exempt as a gather
table. coremltools refuses the conflict rather than resolving it:

ValueError: compression config conflict detected between ops
  %aten_embedding_default_...: gather(x=%p_out_weight_to_fp16, ...)
 and
  %aten_mm_default...: linear(x=..., weight=%p_out_weight_to_fp16, ...)
  ... has config None while ... has OpLinearQuantizerConfig(mode='LINEAR_SYMMETRIC', ...)

So the opt-out does not skip the table. It fails the whole lowering, and a very common LLM
shape cannot be lowered at all with coreml_quantize set.

The change

Gathers whose constant is also consumed by an op of another type are named in
op_name_configs with the same config as the op they share it with, so the constant is seen
one way. Every other gather keeps the opt-out, and a model whose MIL program is not available
returns no names, which leaves the behaviour exactly as it was.

Checking it

The new test in test_coreml_partitioner.py lowers a two-layer model twice, tied and untied.
Against main the tied case raises the error above; with this change both lower. The untied
case passes either way, which is the point — the opt-out still applies where it always did.

Measured on macOS arm64, coremltools 9.0.

The linear quantizer config opts gathers out so embedding tables are not
compressed by it. When a model ties its embedding table to its output
projection, that one constant reaches the compressor configured two ways at
once — quantized as a linear weight, exempt as a gather table — and coremltools
refuses the conflict rather than resolving it. The opt-out then does not skip
the table, it fails the whole lowering, which is how a common LLM shape becomes
unlowerable with coreml_quantize set.

Gathers that share their constant with an op of another type are now configured
the same way as the op they share it with; every other gather keeps the
opt-out. A model whose program is unavailable is left exactly as before.

Fixes pytorch#21856
@john-rocky
john-rocky requested a review from shoumikhin as a code owner August 20, 2026 06:55
@pytorch-bot

pytorch-bot Bot commented Aug 20, 2026

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/21969

Note: Links to docs will display an error until the docs builds have been completed.

⚠️ 14 Awaiting Approval

As of commit 2de6a15 with merge base 7c3cdbb (image):

AWAITING APPROVAL - The following workflows need approval before CI can run:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Aug 20, 2026
@github-actions

Copy link
Copy Markdown

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Core ML coreml_quantize fails on tied embeddings: "compression config conflict detected between ops"

1 participant