Skip to content

Core ML coreml_quantize fails on tied embeddings: "compression config conflict detected between ops" #21856

Description

@john-rocky

get_coreml_partitioner(coreml_quantize=...) cannot lower a model whose embedding
table is also its output projection. The quantizer configures the linear and
leaves the gather alone, then refuses the mismatch:

ValueError: compression config conflict detected between ops
  %aten_embedding_default_cast_fp16...: gather(x=%p_out_weight_to_fp16, ...)
 and
  %aten_mm_default...: linear(x=..., weight=%p_out_weight_to_fp16, ...)
  ... has config None while ... has OpLinearQuantizerConfig(mode='LINEAR_SYMMETRIC', ...)

Repro (executorch 1.4.0, coremltools 9.0, macOS arm64) — it is the tie alone, with
nothing else in the model:

import torch
from executorch.exir import to_edge_transform_and_lower
from executorch.extension.llm.export.partitioner_lib import get_coreml_partitioner


class Tied(torch.nn.Module):
    def __init__(self, vocab=512, dim=64, tie=True):
        super().__init__()
        self.emb = torch.nn.Embedding(vocab, dim)
        self.out = torch.nn.Linear(dim, vocab, bias=False)
        if tie:
            self.out.weight = self.emb.weight

    def forward(self, ids):
        return self.out(self.emb(ids))


ids = torch.zeros(1, 4, dtype=torch.long)
for tie in (True, False):
    part = get_coreml_partitioner(ios=18, coreml_quantize="c4w",
                                  coreml_compute_units="all")
    ep = torch.export.export(Tied(tie=tie).eval(), (ids,))
    to_edge_transform_and_lower(ep, partitioner=[part]).to_executorch()

tie=False lowers; tie=True raises. Same for b4w.

Why it is worth fixing rather than documenting. tie_word_embeddings=True is
the default for most small LLMs — Qwen3.5, LFM2.5, SmolLM2, Gemma and Llama-3.2-1B
all tie — so this is close to universal for the model sizes people actually put on
a phone through Core ML. The failure also gives no hint about the cause: the
message names two MIL ops, not the shared parameter.

Workaround, for anyone else who lands here: quantize the lookup too, so both
uses carry a config.

quantization:
  embedding_quantize: "4,32"
backend:
  coreml:
    quantize: c4w

That changes the model rather than just the pipeline, which is why it reads as a
workaround and not a fix. Two directions that would not: propagate the linear's
config to the gather that shares the parameter, or skip quantizing a linear
whose weight is also used as an embedding.

Related: #21855 (Core ML LLM builds fail at execute on the state binding). The two
are independent — this one reproduces with no mutable buffer in the model.

cc @kimishpatel @YifanShenSZ @cymbalrush @metascroy

Metadata

Metadata

Assignees

Labels

module: coremlIssues related to Apple's Core ML delegation and code under backends/apple/coreml/

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions