get_coreml_partitioner(coreml_quantize=...) cannot lower a model whose embedding
table is also its output projection. The quantizer configures the linear and
leaves the gather alone, then refuses the mismatch:
ValueError: compression config conflict detected between ops
%aten_embedding_default_cast_fp16...: gather(x=%p_out_weight_to_fp16, ...)
and
%aten_mm_default...: linear(x=..., weight=%p_out_weight_to_fp16, ...)
... has config None while ... has OpLinearQuantizerConfig(mode='LINEAR_SYMMETRIC', ...)
Repro (executorch 1.4.0, coremltools 9.0, macOS arm64) — it is the tie alone, with
nothing else in the model:
import torch
from executorch.exir import to_edge_transform_and_lower
from executorch.extension.llm.export.partitioner_lib import get_coreml_partitioner
class Tied(torch.nn.Module):
def __init__(self, vocab=512, dim=64, tie=True):
super().__init__()
self.emb = torch.nn.Embedding(vocab, dim)
self.out = torch.nn.Linear(dim, vocab, bias=False)
if tie:
self.out.weight = self.emb.weight
def forward(self, ids):
return self.out(self.emb(ids))
ids = torch.zeros(1, 4, dtype=torch.long)
for tie in (True, False):
part = get_coreml_partitioner(ios=18, coreml_quantize="c4w",
coreml_compute_units="all")
ep = torch.export.export(Tied(tie=tie).eval(), (ids,))
to_edge_transform_and_lower(ep, partitioner=[part]).to_executorch()
tie=False lowers; tie=True raises. Same for b4w.
Why it is worth fixing rather than documenting. tie_word_embeddings=True is
the default for most small LLMs — Qwen3.5, LFM2.5, SmolLM2, Gemma and Llama-3.2-1B
all tie — so this is close to universal for the model sizes people actually put on
a phone through Core ML. The failure also gives no hint about the cause: the
message names two MIL ops, not the shared parameter.
Workaround, for anyone else who lands here: quantize the lookup too, so both
uses carry a config.
quantization:
embedding_quantize: "4,32"
backend:
coreml:
quantize: c4w
That changes the model rather than just the pipeline, which is why it reads as a
workaround and not a fix. Two directions that would not: propagate the linear's
config to the gather that shares the parameter, or skip quantizing a linear
whose weight is also used as an embedding.
Related: #21855 (Core ML LLM builds fail at execute on the state binding). The two
are independent — this one reproduces with no mutable buffer in the model.
cc @kimishpatel @YifanShenSZ @cymbalrush @metascroy
get_coreml_partitioner(coreml_quantize=...)cannot lower a model whose embeddingtable is also its output projection. The quantizer configures the
linearandleaves the
gatheralone, then refuses the mismatch:Repro (executorch 1.4.0, coremltools 9.0, macOS arm64) — it is the tie alone, with
nothing else in the model:
tie=Falselowers;tie=Trueraises. Same forb4w.Why it is worth fixing rather than documenting.
tie_word_embeddings=Trueisthe default for most small LLMs — Qwen3.5, LFM2.5, SmolLM2, Gemma and Llama-3.2-1B
all tie — so this is close to universal for the model sizes people actually put on
a phone through Core ML. The failure also gives no hint about the cause: the
message names two MIL ops, not the shared parameter.
Workaround, for anyone else who lands here: quantize the lookup too, so both
uses carry a config.
That changes the model rather than just the pipeline, which is why it reads as a
workaround and not a fix. Two directions that would not: propagate the linear's
config to the
gatherthat shares the parameter, or skip quantizing alinearwhose weight is also used as an embedding.
Related: #21855 (Core ML LLM builds fail at execute on the state binding). The two
are independent — this one reproduces with no mutable buffer in the model.
cc @kimishpatel @YifanShenSZ @cymbalrush @metascroy