QuantileType cherry pick from upstream and removal of QuantileQuantized - #19
Draft
vsimion26 wants to merge 423 commits into
Draft
QuantileType cherry pick from upstream and removal of QuantileQuantized#19vsimion26 wants to merge 423 commits into
vsimion26 wants to merge 423 commits into
Conversation
(cherry picked from commit 111219e)
Fixes llvm#153448 (cherry picked from commit 5e57a10)
…lvm#153924) Fixes llvm#153891 (cherry picked from commit a21d17f)
…darwin (llvm#153722) This PR makes sure that when targeting arm64e on darwin platforms the correct flags are set for the userspace platform ABI. (cherry picked from commit 19c4e86)
…lvm#150911) Back-ports additional tests (eb9febb4a6b0, dc697de12792), refactoring (43c9c14577db) and functional change (18f1369297f4) in a single PR. llvm#114990 allowed more aggressive tail duplication for computed-gotos in both pre- and post-regalloc tail duplication. In some cases, performing tail-duplication too early can lead to worse results, especially if we duplicate blocks with a number of phi nodes. This is causing a ~3% performance regression in some workloads using Python 3.12. This patch updates TailDup to delay aggressive tail-duplication for computed gotos to after register allocation. This means we can keep the non-duplicated version for a bit longer throughout the backend, which should reduce compile-time as well as allowing a number of optimizations and simplifications to trigger before drastically expanding the CFG. For the case in llvm#106846, I get the same performance with and without this patch on Skylake. PR: llvm#150911
…#153641) Consider the following code: ```cpp # 1 __FILE__ 1 3 export module a; ``` According to the wording in [P1857R3](https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2020/p1857r3.html): ``` A module directive may only appear as the first preprocessing tokens in a file (excluding the global module fragment.) ``` and the wording in [[cpp.pre]](https://eel.is/c++draft/cpp.pre#nt:module-file) ``` module-file: pp-global-module-fragment[opt] pp-module group[opt] pp-private-module-fragment[opt] ``` `#` is the first pp-token in the translation unit, and it was rejected by clang, but they really should be exempted from this rule. The goal is to not allow any preprocessor conditionals or most state changes, but these don't fit that. State change would mean most semantically observable preprocessor state, particularly anything that is order dependent. Global flags like being a system header/module shouldn't matter. We should exempt a brunch of directives, even though it violates the current standard wording. In this patch, we introduce a `TrivialDirectiveTracer` to trace the **State change** that described above and propose to exempt the following kind of directive: `#line`, GNU line marker, `#ident`, `#pragma comment`, `#pragma mark`, `#pragma detect_mismatch`, `#pragma clang __debug`, `#pragma message`, `#pragma GCC warning`, `#pragma GCC error`, `#pragma gcc diagnostic`, `#pragma OPENCL EXTENSION`, `#pragma warning`, `#pragma execution_character_set`, `#pragma clang assume_nonnull` and builtin macro expansion. Fixes llvm#145274 --------- Signed-off-by: yronglin <yronglin777@gmail.com> (cherry picked from commit e6e874c)
…#153921) The early return for lamda expressions with deduced return types in Sema::ActOnCapScopeReturnStmt meant that we were not actually perform the required return type deduction for such lambdas when in a discarded context. This PR removes that early return allowing the existing return type deduction steps to be performed. Fixes llvm#153884 Fix developed by, and Co-authored-by: Corentin Jabot <corentinjabot@gmail.com> (cherry picked from commit bcab8ac)
…for big-endian (llvm#151565) The patch fixed a bug introduced patch [[PowePC] using MTVSRBMI instruction instead of constant pool in power10+](llvm#144084 (comment)). The issue arose because the layout of vector register elements differs between little-endian and big-endian modes — specifically, the elements appear in reverse order. This led to incorrect behavior when loading constants using MTVSRBMI in big-endian configurations. (cherry picked from commit 23b3203)
…vailable on Darwin (llvm#153912) For backwards compatibility reasons the `ptrauth_qualifier` and `ptrauth_intrinsic` features need to be testable with `__has_feature()` on Apple platforms, but for other platforms this backwards compatibility issue does not exist. This PR resolves these issues by making the `ptrauth_qualifier` and `ptrauth_intrinsic` tests conditional upon a darwin target. This also allows us to revert the ptrauth_qualifier check from an extension to a feature test again, as is required on these platforms. At the same time we introduce a new predefined macro `__PTRAUTH__` that answers the same question as `__has_feature(ptrauth_qualifier)` and `__has_feature(ptrauth_intrinsic)` as those tests are synonymous and only exist separately for compatibility reasons. The requirement to test for the `__PTRAUTH__` macro also resolves the hazard presented by mixing the `ptrauth_qualifier` flag (that impacts ABI and security policies) with `-pedantics-errors`, which makes `__has_extension` return false for all extensions. --------- Co-authored-by: Aaron Ballman <aaron@aaronballman.com> (cherry picked from commit 624b724)
llvm#154109) When parsing a block expression we were not entering a new eval context and as a result when parsing the block body we continue to treat any return statements as discarded so infer a `void` result. This fixes the problem by introducing an evaluation context around the parsing of the body. (cherry picked from commit ec4e6aa)
…152596) This updates the pointer authentication documentation to include a complete description of the existing functionaliy and behaviour, details of the more complex aspects of the semantics and security properties, and the Apple arm64e ABI design. Co-authored-by: Ahmed Bougacha Co-authored-by: Akira Hatanaka Co-authored-by: John Mccall --------- Co-authored-by: Ahmed Bougacha <ahmed@bougacha.org> Co-authored-by: Akira Hatanaka <ahatanak@gmail.com> Co-authored-by: John Mccall <rjmccall@apple.com> (cherry picked from commit 62d2a8e)
…he demanded shuffle mask elts before testing for a matching shuffle (llvm#153554) When lowering using sublane shuffles, we can sometimes end up with the same mask as we started with. We already bail in these occasions, but we weren't fully simplifying the new shuffle mask before testing if it matched. Fixes llvm#153457 (cherry picked from commit c96d0da)
The changes from llvm#136855 missed a change with atomic assignment constraints. This fixes a bug where we'd accidentally drop a non-atomic-to-atomic conversion step. Fixes llvm#154157 co-authored-by: @ahatanak (cherry picked from commit ae434cd)
… in PATH (llvm#149597) The checks for detecting if `clang-cl` and `lld-link` are in `%PATH` were wrong. This fixes the comment in llvm#135446 (comment) (cherry picked from commit 13391ce)
…VTTP2UI nodes without AVX512VL (llvm#154504) Unlike CVTTP2SI, CVTTP2UI is only available on AVX512 targets, so we don't fallback to the AVX1 variant when we split a 512-bit vector, so we can only use the 128/256-bit variants if we have AVX512VL. Fixes llvm#154492 (cherry picked from commit d770567)
Used in follow up to parse slices of buffer. (cherry picked from commit 217f9e5)
It was pointed out in the review that this ended up in the wrong place! Moving it. Note this is a modification of the 21.x release notes, so doesn't have a corresponding trunk change.
…vm#152978) Introduces the use of pointer authentication to protect the invocation, copy and dispose, reference, and descriptor pointers in Objective-C block objects. Resolves llvm#141176
llvm#153700) Pointer auth protection of the block descriptor pointer is only supported in some constrained environments so we do actually need it to be configurable. We had made it non configurable in the first PR to protect block metadata because we believed that was an option but subsequently realised it does need to remain configurable. This PR revives the flags that permit this.
…3849) Auto-generated decoder fails to add the $sgp10 operand because it has no encoding bits. Work around this by adding the missing operand after decoding is complete. Fixes llvm#153829. (cherry picked from commit 76d993b)
(cherry picked from commit 0fff460)
…nateNames (llvm#154837) Fixes: llvm#154595 Prior to commit bbc8346, this flag was set by `insert()` from `addUndefined()`. Set it explicitly now. (cherry picked from commit a6fcd1a)
…PP_COMPRESSED_PAIR (llvm#154559) This patch adds unit tests to catch the regression described in llvm#154146. At the moment, these tests are pinning down the post-break ABI. (cherry picked from commit 2a83cf5)
…PAIR (llvm#154686) LLVM 20 contained an ABI break that can result in the size of `std::unordered_{map,set,multimap,multiset}` and `std::deque` changing when used with an allocator type that is empty and contains a base class that is the same across rebound allocator instantiations (e.g. ``Allocator<int>`` and ``Allocator<char>`` are both empty and contain the same base class). In addition, the layout of a user-defined type that: - contains one of the following containers: `std::unordered_{map,set,multimap,multiset}`, `std::deque`, `std::map`, `std::set`, `std::multimap`, `std::multiset`, `std::list` or `std::vector`, and - passes an empty allocator, comparator or hasher type to that container, and - has a member of that same empty allocator, comparator or hasher type inside the enclosing struct, and - that member is either marked with `[[no_unique_address]]` or optimized out via the EBO (empty base optimization) technique saw its size increase from LLVM 19 to LLVM 20. This was caused by the usage of `[[no_unique_address]]` within some of libc++'s containers in a way that allowed subtle interactions with enclosing objects. This is fixed in LLVM 21 on Clang (returning to the LLVM 19 ABI), however that implies an ABI break from LLVM 20 to LLVM 21. Furthermore, fixing this causes a slight regression to constant evaluation support in `std::unique_ptr`. Specifically, constant evaluation will now fail when the deleter relies on being value-initialized for constant-evaluation admissibility. If a default-initialized deleter can be used during constant evaluation, or if the default constructor is non-trivial, the `unique_ptr` is not affected by this regression. In particular, this regression does not impact any `unique_ptr` using the default deleter. Note that there is currently no way to realistically fix this ABI break on GCC, therefore GCC will remain on the ABI introduced in LLVM 19. That also means that Clang and GCC will have a slightly different ABI for the small subset of types listed above until we are able to apply the same fix we did with Clang on GCC. We fix this regression by surrounding the members of the `_LIBCPP_COMPRESSED_PAIR` with an anonymous struct. This restricts the shifting of empty types to the front of the `_LIBCPP_COMPRESSED_PAIR` instead of throughout the surrounding object. This "frees up" the zero offset to contain another object of the same type, restoring the ability to perform EBO or to elide the storage for a type with `[[no_unique_address]]` in the enclosing (user-defined) struct. Fixes llvm#154146 Co-authored-by: Louis Dionne <ldionne.2@gmail.com>
…tructExpr()` (llvm#154610) When initializing an anonymous struct via an `IndirectFieldDecl`, we create an `APValue` for the struct, but we leave the fields uninitialized. This would later cause the `CXXConstructExpr` that initializes the anonymous struct member to not do anything since its `APValue` already had a value (but the member didn't). Just remove the check for an `APValue` that already has a value from `RecordExprEvaluator::VisitCXXConstructExpr()`. Fixes llvm#154567
…m#147591) The pattern would produce an invalid slice when some dimensions were both sliced and broadcast.
…148666) This PR fixes a use-after-free error that happens when `DistinctAttr` instances are created within a `PassManager` running with crash recovery enabled. The root cause is that `DistinctAttr` storage is allocated in a thread_local allocator, which is destroyed when the crash recovery thread joins, invalidating the storage. Moreover, even without crash reproduction disabling multithreading on the context will destroy the context's thread pool, and in turn delete the threadlocal storage. This means a call to `ctx->disableMulthithreading()` breaks the IR. This PR replaces the thread local allocator with a synchronised allocator that's shared between threads. This persists the lifetime of allocated DistinctAttr storage instances to the lifetime of the context. ### Problem Details: The `DistinctAttributeAllocator` uses a `ThreadLocalCache<BumpPtrAllocator>` for lock-free allocation of `DistinctAttr` storage in a multithreaded context. The issue occurs when a `PassManager` is run with crash recovery (`runWithCrashRecovery`), the pass pipeline is executed on a temporary thread spawned by `llvm::CrashRecoveryContext`. Any `DistinctAttr`s created during this execution have their storage allocated in the thread_local cache of this temporary thread. When the thread joins, the thread_local storage is destroyed, freeing the `DistinctAttr`s' memory. If this attribute is accessed later, e.g. when printing, it results in a use-after-free. As mentioned previously, this is also seen after creating some `DistinctAttr`s and then calling `ctx->disableMulthithreading()`. ### Solution `DistinctAttrStorageAllocator` uses a synchronised, shared allocator instead of one wrapped in a `ThreadLocalCache`. The former is what stores the allocator in transient thread_local storage. ### Testing: A C++ unit test has been added to validate this fix. (I was previously reproducing this failure with `mlir-opt` but I can no longer do so and I am unsure why.) ----- Note: This is a 2nd attempt at my previous PR llvm#128566 that was reverted in llvm#133000. I believe I've addressed the TSAN and race condition concerns.
This PR adds a null check for dyn_cast result before use to prevent crash, and use `isa` instead `dyn_cast` to make code clean. Fixes llvm#148619.
…ARNINGS is ON (llvm#99) Signed-off-by: Zhu, Shaojie <shaojie.zhu@intel.com>
…and-line parser reset When LLVM TableGen flags are globally appended, mlir-src-sharder fails with an unknown argument error because its command-line parser is reset during execution and does not retain the inherited flags.
After the original API change to DefaultTimingManager::setOutput() (see 362aa43), users are forced to provide their own implementation of OutputStrategy. However, default MLIR implementations are usually sufficient. Expose Text and Json strategies via factory-like method to avoid the problem in downstream projects.
…vm#159766) Support custom types (3/N): allow custom tensor and buffer types in function signatures and at call-sites. This is one of the major building blocks to move in the direction of module-level one-shot-bufferization support. To achieve this, `BufferizationOptions::FunctionArgTypeConverterFn` callback is converted to work with tensor-like and buffer-like types, instead of the builtin counterparts. The default behavior for builtins remains unchanged, while custom types by default go through `TensorLikeType::getBufferType()` which is a general conversion interface.
…vm#170) Duplicate of PR merged on npu/release/19.x llvm#151 Cherry-pick from upstream llvm: llvm#147721; adding std::move to getChecked method.
…#167705) Generally, to_tensor and to_buffer already perform sufficient verification. However, there are some unnecessarily strict constraints: * builtin tensor requires its buffer counterpart to always be memref * to_buffer on ranked tensor requires to always return memref These checks are assertions (i.e. preconditions), however, they actually prevent an apparently useful bufferization where builtin tensors could become custom buffers. Lift these assertions, maintaining the verification procedure unchanged, to allow builtin -> custom bufferizations at operation boundary level.
…ted code (llvm#168536) (llvm#181) ODS generate code can be included and used outside of the `mlir` namespace and so references to symbols in the mlir namespace must be fully qualified.
The recent changes in the MLIR TableGen interface for generated
OpTy::build functions involves a new OpTy::create function that is
generated passing arguments without forwarding. This is problematic with
arguments that are move only such as `std::unique_ptr`. My particular
use case involves `std::unique_ptr<mlir::Region>` which is desirable as
the `mlir::OperationState` object accepts calls to
`addRegion(std::unique_ptr<mlir::Region>`.
In Discord, the use of `extraClassDeclarations` was suggested which I
may go with regardless since I still have to define the builder function
anyways, but perhaps you would consider this trivial change as it
supports a broader class of argument types for this approach.
Consider the declaration in TableGen:
```
let builders = [
OpBuilder<(ins "::mlir::Value":$cdr,
"::mlir::ValueRange":$packs,
"std::unique_ptr<::mlir::Region>":$body)>
];
```
Which currently generates:
```cpp
ExpandPacksOp ExpandPacksOp::create(::mlir::OpBuilder &builder, ::mlir::Location location, ::mlir::Value cdr, ::mlir::ValueRange packs, std::unique_ptr<::mlir::Region> body) {
::mlir::OperationState __state__(location, getOperationName());
build(builder, __state__, std::forward<decltype(cdr)>(cdr), std::forward<decltype(packs)>(packs), std::forward<decltype(body)>(body));
auto __res__ = ::llvm::dyn_cast<ExpandPacksOp>(builder.create(__state__));
assert(__res__ && "builder didn't return the right type");
return __res__;
}
```
With this change it will generate:
```cpp
ExpandPacksOp ExpandPacksOp::create(::mlir::OpBuilder &builder, ::mlir::Location location, ::mlir::Value cdr, ::mlir::ValueRange packs, std::unique_ptr<::mlir::Region>&&body) {
::mlir::OperationState __state__(location, getOperationName());
build(builder, __state__, static_cast<decltype(cdr)>(cdr), std::forward<decltype(packs)>(packs), std::forward<decltype(body)>(body));
auto __res__ = ::llvm::dyn_cast<ExpandPacksOp>(builder.create(__state__));
assert(__res__ && "builder didn't return the right type");
return __res__;
}
```
Another option could be to make this function a template but then it
would not be hidden in the generated translation unit. I don't know if
that was the original intent. Thank you for your consideration.
…eConversionsAndLegality (llvm#160344) In a downstream project, there is a need for a type conversion pattern for scf.index_switch operation. A test is added into `mlir/test/Dialect/SparseTensor/scf_1_N_conversion.mlir` (not sure this functionality is really required for sparse tensors, but the test showcase that the new conversion pattern is functional)
…lvm#87948) This commit moves the code responsible for adding newlines and tracking indent, so that it can be used not only for operation printers, but also for attribute and type printers. It could be useful for nested attributes, where proper formatting with newlines and indents would benefit the readability of the IR. Currently, everything is printed on one line, which makes it difficult to read if the attribute is more verbose and there are multiple levels of nesting. Co-authored-by: Andruszkiewicz, Jacenty <andruszkiewicz.jacenty@intel.com>
…torage types in Quant dialect (llvm#152966) (intel-staging#14) Currently, UniformQuantizedType only supports built-in MLIR storage types such as Integer. LLM quantization research introducing feature of using NF4 as a low precision datatype (see https://arxiv.org/pdf/2305.14314). There is a growing need to make the system extensible and maintainable as more types are added. Ensuring that MLIR can natively support NF4 through a clean, extensible interface is essential for both current and future quantization workflows. **Current Approach and Its Limitations:** - The present implementation relies on dynamic checks (e.g., type switches or if-else chains) to determine the storage type and retrieve type-specific information for legality checks. - This approach works for a small, fixed set of types, but as the number of supported types grows, the code becomes harder to read, maintain, and extend. **Proposed Interface-Based Approach:** - Define a StorageTypeInterface that specifies the required methods any storage type must implement to be used in UniformQuantizedType. - Each storage type (Integer, Float8E5M2, Float8E4M3FN, and new types like NF4) would implement this interface, encapsulating their type-specific logic. - When UniformQuantizedType needs to check legality or retrieve information, it can use MLIR’s dyn_cast mechanism to check if the type implements the interface and then call the required methods. - This design decouples UniformQuantizedType from the specifics of each storage type, making it easy to add new types (such as NF4) without modifying the core logic or introducing more type checks. **Benefits:** - Extensibility: New storage types can be added by simply implementing the interface, without touching the core UniformQuantizedType logic. - Readability: The code is cleaner, as it avoids large switch statements or if-else chains. - Maintainability: Type-specific logic is encapsulated within each type, reducing the risk of errors and making the codebase easier to understand and update. Co-authored-by: Roman-Pevnyi <166020069+Roman-Pevnyi@users.noreply.github.com>
Current implementation of MLIRContext's action handling, requires the user to hold the stored memory for any observer or breakpoint manager added to the execution context. Using a getter for the registered action handler, permits the user to store the observers and breakpoints into MLIRContext and modify their state later, by retrieving the action handler and invoking the functor's target. Mainly for attaching new observers later in the compilation pipeline, after the execution context got registered.
vsimion26
force-pushed
the
QuantileTypeCherryPick
branch
2 times, most recently
from
June 15, 2026 09:10
13679c0 to
5c684d6
Compare
ZoranZomborat
approved these changes
Jun 15, 2026
hrotuna
previously approved these changes
Jun 16, 2026
Quantization** Recent [community work](https://discourse.llvm.org/t/rfc-extending-uniformquantizedtype-with-interface-based-support-for-new-storage-types-in-quant-dialect/87803) (RFC by Roman-Pevnyi, Aug 2025) successfully extended UniformQuantizedType with a StorageTypeInterface. This made the quantization framework extensible, allowing new storage types (Integer, Float8E5M2, Float8E4M3FN, NF4) to be plugged in without modifying core quantization logic. QuantileType follows the same interface-driven philosophy but addresses a different level of abstraction: the storage type itself. The observation: Many low-precision storage types (ui4, si8, f8, NF4) can be enhanced with a quantile lookup table. Rather than creating a new complete quantized type for each variant (QuantileQuantizedType, QuantileQuantizedPerAxisType, etc.), we insert an abstraction layer. QuantileType is a builtin that wraps any storage type with quantile metadata: quantile<ui4:f16, {-1.0, -0.696, ..., 1.0}> quantile<si8:f32, {-2.0, -1.0, 0.0, 1.0, 2.0}> quantile<f8E4M3FN:f16, {...}> This storage abstraction then composes naturally with existing quantization: !quant.uniform<quantile<ui4:f16, {...}>:f32, scale:zeropoint> Roman's StorageTypeInterface allows UniformQuantizedType to work with any compliant storage type. QuantileType extends this by making it possible to augment any storage type WITH quantile information, creating composable layers: Builtin QuantileType (unified storage + quantiles) ↓ (implements StorageTypeInterface) UniformQuantizedType (uniform quantization logic) ↓ Hardware-specific lowering 1. SINGLE INTERFACE FOR ALL QUANTILE SCHEMES With QuantileType: One parameterized abstraction quantile<ui4:f16, {nf4_table}> quantile<ui4:f16, {custom_table}> All compose with !quant.uniform naturally [NF4 Type](https://github.com/openvinotoolkit/npu_compiler/blob/90b6098b9ee96055d633dc520354434bae22e336/src/vpux_compiler/include/vpux/compiler/core/types/quantile_float/types.hpp#L62) & [NF4 Table](https://github.com/openvinotoolkit/npu_compiler/blob/90b6098b9ee96055d633dc520354434bae22e336/src/vpux_compiler/src/core/types/quantile_float/types.cpp#L116-L132) 2. EXTENSIBILITY FOR NEW STORAGE TYPES When a new low-fp storage type is added (FP3, FPx, etc.), it automatically works with quantiles: quantile<fpx:f16, {...}> No new dialect types needed. The type just implements StorageTypeInterface and QuantileType wraps it. 3. CLEAN SEPARATION OF CONCERNS StorageTypeInterface: - Defines what storage types must provide (width, signedness, min/max values) QuantileType (builtin abstraction): - Wraps any StorageTypeInterface-compliant type with a lookup table - Acts as a "storage + quantile mapping" layer UniformQuantizedType (quantization logic): - Works with any StorageTypeInterface, including QuantileType In uniform quantization context: !elem_type = !quant.uniform<quantile<ui4:f16, {-1.0, ..., 1.0}>:f32, 0.01:128> QuantileType is a natural extension of the StorageTypeInterface architecture. It: - Provides a unified abstraction for quantile-enhanced storage - Avoids type explosion by parameterizing rather than creating variants - Maintains clean separation between storage concerns and quantization logic - Enables portability and reusability across MLIR consumers
Since the merge of this PR(llvm#190321) there were some issues identified, such as QuantileType not being added in the ByteCode files. This PR focuses on fixing these missing pieces which should make QuantileType a complete and functional type. Signed-off-by: vsimion26 <vlad.simion@intel.com>
vsimion26
force-pushed
the
QuantileTypeCherryPick
branch
from
June 17, 2026 08:40
5c684d6 to
f05e1da
Compare
nikita-kud
previously approved these changes
Jun 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
QuantileType upstream PR - llvm#190321
QuantileType Bytecode patch PR - llvm#203495
JIRA ticket
Related PR in NPU Compiler and/or OpenVINO repository with sub-module update
Other related tickets