Skip to content
This repository was archived by the owner on Oct 28, 2026. It is now read-only.

fix(shader): detect f32 product tininess after rounding, as RDNA2 does - #15

Closed
AlxcBTW wants to merge 1 commit into
mainfrom
fix/shader-tininess-after-rounding-v2
Closed

AlxcBTW wants to merge 1 commit into
mainfrom
fix/shader-tininess-after-rounding-v2

Conversation

@AlxcBTW

@AlxcBTW AlxcBTW commented Oct 10, 2026

Copy link
Copy Markdown
Owner

What

With f32 denormals flushed, flushTinyProduct decides whether a product is tiny. It did that on the exact value, from the exponents and a UMulHi of the significands. RDNA2 detects tininess after rounding to 24 bits with an unbounded exponent, which TechnicalDebt records as measured on gfx1036 for v_mul_f32 and v_mullit_f32: 0x3f7ffffe * 0x00800001 gives 0x00800000 on the hardware and 0 here.

The check now multiplies one factor by 2^64 and multiplies the result by the other factor. Scaling by 2^64 is exact below 2^64, so the second multiply rounds the exact product to 24 bits once, in the normal range. Its magnitude bits are then compared with those of 2^-62 (0x20800000), and a zero product is not tiny, as before. A factor of 2^64 or more cannot give a tiny product, since the other factor is zero or at least 2^-126. NaN and infinity compare as not tiny.

  • 0x3f7ffffe * 0x00800001 now gives 0x00800000, as on RDNA2. F32ConstantFactorTininess pins this lane.
  • The check uses 7 IR instructions per product instead of 20, and uses no UMulHi. hasUnitSignificand was only needed for the carry and is removed.
  • The TechnicalDebt entry for f32 denormals describes the new rule. The fma forms with a zero addend and v_mul_legacy_f32 use it too, but their rounding point is not measured.

Why

The ps5-opengl UI renderer's large pixel shader (VITA5 PPSA39410, Homebrew UI Lab PPSA99050) has 1,587 checked products. On an RTX 5070 Laptop (driver 616.92), creating its graphics pipeline takes 14.7 s with a cold driver cache. UI Lab waits at most 2 s for a submission to retire, so it stops at submission retirement failed on every cold start.

I timed the same SPIR-V module, dumped from Pipeline.cpp, in a standalone Vulkan program. Each variant below replaces one emulation pattern with a pass-through. These are timing experiments only, and the variants are not correct:

Module NVIDIA pipeline creation
as emitted 14.9-16.4 s
without the tiny-product select 8.3 s
without the NaN-result selects 9.8 s
without the denormal-flush selects 12.6 s
without all three 1.4 s

With this change:

before after
VITA5's pipeline in the title, NVIDIA, cold driver cache 14.7 s 10.8 s
the dumped module standalone, NVIDIA 14.9-16.4 s 12.0 s
the dumped module standalone, AMD Radeon 880M 49.9 s 21.4 s
fragment shader SPIR-V 570,063 words 484,495 words

That still exceeds UI Lab's 2 s wait; the NaN-result and denormal-flush selects are the other two large costs.

Tested

Windows 11, WinLibs GCC 15.2.0 r7, NVIDIA RTX 5070 Laptop (616.92) and AMD Radeon 880M (AMDVLK 2.0.353).

  • The 48 agc_driver float tests (ctest -R "^agc_driver_(f32|f16|f64|vop|mix|fma|sin|interpolation|denormal|nan)") pass on the NVIDIA and on the AMD GPU (ANYPS5_GPU=AMD).
  • agc_driver_f32_constant_factor_tininess fails without the source change (lane 2 no float mode v_mul_f32 0x3f7ffffe is 0x00000000, expected the product that rounds to 24 bits at 2^-126 0x00800000) and passes with it.
  • The new rule agrees with rounding the exact product to 24 bits with an unbounded exponent on 297,572 random factor pairs whose product is near 2^-126 (a host-side model, not a GPU run).
  • check_conventions.py passed.

Checklist

  • Based on current main; no other open PR implements the same functions
  • One topic per PR; follow-ups go in a new PR
  • No comments in code except technical debt
  • Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed in TechnicalDebt
  • No notes, investigation .md files or images added to the repository
  • New third-party code is a submodule built from source
  • Generic behaviour, not specific to one title
  • Depends on: none
  • AI-assisted: yes (Claude Code)

🤖 Generated with Claude Code

With f32 denormals flushed, a product was checked for tininess on its
exact value, from the factor exponents and a UMulHi of the significands.
RDNA2 detects tininess after rounding to 24 bits with an unbounded
exponent, as measured on gfx1036 for v_mul_f32 and v_mullit_f32:
0x3f7ffffe * 0x00800001 gives 0x00800000 there and gave 0 here.

One factor is now multiplied by 2^64, which is exact below 2^64, and the
product with the other factor, rounded once to 24 bits in the normal
range, is compared with 2^-62 on its magnitude bits. A zero product is
still not tiny, and a factor of 2^64 or more can't give a tiny product
since the other factor is zero or at least 2^-126.

The check takes 7 IR instructions instead of 20, which makes the
SPIR-V of float-heavy pixel shaders smaller and quicker for drivers to
compile.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@AlxcBTW

AlxcBTW commented Oct 10, 2026

Copy link
Copy Markdown
Owner Author

Replaced by the next PR, rebased onto main be40308 (TechnicalDebt conflict with boykopovar#1290).

@AlxcBTW AlxcBTW closed this Oct 10, 2026
@AlxcBTW
AlxcBTW deleted the fix/shader-tininess-after-rounding-v2 branch October 10, 2026 11:22
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant