Skip to content

fix(shader): make v_sin and v_cos linear within 2^-12 of a turn from a zero - #2380

Merged
adriadam10 merged 3 commits into
boykopovar:mainfrom
SleglJan:fix/shader-trig-quarter-fold
Oct 10, 2026
Merged

adriadam10 merged 3 commits into
boykopovar:mainfrom
SleglJan:fix/shader-trig-quarter-fold

Conversation

@SleglJan

@SleglJan SleglJan commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

What

Follow-up to #1979, which makes the cycle exact. Two errors remain after it, both in the step from the cycle to the driver's sine and cosine of the radians:

  • A tiny cycle is multiplied by 2π and handed to the driver's sin, whose absolute error (about 2^-22 on NVIDIA) swallows the result: through the sweep below on an RTX 3080 (615.71.09), 12,930 of 32,832 v_sin_f32 inputs come back as zero for a finite hardware result (6,465 of each sign), and v_sin_f32 of −2^-126 is −0 where the hardware returns 0x81c90fd5 (−7.4e-38).
  • Near the zeros of sine at k + 0.5 and of cosine at k ± 0.25 the f32 radians drop the low bits of the cycle: v_cos_f32 of 0x3e7fffff (0.24999999) is 0 on the RTX 3080 where the hardware returns 0x33c90fd9 (9.4e-8), and with correctly rounded radians the result is still up to 51% off, as fix(shader): reduce v_sin_f32 and v_cos_f32 inputs exactly #1979's description says.

Measured on RDNA2 (gfx1036, Ryzen 7950X3D iGPU) with tools/hw-oracle (Linux, CachyOS kernel 7.2.9, hsa-rocr 7.2.4, clang 23.1.1 as the assembler; IEEE 0, f32 denormals flushed and also kept, f16 kept, DX10_CLAMP 1, RNE) over every f16 input, 32,832 f32 rows (every exponent with 64 mantissas of both signs, 13 specials, 26 cycle counts around 256 and beyond, 25 zero rows of padding), a dense sample of 20,480 inputs in [0, 0.25], and 1,920 inputs at 2^-k (1 + m/16) off the half and quarter turns:

  • below 2^-12 of a turn the hardware returns the input times 6.2831826 (0x40c90fd5, six ULP below the f32 2π): 16 of 16 mantissas at every exponent from 2^-13 down to 2^-22, 8 of 16 at 2^-12, never the f32 2π;
  • at a distance q below 2^-12 from the half and quarter turns its result is 4 to 14 ULP above q × 0x40c90fd5, so within 7 ULP of q × 2π, and no single constant reproduces it;
  • it gives sin 0 and cos 1 for |x| ≥ 2^23, differs from the exact fraction by up to 10 ULP just below integers and up to 15 ULP beyond ±256 cycles, and no odd polynomial of degree 9 to 13 fitted to the dense sample by least squares reproduces its sine (9 ULP off).

Within 2^-12 of a turn from a zero of the function the lowering now returns the linear term of the exact distance to that zero (0.5 − |r| for the sine and 0.25 − |r| for the cosine, both exact by Sterbenz's lemma, |r| itself at the integers), with the hardware's constant at the integers and the f32 2π at the half and quarter turns; the sine's sign is applied on the bits. Everywhere else the driver's sine or cosine of the radians stays as it is, so RADV keeps selecting its native v_sin_f32 and v_cos_f32 on the cycle (RADV_DEBUG=shaders on the new test shows v_rndne_f32, v_sin_f32 and v_cos_f32, no v_fract_f32). The translation's zero, infinite and NaN picks are unchanged. A fold of the whole quarter turn was measured too and rejected: with the same 2^-12 band it moves 848 of the 20,480 dense cosines on RADV by up to 15 ULP (the hardware's cosine is not its sine of the complement) and gains the RTX 3080 nothing outside the band (1,166 against 1,149 ULP).

The sweep through the recompiler (f32 with denormals flushed), results equal to the hardware bits and the largest ULP distance, #1979 → this PR:

RADV (Mesa 26.2.4, the iGPU) RTX 3080 (615.71.09)
v_sin_f16 65,536 of 65,536 → 65,536 63,086 of 65,536, max 4 → 65,230, max 1
v_cos_f16 65,536 → 65,536 65,512, max 1 → 65,514, max 1
v_sin_f32 32,424 of 32,832, max 15 → 32,422, max 15 14,864, max 8,388,062, 12,930 zero for a finite value → 29,456, max 1,166, none
v_cos_f32 32,488, max 15 → 32,486, max 15 29,092, max 163, 2 zero for a finite value → 29,092, max 163, none

Input by input: on RADV the change touches 4 of the 32,832 f32 inputs (the two sines within 2^-14 of ±0.5 by 1 ULP, the two cosines at ±0.24999999 by 2 ULP) and 10 of the 20,480 dense cosines by up to 5 ULP, all inside the bands; on the RTX 3080 14,618 sine and 2 cosine results move closer to the hardware and none away (14,592 sines become the hardware's bits), and 2,144 f16 sines become exact. What remains on the RTX 3080 is the driver's: its sine at the edge of the band (1,166 ULP at 0xc57fffff, whose cycle is exactly 2^-12, the first value outside it) and its cosine inside the quarter turn (1,149 ULP on the dense sample at 0x3e7fbc00). For reference, at main before #1979: RADV v_sin_f32 24,529 exact with a maximum of 855,638,025 ULP and v_cos_f32 32,285 with 202; RTX 3080 v_sin_f32 14,752 exact, 6,466 zero for a finite value, maximum 847,249,963, and v_cos_f32 28,864 with 1,746,018,777.

The other transcendental ops are unchanged and recorded in the same TechnicalDebt entry: RADV returns the hardware's bits for every f16 and f32 input of rcp, rsq, sqrt and log and all but one of exp (f16 0x11c5, 1 ULP); NVIDIA stays within 2 ULP except its log2 between 0.52 and 1.32 (40 inputs over 2 ULP, up to 24 million at 0x3f7fffff).

Data (the sweeps, the hosts' summary at main and their results at #1979 and with this PR input by input, the dense sample and the fold measurements, scripts): https://github.com/SleglJan/AnyPS5/tree/audit/oracle-2026-10-08/oracle-audit/transcendentals.

Tested

Linux (CachyOS, kernel 7.2.9, GCC 16.2.1), Release, ANYPS5_ENABLE_SPIRV_TOOLS=ON, RTX 3080 and RADV, on this branch (#1979 at e6eb64a plus one commit).

  • New agc_driver_sin_cos_near_zero: 32 inputs per type in each lane (f32 and f16: tiny and small negatives, ±6.1e-5, the neighbours of ±0.5 and ±0.25, the cardinal points, 255.5, 256.25, −1000.5, 3.4e38, ±0, inf, NaN, eight ordinary values), pinned on the oracle in the titles' mode and run in FLOAT_MODE 0xc0; zero, infinite and NaN results must match the bits, the others are compared in ULPs with a per-lane tolerance: 0 for 18 lanes (7 of which take ±1 from the driver's sine or cosine of the f32 π/2 and π, exact on RADV, the RTX 3080 and lavapipe), 2 for the ±6.1e-5 lanes (the cosine's maximum, where RADV returns the hardware's 0x3f7ffffe, NVIDIA 0x3f7ffffd and Intel ANV 0x3f800000), 1 for the ±0.49999997 lanes, 2 for the ±0.24999999 lanes, 12 for the eight ordinary lanes (measured maxima there: RADV 0, NVIDIA 9). Written first: on fix(shader): reduce v_sin_f32 and v_cos_f32 inputs exactly #1979 alone it passes on RADV, whose native instruction already covers the bands, and fails on the RTX 3080 at lane 0 (v_sin_f32 of −2^-126 is 0x80000000, expected 0x81c90fd5); with the change it passes, three repeats on each GPU.
  • agc_driver_sin_cos_reduction (fix(shader): reduce v_sin_f32 and v_cos_f32 inputs exactly #1979's test), agc_driver_vop1_float_unary, agc_driver_float16_misc, agc_driver_float16_result_modifiers, agc_driver_formatted_store, agc_driver_image_sample_unnormalized and agc_driver_typed_heaps pass on both GPUs, three repeats; full ctest 499/499 on each GPU. (Unrelated to this branch: on main d70b899 agc_driver_dcc_keys_first_write fails on the RTX 3080 and agc_driver_storage_cache_budget on the iGPU without any change.)
  • tools/check_conventions.py (against fix(shader): reduce v_sin_f32 and v_cos_f32 inputs exactly #1979's head) and tools/check_test_registration.py pass.
  • Fork CI on this branch: https://github.com/SleglJan/AnyPS5/actions/runs/37960569207 at e58e1a6 (this code; the later commits only reword the TechnicalDebt entry and widen the ±6.1e-5 lanes to 2 ULP, 036e825): Linux with lavapipe 518/518 including agc_driver_sin_cos_near_zero, Windows 509/509 with the AGC GPU tests skipped (no Vulkan device), relinker matrix green.
  • Native Windows (popcoder565, RTX 3060 Laptop, NVIDIA 616.92, head 036e825 on main 8b393b6): agc_driver_sin_cos_near_zero and agc_driver_sin_cos_reduction pass 5 of 5 each; full suite 513 of 513.

Checklist

  • Based on current main; no other open PR implements the same functions
  • One topic per PR; follow-ups go in a new PR
  • No comments in code except technical debt
  • Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed in TechnicalDebt
  • No notes, investigation .md files or images added to the repository (attach them to this PR)
  • New third-party code is a submodule built from source
  • Generic behaviour, not specific to one title
  • Depends on: fix(shader): reduce v_sin_f32 and v_cos_f32 inputs exactly #1979
  • AI-assisted: yes (Claude Code)

🤖 Generated with Claude Code

https://claude.ai/code/session_012GLfKH7Sqe51wX254k1XsV

@Kikoarl

Kikoarl commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

Linux run: Linux 7.2 (x86_64), Intel(R) Graphics (RPL-S) (Mesa 26.3.0-devel, Vulkan 1.4.363), GCC 16.2.1, Release, -DBUILD_TESTING=ON, #2380 (61d84dd5) tested on current main (0fdda7be).

Found a tolerance failure on Intel Mesa Vulkan (ANV driver):

  • tests/agc_driver_sin_cos_near_zero_tests: fails at lane 6:
    AGC graphics: sin cos near zero: lane 6 v_cos_f32 is 0x3f800000, expected 0x3f7ffffe (tolerance 1 ulp)
    
    On Intel ANV, v_cos_f32 in lane 6 rounds to exactly 1.0f (0x3f800000), which is 2 ULPs away from 0x3f7ffffe. Bumping the tolerance for lane 6 to 2 ULPs would allow passing cleanly on Intel Vulkan implementations alongside RADV and NVIDIA.
  • tests/agc_driver_sin_cos_reduction_tests: passes cleanly.
  • tools/check_conventions.py: passes cleanly.

SleglJan added a commit to SleglJan/AnyPS5 that referenced this pull request Oct 9, 2026
…near_zero

The ±6.1e-5 lanes take the driver's cosine of a tiny angle, where RADV
returns the hardware's 0x3f7ffffe, NVIDIA 0x3f7ffffd and Intel ANV
0x3f800000 (reported by Kikoarl on boykopovar#2380), so the lanes tolerate 2 ULP
instead of 1.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012GLfKH7Sqe51wX254k1XsV
@SleglJan

SleglJan commented Oct 9, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the Intel run. Lanes 6 and 7 (±6.1e-5) take the driver's cosine of a tiny angle, where the three drivers now measured give three neighbouring values: RADV the hardware's 0x3f7ffffe, the RTX 3080 0x3f7ffffd, ANV 0x3f800000. Pushed 036e825 with tolerance 2 for those two lanes; the other lanes are unchanged. If the test gets past lane 7 on ANV, the remaining risk is the lanes that expect ±1.0 from the driver's sine of the f32 π/2 (tolerance 0, exact on RADV, NVIDIA and lavapipe); a re-run would tell.

AI-assisted: yes (Claude Code).

@popcoder565

Copy link
Copy Markdown
Contributor

Native Windows run on another NVIDIA card, since the measurements above used an RTX 3080 on Linux. Setup: Windows 10 (19045), RTX 3060 Laptop (NVIDIA 616.92), MinGW-w64 GCC 15.2.0 (WinLibs r7), Release. Head 036e825a (with #1979's e6eb64aa), merged onto main 8b393b6.

  • agc_driver_sin_cos_near_zero and agc_driver_sin_cos_reduction each pass 5 of 5.
  • The full suite passes in that build (513 of 513).

No title was run.

🤖 Addressed by Claude Code

SleglJan and others added 3 commits October 10, 2026 09:39
…a zero

After boykopovar#1979 the cycle is exact, but the driver's sine of the radians loses
tiny results to its absolute error, and near the zeros of sine at k + 0.5
and of cosine at k ± 0.25 the f32 radians drop the low bits of the cycle.
Measured on RDNA2 (gfx1036), the hardware returns the input times 6.2831826
for every input below 2^-12 of a turn and stays within 7 ULP of the f32 2π
times the distance at the half and quarter turns.

Within 2^-12 of a turn from a zero the lowering now returns the linear term
of the exact distance to that zero, with the hardware's constant at the
integers and the f32 2π elsewhere; everywhere else it keeps the driver's
sine or cosine of the radians. agc_driver_sin_cos_near_zero pins 32 inputs
per type from the oracle with per-lane tolerances.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012GLfKH7Sqe51wX254k1XsV
…near_zero

The ±6.1e-5 lanes take the driver's cosine of a tiny angle, where RADV
returns the hardware's 0x3f7ffffe, NVIDIA 0x3f7ffffd and Intel ANV
0x3f800000 (reported by Kikoarl on boykopovar#2380), so the lanes tolerate 2 ULP
instead of 1.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012GLfKH7Sqe51wX254k1XsV
Co-Authored-By: Claude <noreply@anthropic.com>
@adriadam10
adriadam10 force-pushed the fix/shader-trig-quarter-fold branch from 036e825 to 64193dd Compare October 10, 2026 07:43
@adriadam10

Copy link
Copy Markdown
Collaborator

Checked on the HW oracle (RDNA2, IEEE 0, f32 denormals flushed, f16 kept):

  • All 32 rows × 4 results of SinCosNearZero are bit-identical with the hardware.
  • Inside the bands against this PR's formula: the sine within 2^-12 of 0 is exact (136/136), the half and quarter turns are within 6 ULP, and every f16 input in the bands is exact.
  • Within 2^-12 of non-zero integers (1, 2, 3, 17, 100, 255, 256, 300, 1000, on both sides) the hardware is up to 14 ULP above the linear term. That's its own reduction, the same as after fix(shader): reduce v_sin_f32 and v_cos_f32 inputs exactly #1979 and unchanged on RADV, but the TechnicalDebt numbers said "10 ULP just below non-zero integers" and "5 ULP inside the bands".
  • The new test passes without the change on RADV, as you say, but fails on CI's lavapipe (Mesa 25.2.8, lane 0 0x81c90fdb, expected 0x81c90fd5), so CI covers it. With the change it passes on RADV and on CI's lavapipe.

Added:

  • Rebased on main. The only conflict was the TechnicalDebt entry that fix(shader): reduce v_sin_f32 and v_cos_f32 inputs exactly #1979 rewrote (the "Per-stage descriptor count" line was already gone from main).
  • docs(technical-debt): up to 14 ULP on either side of non-zero integers and up to 6 ULP in the half- and quarter-turn bands, as measured above.

Scoped ctest is green after the rebase (326/326). Merging.

@adriadam10
adriadam10 merged commit f70a091 into boykopovar:main Oct 10, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agc libSceAgc* docs needs rebase Conflicts with main, rebase needed shader core/shader

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants