Repository navigation
Conversation
|
This pull request is labeled |
|
Windows and NVIDIA Blackwell run, for head
Both failures fail on The probe on this device. Every device creation prints, with the time varying from run to run: The count is the same in every probe line of these runs, with and without the layer. So this card keeps the selection and the host path isn't exercised here. To see where the results differ, I added a local print of each mismatch to
Probe cost here. I alternated runs on the merged build without and with
No title was run. AI-assisted: yes. Claude Code ran the builds and tests and drafted this comment; the numbers are from the runs on this machine. |
❌ NVIDIA GPU test: failedPR head
New validation errorsThis PR changes test files, so each test below was also run on main with only the PR's test changes applied; 0 there means the PR's other changes cause the error.
…truncated; see the full logs. Automated build and test on a contributor's NVIDIA machine; it does not replace upstream CI. A new comment is posted for each new push. Full logs. |
✅ AMD GPU test: passedPR head
Unstable on main (1, these also failed in recent main runs on this machine; not counted against this PR)Failing on main too (7, not counted against this PR)Validated but not compared (1, not in the main baseline's validation pass)Validation IDs already present on main (8 IDs in 546 tests)Skipped (60)ReproduceAutomated build and test on a contributor's AMD machine; it does not replace upstream CI. A new comment is posted for each new push. Full logs. |
|
About the NVIDIA run above and its 1,084 new IDs (
Here, the RTX 5080 (driver 621.14) offers
The AI-assisted: yes. Claude Code read the bot's logs, ran the validation runs and drafted this comment. |
e7ef41a to
694649b
Compare
❌ NVIDIA GPU test: failedPR head
New validation errorsThis PR changes test files, so each test below was also run on main with only the PR's test changes applied; 0 there means the PR's other changes cause the error.
…truncated; see the full logs. Automated build and test on a contributor's NVIDIA machine; it does not replace upstream CI. A new comment is posted for each new push. Full logs. |
…when a startup probe matches RDNA
694649b to
c51af50
Compare
What
A startup-probe step of #11183, on top of #15718 (the scalar fma PR). With
IEEE_MODEclear, the native packed f16 add, mul, fma and fmac keep the device's NaN result when a startup probe shows that it is already the one RDNA selects. So do the native scalar fma forms.Today every native
v_pk_add_f16,v_pk_mul_f16,v_pk_fma_f16andv_pk_fmac_f16picks its NaN result on the bits afterwards, in each lane: the first NaN source (quieted only withIEEE_MODEset), with the addend skipped after an inf * 0 product, or the default NaN 0xfe00. That is three or four compares and selects per lane on top of the one native instruction. The native scalar fma forms test for the default NaN of an invalid operation in the same way.On a Radeon RX 7900 XTX (AMD Software 26.10.1) the device already returns exactly those bits when
IEEE_MODEis clear, with one exception. When both operands are NaNs, the driver may swap the operands of an add (it did forop_selandnegoperands), and then returns the second NaN. So the host path replaces the second operand by the first when the first is a NaN: one compare and one select per lane, and the order no longer matters. WithIEEE_MODEset, the device returns a signalling NaN source unchanged where RDNA quiets it, so that mode keeps the selection.ProbeFloat16HostNans(Execution/src/Float16NanProbe.cpp) runs when the device is created, after the f32 denormal flush and preserve probes, on devices with native f16 math. It compiles the instruction templates ofagc_driver_packed_float16_native(add, mul, fma, fmac withop_sel,negand clamp) and the scalar fma forms ofagc_driver_float16_nativetwice: with the selection and with the host path.IEEE_MODEclear withFP16_OVFL, with and withoutDX10_CLAMP.docs/dev/debt/shader.md).SpirvTarget::float16HostNansis set only when nothing differs, and it is part of the cache key. Recompile requests saved withAPS5_DUMP_SHADERSdo not carry it (perf(shader): keep native f16 conversion NaN results when a startup probe matches RDNA #11351 and fix(shader): declare f16 and f64 DenormPreserve together when their modes are shared #11483 already queue changes to the request format);agc_shader_replay --f16-host-nanssets it on a replay.APS5_NO_HOST_F16_NANSskips the probe.New test
agc_driver_float16_nan_probe:_spirv(no device): with the flag andIEEE_MODEclear, both programs have fewer selects; withIEEE_MODEset, the module is the same with and without the flag.APS5_NO_HOST_F16_NANSis set.The existing
agc_driver_packed_float16_native(589,824 halves against the f32 path, bit for bit) andagc_driver_float16_nativerun with the device's target, so on a device where the probe passes they check the host path too.Tested
Windows 11 Pro 26300, Radeon RX 7900 XTX (AMD Software 26.10.1), Ryzen 7 7800X3D, MinGW-w64 GCC 15.2, Ninja, Release,
BUILD_TESTING=ON, this head on #15718 and mainb11c7bf84f,ANYPS5_REQUIRE_VULKAN=1,ANYPS5_NO_SHADER_CACHE=1.Focused tests (
ctest -R "fma|float16|f16|packed_alu|dx10_clamp|dpp8_pk_fmac|dpp16_operands|request_serializer|agc_shader_memory", 51 tests including the two new ones): 50 pass andagc_driver_float16_unused_nan_payloadsskips, as it does onmainon this device. The same withVK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation: 44 layer insertions withVK_LOADER_DEBUG=layer, nowas not foundline, noVUID-line.Probe on the RX 7900 XTX:
[float-controls] f16 NaN results with IEEE_MODE clear: host (0 of 270336 probe results differ from the selected NaN, 36 ms).agc_driver_packed_float16_native:packed f16 native tests passed (589824 halves, fused f16 fma).^agc(-E "^agc_driver_bda_device$",-j12): 1363 of 1366.agc_driver_direct_write_completionandagc_driver_sample_order_grid_referencefail onmainb11c7bf84ftoo.agc_driver_dumped_constant_captureis intermittent and not changed by this PR: run alone it failed 3 of 127 times on this head, 3 of 134 withAPS5_NO_HOST_F16_NANS=1and 3 of 111 onmain(the draw after a CPU write over a dumped constant still read the dumped 1.0).Head
c51af50be5(rebased onmaine8b1bd3d7f; the probe stops at the first differing batch): the probe tests pass (agc_driver_float16_nan_probereportshost (0 of 270336 probe results differ),agc_driver_float16_nan_probe_spirv), and the cache key, request serializer and disk cache tests pass 26 of 26.ctest -R "float16|f16|nan|NaN|probe|ieee"passes 67 of 74 with 1 device skip. The 6 failures (agc_driver_f16_conversion_float_mode,agc_driver_float16_misc,agc_driver_float16_modifiers,agc_driver_float16_rounding,agc_driver_ldexp_f16_inline_constants,agc_driver_sdwa_ldexp_f16, each +0 where -0 is expected) fail the same way onmaine8b1bd3d7f; they pass withAPS5_NO_NATIVE_F16_NANS=1, so they come from main's native f32-to-f16 conversion probe on this RX 7900 XTX.Before the change,
agc_driver_packed_float16_nativewithout the selection and without the operand replacement: 0 mismatches withIEEE_MODEclear exceptv_pk_add_f16 ... op_sel neg(40-54 per mode, both operands NaN, second NaN returned); 400-560 per instruction withIEEE_MODEset (signalling NaN not quieted). With the replacement: 0 in everyIEEE_MODEclear mode.Size, PPSA04264 compute shaders (
agc_shader_replay --spv --shader-fma,IEEE_MODEclear):--f16-host-nans)0x25b16a200(the 1440p to 2160p upscale)0x25b16ca00(RCAS)In game, PPSA04264 (city gameplay, 3840x2160 output,
APS5_PROFILE_GPU, this head's driver), measured before the rebase:[float-controls] f16 NaN results with IEEE_MODE clear: host (0 of 270336 probe results differ from the selected NaN, 41 ms). The upscale takes 2.29-2.42 ms of GPU per dispatch (4.44-4.63 ms with perf(shader): compute v_fma_f16, v_fmac_f16, v_fmamk_f16 and v_fmaak_f16 with OpFmaKHR #15718 alone), RCAS 0.80 ms (0.86 ms).Frames, within one run (
APS5_AB, 30 s phases, 20 phases, B against the neighbouring A): on a local build, dispatching the upscale four times instead of once (about +7 ms of GPU) costs 11.0 % of the presents (-12.6 +- 5.8 per 10 s) and adds 10.9 +- 5.2 ms per frame. Skipping it costs nothing at 2.4 ms (-2.7 +- 5.1 %), where it gave +11.6 % (SE 3.7) at 8.9 ms. So in this scene the upscale's GPU time is on the frame's path, and perf(shader): compute v_pk_fma_f16 and v_pk_fmac_f16 with OpFmaKHR #11265, perf(shader): compute v_fma_f16, v_fmac_f16, v_fmamk_f16 and v_fmaak_f16 with OpFmaKHR #15718 and this PR together are worth about +10 % presents. Comparing whole boots instead (three interleaved pairs, warm caches) only shows the GPU part: GPU busy 39.2 -> 32.8 ms per present, and FPS 7.13 -> 7.26, inside the boot-to-boot noise of that scene.Performance report (Performance report form): PPSA04264 open-world gameplay after the opening, 3840x2160 output, Performance graphics mode,
APS5_TRACE_SECONDS=300:30, no otherAPS5_TRACE_*orAPS5_PROFILE_*switch, the log not redirected. Each build ran the scene once and the title was restarted before its captures, and all captures come from this machine in one session. The base is #15718.queue worker 0x0busy 97.4-97.7 %), so the frame time follows the scene rather than the shaders.gpu_busy_ms_per_presentis 27.87 ms with perf(shader): compute v_fma_f16, v_fmac_f16, v_fmamk_f16 and v_fmaak_f16 with OpFmaKHR #15718 and 24.68 ms in this PR's second capture; the GPU time per dispatch above is the cleaner measure of the change.Driver.Draw68.3 ms per frame, against 44.5 ms in the second capture and 45.6 ms with perf(shader): compute v_fma_f16, v_fmac_f16, v_fmamk_f16 and v_fmaak_f16 with OpFmaKHR #15718), with 299 presents and 30.80 ms of GPU busy per present. Its line is the second one below.trace_dropped, withtotals_fullfalse), so the summary lines are complete while the traces hold the first 6-15 s of those threads.{"commit":"476df725ba14","dirty":false,"env":{"APS5_LOG_TIMESTAMPS":"1","APS5_NO_PAD_INPUT":"1","APS5_PAD_INPUT_FILE":"<path>","APS5_TRACE_DIR":"<path>","APS5_TRACE_SECONDS":"300:30"},"gpu":"AMD Radeon RX 7900 XTX","driver":"AMD proprietary driver 26.10.1 (LLPC)","os":"Windows 10.0.26300","cpu":"AMD Ryzen 7 7800X3D 8-Core Processor","cpu_threads":16,"memory_mib":31860,"title_id":"PPSA04264","title":"Grand Theft Auto V","content_version":"01.011.001","resolution":"3840x2160","window":{"spec":"seconds 300:30","first_present":4887,"ms":30006.986},"presents":390,"frame_ms":{"p50":75.630,"p90":80.178,"p99":114.688,"max":139.883},"gpu_busy_ms_per_present":27.870,"worker_busy_pct":{"queue worker 0x0":97.719},"top_phases_ms_per_frame":{"Driver.Packet":71.331,"Graphics.Draw":49.609,"Driver.Packet.draw":46.031,"Driver.Draw":45.594,"Graphics.Draw.draw_and_resource_release":43.795,"Driver.Draw.shader_compile_and_link":26.977,"Shader.DrawStage":20.800,"Driver.Packet.dispatch":16.838},"trace_dropped":true,"trace_dropped_threads":{"queue worker 0x0":{"events_full_after_s":6.614,"events_lost":911231,"totals_full":false},"draw committer":{"events_full_after_s":14.926,"events_lost":253977,"totals_full":false}},"trace":"aps5-trace-476df725ba14-26272.json.gz"} {"commit":"e7ef41a7add1","dirty":false,"env":{"APS5_LOG_TIMESTAMPS":"1","APS5_NO_PAD_INPUT":"1","APS5_PAD_INPUT_FILE":"<path>","APS5_TRACE_DIR":"<path>","APS5_TRACE_SECONDS":"300:30"},"gpu":"AMD Radeon RX 7900 XTX","driver":"AMD proprietary driver 26.10.1 (LLPC)","os":"Windows 10.0.26300","cpu":"AMD Ryzen 7 7800X3D 8-Core Processor","cpu_threads":16,"memory_mib":31860,"title_id":"PPSA04264","title":"Grand Theft Auto V","content_version":"01.011.001","resolution":"3840x2160","window":{"spec":"seconds 300:30","first_present":5191,"ms":30041.302},"presents":299,"frame_ms":{"p50":107.511,"p90":113.276,"p99":134.383,"max":167.236},"gpu_busy_ms_per_present":30.801,"worker_busy_pct":{"queue worker 0x0":97.435},"top_phases_ms_per_frame":{"Driver.Packet":92.893,"Graphics.Draw":71.279,"Driver.Packet.draw":68.987,"Driver.Draw":68.333,"Graphics.Draw.draw_and_resource_release":62.897,"Driver.Draw.shader_compile_and_link":41.868,"Shader.DrawStage":32.931,"Driver.Draw.draw_enqueued":16.416},"trace_dropped":true,"trace_dropped_threads":{"queue worker 0x0":{"events_full_after_s":6.226,"events_lost":1139325,"totals_full":false},"draw committer":{"events_full_after_s":12.852,"events_lost":390826,"totals_full":false}},"trace":"aps5-trace-e7ef41a7add1-17208.json.gz"} {"commit":"e7ef41a7add1","dirty":false,"env":{"APS5_LOG_TIMESTAMPS":"1","APS5_NO_PAD_INPUT":"1","APS5_PAD_INPUT_FILE":"<path>","APS5_TRACE_DIR":"<path>","APS5_TRACE_SECONDS":"300:30"},"gpu":"AMD Radeon RX 7900 XTX","driver":"AMD proprietary driver 26.10.1 (LLPC)","os":"Windows 10.0.26300","cpu":"AMD Ryzen 7 7800X3D 8-Core Processor","cpu_threads":16,"memory_mib":31860,"title_id":"PPSA04264","title":"Grand Theft Auto V","content_version":"01.011.001","resolution":"3840x2160","window":{"spec":"seconds 300:30","first_present":5086,"ms":30041.236},"presents":417,"frame_ms":{"p50":68.603,"p90":82.234,"p99":91.440,"max":124.561},"gpu_busy_ms_per_present":24.683,"worker_busy_pct":{"queue worker 0x0":97.506},"top_phases_ms_per_frame":{"Driver.Packet":65.869,"Graphics.Draw":47.574,"Driver.Packet.draw":44.935,"Driver.Draw":44.475,"Graphics.Draw.draw_and_resource_release":41.933,"Driver.Draw.shader_compile_and_link":26.907,"Shader.DrawStage":20.735,"Driver.Packet.dispatch":12.367},"trace_dropped":true,"trace_dropped_threads":{"queue worker 0x0":{"events_full_after_s":6.552,"events_lost":963457,"totals_full":false},"draw committer":{"events_full_after_s":14.967,"events_lost":289464,"totals_full":false}},"trace":"aps5-trace-e7ef41a7add1-1056.json.gz"}Traces:
aps5-trace-476df725ba14-26272.json.gz(#15718),aps5-trace-e7ef41a7add1-17208.json.gzandaps5-trace-e7ef41a7add1-1056.json.gz(this PR, first and second capture), attached below.tools/trace_digest.pyof the #15718 captureseconds 300:30 from present 4887: 390 presents, frame p50 75.6 / p90 80.2 / p99 114.7 / max 139.9 ms, GPU busy 27.9 ms per present
AMD Radeon RX 7900 XTX, AMD proprietary driver 26.10.1 (LLPC), Windows 10.0.26300, 3840x2160, commit 476df72, dropped: yes
{"commit":"476df725ba14","dirty":false,"env":{"APS5_LOG_TIMESTAMPS":"1","APS5_NO_PAD_INPUT":"1","APS5_PAD_INPUT_FILE":"<path>","APS5_TRACE_DIR":"<path>","APS5_TRACE_SECONDS":"300:30"},"gpu":"AMD Radeon RX 7900 XTX","driver":"AMD proprietary driver 26.10.1 (LLPC)","os":"Windows 10.0.26300","cpu":"AMD Ryzen 7 7800X3D 8-Core Processor","cpu_threads":16,"memory_mib":31860,"title_id":"PPSA04264","title":"Grand Theft Auto V","content_version":"01.011.001","resolution":"3840x2160","window":{"spec":"seconds 300:30","first_present":4887,"ms":30006.986},"presents":390,"frame_ms":{"p50":75.63,"p90":80.178,"p99":114.688,"max":139.883},"gpu_busy_ms_per_present":27.87,"worker_busy_pct":{"queue worker 0x0":97.719},"top_phases_ms_per_frame":{"Driver.Packet":71.331,"Graphics.Draw":49.609,"Driver.Packet.draw":46.031,"Driver.Draw":45.594,"Graphics.Draw.draw_and_resource_release":43.795,"Driver.Draw.shader_compile_and_link":26.977,"Shader.DrawStage":20.8,"Driver.Packet.dispatch":16.838},"trace_dropped":true,"trace_dropped_threads":{"queue worker 0x0":{"events_full_after_s":6.614,"events_lost":911231,"totals_full":false},"draw committer":{"events_full_after_s":14.926,"events_lost":253977,"totals_full":false}},"trace":"aps5-trace-476df725ba14-26272.json.gz"}Incomplete trace: recorded data was lost. Slice-based timings, waits and batch inventories cover retained events only; missing slices are not idle time. Summary statistics are reported separately.
Slowest frames (largest slices inside each, inclusive)
Persistent spikes (frame time above the fastest frame that separates it from a longer frame; one entry per stall, not per frame)
Threads (ms per frame)
Largest slices (ms per frame, inclusive, all threads): batch 27.9, Graphics.Draw 17.3, Graphics.Draw.draw_and_resource_release 14.2, Driver.Packet 11.2, VideoOut.Flip 10.0, Driver.Present 9.9, VideoOut.Flip.present 9.9, Driver.Packet.draw 6.9, Driver.Draw 6.8, Recorder.Sync 4.8
Longest waits
Largest GPU batches (GPU ranges inside, and the CPU slices that recorded each, inclusive)
tools/trace_digest.pyof this PR's first captureseconds 300:30 from present 5191: 299 presents, frame p50 107.5 / p90 113.3 / p99 134.4 / max 167.2 ms, GPU busy 30.8 ms per present
AMD Radeon RX 7900 XTX, AMD proprietary driver 26.10.1 (LLPC), Windows 10.0.26300, 3840x2160, commit e7ef41a, dropped: yes
{"commit":"e7ef41a7add1","dirty":false,"env":{"APS5_LOG_TIMESTAMPS":"1","APS5_NO_PAD_INPUT":"1","APS5_PAD_INPUT_FILE":"<path>","APS5_TRACE_DIR":"<path>","APS5_TRACE_SECONDS":"300:30"},"gpu":"AMD Radeon RX 7900 XTX","driver":"AMD proprietary driver 26.10.1 (LLPC)","os":"Windows 10.0.26300","cpu":"AMD Ryzen 7 7800X3D 8-Core Processor","cpu_threads":16,"memory_mib":31860,"title_id":"PPSA04264","title":"Grand Theft Auto V","content_version":"01.011.001","resolution":"3840x2160","window":{"spec":"seconds 300:30","first_present":5191,"ms":30041.302},"presents":299,"frame_ms":{"p50":107.511,"p90":113.276,"p99":134.383,"max":167.236},"gpu_busy_ms_per_present":30.801,"worker_busy_pct":{"queue worker 0x0":97.435},"top_phases_ms_per_frame":{"Driver.Packet":92.893,"Graphics.Draw":71.279,"Driver.Packet.draw":68.987,"Driver.Draw":68.333,"Graphics.Draw.draw_and_resource_release":62.897,"Driver.Draw.shader_compile_and_link":41.868,"Shader.DrawStage":32.931,"Driver.Draw.draw_enqueued":16.416},"trace_dropped":true,"trace_dropped_threads":{"queue worker 0x0":{"events_full_after_s":6.226,"events_lost":1139325,"totals_full":false},"draw committer":{"events_full_after_s":12.852,"events_lost":390826,"totals_full":false}},"trace":"aps5-trace-e7ef41a7add1-17208.json.gz"}Incomplete trace: recorded data was lost. Slice-based timings, waits and batch inventories cover retained events only; missing slices are not idle time. Summary statistics are reported separately.
Slowest frames (largest slices inside each, inclusive)
Persistent spikes (frame time above the fastest frame that separates it from a longer frame; one entry per stall, not per frame)
Threads (ms per frame)
Largest slices (ms per frame, inclusive, all threads): batch 30.8, Graphics.Draw 21.3, Graphics.Draw.draw_and_resource_release 17.2, Driver.Packet 13.8, Driver.Packet.draw 9.0, Driver.Draw 8.8, VideoOut.Flip 7.4, Driver.Present 7.2, VideoOut.Flip.present 7.2, Driver.Packet.dispatch 3.9
Longest waits
Largest GPU batches (GPU ranges inside, and the CPU slices that recorded each, inclusive)
tools/trace_digest.pyof this PR's second captureseconds 300:30 from present 5086: 417 presents, frame p50 68.6 / p90 82.2 / p99 91.4 / max 124.6 ms, GPU busy 24.7 ms per present
AMD Radeon RX 7900 XTX, AMD proprietary driver 26.10.1 (LLPC), Windows 10.0.26300, 3840x2160, commit e7ef41a, dropped: yes
{"commit":"e7ef41a7add1","dirty":false,"env":{"APS5_LOG_TIMESTAMPS":"1","APS5_NO_PAD_INPUT":"1","APS5_PAD_INPUT_FILE":"<path>","APS5_TRACE_DIR":"<path>","APS5_TRACE_SECONDS":"300:30"},"gpu":"AMD Radeon RX 7900 XTX","driver":"AMD proprietary driver 26.10.1 (LLPC)","os":"Windows 10.0.26300","cpu":"AMD Ryzen 7 7800X3D 8-Core Processor","cpu_threads":16,"memory_mib":31860,"title_id":"PPSA04264","title":"Grand Theft Auto V","content_version":"01.011.001","resolution":"3840x2160","window":{"spec":"seconds 300:30","first_present":5086,"ms":30041.236},"presents":417,"frame_ms":{"p50":68.603,"p90":82.234,"p99":91.44,"max":124.561},"gpu_busy_ms_per_present":24.683,"worker_busy_pct":{"queue worker 0x0":97.506},"top_phases_ms_per_frame":{"Driver.Packet":65.869,"Graphics.Draw":47.574,"Driver.Packet.draw":44.935,"Driver.Draw":44.475,"Graphics.Draw.draw_and_resource_release":41.933,"Driver.Draw.shader_compile_and_link":26.907,"Shader.DrawStage":20.735,"Driver.Packet.dispatch":12.367},"trace_dropped":true,"trace_dropped_threads":{"queue worker 0x0":{"events_full_after_s":6.552,"events_lost":963457,"totals_full":false},"draw committer":{"events_full_after_s":14.967,"events_lost":289464,"totals_full":false}},"trace":"aps5-trace-e7ef41a7add1-1056.json.gz"}Incomplete trace: recorded data was lost. Slice-based timings, waits and batch inventories cover retained events only; missing slices are not idle time. Summary statistics are reported separately.
Slowest frames (largest slices inside each, inclusive)
Persistent spikes (frame time above the fastest frame that separates it from a longer frame; one entry per stall, not per frame)
Threads (ms per frame)
Largest slices (ms per frame, inclusive, all threads): batch 24.7, Graphics.Draw 15.7, Graphics.Draw.draw_and_resource_release 12.7, Driver.Packet 9.8, VideoOut.Flip 7.9, Driver.Present 7.8, VideoOut.Flip.present 7.8, Driver.Packet.draw 6.2, Driver.Draw 6.1, Recorder.Sync 3.3
Longest waits
Largest GPU batches (GPU ranges inside, and the CPU slices that recorded each, inclusive)
Checklist
main; no other open PR implements the same functionsNotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file.mdfiles or images added to the repository (attach them to this PR)