Repository navigation
perf(shader): load from buffers without Coherent when the load has no GLC or DLC - #2369
Conversation
… GLC or DLC A program with one GLC or DLC buffer load or store decorated the whole guest buffer array Coherent, so its other loads were coherent too. On NVIDIA that keeps them out of the L1 cache: one GLC store made the re-reads of a 4 KiB window about 2.7 times slower. RDNA keeps those loads incoherent. A load without GLC or DLC can hit the per-CU L0 and per-SA L1 caches, which are not coherent between work-groups, and code that needs another work-group's data reads it with GLC/DLC or invalidates the caches first (LLVM AMDGPU memory model, GFX10: an acquiring load is followed by buffer_gl1_inv; buffer_gl0_inv). The program metadata now records whether a program invalidates a cache (buffer_gl0_inv, buffer_gl1_inv, s_gl1_inv, s_dcache_inv). Without one, a program with GLC/DLC accesses gets a second array on the same binding, buffers_coherent, decorated Coherent, and both arrays are Aliased as buffers_u64 already is. Loads without GLC or DLC read buffers; GLC/DLC loads, stores and atomics use buffers_coherent. A program that invalidates a cache keeps the whole array Coherent.
|
Linux run: Linux 7.2 (x86_64), GCC 16.2.1, Release,
|
|
Checked on Linux (merged onto current main): build + This changes memory behaviour, so a design call before merging:
@oneandonlydean could you run Before it can land:
|
…alias # Conflicts: # core/shader/recompiler/IntermediateRepresentation/include/IntermediateRepresentation/IrMetadata/ProgramMetadata.hpp
|
Thanks for checking it. I merged current
Retested at 2069fd1 (Windows 11, MinGW-w64 GCC 15.2.0, Release):
The design question for @boykopovar and the RTX 4090 run from @oneandonlydean are still open; I'll follow whatever you decide. |
|
Checked merged onto current main: libSceAgc/libSceAgcDriver build + ctest green (402 tests), The design question is settled on our side: plain loads in these programs follow the GFX10 memory model (acquire = GLC+DLC load plus |
The SPIR-V emitter declares extra views of the guest buffers binding: buffers_coherent for GLC or DLC accesses (boykopovar#2369) and buffers_u64 for 64-bit atomics. Each is another StorageBuffer variable on the same set and binding, decorated Aliased, which Vulkan allows. ShaderValidation, which checks the stages of every draw, accepted one StorageBuffer variable per binding with u32 elements only, so such a graphics shader was refused with "duplicate SPIR-V resource binding". A second variable is now accepted when the binding is the guest buffers array and every variable on it is Aliased; an aliased view may hold u64 elements. Other duplicates are still refused. agc_driver_graphics validates pixel shaders with a GLC store and a 64-bit atomic, and checks that the same modules without Aliased are refused.
What
A program with a single GLC or DLC buffer load or store (28b09d4) decorates the whole guest
buffersdescriptor arrayCoherent. Every other buffer access in the program is then coherent as well, including the loads the guest made without GLC or DLC. On NVIDIA the SM's L1 cache is not coherent between SMs, and the coherent loads lose it: in a compute program that re-reads a 4 KiB window, one GLC store at the end makes it about 2.7 times slower (Tested). GLC/DLC buffer accesses come fromgloballycoherentbuffers and volatile accesses, e.g. decoupled look-back scans (the radix sort behind 28b09d4) and GPU-driven work queues, and these programs usually read a lot of other buffer data with plain loads.Why the loads without GLC or DLC don't need to be coherent
RDNA does not keep them coherent either. A vector load without GLC or DLC can hit the per-CU L0 and the per-shader-array L1 caches, and neither is coherent between work-groups. The LLVM AMDGPU memory model for GFX10 (https://llvm.org/docs/AMDGPUUsage.html, "Memory Model GFX10-GFX11") describes it:
buffer_gl0_inv"is also required for coherence between wavefronts executing in different work-groups", and in WGP mode for wavefronts of the same work-group.buffer/global_load glc=1 dlc=1,s_waitcnt vmcnt(0), thenbuffer_gl1_inv; buffer_gl0_inv, which "Ensures that following loads will not see stale global data".slc=1only.So guest code that needs another work-group's data either reads it with GLC/DLC, or invalidates the caches and then reads it with plain loads. Only the second case needs the plain loads to be coherent on the host, and it is visible in the program as a cache invalidation.
Change
buffer_gl0_inv,buffer_gl1_inv,s_gl1_invands_dcache_inv. They still emit nothing.buffers_coherent, on the same binding asbuffers, decoratedCoherent. All buffer arrays of the program are decoratedAliased, asbuffersandbuffers_u64already are for 64-bit atomics.buffers. GLC/DLC loads, every store and the 32-bit atomics go throughbuffers_coherent, so the program's writes stay coherent as before.buffers_u64is unchanged and staysCoherent.buffersarray isCoherent.buffers, not coherentbuffersCoherentfor every accessbuffers; GLC/DLC loads, stores, atomics:buffers_coherent(Coherent,Aliased)buffersCoherentPlain loads in programs without GLC or DLC accesses were already incoherent; this treats plain loads in programs with GLC/DLC accesses the same way. The
Volatilememory operand on GLC/DLC accesses and the BDA path (GPU-selected V#s,SpirvBdaRead.cpp) are unchanged.Which GPUs gain
The gain comes from plain loads staying in L1. It was measured on Ada (RTX 4060) only. The CUDA C++ Programming Guide says that on compute capability 5.x and 6.x (Maxwell, Pascal) global loads of data that is not read-only are not cached in L1 by default, so those GPUs may gain less. Other vendors were not measured.
Tested
Windows 11, MinGW-w64 GCC 15.2.0 (WinLibs r7), Release,
ANYPS5_ENABLE_SPIRV_TOOLS=ON,mainat d70b899. GPU tests on an NVIDIA GeForce RTX 4060 Laptop GPU (591.86, subgroup 32) and on lavapipe (Mesa 26.2.4 for Windows,GALLIUM_OVERRIDE_CPU_CAPS=avx).New check in
agc_shader_memory(no GPU needed): four wave32 compute programs that load from one guest buffer and store to another, inspected in the emitted SPIR-V:buffersarray, notCoherent, used by the load and the store;bufferswithoutCoherentandbuffers_coherentwith it, bothAliased; the load goes throughbuffersand the store throughbuffers_coherent;buffers_coherent;buffer_gl0_inv: onebuffersarray,Coherent, as onmain.Negative checks, each fails the new check:
coherent buffers: the two buffer arrays are not decorated Aliased(buffersis then unused and no longer in the module);coherent buffers: a program that invalidates the vector caches does not keep every buffer access coherent;coherent buffers: a GLC store has no coherent buffer array.Every shader test (
agc_driver_*,agc_shader_*,agc_spirv_*,vector_*,agc_masked_select,agc_validate_program, 234 tests):agc_driver_storage_cache_budget). This includesagc_driver_look_back_scan, a decoupled look-back scan across 4096 work-groups with GLC pair loads, atomics and plain loads and no cache invalidation, which now takes the new lowering (checked in its SPIR-V: plain loads onbuffers, the GLC loads and stores onbuffers_coherent).look_back_scanamong them); all 26 pass on the RTX 4060.Full ctest after
cmake --build buildand--target libs, serial,--timeout 120, RTX 4060: 505 of 511 pass, 2 skipped (agc_driver_storage_cache_budget,guest_video_out_wqhd_detection_no_vulkan), 4 fail. All four are relinker tests, whose build contains no recompiler sources:windows_import_modules(Windows Error Reporting keeps its GUI fixture open, test(relinker): launch the GUI startup fixtures without Windows Error Reporting #2227), andtls_function_coverage,windows_guiandwindows_icon, which fail to open an executable they just wrote to the temporary directory because Windows Defender removes it (Defender logged the detections during the run).Throughput on the RTX 4060, not committed: a wave32 compute program whose 32 lanes read 512 dwords each from a 4 KiB window of one buffer (64 batches of 8
buffer_load_dword, L1-resident after the first pass), sum them and write the result with onebuffer_store_dwordx4; 8192 work-groups, timed per dispatch including submit and wait, best of 8 dispatches. All variants run in one process, in a rotated order each round, because the laptop's clocks change by up to 4x over a few seconds and separate runs could not be compared. Abuffer_gl0_invat the start selects the previous lowering for the same program (the instruction itself emits nothing), so the last column is whatmaindoes for this program:buffer_gl0_invbuffer_gl0_inv(main)The previous lowering is the slowest variant in every round, 1.7 to 4.8 times the GLC store with this change (2.7 times in the best rounds: 3.09 ms against 1.14 ms); with the change, the GLC program runs about as fast as the program without GLC. All variants produce the same output (same checksum). An earlier in-process run with a temporary switch that dropped the
Volatilememory operand from the GLC store showed no measurable difference, with either lowering; the difference is theCoherentdecoration on the loads.Checklist
main; no other open PR implements the same functionsNotImplemented_nid_no_patch); silent stubs are listed in TechnicalDebt.mdfiles or images added to the repository (attach them to this PR)