Skip to content

perf(agc): order a storage flush behind its GPU writer when the write-back is a GPU copy - #15569

Open
milescoviello wants to merge 7 commits into
boykopovar:mainfrom
milescoviello:perf/storage-flush-gpu-ordered
Open

milescoviello wants to merge 7 commits into
boykopovar:mainfrom
milescoviello:perf/storage-flush-gpu-ordered

Conversation

@milescoviello

@milescoviello milescoviello commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

What

A label's readback store (StoreReadbackBeforeLabel) and a storage refresh (StorageTexture::refresh) flush the pending storage images over their range before they go on. FlushPending waited on the CPU for the images' GPU writers first. When every flushed image lives in a host import, its write-back is a copy the recorder records into the import. That copy runs after the image's writer and before anything recorded later, including the label, so the wait adds nothing to the order. Ghost of Tsushima (PPSA02225) paid about 55 ms for it once a frame. A sampled 4K storage surface overlapped six pending images of a transient heap, and its refresh drained the whole recorder.

How

  • StoreReadbackBeforeLabel and an eligible refresh set a thread-local scope. Inside it, FlushPending skips the wait for the GPU writer when the recorder is active, the call is not in a completion, and every image it flushes has a host import (HostImportFor). The write-backs are then recorded behind their writers as before.
  • A refresh is eligible when it is not scoped, has no alias or key hand-over, no DCC metadata, no layer still pending, and its own image has a host import. When the flush skipped the wait, the guest bytes the refresh would compare are still in flight, so the refresh doesn't compare them. It marks every tracked layer changed and re-uploads the layers from the import, a GPU copy recorded after the write-backs.
  • APS5_NO_FLUSH_GPU_ORDERED=1 and APS5_NO_REFRESH_GPU_ORDERED=1 restore the waits.

Tested

  • New agc_driver_label_ordered_flush: a compute image_store into an imported 128x128 R32UI surface, a small storage image (64 KiB, the size a label stores before it), followed directly by a RELEASE_MEM label while the store is still pending on the GPU. When the label lands, the stored texels must be in guest memory. 8 runs per test; the test checks that the surface is small.
  • Since b6007f69a9, StorageTexture::TakeOrderedSkipCounts() counts the skips taken, and the tests check them:
    • agc_driver_label_ordered_flush covers the label half (at least one flush skip) and the refresh half. For the refresh half, a 256x64 linear R32UI view of the block reads after a 128x128 tiled store over the whole block is left pending. The view isn't an alias, so its refresh flushes the store, and every texel read must equal the stored value and guest memory. The half requires at least one refresh skip.
    • agc_driver_label_ordered_flush_disabled (APS5_NO_FLUSH_GPU_ORDERED=1 APS5_NO_REFRESH_GPU_ORDERED=1) and _unimported (APS5_NO_HOST_IMPORT=1) run the same scenarios. They require zero skips and the same results.
    • agc_driver_ordered_refresh_failure injects a FlushPending failure inside an armed refresh (SetFlushFailureForTest). It checks that the flag is cleared on the thread and that the next flush waits.
  • GoT, local stack (the same change on an older base, with other GoT experiments), warm, same route, 1200 s runs:
    • Before: 2.38-2.45 fps, one ~55 ms Recorder::Sync per frame from StorageTexture::refresh -> FlushPending (APS5_TRACE_LONG_SYNCS).
    • Run 265, with both: 2.87 fps, p50 346 ms. 5 syncs of 50 ms or more in 785 s, against one per frame. Run 264, with only the flush half: 2.53 fps. Run 271, a clean measure with no traces: 2.90 fps, p50 343 ms.
    • Frames unchanged against the runs without it.
  • ASTRO BOT (PPSA21567 01.019.000), Crash Site, main 086c0686c8 against the same main + this PR, measured by @TripyFox (comment). Windows 11, RTX 5090 (NVIDIA 621.01), 3840x2160, warm shader cache, three interleaved 240 s pairs with APS5_TRACE_SUMMARY_EVERY=10 APS5_TRACE_BENCH=1:
    • p50 92.7 -> 84.2 ms (-9.2 %), p99 108.5 -> 92.0 ms, presents 108.0 -> 120.3 per 10 s (+11.4 %), GPU busy 73.6 -> 72.8 ms per present. The fastest main run is slower than the slowest PR run.
    • Recorder syncs per 10 s fall from 4209 to 3157, and pending-write syncs from 1792 to 472. The storage write-back and upload volumes are unchanged.
    • Most of the gain comes from the label half. APS5_NO_FLUSH_GPU_ORDERED=1 gives back most of it; APS5_NO_REFRESH_GPU_ORDERED=1 gives back about 1 ms.
    • A confirmation pair on head 2da15973ce gives the same size: p50 -6 ms in every compared interval.
    • On the current head, @TripyFox measured main b11c7bf84f against b6007f6 merged onto it (comment). Only libSceAgcDriver.prx differs between the builds. The setup was 240 s runs with a warm cache, two interleaved rounds, and medians of the 10 s intervals from 40 s to 240 s, with APS5_TRACE_SUMMARY_EVERY=10 APS5_TRACE_BENCH=1:
Run presents / 10 s frame p50 ms p99 ms GPU busy / present ms
main, round 1 110.8 90.01 108.88 70.38
change, round 1 122.7 82.79 92.06 70.45
main, round 2 110.5 91.06 104.54 72.81
change, round 2 120.5 84.39 90.58 72.73
  • tools/performance_compare.py on the two summary lines below (interval from 100 s, round 1): frame p50 -8.3 ms (-9.1 %), p99 -22.9 ms, GPU busy -0.6 ms. The only configuration difference it lists is window.first_present.
  • Summary line, main:
{"commit":"b11c7bf84f5d","dirty":false,"env":{"APS5_NO_PAD_INPUT":"1","APS5_TRACE_BENCH":"1","APS5_TRACE_DIR":"<path>","APS5_TRACE_SUMMARY_EVERY":"10"},"gpu":"NVIDIA GeForce RTX 5090","driver":"NVIDIA 621.01","os":"Windows 10.0.26200","cpu":"AMD Ryzen 7 9800X3D 8-Core Processor","cpu_threads":16,"memory_mib":65140,"title_id":"PPSA21567","title":"ASTRO BOT","content_version":"01.019.000","resolution":"3840x2160","window":{"spec":"summary_every 10","first_present":1641,"ms":10040.788},"interval_start_s":100.904,"interval_end_s":110.945,"presents":110,"frame_ms":{"p50":90.588,"p90":100.512,"p99":114.130,"max":118.423},"gpu_busy_ms_per_present":70.925,"worker_busy_pct":{"queue worker 0x0":97.787,"queue worker 0x40":99.954,"queue worker 0x48":100.058,"queue worker 0x46":2.206},"top_phases_ms_per_frame":{"Driver.Packet":272.861,"Driver.Packet.memory_wait":159.319,"Driver.Packet.draw":55.195,"Driver.Draw":54.820,"Graphics.Draw":46.451,"Graphics.Draw.draw_and_resource_release":37.184,"Driver.Draw.shader_compile_and_link":29.343,"Shader.DrawStage":24.561},"trace_dropped":false,"trace":""}
  • Summary line, this change:
{"commit":"78693efa462b","dirty":false,"env":{"APS5_NO_PAD_INPUT":"1","APS5_TRACE_BENCH":"1","APS5_TRACE_DIR":"<path>","APS5_TRACE_SUMMARY_EVERY":"10"},"gpu":"NVIDIA GeForce RTX 5090","driver":"NVIDIA 621.01","os":"Windows 10.0.26200","cpu":"AMD Ryzen 7 9800X3D 8-Core Processor","cpu_threads":16,"memory_mib":65140,"title_id":"PPSA21567","title":"ASTRO BOT","content_version":"01.019.000","resolution":"3840x2160","window":{"spec":"summary_every 10","first_present":1452,"ms":10065.337},"interval_start_s":100.957,"interval_end_s":111.022,"presents":124,"frame_ms":{"p50":82.300,"p90":86.826,"p99":91.225,"max":94.329},"gpu_busy_ms_per_present":70.315,"worker_busy_pct":{"queue worker 0x0":97.584,"queue worker 0x40":99.964,"queue worker 0x48":99.992,"queue worker 0x46":1.450},"top_phases_ms_per_frame":{"Driver.Packet":241.869,"Driver.Packet.memory_wait":142.862,"Graphics.Draw":47.769,"Driver.Packet.draw":47.312,"Driver.Draw":46.912,"Graphics.Draw.draw_and_resource_release":38.622,"Driver.Draw.shader_compile_and_link":26.900,"Driver.Packet.prepare_memory":22.820},"trace_dropped":false,"trace":""}
  • These runs used the bench summaries, so no .json.gz traces are attached.
  • Focused tests at b6007f69a9: the four tests above plus agc_driver_direct_write_completion, agc_driver_linear_target_direct_write_completion, agc_driver_fill_completion_labels, agc_driver_linear_target_completion_points and agc_driver_memory_value_fill.
Where Result
Linux, RTX 4070 Ti, driver 615.71.09 9 of 9 pass in each of 3 runs
Windows 11 VM, GT 1030, driver 582.78 at b6007f69a9: 8 of 9 pass in each of 3 runs, including all four new tests. The failure is agc_driver_direct_write_completion, which also fails on main 1c8c0efc8c there in each of 3 runs (#15564, fixed by #15610)
Khronos validation 1.4.363, base 1c8c0efc8c vs head b6007f69a9 5 of 5 and 9 of 9 pass, the layer loaded in each, 9 and 16 Vulkan devices created, no VUID-

Overlap: #12070 also changes FlushPending's writer wait: it skips waiting for an image's own stamped writes. Both skip the wait on different conditions, so they compose. The second to land rebases the wait condition. No open PR changes StoreReadbackBeforeLabel or the refresh compare.

Checklist

  • Based on current main; no other open PR implements the same functions
  • One topic per PR; follow-ups go in a new PR
  • No comments in code except technical debt
  • Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt
  • No notes, investigation .md files or images added to the repository (attach them to this PR)
  • New third-party code is a submodule built from source
  • Generic behaviour, not specific to one title
  • Depends on: none
  • Fixes: none
  • Unblocks: none
  • GPUs tested: NVIDIA GeForce RTX 4070 Ti, driver 615.71.09, Ubuntu 26.04.1; NVIDIA GeForce GT 1030, driver 582.78, Windows 11 VM
  • AI-assisted: yes (Claude Code)

🤖 Generated with Claude Code

milescoviello and others added 2 commits November 3, 2026 14:00
…-back is a GPU copy

A label's readback store and a storage refresh flush pending storage images before going on. When every flushed image lives in a host import, its write-back is a copy the recorder records into the import, so it runs after the image's writer and before anything recorded later, including the label. The CPU wait for the writer is skipped then. A refresh whose flush skipped the wait re-uploads its layers from the import on the GPU instead of comparing guest bytes the copy has not reached yet. APS5_NO_FLUSH_GPU_ORDERED=1 and APS5_NO_REFRESH_GPU_ORDERED=1 restore the waits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…dered_flush

WAIT_REG_MEM is evaluated by the queue worker, so a gate in front of the store kept it from being recorded at all. The label now follows the store directly, while the store is still pending on the GPU, which is the case the ordered flush handles.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added perf Performance change perf numbers missing perf pull request without a before and after summary line unblocks title Names a title ID on the Unblocks line agc libSceAgc* labels Nov 3, 2026
@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown

This pull request is labeled perf, so it needs the summary line of a capture on main and one with the change, from the same title, scene and machine, as the Performance report form asks. Please paste both aps5-summary.jsonl lines, attach the two traces and add the output of tools/trace_digest.py.

milescoviello and others added 2 commits November 3, 2026 16:41
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A label stores pending linear and small (64 KiB or less) storage images before it; a 256 KiB image is stored there only when it was written occasionally, so the test checked an order the driver does not promise. The surface is now 128x128 R32UI, one 64 KiB block, and the test requires that size.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@milescoviello
milescoviello marked this pull request as ready for review November 3, 2026 22:12
@EagleBearerYS

Copy link
Copy Markdown
Contributor

Hi, a note on how this PR and #13251 fit together. Both change the FlushPending call in StorageTexture::refresh (Texture.cpp), and git can't merge the two on its own.

  • This PR wraps that call in the GPU-ordered flush scope, when the refresh is eligible: no alias, no key hand-over, no DCC metadata, no layer still pending, and a host import for its own image.
  • perf(agc): take an aliased storage image's units on the device without a host import #13251 lets a refresh take another storage image's pending units on the device without a host import. It keeps a list of alias plans (aliases) where main keeps one alias. While the refresh runs, those aliases are left out of FlushPending (refreshingAliases), so the call passes nullptr for the alias. Just before the call, the refresh flushes the alias units it stores itself (aliasStores), so flushed is already set and the call ORs into it: flushed = (...) || flushed.

A proposed composition, for whichever PR goes second:

A refresh with an alias plan stays ineligible, as one with an alias is here, so the two paths don't meet. Either order works. If this PR lands first, #13251 takes the composition in its next merge of main.

@milescoviello

Copy link
Copy Markdown
Contributor Author

@EagleBearerYS thanks, that composition looks right to me. A refresh with an alias plan stays out of the ordered path, as one with an alias is here, so orderedEligible reading aliases.empty(), the ordered scope around #13251's call, and #13251's flushed are all it takes. If #13251 lands first I'll merge it into this branch that way and rerun this PR's tests (agc_driver_label_ordered_flush and the completion tests in the description). If this one lands first, your plan for #13251 works as written.

AI-assisted: yes (Claude Code)

Resolves Texture.cpp: the refresh keeps its GPU-ordered flush scope and calls main's pruneAdoptedSources() after the flush.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added the docs label Nov 3, 2026
@TechLevelZero

Copy link
Copy Markdown
Contributor

✅ NVIDIA GPU test: passed

PR head d3f7307 merged onto main 1e65852 · tested 2026-11-03 22:37 UTC · 11 min

GPU NVIDIA RTX 2000 Ada Generation, driver 595.99.02, Vulkan 1.4.329
Host Ubuntu 26.04.1 LTS, kernel 7.0.0-38-generic, container ubuntu:24.04
Logs build and test logs for this run
Build ✅ passed (Release, -DANYPS5_ENABLE_SPIRV_TOOLS=ON)
Tests ✅ 2,696 passed · 3 failed (0 new, 3 also fail on main) · 48 skipped (ANYPS5_REQUIRE_VULKAN=1)
Test setup CPU-only tests without a GPU; GPU tests (and any CPU-only test that failed or skipped without one) once on the GPU with VK_LAYER_KHRONOS_validation and ANYPS5_NO_SHADER_CACHE=1; main ran the same way
Test selection 2,747 of 3,818 tests run: the ones this change can reach, chosen by tools/select_tests.py from main's build graph
Vulkan validation ✅ no new validation IDs vs main (929 tests run on the GPU)
Slow tests (>30 s, GPU tests timed with validation layers) ⚠️ agc_driver_pipeline_journal 107.3 s
Failing on main too (3, not counted against this PR)
agc_driver_image_table
guest_mmap_executable
guest_socket_options_multicast
Validated but not compared (1, not in the main baseline's validation pass)
agc_driver_label_ordered_flush
Validation IDs already present on main (4 IDs in 517 tests)
VUID-VkDeviceCreateInfo-pNext-pNext (517 tests): agc_driver, agc_driver_address_passes, agc_driver_alias_copy_label, agc_driver_alpha_to_coverage, agc_driver_array_layer_target, agc_driver_bc_block_copy, agc_driver_bda_anchor_reads, agc_driver_bda_span_reads, and 509 more
VUID-vkCmdCopyImage-srcImage-01548 (7 tests): agc_driver_depth_clear_hle, agc_driver_depth_plane_buffer_memory, agc_driver_depth_plane_partial_store, agc_driver_depth_plane_sampling, agc_driver_depth_storage_dispatch, agc_driver_depth_surface_from_storage, agc_driver_dispatch_move_depth
VUID-vkCmdCopyImage-srcImage-01551 (7 tests): agc_driver_depth_clear_hle, agc_driver_depth_plane_buffer_memory, agc_driver_depth_plane_partial_store, agc_driver_depth_plane_sampling, agc_driver_depth_storage_dispatch, agc_driver_depth_surface_from_storage, agc_driver_dispatch_move_depth
VUID-VkPipelineColorBlendStateCreateInfo-pAttachments-00605 (2 tests): agc_driver_prepared_exports, agc_driver_srgb8_color_target
Skipped (48)
agc_driver_bda_reads_shadowed_results: skipped, the device reports watched host memory written when it imports or reads it
agc_driver_bda_reads_shadowed_results_per_build: skipped, the device reports watched host memory written when it imports or reads it
agc_driver_sample_order_grid_reference: skipped, the device does not keep depth drawn at custom sample locations at those locations, even after a transition that names them: the composite of uniform renders holds the guest's depth in 0 of 3005 sloped samples, so no path can be compared with it bit for bit
agc_driver_scattered_runs: skipped, host imports are unavailable: /dev/udmabuf is not accessible
agc_driver_sparse_dma_buf_import: skipped, host imports are unavailable: /dev/udmabuf is not accessible
agc_driver_stencil_ref_export: skipped, the device lacks VK_EXT_shader_stencil_export
agc_host_import_sparse: skipped, host imports are unavailable: /dev/udmabuf is not accessible
agc_import_settle_initial_dcc: skipped, the device has no deferred settle of host imports: imports under 1 MiB wait for one only on Windows
agc_unit_shadow_completion_store: skipped, the device reports watched host memory written when it imports or reads it
guest_http_uri_locale: 
guest_net_ctl: skipped, the host has no IPv4 address
guest_video_out_color_space: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_hdr: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_high_frame_rate: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_output_mode: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_resolution_status: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_wqhd_detection: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
video_out_flip_aftergpu: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_backtoback: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_controls: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_cpuflip: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_cpuflip_in_order: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_dcc: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_deferred: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_eop: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_eop_aftergpu: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_eop_main: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not co
Reproduce
git fetch origin d3f7307afdf8752297fc8219277117a2d44e9dc5 1e658521a0a599872612bb91949311e2956b5283
git checkout 1e658521a0a599872612bb91949311e2956b5283 && git merge --no-ff d3f7307afdf8752297fc8219277117a2d44e9dc5
git submodule update --init --recursive
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=ON -DANYPS5_ENABLE_SPIRV_TOOLS=ON
cmake --build build && cmake --build build --target libs
VK_DRIVER_FILES=/nonexistent/icd.json ctest --test-dir build -E '^(agc|shader|recompiler|videoout|video_out|guest_video_out|vulkan)|gpu' --timeout 120 --parallel 32 --output-on-failure
ANYPS5_REQUIRE_VULKAN=1 ANYPS5_NO_SHADER_CACHE=1 VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation \
  ctest --test-dir build -R '^(agc|shader|recompiler|videoout|video_out|guest_video_out|vulkan)|gpu' --timeout 120 --parallel 4 --output-on-failure
# failures are then re-run alone: --timeout 360 --parallel 2; new validation IDs: the test alone, twice, --parallel 1

Automated build and test on a contributor's NVIDIA machine; it does not replace upstream CI. A new comment is posted for each new push. Full logs.

@TechLevelZero

Copy link
Copy Markdown
Contributor

✅ AMD GPU test: passed

PR head d3f7307 merged onto main 1e65852 · tested 2026-11-03 22:43 UTC · 16 min

GPU AMD Radeon RX 6300 (RADV NAVI24), driver Mesa 25.2.8-0ubuntu0.24.04.4, Vulkan 1.4.318
Host Ubuntu 26.04.1 LTS, kernel 7.0.0-42-generic, container ubuntu:24.04
Logs build and test logs for this run
Build ✅ passed (Release, -DANYPS5_ENABLE_SPIRV_TOOLS=ON)
Tests ✅ 2,680 passed · 10 failed (0 new, 9 also fail on main, 1 unstable on main) · 57 skipped (ANYPS5_REQUIRE_VULKAN=1)
Test setup CPU-only tests without a GPU; GPU tests (and any CPU-only test that failed or skipped without one) once on the GPU with VK_LAYER_KHRONOS_validation and ANYPS5_NO_SHADER_CACHE=1; main ran the same way
Test selection 2,747 of 3,818 tests run: the ones this change can reach, chosen by tools/select_tests.py from main's build graph
Vulkan validation ✅ no new validation IDs vs main (929 tests run on the GPU)
Slow tests (>30 s, GPU tests timed with validation layers) ⚠️ agc_driver_pipeline_journal 120.2 s, ⚠️ agc_driver_lazy_resolved_abi 55.9 s, ⚠️ agc_driver_image_sample_unnormalized 51.3 s, ⚠️ public_headers 37.1 s, ⚠️ agc_driver_sample_multisample_target 36.7 s
Unstable on main (1, these also failed in recent main runs on this machine; not counted against this PR)
agc_driver_pipeline_journal: failed in 7 of the last 9 main runs
Failing on main too (9, not counted against this PR)
agc_driver_image_table
agc_driver_native_depth_bda_clear
agc_driver_native_multisample_depth
agc_driver_store_over_untracked
agc_driver_submit_progress
agc_submit_progress
agc_submit_progress_no_timeline
guest_mmap_executable
guest_socket_options_multicast
Validated but not compared (1, not in the main baseline's validation pass)
agc_driver_label_ordered_flush
Validation IDs already present on main (4 IDs in 517 tests)
VUID-VkDeviceCreateInfo-pNext-pNext (517 tests): agc_driver, agc_driver_address_passes, agc_driver_alias_copy_label, agc_driver_alpha_to_coverage, agc_driver_array_layer_target, agc_driver_bc_block_copy, agc_driver_bda_anchor_reads, agc_driver_bda_span_reads, and 509 more
VUID-vkCmdCopyImage-srcImage-01548 (4 tests): agc_driver_depth_clear_hle, agc_driver_depth_storage_dispatch, agc_driver_depth_surface_from_storage, agc_driver_dispatch_move_depth
VUID-vkCmdCopyImage-srcImage-01551 (4 tests): agc_driver_depth_clear_hle, agc_driver_depth_storage_dispatch, agc_driver_depth_surface_from_storage, agc_driver_dispatch_move_depth
VUID-VkPipelineColorBlendStateCreateInfo-pAttachments-00605 (2 tests): agc_driver_prepared_exports, agc_driver_srgb8_color_target
Skipped (57)
agc_driver_bda_reads_shadowed_results: skipped, the device reports watched host memory written when it imports or reads it
agc_driver_bda_reads_shadowed_results_per_build: skipped, the device reports watched host memory written when it imports or reads it
agc_driver_copied_writer_label_order: skipped, the device reports a minStorageBufferOffsetAlignment of 4, so the dispatch binds its record at +0x1004 in place and nothing is written back; the test needs an alignment above 4
agc_driver_float16_unused_nan_payloads: skipped, the device has no native f16 conversions with RTE rounding and kept f16 denormals
agc_driver_image_table_vouched_lookups: skipped, the device reports 4 of 4 host-imported pages written after a GPU read, so a watched imported table cannot be vouched for
agc_driver_linear_target_direct_write_completion: skipped, the device does not keep host imports write-watched, so no completion stamp over the target is tracked
agc_driver_ngg_vertex_wave_reduction: skipped, s3 describes a 32-lane wave only on 32-lane subgroups, and the device has 64 lanes
agc_driver_scattered_runs: skipped, host imports are unavailable: /dev/udmabuf is not accessible
agc_driver_sparse_dma_buf_import: skipped, host imports are unavailable: /dev/udmabuf is not accessible
agc_driver_unit_shadow: skipped, the device does not import watched host memory for unit shadows
agc_driver_unit_shadow_budget: skipped, the device does not import watched host memory for unit shadows
agc_host_import_sparse: skipped, host imports are unavailable: /dev/udmabuf is not accessible
agc_idle_publish_wait: skipped, the device reports 4 of 4 host-imported pages written after a GPU read, so a forced watch of the import reads its submissions as CPU stores
agc_import_settle_initial_dcc: skipped, the device has no deferred settle of host imports: imports under 1 MiB wait for one only on Windows
agc_shadowed_key_stores: skipped, the device imports no watched host memory under a unit shadow
agc_small_tiled_shadow_label: skipped, host imports are unavailable: the test block is not imported and watched
agc_unit_shadow: skipped, the device does not import watched host memory for unit shadows
agc_unit_shadow_completion_store: skipped, the device reports watched host memory written when it imports or reads it
guest_http_uri_locale: 
guest_net_ctl: skipped, the host has no IPv4 address
guest_video_out_color_space: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_hdr: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_high_frame_rate: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_output_mode: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_resolution_status: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_wqhd_detection: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
video_out_flip_aftergpu: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_backtoback: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_controls: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_cpuflip: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscree
Reproduce
git fetch origin d3f7307afdf8752297fc8219277117a2d44e9dc5 1e658521a0a599872612bb91949311e2956b5283
git checkout 1e658521a0a599872612bb91949311e2956b5283 && git merge --no-ff d3f7307afdf8752297fc8219277117a2d44e9dc5
git submodule update --init --recursive
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=ON -DANYPS5_ENABLE_SPIRV_TOOLS=ON
cmake --build build && cmake --build build --target libs
VK_DRIVER_FILES=/nonexistent/icd.json ctest --test-dir build -E '^(agc|shader|recompiler|videoout|video_out|guest_video_out|vulkan)|gpu' --timeout 120 --parallel 12 --output-on-failure
ANYPS5_REQUIRE_VULKAN=1 ANYPS5_NO_SHADER_CACHE=1 VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation \
  ctest --test-dir build -R '^(agc|shader|recompiler|videoout|video_out|guest_video_out|vulkan)|gpu' --timeout 120 --parallel 16 --output-on-failure
# failures are then re-run alone: --timeout 360 --parallel 2; new validation IDs: the test alone, twice, --parallel 1

Automated build and test on a contributor's AMD machine; it does not replace upstream CI. A new comment is posted for each new push. Full logs.

@XairRrR XairRrR left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed — changes requested

The ordered flush is correct where I could check it, and the new test reaches the new path. The PR is a perf change, though, and it lacks the summary lines from main and from the change and the traces that the rules require.

What I ran

Head d3f7307afdf8752297fc8219277117a2d44e9dc5 merged into main 49021a4c907b7f493429dfe269416bbec2de96e1 with --no-ff (merged tree 13013875e03d5a12b8def62b4a51e6ebbea6e6fb). Debian 13 VM, GCC 14.2.0, Release, Ninja, lavapipe (Mesa 26.1.6, LLVM 19).

git diff --check 49021a4c907b7f493429dfe269416bbec2de96e1...d3f7307afdf8752297fc8219277117a2d44e9dc5
python3 tools/check_conventions.py --base 49021a4c907b7f493429dfe269416bbec2de96e1
ctest --test-dir build -R '^agc_driver_label_ordered_flush$' --output-on-failure
env VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation ANYPS5_NO_SHADER_CACHE=1 ANYPS5_REQUIRE_VULKAN=1 \
  ctest --test-dir build -j8 --test-load 32 --output-on-failure \
  -R '^agc_driver_(label_ordered_flush|direct_write_completion|linear_target_direct_write_completion|fill_completion_labels|linear_target_completion_points|memory_value_fill)$'
env VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation ANYPS5_NO_SHADER_CACHE=1 ANYPS5_REQUIRE_VULKAN=1 VK_LOADER_DEBUG=error,warn,layer \
  ctest --test-dir build -V -R '^agc_driver_label_ordered_flush$'
# negative control and probe (see below)
git show 49021a4c907b7f493429dfe269416bbec2de96e1:core/libs/prx/libSceAgcDriver/Graphics/src/Texture.cpp > core/libs/prx/libSceAgcDriver/Graphics/src/Texture.cpp
ninja -C build -j32 -l32 tests/agc_driver_label_ordered_flush_tests
ctest --test-dir build -R '^agc_driver_label_ordered_flush$' --output-on-failure --repeat until-fail:3
git show 13013875e03d5a12b8def62b4a51e6ebbea6e6fb:core/libs/prx/libSceAgcDriver/Graphics/src/Texture.cpp > core/libs/prx/libSceAgcDriver/Graphics/src/Texture.cpp
git status --porcelain

Results

Check Result
git diff --check clean
check_conventions.py All conventions checks passed
Configure / build / libs (script) passed
Selected tests (script, 2176 run) 0 new failures, 0 timeouts; the 7 known failures also fail on main (agc_driver_division, denormal-mode tests, vector_constant_factor_tininess, one unwrap_self test)
agc_driver_label_ordered_flush passes, 0.38 s
The 6 focused tests from the description, with the validation layer 5 passed, agc_driver_linear_target_direct_write_completion skipped on lavapipe
Validation, agc_driver_label_ordered_flush layer loaded (inserted, no "was not found but was requested"), 0 VUID- lines
Script validation comparison 783/784 tests with the layer loaded. New VUID-VkSamplerCreateInfo-addressMode* lines only in agc_driver_image_gather_volume, which was not in main's baseline pass. The PR does not touch sampler creation, so I don't count them against it
Skip actually taken in the new test (temporary probe in FlushPending) ordered=1 active=1 skip=1 n=1 in all 8 runs; label landed after the stored texels
Probe output (temporary fprintf before if (orderedSkip), removed afterwards)
      8 [probe] ordered=1 active=1 skip=1 n=1
      1 label ordered flush tests passed

Negative control

I restored core/libs/prx/libSceAgcDriver/Graphics/src/Texture.cpp from main, rebuilt agc_driver_label_ordered_flush_tests and ran it three times: it passes on main. That is expected. On main the CPU waits for the writer, so the label order already holds. The test guards against the skip breaking the order; it doesn't fail without the change. A temporary probe in the merged build showed that the test reaches the new skip in each of its 8 runs. I then restored the merged file and rebuilt; git status --porcelain is empty.

Findings

Blocking

  • Perf evidence missing (project rule). The PR is titled perf(agc) and still carries perf numbers missing. CONTRIBUTING.md requires the aps5-summary.jsonl lines of main and of the change from the same machine and scene, the two traces, and tools/trace_digest.py output. The bot comment asks for the same. The description gives fps and p50 from a "local stack (the same change on an older base, with other GoT experiments)", which doesn't isolate this change. Fix: capture PPSA02225 on current main and on this branch the way the Performance report form describes, then paste both summary lines, attach the traces and add the digest.

Non-blocking

  • The skip doesn't check the image's device against the active recorder. core/libs/prx/libSceAgcDriver/Graphics/src/Texture.cpp:4690 requires Recorder::Active() != nullptr and a host import for each flushed image. The write-back records into the recorder only when recorderFor(context) matches the image's device (Texture.cpp:70-72). Otherwise it goes to a standalone CommandBatch (Texture.cpp:6783-6807), which isn't ordered behind the writer. The first range loop in StoreReadbackBeforeLabel doesn't filter by device, and the tests create several Vulkan devices. Suggest adding texture->context.device == Recorder::Active()->Device() to the all_of predicate, so a cross-device flush keeps the wait.
  • Checklist Unblocks line. The rule's form is Unblocks: PPSA01325 #3232, with the Game bring-up discussion. The line here names no discussion, and it describes a speed-up rather than a stop or a wrong result. Add the discussion number, or use none if no bring-up entry tracks this.
  • Head moved since the reviewed commit. The PR head is now f37b080 (a merge of upstream main that resolves Texture.cpp around pruneAdoptedSources()). I reviewed d3f7307 merged into main, as prepared, so that conflict resolution was not reviewed here.

Not verified

  • Hardware GPUs: only lavapipe here. The description's RTX 4070 Ti and GT 1030 runs, and the NVIDIA CI comment, were not reproduced.
  • The GoT numbers (I can't run titles) and the claim that frames are unchanged.
  • A refresh where the skip is taken over memory outside the write watch. There, the skipped write-backs' CPU-side checks (cpuLeftUnchanged, ChangedBlocks) may read guest bytes that an earlier recorded write-back is still copying into. I found no test covering this.
  • The negative control can't show that the test detects a misordering, because main is ordered by the CPU wait.

To get this to a good state

  1. Add the two summary lines (main and this change, same scene and machine), the traces and the trace_digest.py output, so the perf numbers missing label clears.
  2. Optionally restrict the skip to images on the active recorder's device, and fix the Unblocks line.
  3. Rerun the focused tests on the new head f37b080 after the merge resolution.

…der's device

The write-backs are recorded into the active recorder, so an image of another device keeps the CPU wait.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@XairRrR XairRrR left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-checked — changes still requested

Since my last review, the author merged upstream main into the branch (f37b080). That merge resolves the Texture.cpp conflict around pruneAdoptedSources() correctly, and the tests pass. The perf evidence is still missing. The newest head, 2da1597 (a device check on the ordered skip), was not part of the tree prepared for this review.

What changed since my last review

  • f37b080: a merge of upstream main. In StorageTexture::refresh, the ordered-flush scope wraps FlushPending, and pruneAdoptedSources() follows the scope. That is the same position as on main, where it directly follows FlushPending (core/libs/prx/libSceAgcDriver/Graphics/src/Texture.cpp:2339). The four earlier commits are unchanged (delta.patch).
  • 2da1597 ("take the ordered flush only for images on the active recorder's device") is listed in pr.md as the current head. It was not in the prepared tree, and its object is not in the repository here, so I could not review it.
  • No author reply to my review. The description is unchanged in the parts that matter: there are still no summary lines and no traces.

What I ran

Head f37b080fed2a31fe64cf1ecd3e88778a3ba6593f merged into main 1c8c0efc8c41314402e6966da5bef6ea3d639d10 with --no-ff (merged tree 16285379dde5714fadf0c8bb93a6248ad1222d13). Debian 13 VM, GCC 14.2.0, Release, Ninja, lavapipe (Mesa 26.1.6, LLVM 19).

git diff --check 1c8c0efc8c41314402e6966da5bef6ea3d639d10...f37b080fed2a31fe64cf1ecd3e88778a3ba6593f
python3 tools/check_conventions.py --base 1c8c0efc8c41314402e6966da5bef6ea3d639d10
env VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation ANYPS5_NO_SHADER_CACHE=1 ANYPS5_REQUIRE_VULKAN=1 VK_LOADER_DEBUG=error,warn,layer \
  ctest --test-dir build -V -R '^agc_driver_label_ordered_flush$'
env VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation ANYPS5_NO_SHADER_CACHE=1 ANYPS5_REQUIRE_VULKAN=1 \
  ctest --test-dir build -j8 --test-load 32 --output-on-failure \
  -R '^agc_driver_(label_ordered_flush|direct_write_completion|linear_target_direct_write_completion|fill_completion_labels|linear_target_completion_points|memory_value_fill)$'
git show 1c8c0efc8c41314402e6966da5bef6ea3d639d10:core/libs/prx/libSceAgcDriver/Graphics/src/Texture.cpp > core/libs/prx/libSceAgcDriver/Graphics/src/Texture.cpp
ninja -C build -j32 -l32 tests/agc_driver_label_ordered_flush_tests
ctest --test-dir build -R '^agc_driver_label_ordered_flush$' --output-on-failure --repeat until-fail:3
git show 16285379dde5714fadf0c8bb93a6248ad1222d13:core/libs/prx/libSceAgcDriver/Graphics/src/Texture.cpp > core/libs/prx/libSceAgcDriver/Graphics/src/Texture.cpp
ninja -C build -j32 -l32 tests/agc_driver_label_ordered_flush_tests
ctest --test-dir build -R '^agc_driver_label_ordered_flush$'
git status --porcelain
Check Result
git diff --check clean
check_conventions.py All conventions checks passed
Configure / build / libs (script) passed
Selected tests (script, 2177 run) 0 new failures, 0 timeouts. The 7 known failures also fail on main. agc_driver_alpha_to_coverage_sample_mask_export passed on a rerun
agc_driver_label_ordered_flush with validation passes. The layer was inserted, with no "was not found but was requested" message and 0 VUID- lines
The 6 focused tests from the description, with validation 5 passed, and agc_driver_linear_target_direct_write_completion was skipped on lavapipe
Script validation comparison 784 of 785 tests loaded the layer. New VUID-VkSamplerCreateInfo-addressMode* lines appear only in agc_driver_image_gather_volume, which main's baseline did not run. The PR doesn't touch sampler creation, so I don't count them against it
Negative control with Texture.cpp restored from main, the test passes 3 times out of 3, as expected (see last review). After restoring the merged file it passes again, and git status --porcelain is empty

Previous findings

  • Blocking: perf evidence missing. Still open. The description still has no aps5-summary.jsonl lines from main and from the change, no traces and no trace_digest.py output, and the PR still carries perf numbers missing.
  • Non-blocking: the skip doesn't check the image's device against the active recorder. Still open in the reviewed tree: the all_of predicate at core/libs/prx/libSceAgcDriver/Graphics/src/Texture.cpp:4696 tests only HostImportFor. The author pushed 2da1597, whose message says it adds this check. I couldn't see that commit, so I can't confirm it.
  • Non-blocking: checklist Unblocks line. Still open. It names no Game bring-up discussion, and it describes a speed-up rather than a stop or a wrong result.
  • Non-blocking: head moved, merge resolution not reviewed. Resolved for f37b080. The resolution keeps pruneAdoptedSources() right after the flush, as on main, and the focused tests pass on the merged tree.

New findings

Blocking

None.

Non-blocking

None.

Not verified

  • Commit 2da1597: it was not available in the prepared tree.
  • Hardware GPUs: only lavapipe here.
  • The GoT numbers, and the claim that frames are unchanged.
  • Whether the test catches a misordering: on main the CPU wait already provides the order, so the negative control passes there.

To get this to a good state

  1. Capture PPSA02225 on current main and on this branch, on the same machine and scene, following the Performance report form. Paste both summary lines, attach the traces and add the trace_digest.py output.
  2. Add the Game bring-up discussion to the Unblocks line, or write none.
  3. I'll check the device-check commit 2da1597 in the next pass.

@github-actions github-actions Bot removed the unblocks title Names a title ID on the Unblocks line label Nov 3, 2026
@milescoviello

Copy link
Copy Markdown
Contributor Author

Thanks for the re-check.

  • Unblocks is now none.
  • 2da1597 is pushed to the PR branch. It adds texture->context.device == orderedRecorder->Device() to the skip's all_of predicate, so images on another device take the old CPU-wait path. agc_driver_label_ordered_flush and the 5 other focused tests pass on it here (RTX 4070 Ti). A Windows run on the GT 1030 is in progress.
  • Perf evidence: the PPSA02225 captures are queued on one machine (RTX 4070 Ti, driver 615.71.09, Ubuntu 26.04.1).

AI-assisted: yes (Claude Code)

@XairRrR XairRrR left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-checked — changes still requested

2da1597 adds the device check on the ordered skip, and Unblocks now reads none. Both earlier non-blocking findings are resolved. The perf evidence is still missing.

What changed since my last review

  • 2da1597: FlushPending keeps the active recorder in orderedRecorder and adds texture->context.device == orderedRecorder->Device() to the skip's all_of predicate (core/libs/prx/libSceAgcDriver/Graphics/src/Texture.cpp:4696-4697). Recorder::Device() returns the recorder's context.device (core/libs/prx/libSceAgcDriver/Graphics/include/Recorder.hpp:457). The four earlier commits and the merge are unchanged (delta.patch).
  • Description: the Unblocks line is now none.
  • Author comment: the captures are queued on an RTX 4070 Ti. The base is main 1c8c0efc8c plus #9742, and the plan is three base runs alternating with three runs of the change. The summary lines, traces and digest are not in the description yet.

What I ran

Head 2da15973ce0a3fa5e4d9e574d6ee1df640544dad merged into main 1c8c0efc8c41314402e6966da5bef6ea3d639d10 with --no-ff (merged tree 8299001b2297bfe2faec19be0c646a7b41e71798). Debian 13 VM, GCC 14.2.0, Release, Ninja, lavapipe (Mesa 26.1.6, LLVM 19).

git diff --check 1c8c0efc8c41314402e6966da5bef6ea3d639d10...2da15973ce0a3fa5e4d9e574d6ee1df640544dad
python3 tools/check_conventions.py --base 1c8c0efc8c41314402e6966da5bef6ea3d639d10
env VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation ANYPS5_NO_SHADER_CACHE=1 ANYPS5_REQUIRE_VULKAN=1 VK_LOADER_DEBUG=error,warn,layer \
  ctest --test-dir build -j8 --test-load 32 -V \
  -R '^agc_driver_(label_ordered_flush|direct_write_completion|linear_target_direct_write_completion|fill_completion_labels|linear_target_completion_points|memory_value_fill)$'
env VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation ANYPS5_NO_SHADER_CACHE=1 ANYPS5_REQUIRE_VULKAN=1 VK_LOADER_DEBUG=error,warn,layer \
  ctest --test-dir build -V -R '^agc_driver_label_ordered_flush$'
env APS5_TRACE_FLUSH=1 ANYPS5_REQUIRE_VULKAN=1 ctest --test-dir build -V -R '^agc_driver_label_ordered_flush$'
env APS5_NO_FLUSH_GPU_ORDERED=1 ANYPS5_REQUIRE_VULKAN=1 ctest --test-dir build -R '^agc_driver_label_ordered_flush$' --output-on-failure
git status --porcelain
Check Result
git diff --check clean
check_conventions.py All conventions checks passed
Configure, build and libs (script) passed
Selected tests (script, 2177 run) 0 new failures and 0 timeouts. The 7 known failures also fail on main. agc_driver_alpha_to_coverage_sample_mask_export passed on a rerun
The 6 focused tests, with validation 5 passed. agc_driver_linear_target_direct_write_completion was skipped on lavapipe. 0 VUID- lines, and no "was not found but was requested" message
agc_driver_label_ordered_flush, with validation passed. The loader added VK_LAYER_KHRONOS_validation, with 0 VUID- lines
APS5_TRACE_FLUSH=1 each run logs a [flush] image …+0x10000 for readback before label, so the test reaches the ordered path
APS5_NO_FLUSH_GPU_ORDERED=1 passed (the CPU-wait path)
Script validation comparison 784 of 785 tests loaded the layer. New VUID-VkSamplerCreateInfo-addressMode* lines appear only in agc_driver_image_gather_volume. main's baseline did not run that test, and the PR doesn't touch sampler creation, so I don't count them against it (same as last time)
git status --porcelain empty

The test is unchanged since my last review, so I did not repeat the negative control. That control passed with main's Texture.cpp, as expected, because there the CPU wait already provides the order.

Previous findings

  • Blocking: perf evidence missing. Still open. The description has no aps5-summary.jsonl lines from a base and from the change, no traces and no trace_digest.py output, and the PR still carries perf numbers missing. The author says the captures are in progress.
  • Non-blocking: the skip doesn't check the image's device against the active recorder. Resolved by 2da1597. The predicate now requires texture->context.device == orderedRecorder->Device(). An image of another device falls through to the old wait loop, and the refresh's orderedSkipped is only set when FlushPending actually skipped.
  • Non-blocking: checklist Unblocks line. Resolved. It now reads none.

New findings

Blocking

None.

Non-blocking

None.

Not verified

  • Hardware GPUs: only lavapipe here. The author reports that 2da1597 passes on an RTX 4070 Ti; the GT 1030 run was still in progress.
  • The GoT frame-rate numbers, and the claim that frames are unchanged.
  • Whether the test catches a misordering: on main the CPU wait already provides the order, so the test passes there too.

To get this to a good state

  1. Add the PPSA02225 summary lines for the base and the change to the description, from the same machine and scene, as the Performance report form asks. Attach the traces and add the trace_digest.py output. The base the author describes (main + #9742, with only this PR's 3 files differing) is fine, as long as the two lines carry different commits.

@TripyFox

TripyFox commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

Windows/NVIDIA data point: ASTRO BOT (PPSA21567 01.019.000), Crash Site.

Setup

  • RTX 5090 (NVIDIA 621.01), Ryzen 7 9800X3D, Windows 11 26200, 3840x2160, warm shader cache, no input.
  • A is main 086c068. B is the same main with this PR merged cleanly. The perf numbers below were measured on head 849a669 (merge commit 8021c53cb2). A confirmation pair and the tests were run on the current head 2da1597 (see below). Both trees are Release, MinGW GCC 15.2. Only libSceAgcDriver.prx differs between the two run folders, and astrobot.exe is byte-identical.
  • Each run is 240 s with -lvl hub_crashsite, APS5_TRACE_SUMMARY_EVERY=10 APS5_TRACE_BENCH=1 and nothing else set. The order was A1 B1 A2 B2, then A3 B3. The steady window is intervals 8-22 (about 70-220 s, 15 intervals per run). Each value below is the median over that window.
run presents / 10 s p50 ms p90 ms p99 ms GPU busy ms / present
A1 104 96.2 101.4 111.3 75.6
A2 111 90.4 94.6 102.7 72.4
A3 109 91.5 97.6 111.5 72.8
B1 121 83.6 87.7 89.9 72.3
B2 121 83.8 87.4 90.1 72.5
B3 119 85.2 89.1 95.9 73.5
| Metric | Baseline | Candidate | Change | Change (%) |
| --- | ---: | ---: | ---: | ---: |
| frame_ms.p50 | 90.162 | 83.770 | -6.392 | -7.089 |
| frame_ms.p99 | 95.683 | 89.286 | -6.397 | -6.686 |
| mean_frame_ms | 89.664 | 82.753 | -6.910 | -7.707 |

What changes underneath (one 200 s counter run per arm, APS5_PROFILE_GPU=1 APS5_PROFILE_DRAW=1, 12 steady 10 s samples; APS5_PROFILE_DRAW changes the draw path, so these runs are not used for the frame times above)

  • Recorder syncs drop from 4209 to 3157 per 10 s, and pending-write syncs from 1792 to 472. The busiest sync site in A (1374 syncs, 742 ms per 10 s) is gone in B. Those are many short waits of about 0.5 ms, not a few long ones.
  • The data moved per present stays the same.
    • storage refresh write-backs: 25.8 vs 25.9 per present, 98.2 vs 98.4 MiB.
    • Storage uploads: 262 vs 263 MiB per present.
    • The storage-writeback GPU class: 18.2 vs 18.4 ms per present.
    • The extra re-uploads that an ordered-skipped refresh could cause do not show up in this title.

Which half helps here (one 240 s run each on B, same method)

arm p50 ms p90 ms p99 ms presents / 10 s
A, mean of 3 92.7 97.9 108.5 108
B, mean of 3 84.2 88.1 92.0 120
B + APS5_NO_REFRESH_GPU_ORDERED=1 85.2 89.0 98.7 119
B + APS5_NO_FLUSH_GPU_ORDERED=1 91.1 95.1 101.2 111
B + APS5_NO_LABEL_READBACK_STORE=1 (diagnostic switch only, not a faithful mode) 76.7 80.9 93.8 131
  • In this title the label half carries the gain: with only the label half, B keeps about 88 % of it. The refresh half alone is worth about 1.6 ms, which is within main's run-to-run noise.
  • With the label readback store switched off entirely on top of this PR, GPU busy drops from about 72.8 to 54.6 ms per present and p50 drops another 7.6 ms. So on this title the label store's GPU work, which this PR leaves in place, is still worth about as much again. That's a reference point for the planned asynchronous label store (perf: ASTRO BOT's Crash Site is about 47 ms slower per frame since #6948's label readback store #10364), not a proposal to drop the store.
  • 20 refreshes per frame store other images. 12 of them are of DCC images, which don't qualify for the ordered path. The other 8 qualify.

Confirmation on the current head 2da1597 (main 086c068 with 2da1597 merged cleanly, merge commit 22ddeff52d; one A/B pair, same method)

run presents / 10 s p50 ms p90 ms p99 ms GPU busy ms / present
A4 (main) 110 91.3 95.6 105.7 72.9
B4 (2da1597) 119 85.4 89.0 98.2 73.1

Tests (Windows, RTX 5090, Khronos validation 1.4.363, head 2da1597)

  • Run on main 086c068 and on the merge of 2da1597, BUILD_TESTING=ON, with ANYPS5_REQUIRE_VULKAN=1 ANYPS5_NO_SHADER_CACHE=1 VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation. VK_LOADER_DEBUG=layer shows the layer inserted and no "not found" line.
  • agc_driver_label_ordered_flush passes in 8 of 8 runs.
  • The same ctest pattern on both trees, covering 26 storage and label execution tests: agc_driver_storage*, pending_exact, dcc_key, copied_writer, finisher_label, unregistered/unimported_label, label_after_import, linear_readback_label, same_queue_label, pipelined_predicated_label, alias_copy_label, fill_completion_labels, two_device_pending, depth_surface_last_writer, staged_clear_sampled, sampled_clear_keys. All pass on both trees, with 0 VUID- lines in either log. The only validation warning is WARNING-VkImageSubresourceRange-layerCount-compatibility from agc_driver_storage_write_back_no_window, identical on both.

No crashes, device losses or FATAL/ASSERT lines in any A/B or split run of this PR's build or main. One launch with the diagnostic APS5_NO_LABEL_READBACK_STORE=1 crashed shortly after the level started (access violation), and its rerun was clean.

On the perf numbers missing note: these runs used the APS5_TRACE_SUMMARY_EVERY=10 APS5_TRACE_BENCH=1 bench summaries, so I have no .json.gz traces to attach. If it would help, I can record the performance-form capture (APS5_TRACE_FRAMES, no other switch) on both arms and post the two aps5-summary.jsonl lines, the traces and the tools/trace_digest.py output.

AI-assisted: yes (Claude Code)

@TechLevelZero

Copy link
Copy Markdown
Contributor

✅ NVIDIA GPU test: passed

PR head 2da1597 merged onto main 1c8c0ef · tested 2026-11-04 01:19 UTC · 25 min

GPU NVIDIA RTX 2000 Ada Generation, driver 595.99.02, Vulkan 1.4.329
Host Ubuntu 26.04.1 LTS, kernel 7.0.0-38-generic, container ubuntu:24.04
Logs build and test logs for this run
Build ✅ passed (Release, -DANYPS5_ENABLE_SPIRV_TOOLS=ON)
Tests ✅ 2,712 passed · 5 failed (0 new, 3 also fail on main, 2 unstable on main) · 50 skipped (ANYPS5_REQUIRE_VULKAN=1)
Test setup CPU-only tests without a GPU; GPU tests (and any CPU-only test that failed or skipped without one) once on the GPU with VK_LAYER_KHRONOS_validation and ANYPS5_NO_SHADER_CACHE=1; main ran the same way
Test selection 2,767 of 3,842 tests run: the ones this change can reach, chosen by tools/select_tests.py from main's build graph
Vulkan validation ✅ no new validation IDs vs main (945 tests run on the GPU)
Slow tests (>30 s, GPU tests timed with validation layers) ⚠️ agc_driver_pipeline_journal 120.1 s
Unstable on main (2, these also failed in recent main runs on this machine; not counted against this PR)
agc_driver_pipeline_journal: failed in 4 of the last 10 main runs
agc_driver_submit_progress: failed in 4 of the last 10 main runs
Failing on main too (3, not counted against this PR)
agc_driver_image_table
guest_mmap_executable
guest_socket_options_multicast
Validated but not compared (1, not in the main baseline's validation pass)
agc_driver_label_ordered_flush
Validation IDs already present on main (8 IDs in 527 tests)
VUID-VkDeviceCreateInfo-pNext-pNext (527 tests): agc_driver, agc_driver_address_passes, agc_driver_alias_copy_label, agc_driver_alpha_to_coverage, agc_driver_array_layer_target, agc_driver_bc_block_copy, agc_driver_bda_anchor_reads, agc_driver_bda_span_reads, and 519 more
VUID-vkCmdCopyImage-srcImage-01548 (7 tests): agc_driver_depth_clear_hle, agc_driver_depth_plane_buffer_memory, agc_driver_depth_plane_partial_store, agc_driver_depth_plane_sampling, agc_driver_depth_storage_dispatch, agc_driver_depth_surface_from_storage, agc_driver_dispatch_move_depth
VUID-vkCmdCopyImage-srcImage-01551 (7 tests): agc_driver_depth_clear_hle, agc_driver_depth_plane_buffer_memory, agc_driver_depth_plane_partial_store, agc_driver_depth_plane_sampling, agc_driver_depth_storage_dispatch, agc_driver_depth_surface_from_storage, agc_driver_dispatch_move_depth
VUID-VkPipelineColorBlendStateCreateInfo-pAttachments-00605 (2 tests): agc_driver_prepared_exports, agc_driver_srgb8_color_target
VUID-VkSamplerCreateInfo-addressModeU-01079 (1 test): agc_driver_image_gather_volume
VUID-VkSamplerCreateInfo-addressModeU-parameter (1 test): agc_driver_image_gather_volume
VUID-VkSamplerCreateInfo-addressModeV-parameter (1 test): agc_driver_image_gather_volume
VUID-VkSamplerCreateInfo-addressModeW-parameter (1 test): agc_driver_image_gather_volume
Skipped (50)
agc_driver_bda_reads_shadowed_results: skipped, the device reports watched host memory written when it imports or reads it
agc_driver_bda_reads_shadowed_results_per_build: skipped, the device reports watched host memory written when it imports or reads it
agc_driver_gcn_display_present: skipped, no display: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
agc_driver_sample_order_grid_reference: skipped, the device does not keep depth drawn at custom sample locations at those locations, even after a transition that names them: the composite of uniform renders holds the guest's depth in 0 of 3005 sloped samples, so no path can be compared with it bit for bit
agc_driver_scattered_runs: skipped, host imports are unavailable: /dev/udmabuf is not accessible
agc_driver_sparse_dma_buf_import: skipped, host imports are unavailable: /dev/udmabuf is not accessible
agc_driver_stencil_ref_export: skipped, the device lacks VK_EXT_shader_stencil_export
agc_driver_vertex_elected_atomic_max: skipped, the device runs vertex programs in subgroups, so they keep their cross-lane reductions
agc_host_import_sparse: skipped, host imports are unavailable: /dev/udmabuf is not accessible
agc_import_settle_initial_dcc: skipped, the device has no deferred settle of host imports: imports under 1 MiB wait for one only on Windows
agc_unit_shadow_completion_store: skipped, the device reports watched host memory written when it imports or reads it
guest_http_uri_locale: 
guest_net_ctl: skipped, the host has no IPv4 address
guest_video_out_color_space: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_hdr: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_high_frame_rate: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_output_mode: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_resolution_status: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_wqhd_detection: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
video_out_flip_aftergpu: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_backtoback: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_controls: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_cpuflip: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_cpuflip_in_order: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_dcc: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_deferred: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_eop: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
vide
Reproduce
git fetch origin 2da15973ce0a3fa5e4d9e574d6ee1df640544dad 1c8c0efc8c41314402e6966da5bef6ea3d639d10
git checkout 1c8c0efc8c41314402e6966da5bef6ea3d639d10 && git merge --no-ff 2da15973ce0a3fa5e4d9e574d6ee1df640544dad
git submodule update --init --recursive
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=ON -DANYPS5_ENABLE_SPIRV_TOOLS=ON
cmake --build build && cmake --build build --target libs
VK_DRIVER_FILES=/nonexistent/icd.json ctest --test-dir build -E '^(agc|shader|recompiler|videoout|video_out|guest_video_out|vulkan)|gpu' --timeout 120 --parallel 32 --output-on-failure
ANYPS5_REQUIRE_VULKAN=1 ANYPS5_NO_SHADER_CACHE=1 VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation \
  ctest --test-dir build -R '^(agc|shader|recompiler|videoout|video_out|guest_video_out|vulkan)|gpu' --timeout 120 --parallel 4 --output-on-failure
# failures are then re-run alone: --timeout 360 --parallel 2; new validation IDs: the test alone, twice, --parallel 1

Automated build and test on a contributor's NVIDIA machine; it does not replace upstream CI. A new comment is posted for each new push. Full logs.

@milescoviello

Copy link
Copy Markdown
Contributor Author

@TripyFox thank you, this is very thorough, and the split by half is useful too. Yes, please record the Performance-form captures on both arms (APS5_TRACE_FRAMES, no other switch). That's the one thing still blocking the perf numbers missing label.

@XairRrR here's an update on the GoT captures I promised. PPSA02225 doesn't run on main yet, even with #9742 merged locally:

  • Base main 1c8c0efc8c + fix(agc): accept a colour alias over a native multisample depth plane when HTILE proves its clear #9742 b3a54b169f, change = that + this PR. On this machine (RTX 4070 Ti, 615.71.09) the first run aborted at 149 s: compute shader 0x20001f3500: AGC graphics: native multisample depth publication over a resident colour target is not implemented.
  • The second run failed at 25 s on vkAllocateMemory sampled depth shadow: Vulkan result -2.
  • So a GoT pair on main isn't possible yet. TripyFox's ASTRO BOT A/B above (main vs main + this PR, three interleaved pairs, p50 92.7 -> 84.2 ms) is the scene that runs on both trees. Once their form captures are posted, I'll add both summary lines, the traces and the trace_digest.py output to the description.

On the "not verified" refresh case, where the skip is taken over memory outside the write watch, I traced the CPU-side reads that follow the skip:

  • StorageTexture::upload re-uploads from the host import (Texture.cpp, the HostImportFor branch).
  • For a watched surface it clears layerOriginal and reads no guest bytes.
  • For an unwatched surface it calls captureGuestBytes, which runs GuestMemory::FlushGpuWrites first. That reaches the recorder's FlushForAccess hook (Recorder.cpp, SetFlushHook(&FlushForAccess)), and the hook syncs whenever PendingWriteOverlaps covers the range. The skipped write-backs were recorded as pending writes of the open batch, so the capture reads the bytes after they land.
  • Over unwatched memory the skip therefore costs a CPU wait at that point, not a stale original. I haven't added a test for it.

AI-assisted: yes (Claude Code)

@oneandonlydean

Copy link
Copy Markdown
Collaborator

Hi milescoviello,

Thanks, the idea is sound: the label is stored on the GPU after the recorded write-back copy, so the CPU wait adds no order there. A few points before merge, reviewed at d3f7307afd:

  1. Exception safety. In Texture.cpp:2326-2333 the refresh scope restores GpuOrderedFlush and GpuOrderedSkipped only after FlushPending returns normally. That call can rethrow a writer-wait failure or a saved write-back exception (line 4755), leaving the thread-local opt-in enabled, so a later ordinary flush on that thread can skip its wait. Please restore both flags on exceptional exits too (the label scope already uses a destructor), keeping the nested skipped-result propagation, with a fault-injection case that catches a flush failure and then checks that an unrelated flush keeps its normal wait.
  2. Publish side. The wait FlushPending skips also covers publishing: with scope != PublishScope::None it waits on LiveShadowedRanges(address, bytes, scope) and PendingLeaseSerial before PublishShadow copies the unit shadows into the import. StoreReadbackBeforeLabel uses the default PublishScope::Whole, and orderedSkip only checks for host imports, so with a live shadow or lease over the range the label path now publishes without that wait. Please skip only when there is no live shadowed range or lease, or explain why PublishShadow cannot see an unfinished writer.
  3. Nested scopes. GpuOrderedFlush() = previousOrdered || orderedEligible lets a non-eligible refresh (scoped, aliased, DCC, layers pending) inherit an outer ordered scope: its flush skips the wait, but orderedSkipped stays false, so it then compares guest bytes the GPU copy has not written yet. If no refresh can run inside the scope, please say so; otherwise set the flag to orderedEligible for the refresh.
  4. Coverage. agc_driver_label_ordered_flush covers the label half only, and main's synchronous path can satisfy the same assertion. Please add tests that prove each optimized branch ran and differ from the disabled/main path: host-import eligibility, and the refresh half (a pending store over an imported image, refreshed with APS5_NO_REFRESH_GPU_ORDERED set and unset, sampled the same), plus the non-import fallback. Please also state the cost of the full re-upload for a large image that used to compare unchanged.
  5. Performance. The figures in the body come from an older-base experimental stack, so they don't bind this head. Could you attach one matched warm pair (same scene and duration) with summary JSONL and traces, full baseline/candidate revisions and stack/configuration? Keep traced diagnostic runs separate from throughput measurements.

@TechLevelZero

Copy link
Copy Markdown
Contributor

✅ AMD GPU test: passed

PR head 2da1597 merged onto main 1c8c0ef · tested 2026-11-04 03:21 UTC · 15 min

GPU AMD Radeon RX 6300 (RADV NAVI24), driver Mesa 25.2.8-0ubuntu0.24.04.4, Vulkan 1.4.318
Host Ubuntu 26.04.1 LTS, kernel 7.0.0-42-generic, container ubuntu:24.04
Logs build and test logs for this run
Build ✅ passed (Release, -DANYPS5_ENABLE_SPIRV_TOOLS=ON)
Tests ✅ 2,698 passed · 9 failed (0 new, 7 also fail on main, 2 unstable on main) · 60 skipped (ANYPS5_REQUIRE_VULKAN=1)
Test setup CPU-only tests without a GPU; GPU tests (and any CPU-only test that failed or skipped without one) once on the GPU with VK_LAYER_KHRONOS_validation and ANYPS5_NO_SHADER_CACHE=1; main ran the same way
Test selection 2,767 of 3,842 tests run: the ones this change can reach, chosen by tools/select_tests.py from main's build graph
Vulkan validation ✅ no new validation IDs vs main (945 tests run on the GPU)
Slow tests (>30 s, GPU tests timed with validation layers) ⚠️ agc_driver_pipeline_journal 120.1 s, ⚠️ public_headers 45.2 s, ⚠️ agc_shader_sampler_limits 34.3 s, ⚠️ agc_driver_image_sample_unnormalized 33.9 s
Unstable on main (2, these also failed in recent main runs on this machine; not counted against this PR)
agc_driver_pipeline_journal: failed in 8 of the last 10 main runs
agc_driver_submit_progress: failed in 8 of the last 10 main runs
Failing on main too (7, not counted against this PR)
agc_driver_image_table
agc_driver_native_depth_bda_clear
agc_driver_native_multisample_depth
agc_submit_progress
agc_submit_progress_no_timeline
guest_mmap_executable
guest_socket_options_multicast
Validated but not compared (1, not in the main baseline's validation pass)
agc_driver_label_ordered_flush
Validation IDs already present on main (8 IDs in 527 tests)
VUID-VkDeviceCreateInfo-pNext-pNext (527 tests): agc_driver, agc_driver_address_passes, agc_driver_alias_copy_label, agc_driver_alpha_to_coverage, agc_driver_array_layer_target, agc_driver_bc_block_copy, agc_driver_bda_anchor_reads, agc_driver_bda_span_reads, and 519 more
VUID-vkCmdCopyImage-srcImage-01548 (4 tests): agc_driver_depth_clear_hle, agc_driver_depth_storage_dispatch, agc_driver_depth_surface_from_storage, agc_driver_dispatch_move_depth
VUID-vkCmdCopyImage-srcImage-01551 (4 tests): agc_driver_depth_clear_hle, agc_driver_depth_storage_dispatch, agc_driver_depth_surface_from_storage, agc_driver_dispatch_move_depth
VUID-VkPipelineColorBlendStateCreateInfo-pAttachments-00605 (2 tests): agc_driver_prepared_exports, agc_driver_srgb8_color_target
VUID-VkSamplerCreateInfo-addressModeU-01079 (1 test): agc_driver_image_gather_volume
VUID-VkSamplerCreateInfo-addressModeU-parameter (1 test): agc_driver_image_gather_volume
VUID-VkSamplerCreateInfo-addressModeV-parameter (1 test): agc_driver_image_gather_volume
VUID-VkSamplerCreateInfo-addressModeW-parameter (1 test): agc_driver_image_gather_volume
Skipped (60)
agc_driver_bda_reads_shadowed_results: skipped, the device reports watched host memory written when it imports or reads it
agc_driver_bda_reads_shadowed_results_per_build: skipped, the device reports watched host memory written when it imports or reads it
agc_driver_copied_writer_label_order: skipped, the device reports a minStorageBufferOffsetAlignment of 4, so the dispatch binds its record at +0x1004 in place and nothing is written back; the test needs an alignment above 4
agc_driver_float16_unused_nan_payloads: skipped, the device has no native f16 conversions with RTE rounding and kept f16 denormals
agc_driver_gcn_display_present: skipped, no display: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
agc_driver_image_table_vouched_lookups: skipped, the device reports 4 of 4 host-imported pages written after a GPU read, so a watched imported table cannot be vouched for
agc_driver_linear_target_direct_write_completion: skipped, the device does not keep host imports write-watched, so no completion stamp over the target is tracked
agc_driver_ngg_vertex_wave_reduction: skipped, s3 describes a 32-lane wave only on 32-lane subgroups, and the device has 64 lanes
agc_driver_scattered_runs: skipped, host imports are unavailable: /dev/udmabuf is not accessible
agc_driver_sparse_dma_buf_import: skipped, host imports are unavailable: /dev/udmabuf is not accessible
agc_driver_unit_shadow: skipped, the device does not import watched host memory for unit shadows
agc_driver_unit_shadow_budget: skipped, the device does not import watched host memory for unit shadows
agc_driver_vertex_elected_atomic_max: skipped, the device runs vertex programs in subgroups, so they keep their cross-lane reductions
agc_host_import_sparse: skipped, host imports are unavailable: /dev/udmabuf is not accessible
agc_idle_publish_wait: skipped, the device reports 4 of 4 host-imported pages written after a GPU read, so a forced watch of the import reads its submissions as CPU stores
agc_import_settle_initial_dcc: skipped, the device has no deferred settle of host imports: imports under 1 MiB wait for one only on Windows
agc_shadowed_key_stores: skipped, the device imports no watched host memory under a unit shadow
agc_small_tiled_shadow_label: skipped, host imports are unavailable: the test block is not imported and watched
agc_storage_clear_hand_over: skipped, the device imports no watched host memory for the clear-key hand-over
agc_unit_shadow: skipped, the device does not import watched host memory for unit shadows
agc_unit_shadow_completion_store: skipped, the device reports watched host memory written when it imports or reads it
guest_http_uri_locale: 
guest_net_ctl: skipped, the host has no IPv4 address
guest_video_out_color_space: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_hdr: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_high_frame_rate: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_output_mode: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_resolution_status: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_wqhd_detection: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
video_out_flip_aftergpu: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_backtoback: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SD
Reproduce
git fetch origin 2da15973ce0a3fa5e4d9e574d6ee1df640544dad 1c8c0efc8c41314402e6966da5bef6ea3d639d10
git checkout 1c8c0efc8c41314402e6966da5bef6ea3d639d10 && git merge --no-ff 2da15973ce0a3fa5e4d9e574d6ee1df640544dad
git submodule update --init --recursive
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=ON -DANYPS5_ENABLE_SPIRV_TOOLS=ON
cmake --build build && cmake --build build --target libs
VK_DRIVER_FILES=/nonexistent/icd.json ctest --test-dir build -E '^(agc|shader|recompiler|videoout|video_out|guest_video_out|vulkan)|gpu' --timeout 120 --parallel 12 --output-on-failure
ANYPS5_REQUIRE_VULKAN=1 ANYPS5_NO_SHADER_CACHE=1 VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation \
  ctest --test-dir build -R '^(agc|shader|recompiler|videoout|video_out|guest_video_out|vulkan)|gpu' --timeout 120 --parallel 16 --output-on-failure
# failures are then re-run alone: --timeout 360 --parallel 2; new validation IDs: the test alone, twice, --parallel 1

Automated build and test on a contributor's AMD machine; it does not replace upstream CI. A new comment is posted for each new push. Full logs.

@TripyFox

TripyFox commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Performance-form captures on both arms, as requested. The traced runs and the throughput runs are separate launches.

Title and scene

  • PPSA21567 ASTRO BOT 01.019.000, launched with -lvl hub_crashsite (Crash Site).
  • No input, default settings, 3840x2160.
  • The trace window is presents 1200-1249. That is about 85-95 s into each run, after the level has settled.

Revisions

  • Baseline A is main 1c8c0efc8c41314402e6966da5bef6ea3d639d10, the main the AMD CI run used.
  • Candidate B is b830a33eb5e71cd0dedb0bc396ac08a9c3716677, a local git merge --no-ff of head 2da15973ce0a3fa5e4d9e574d6ee1df640544dad onto that main, the same way the CI reproduces it.
    • The parents are 1c8c0efc8c41314402e6966da5bef6ea3d639d10 and 2da15973ce0a3fa5e4d9e574d6ee1df640544dad.
    • The merge was clean and changes only Texture.cpp, LabelOrderedFlush.cpp and ENVIRONMENT.md.
    • The merge commit is local only. Its 12-character prefix b830a33eb5e7 is the commit in B's files.
  • Both trees were clean (dirty: false in every line).

Stack and configuration

  • RTX 5090 with NVIDIA 621.01, Ryzen 7 9800X3D, 64 GB, Windows 11 (10.0.26200).
  • Release build with Ninja, MinGW GCC 15.2 and BUILD_TESTING=OFF. The CMake options are identical in both trees.
  • Each arm has its own game folder with the same title files. astrobot.exe is relinked from each tree and is byte-identical across the two.
  • libSceAgcDriver.prx is the only library whose size differs.
  • Both arms used the same machine and session, interleaved under one lock: warm A, warm B, trace A, trace B, bench A, bench B.

Capture

  • Each arm first ran the scene once for 150 s with no APS5_* switch, then the title was restarted. So the shader and pipeline caches were warm.
  • Trace runs set only APS5_TRACE_FRAMES=1200:50. Bench runs set only APS5_TRACE_SUMMARY_EVERY=10 APS5_TRACE_BENCH=1.
  • No other APS5_* switch was set, the log was not redirected, and the output went to the working directory.
  • The window is 50 presents so that no thread buffer fills at the default APS5_TRACE_EVENTS. Queue worker 0x0 filled after about 70 presents in earlier captures of this scene. trace_dropped is false on both.

Traced runs (diagnostic, 50 presents each)

presents window ms p50 p90 p99 GPU busy / present queue worker 0x0 / 0x40 / 0x48 busy %
A 1c8c0efc8c41 50 4502.6 89.4 95.7 98.5 71.9 97.3 / 98.6 / 98.1
B b830a33eb5e7 50 4191.4 84.9 89.8 101.4 72.8 96.8 / 98.2 / 98.0
  • The mean frame is 90.05 -> 83.83 ms (-6.9 %).
  • A p99 over 50 frames is a single frame, so it isn't meaningful here. The throughput runs below give the p99.
  • Driver.Packet.memory_wait goes from 149.7 to 140.0 ms per frame (summed across workers), and Driver.Packet from 265.3 to 246.5.

Summary lines:

{"commit":"1c8c0efc8c41","dirty":false,"env":{"APS5_TRACE_FRAMES":"1200:50"},"gpu":"NVIDIA GeForce RTX 5090","driver":"NVIDIA 621.01","os":"Windows 10.0.26200","cpu":"AMD Ryzen 7 9800X3D 8-Core Processor","cpu_threads":16,"memory_mib":65140,"title_id":"PPSA21567","title":"ASTRO BOT","content_version":"01.019.000","resolution":"3840x2160","window":{"spec":"frames 1200:50","first_present":1200,"ms":4502.648},"presents":50,"frame_ms":{"p50":89.365,"p90":95.693,"p99":98.460,"max":98.460},"gpu_busy_ms_per_present":71.899,"worker_busy_pct":{"queue worker 0x0":97.293,"queue worker 0x40":98.621,"queue worker 0x48":98.148,"queue worker 0x46":1.588},"top_phases_ms_per_frame":{"Driver.Packet":265.310,"Driver.Packet.memory_wait":149.718,"Driver.Packet.draw":58.025,"Driver.Draw":57.632,"Graphics.Draw":45.145,"Graphics.Draw.draw_and_resource_release":36.134,"Driver.Draw.draw_enqueued":26.902,"Driver.Draw.shader_compile_and_link":24.790},"trace_dropped":false,"trace":"aps5-trace-1c8c0efc8c41-26308.json.gz"}
{"commit":"b830a33eb5e7","dirty":false,"env":{"APS5_TRACE_FRAMES":"1200:50"},"gpu":"NVIDIA GeForce RTX 5090","driver":"NVIDIA 621.01","os":"Windows 10.0.26200","cpu":"AMD Ryzen 7 9800X3D 8-Core Processor","cpu_threads":16,"memory_mib":65140,"title_id":"PPSA21567","title":"ASTRO BOT","content_version":"01.019.000","resolution":"3840x2160","window":{"spec":"frames 1200:50","first_present":1200,"ms":4191.368},"presents":50,"frame_ms":{"p50":84.941,"p90":89.842,"p99":101.363,"max":101.363},"gpu_busy_ms_per_present":72.803,"worker_busy_pct":{"queue worker 0x0":96.799,"queue worker 0x40":98.180,"queue worker 0x48":98.010,"queue worker 0x46":2.133},"top_phases_ms_per_frame":{"Driver.Packet":246.538,"Driver.Packet.memory_wait":139.971,"Driver.Packet.draw":52.793,"Driver.Draw":52.400,"Graphics.Draw":46.792,"Graphics.Draw.draw_and_resource_release":37.914,"Driver.Packet.prepare_memory":25.616,"Driver.Draw.shader_compile_and_link":25.597},"trace_dropped":false,"trace":"aps5-trace-b830a33eb5e7-28056.json.gz"}

Traces: aps5-trace-1c8c0efc8c41-26308.json.gz (A) and aps5-trace-b830a33eb5e7-28056.json.gz (B); they open in ui.perfetto.dev. The python tools/trace_digest.py <trace> output for each is digest-traceA.txt and digest-traceB.txt. I can share the traces, digests and full summary JSONL if you want them.

Throughput runs (untraced, 240 s each)

APS5_TRACE_SUMMARY_EVERY=10 APS5_TRACE_BENCH=1 only. The steady window is intervals 8-22 (70-221 s, 15 intervals per run). Each value is the median over that window.

presents / 10 s p50 p90 p99 GPU busy / present busiest worker %
A 1c8c0efc8c41 109 91.6 96.6 110.1 72.7 100.0
B b830a33eb5e7 120 84.5 89.0 91.1 73.0 100.0

tools/performance_compare.py on matched intervals, B minus A:

interval p50 p99 mean frame GPU busy / present
#8 91.57 -> 84.00 (-7.6) 112.31 -> 90.41 (-21.9) -8.4 +0.1
#12 91.63 -> 84.87 (-6.8) 105.64 -> 90.48 (-15.2) -7.6 +1.1
#16 92.16 -> 85.17 (-7.0) 110.01 -> 92.43 (-17.6) -7.5 +0.3
#20 91.50 -> 84.25 (-7.2) 110.52 -> 91.08 (-19.4) -8.2 -0.5
  • This matches my earlier interleaved runs on 086c0686c8 (p50 92.7 -> 84.2 ms). The gain is on the CPU side: GPU busy per present is unchanged within noise, and the busiest queue worker is at 100 % in both arms.
  • The full interval lines (aps5-summary.jsonl per arm) and the bench reports aps5-bench-1c8c0efc8c41-16404.json.gz and aps5-bench-b830a33eb5e7-21620.json.gz are available too.

None of the six runs (two warm-up runs, two traced, two throughput) crashed or ended early. Each ran to its time cap.

AI-assisted: yes (Claude Code)

…without a live shadow or lease; cover each branch

Review of 2da1597 (oneandonlydean):
- The refresh's ordered scope is a destructor-restored guard, so a FlushPending that throws leaves no armed flag on its thread, and a refresh sets the flag to its own eligibility instead of inheriting an outer label scope.
- FlushPending takes the ordered skip only when its publish scope is None or no live shadowed range or pending lease covers the range, since that wait also orders PublishShadow.
- StorageTexture::TakeOrderedSkipCounts counts the skips taken. agc_driver_label_ordered_flush checks the label half and a storage refresh of a linear view over a pending tiled store, and its _disabled and _unimported runs check that neither skip is taken with the switches set or without a host import, with the same results.
- agc_driver_ordered_refresh_failure injects a flush failure inside an armed refresh (StorageTexture::SetFlushFailureForTest) and checks that the flag is restored and that the next flush waits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@milescoviello

milescoviello commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor Author

@oneandonlydean thank you, all five points hold. Head b6007f69a9 addresses 1-4:

  1. Exception safety.
    • The refresh's ordered scope is now a guard object whose destructor restores GpuOrderedFlush and propagates GpuOrderedSkipped (previous || skipped), the same way the label scope does.
    • New agc_driver_ordered_refresh_failure:
      • It loads a linear storage view over a pending tiled store with a FlushPending failure injected (StorageTexture::SetFlushFailureForTest), and requires that the failure fired inside an armed ordered scope (TakeFlushFailureForTest returns 2). The caller may catch that exception (the sampled-texture path does), so the test doesn't depend on it reaching the test.
      • It then checks that the flag is clear on the thread (OrderedFlushArmedForTest), that the pending image survived, and that an unrelated FlushPending afterwards takes the normal wait: no ordered skip is counted, and guest memory holds the store after the wait.
  2. Publish side. The skip is taken only when the publish scope is None or when LiveShadowedRanges(address, bytes, scope) is empty and PendingLeaseSerial over the same 64 KiB-rounded range is 0. Otherwise the existing wait runs.
  3. Nested scopes. A refresh now sets the flag to its own orderedEligible instead of inheriting an outer label scope.
  4. Coverage.
    • StorageTexture::TakeOrderedSkipCounts() counts the flush and refresh skips taken.
    • agc_driver_label_ordered_flush now checks both halves:
      • The label half asserts at least one flush skip.
      • The refresh half asserts at least one refresh skip. It reads a 256x64 linear R32UI view of a 64 KiB host-imported block after a 128x128 tiled store over the whole block is left pending. The linear view has a different shape, so it isn't an alias and its refresh flushes the store. The test requires every texel read to equal both the stored value and guest memory.
    • Two more registrations run the same scenarios:
      • _disabled (APS5_NO_FLUSH_GPU_ORDERED=1 APS5_NO_REFRESH_GPU_ORDERED=1) requires zero skips with the same results.
      • _unimported (APS5_NO_HOST_IMPORT=1, the non-import fallback) also requires zero skips with the same results.
    • Full re-upload cost: an ordered-skipped refresh marks every layer changed and re-uploads the whole image from its host import as a GPU copy behind the write-back, instead of comparing guest bytes against original. On this machine a vkCmdCopyBuffer of 32 MiB from a host-imported buffer into device-local memory takes 1.8-2.1 ms (16.2-18.6 GB/s over 8 copies from distinct slices of a 256 MiB import, so nothing is served from L2). So a 3840x2160 RGBA8 surface (31.6 MiB) that would have compared unchanged now costs about 1.9 ms of GPU copy. In exchange there's no CPU wait for its writer; in GoT that wait was about 55 ms per frame. On ASTRO BOT, TripyFox measured the same storage upload volume with and without the PR (262 vs 263 MiB per present).
  5. Performance. TripyFox's ASTRO BOT pair above is main 086c0686c8 vs that main + this PR, with the same scene, duration and configuration. They've offered the Performance-form captures (summary JSONL and traces, no other switch) on both arms. I'll put those in the description as the binding numbers, kept apart from their counter runs.

Focused tests on the new head (Linux, RTX 4070 Ti, 615.71.09): all 9 pass in each of 3 runs: agc_driver_label_ordered_flush and its _disabled and _unimported registrations, agc_driver_ordered_refresh_failure, and the five completion and fill tests from the description. A Windows VM run on the new head is queued.

AI-assisted: yes (Claude Code)

@github-actions github-actions Bot added the build label Nov 4, 2026
@TechLevelZero

Copy link
Copy Markdown
Contributor

✅ NVIDIA GPU test: passed

PR head b6007f6 merged onto main 5b1d381 · tested 2026-11-04 10:10 UTC · 17 min

GPU NVIDIA RTX 2000 Ada Generation, driver 595.99.02, Vulkan 1.4.329
Host Ubuntu 26.04.1 LTS, kernel 7.0.0-38-generic, container ubuntu:24.04
Logs build and test logs for this run
Build ✅ passed (Release, -DANYPS5_ENABLE_SPIRV_TOOLS=ON)
Tests ✅ 2,837 passed · 4 failed (0 new, 3 also fail on main, 1 unstable on main) · 49 skipped (ANYPS5_REQUIRE_VULKAN=1)
Test setup CPU-only tests without a GPU; GPU tests (and any CPU-only test that failed or skipped without one) once on the GPU with VK_LAYER_KHRONOS_validation and ANYPS5_NO_SHADER_CACHE=1; main ran the same way
Test selection 2,890 of 3,883 tests run: the ones this change can reach, chosen by tools/select_tests.py from main's build graph
Vulkan validation ✅ no new validation IDs vs main (978 tests run on the GPU)
Slow tests (>30 s, GPU tests timed with validation layers) ⚠️ agc_driver_pipeline_journal 120.1 s
Unstable on main (1, these also failed in recent main runs on this machine; not counted against this PR)
agc_driver_pipeline_journal: failed in 3 of the last 10 main runs
Failing on main too (3, not counted against this PR)
agc_driver_image_table
guest_mmap_executable
guest_socket_options_multicast
Validated but not compared (2, not in the main baseline's validation pass)
agc_driver_label_ordered_flush
agc_driver_ordered_refresh_failure
Validation IDs already present on main (13 IDs in 542 tests)
VUID-VkDeviceCreateInfo-pNext-pNext (542 tests): agc_driver, agc_driver_address_passes, agc_driver_alias_copy_label, agc_driver_alpha_to_coverage, agc_driver_array_layer_target, agc_driver_bc_block_copy, agc_driver_bda_anchor_reads, agc_driver_bda_span_reads, and 534 more
VUID-VkShaderModuleCreateInfo-pCode-08737 (8 tests): agc_driver_dpp16_operands, agc_driver_dpp8_pk_fmac, agc_driver_dx10_clamp, agc_driver_float16_fma_rounding, agc_driver_float16_unused_nan_payloads, agc_driver_packed_alu, agc_driver_packed_f16_denormal_modes, agc_driver_packed_float16_native
VUID-VkShaderModuleCreateInfo-pCode-08739 (8 tests): agc_driver_dpp16_operands, agc_driver_dpp8_pk_fmac, agc_driver_dx10_clamp, agc_driver_float16_fma_rounding, agc_driver_float16_unused_nan_payloads, agc_driver_packed_alu, agc_driver_packed_f16_denormal_modes, agc_driver_packed_float16_native
VUID-VkShaderModuleCreateInfo-pCode-08741 (8 tests): agc_driver_dpp16_operands, agc_driver_dpp8_pk_fmac, agc_driver_dx10_clamp, agc_driver_float16_fma_rounding, agc_driver_float16_unused_nan_payloads, agc_driver_packed_alu, agc_driver_packed_f16_denormal_modes, agc_driver_packed_float16_native
VUID-vkCmdCopyImage-srcImage-01548 (7 tests): agc_driver_depth_clear_hle, agc_driver_depth_plane_buffer_memory, agc_driver_depth_plane_partial_store, agc_driver_depth_plane_sampling, agc_driver_depth_storage_dispatch, agc_driver_depth_surface_from_storage, agc_driver_dispatch_move_depth
VUID-vkCmdCopyImage-srcImage-01551 (7 tests): agc_driver_depth_clear_hle, agc_driver_depth_plane_buffer_memory, agc_driver_depth_plane_partial_store, agc_driver_depth_plane_sampling, agc_driver_depth_storage_dispatch, agc_driver_depth_surface_from_storage, agc_driver_dispatch_move_depth
VUID-VkPipelineColorBlendStateCreateInfo-pAttachments-00605 (2 tests): agc_driver_prepared_exports, agc_driver_srgb8_color_target
VUID-VkPipelineShaderStageCreateInfo-module-parameter (1 test): agc_driver_float16_unused_nan_payloads
VUID-VkPipelineShaderStageCreateInfo-pSpecializationInfo-06849 (1 test): agc_driver_float16_unused_nan_payloads
VUID-VkSamplerCreateInfo-addressModeU-01079 (1 test): agc_driver_image_gather_volume
VUID-VkSamplerCreateInfo-addressModeU-parameter (1 test): agc_driver_image_gather_volume
VUID-VkSamplerCreateInfo-addressModeV-parameter (1 test): agc_driver_image_gather_volume
VUID-VkSamplerCreateInfo-addressModeW-parameter (1 test): agc_driver_image_gather_volume
Skipped (49)
agc_driver_bda_reads_shadowed_results: skipped, the device reports watched host memory written when it imports or reads it
agc_driver_bda_reads_shadowed_results_per_build: skipped, the device reports watched host memory written when it imports or reads it
agc_driver_gcn_display_present: skipped, no display: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
agc_driver_sample_order_grid_reference: skipped, the device does not keep depth drawn at custom sample locations at those locations, even after a transition that names them: the composite of uniform renders holds the guest's depth in 0 of 3005 sloped samples, so no path can be compared with it bit for bit
agc_driver_scattered_runs: skipped, host imports are unavailable: /dev/udmabuf is not accessible
agc_driver_sparse_dma_buf_import: skipped, host imports are unavailable: /dev/udmabuf is not accessible
agc_driver_stencil_ref_export: skipped, the device lacks VK_EXT_shader_stencil_export
agc_host_import_sparse: skipped, host imports are unavailable: /dev/udmabuf is not accessible
agc_import_settle_initial_dcc: skipped, the device has no deferred settle of host imports: imports under 1 MiB wait for one only on Windows
agc_unit_shadow_completion_store: skipped, the device reports watched host memory written when it imports or reads it
guest_http_uri_locale: 
guest_net_ctl: skipped, the host has no IPv4 address
guest_video_out_color_space: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_hdr: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_high_frame_rate: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_output_mode: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_resolution_status: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_wqhd_detection: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
video_out_flip_aftergpu: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_backtoback: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_controls: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_cpuflip: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_cpuflip_in_order: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_dcc: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_deferred: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_eop: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_eop_aftergpu: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not av
Reproduce
git fetch origin b6007f69a9851871f24fd9d222220509562ef46e 5b1d381d5ad5d9f923326b18479c3bd889cdd588
git checkout 5b1d381d5ad5d9f923326b18479c3bd889cdd588 && git merge --no-ff b6007f69a9851871f24fd9d222220509562ef46e
git submodule update --init --recursive
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=ON -DANYPS5_ENABLE_SPIRV_TOOLS=ON
cmake --build build && cmake --build build --target libs
VK_DRIVER_FILES=/nonexistent/icd.json ctest --test-dir build -E '^(agc|shader|recompiler|videoout|video_out|guest_video_out|vulkan)|gpu' --timeout 120 --parallel 32 --output-on-failure
ANYPS5_REQUIRE_VULKAN=1 ANYPS5_NO_SHADER_CACHE=1 VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation \
  ctest --test-dir build -R '^(agc|shader|recompiler|videoout|video_out|guest_video_out|vulkan)|gpu' --timeout 120 --parallel 4 --output-on-failure
# failures are then re-run alone: --timeout 360 --parallel 2; new validation IDs: the test alone, twice, --parallel 1

Automated build and test on a contributor's NVIDIA machine; it does not replace upstream CI. A new comment is posted for each new push. Full logs.

@TechLevelZero

Copy link
Copy Markdown
Contributor

✅ AMD GPU test: passed

PR head b6007f6 merged onto main 5b1d381 · tested 2026-11-04 10:19 UTC · 37 min

GPU AMD Radeon RX 6300 (RADV NAVI24), driver Mesa 25.2.8-0ubuntu0.24.04.4, Vulkan 1.4.318
Host Ubuntu 26.04.1 LTS, kernel 7.0.0-42-generic, container ubuntu:24.04
Logs build and test logs for this run
Build ✅ passed (Release, -DANYPS5_ENABLE_SPIRV_TOOLS=ON)
Tests ✅ 2,822 passed · 8 failed (0 new, 7 also fail on main, 1 unstable on main) · 60 skipped (ANYPS5_REQUIRE_VULKAN=1)
Test setup CPU-only tests without a GPU; GPU tests (and any CPU-only test that failed or skipped without one) once on the GPU with VK_LAYER_KHRONOS_validation and ANYPS5_NO_SHADER_CACHE=1; main ran the same way
Test selection 2,890 of 3,883 tests run: the ones this change can reach, chosen by tools/select_tests.py from main's build graph
Vulkan validation ✅ no new validation IDs vs main (978 tests run on the GPU)
Slow tests (>30 s, GPU tests timed with validation layers) ⚠️ agc_driver_pipeline_journal 121.9 s, ⚠️ public_headers 69.2 s, ⚠️ agc_driver_image_sample_unnormalized 57.7 s, ⚠️ agc_driver_lazy_resolved_abi 47.8 s, ⚠️ agc_driver_sample_multisample_target 45.7 s
Unstable on main (1, these also failed in recent main runs on this machine; not counted against this PR)
agc_driver_pipeline_journal: failed in 8 of the last 10 main runs
Failing on main too (7, not counted against this PR)
agc_driver_image_table
agc_driver_native_depth_bda_clear
agc_driver_native_multisample_depth
agc_submit_progress
agc_submit_progress_no_timeline
guest_mmap_executable
guest_socket_options_multicast
Passed on retry (1, failed in the parallel run, passed when re-run with a 360 s timeout; not counted against this PR)
agc_driver_recorder: Timeout in the parallel run; also passed only on retry in 17 other runs in the last 7 days
Validated but not compared (2, not in the main baseline's validation pass)
agc_driver_label_ordered_flush
agc_driver_ordered_refresh_failure
Validation IDs already present on main (8 IDs in 542 tests)
VUID-VkDeviceCreateInfo-pNext-pNext (542 tests): agc_driver, agc_driver_address_passes, agc_driver_alias_copy_label, agc_driver_alpha_to_coverage, agc_driver_array_layer_target, agc_driver_bc_block_copy, agc_driver_bda_anchor_reads, agc_driver_bda_span_reads, and 534 more
VUID-vkCmdCopyImage-srcImage-01548 (4 tests): agc_driver_depth_clear_hle, agc_driver_depth_storage_dispatch, agc_driver_depth_surface_from_storage, agc_driver_dispatch_move_depth
VUID-vkCmdCopyImage-srcImage-01551 (4 tests): agc_driver_depth_clear_hle, agc_driver_depth_storage_dispatch, agc_driver_depth_surface_from_storage, agc_driver_dispatch_move_depth
VUID-VkPipelineColorBlendStateCreateInfo-pAttachments-00605 (2 tests): agc_driver_prepared_exports, agc_driver_srgb8_color_target
VUID-VkSamplerCreateInfo-addressModeU-01079 (1 test): agc_driver_image_gather_volume
VUID-VkSamplerCreateInfo-addressModeU-parameter (1 test): agc_driver_image_gather_volume
VUID-VkSamplerCreateInfo-addressModeV-parameter (1 test): agc_driver_image_gather_volume
VUID-VkSamplerCreateInfo-addressModeW-parameter (1 test): agc_driver_image_gather_volume
Skipped (60)
agc_driver_bda_reads_shadowed_results: skipped, the device reports watched host memory written when it imports or reads it
agc_driver_bda_reads_shadowed_results_per_build: skipped, the device reports watched host memory written when it imports or reads it
agc_driver_copied_writer_label_order: skipped, the device reports a minStorageBufferOffsetAlignment of 4, so the dispatch binds its record at +0x1004 in place and nothing is written back; the test needs an alignment above 4
agc_driver_float16_unused_nan_payloads: skipped, the device has no native f16 conversions with RTE rounding and kept f16 denormals
agc_driver_gcn_display_present: skipped, no display: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
agc_driver_image_table_vouched_lookups: skipped, the device reports 4 of 4 host-imported pages written after a GPU read, so a watched imported table cannot be vouched for
agc_driver_linear_target_direct_write_completion: skipped, the device does not keep host imports write-watched, so no completion stamp over the target is tracked
agc_driver_ngg_vertex_wave_reduction: skipped, s3 describes a 32-lane wave only on 32-lane subgroups, and the device has 64 lanes
agc_driver_scattered_runs: skipped, host imports are unavailable: /dev/udmabuf is not accessible
agc_driver_sparse_dma_buf_import: skipped, host imports are unavailable: /dev/udmabuf is not accessible
agc_driver_unit_shadow: skipped, the device does not import watched host memory for unit shadows
agc_driver_unit_shadow_budget: skipped, the device does not import watched host memory for unit shadows
agc_host_import_sparse: skipped, host imports are unavailable: /dev/udmabuf is not accessible
agc_idle_publish_wait: skipped, the device reports 4 of 4 host-imported pages written after a GPU read, so a forced watch of the import reads its submissions as CPU stores
agc_import_settle_initial_dcc: skipped, the device has no deferred settle of host imports: imports under 1 MiB wait for one only on Windows
agc_shadowed_key_stores: skipped, the device imports no watched host memory under a unit shadow
agc_small_tiled_shadow_label: skipped, host imports are unavailable: the test block is not imported and watched
agc_storage_clear_hand_over: skipped, the device imports no watched host memory for the clear-key hand-over
agc_storage_pending_key_refresh: skipped, refreshes over a pending key store are not testable here
agc_unit_shadow: skipped, the device does not import watched host memory for unit shadows
agc_unit_shadow_completion_store: skipped, the device reports watched host memory written when it imports or reads it
guest_http_uri_locale: 
guest_net_ctl: skipped, the host has no IPv4 address
guest_video_out_color_space: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_hdr: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_high_frame_rate: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_output_mode: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_resolution_status: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_wqhd_detection: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
video_out_flip_aftergpu: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_backtoback: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platf
Reproduce
git fetch origin b6007f69a9851871f24fd9d222220509562ef46e 5b1d381d5ad5d9f923326b18479c3bd889cdd588
git checkout 5b1d381d5ad5d9f923326b18479c3bd889cdd588 && git merge --no-ff b6007f69a9851871f24fd9d222220509562ef46e
git submodule update --init --recursive
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=ON -DANYPS5_ENABLE_SPIRV_TOOLS=ON
cmake --build build && cmake --build build --target libs
VK_DRIVER_FILES=/nonexistent/icd.json ctest --test-dir build -E '^(agc|shader|recompiler|videoout|video_out|guest_video_out|vulkan)|gpu' --timeout 120 --parallel 12 --output-on-failure
ANYPS5_REQUIRE_VULKAN=1 ANYPS5_NO_SHADER_CACHE=1 VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation \
  ctest --test-dir build -R '^(agc|shader|recompiler|videoout|video_out|guest_video_out|vulkan)|gpu' --timeout 120 --parallel 16 --output-on-failure
# failures are then re-run alone: --timeout 360 --parallel 2; new validation IDs: the test alone, twice, --parallel 1

Automated build and test on a contributor's AMD machine; it does not replace upstream CI. A new comment is posted for each new push. Full logs.

@TripyFox

TripyFox commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

Main-vs-change data for the current head b6007f6, ASTRO BOT (PPSA21567 01.019.000) at Crash Site (-lvl hub_crashsite), as the perf rules ask.

  • Machine: RTX 5090 (driver 621.01), Ryzen 7 9800X3D, Windows 11 (10.0.26200), 3840x2160.
  • Baseline: main b11c7bf84f. Change: PR head b6007f6 merged onto that main with --no-ff (local merge commit 78693efa462b, clean merge). Only libSceAgcDriver.prx differs between the two builds.
  • Method: Release builds, APS5_TRACE_SUMMARY_EVERY=10 APS5_TRACE_BENCH=1, no pad input, 240 s per run with a warm shader cache. Two interleaved rounds (main, change, …, change, main). The numbers are medians of the 19 ten-second intervals between 40 s and 240 s in each run.
Run presents / 10 s frame p50 p99 GPU busy / present
main, round 1 110.8 90.01 108.88 70.38
change, round 1 122.7 82.79 92.06 70.45
main, round 2 110.5 91.06 104.54 72.81
change, round 2 120.5 84.39 90.58 72.73

The change lowers frame p50 by 7.2 ms in round 1 and by 6.7 ms in round 2, and lowers p99 by 14–17 ms. GPU busy per present does not change (±0.1 ms), so the gain is on the CPU side. Averaged over both rounds, the largest phase changes are Driver.Packet −24.8 ms/frame, Driver.Packet.memory_wait −11.4, and Driver.Draw / Driver.Packet.draw −7.3. Phase times are summed across threads. Neither build logged a crash, device loss or assert.

Summary lines (interval starting at 100 s, round 1)

main:

{"commit":"b11c7bf84f5d","dirty":false,"env":{"APS5_NO_PAD_INPUT":"1","APS5_TRACE_BENCH":"1","APS5_TRACE_DIR":"<path>","APS5_TRACE_SUMMARY_EVERY":"10"},"gpu":"NVIDIA GeForce RTX 5090","driver":"NVIDIA 621.01","os":"Windows 10.0.26200","cpu":"AMD Ryzen 7 9800X3D 8-Core Processor","cpu_threads":16,"memory_mib":65140,"title_id":"PPSA21567","title":"ASTRO BOT","content_version":"01.019.000","resolution":"3840x2160","window":{"spec":"summary_every 10","first_present":1641,"ms":10040.788},"interval_start_s":100.904,"interval_end_s":110.945,"presents":110,"frame_ms":{"p50":90.588,"p90":100.512,"p99":114.130,"max":118.423},"gpu_busy_ms_per_present":70.925,"worker_busy_pct":{"queue worker 0x0":97.787,"queue worker 0x40":99.954,"queue worker 0x48":100.058,"queue worker 0x46":2.206},"top_phases_ms_per_frame":{"Driver.Packet":272.861,"Driver.Packet.memory_wait":159.319,"Driver.Packet.draw":55.195,"Driver.Draw":54.820,"Graphics.Draw":46.451,"Graphics.Draw.draw_and_resource_release":37.184,"Driver.Draw.shader_compile_and_link":29.343,"Shader.DrawStage":24.561},"trace_dropped":false,"trace":""}

change:

{"commit":"78693efa462b","dirty":false,"env":{"APS5_NO_PAD_INPUT":"1","APS5_TRACE_BENCH":"1","APS5_TRACE_DIR":"<path>","APS5_TRACE_SUMMARY_EVERY":"10"},"gpu":"NVIDIA GeForce RTX 5090","driver":"NVIDIA 621.01","os":"Windows 10.0.26200","cpu":"AMD Ryzen 7 9800X3D 8-Core Processor","cpu_threads":16,"memory_mib":65140,"title_id":"PPSA21567","title":"ASTRO BOT","content_version":"01.019.000","resolution":"3840x2160","window":{"spec":"summary_every 10","first_present":1452,"ms":10065.337},"interval_start_s":100.957,"interval_end_s":111.022,"presents":124,"frame_ms":{"p50":82.300,"p90":86.826,"p99":91.225,"max":94.329},"gpu_busy_ms_per_present":70.315,"worker_busy_pct":{"queue worker 0x0":97.584,"queue worker 0x40":99.964,"queue worker 0x48":99.992,"queue worker 0x46":1.450},"top_phases_ms_per_frame":{"Driver.Packet":241.869,"Driver.Packet.memory_wait":142.862,"Graphics.Draw":47.769,"Driver.Packet.draw":47.312,"Driver.Draw":46.912,"Graphics.Draw.draw_and_resource_release":38.622,"Driver.Draw.shader_compile_and_link":26.900,"Driver.Packet.prepare_memory":22.820},"trace_dropped":false,"trace":""}

tools/performance_compare.py on these two lines reports frame p50 −8.3 ms (−9.1 %), p99 −22.9 ms, GPU busy −0.6 ms. The only configuration difference it lists is window.first_present.

AI-assisted: yes (Claude Code)

@milescoviello

milescoviello commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor Author

@TripyFox thank you. I've put both summary lines, the table and the performance_compare.py result in the description's Tested section, credited to you.

@oneandonlydean @XairRrR the description now has the two summary lines from the same machine and scene: main b11c7bf84f vs this head merged onto it, both from the interval starting at 100 s. p50 drops 6.7-7.2 ms per round and p99 14-17 ms, with GPU busy unchanged. These are bench-summary runs, so there are no .json.gz traces. If you want the traces too, I'll ask for a matched APS5_TRACE_FRAMES pair.

AI-assisted: yes (Claude Code)

@LeoFeitosa

Copy link
Copy Markdown
Contributor

Another ASTRO BOT data point, in a heavier scene than Crash Site: the opening desert level, where on main the label readback store's GPU wait shows up on the draw committer. Same direction as @TripyFox's numbers.

Setup. PPSA21564 (ASTRO BOT 01.007.000), opening level from a new game, scripted pad input (APS5_PAD_INPUT_FILE), 3840x2160. Windows 11 10.0.26300, RTX 5070 Ti (NVIDIA 621.01), Ryzen 9 7900X3D, WinLibs GCC 15.2.0 Release. main b11c7bf against b6007f6 merged onto it with --no-ff (clean merge); only libSceAgcDriver.prx and the relinked executable differ. Four runs, main/PR/PR/main, 20 s trace window at 1080 s (APS5_TRACE_SECONDS=1080:20), no APS5_PROFILE_*, no other driver switch.

run presents / 20 s frame p50 p90 p99 GPU busy per present FPS, minutes 20-30
main 1 245 81.9 84.8 87.0 61.3 13.7
change 1 259 75.3 84.7 114.0 67.4 14.9
change 2 252 78.7 85.8 93.1 70.9 14.8
main 2 238 84.1 86.8 91.9 64.2 13.5

(ms) Frame p50 83.0 -> 77.0 ms (-7 %), FPS over minutes 20-30 13.6 -> 14.9 (+9 %), and GPU busy per present goes up (62.8 -> 69.2 ms), so the GPU is fed more. The p99 of the first change run comes from one stretch (max 205.5 ms); the second change run has p99 93.1 ms. None of the four runs logged a FATAL or a device loss.

Why it shows here: a sampled run on main b11c7bf in the same scene (APS5_SAMPLE_COMMITTER, APS5_DRAW_PIPELINE=1) had about 14 % of the committer's samples in vkWaitForFences under Driver::commitLabel > VulkanDevice::WriteLabelOnGpu > StorageTexture::StoreReadbackBeforeLabel > FlushPending > Recorder::FinishUpTo / Recorder::Sync > Recorder::finish, under the GPU mutex, while queue worker 0 spent 22.6 % in DrawPipeline::Enqueue waiting for room.

@dantdmdarealguy

Copy link
Copy Markdown
Contributor

Cross-test on GTA V (PPSA04264 01.005.000), North Yankton prologue, Windows 11 26300, RTX 5080 (NVIDIA 621.01), 3840x2160, warm caches. main 0540ef0 merged with b6007f6 against plain main; same executable, APS5_TRACE_SECONDS=300:120.

  • Frame: 22.3 presents per 10 s, p50 438 ms, against main's 22.2-24.5 over 4 runs, so no change on this scene.
  • Why: steady-state APS5_SAMPLE_COMMITTER with this PR still puts 2185 of 2365 FlushPending → FinishUpTo wait samples under StoreReadbackBeforeLabel, so GTA's label-time stores don't take the ordered skip. [storage] reports about 430 "readback before label" write-backs per 10 s here, with live unit shadows (shadow: retiled ~1000, detiled from shadow ~800 per 10 s). That suggests the publishesNothing (live shadow or lease) or host-import condition refuses them.
  • No regression: no fatal error and no visible difference in the scene.

Traces available on request.

@milescoviello

Copy link
Copy Markdown
Contributor Author

@LeoFeitosa thanks for the desert-level runs. Frame p50 83.0 -> 77.0 ms with more GPU busy per present is the direction the committer samples pointed to.

@dantdmdarealguy thanks for the GTA V cross-test. That matches what the head does: the skip is refused when a live unit shadow covers the flushed range (LiveShadowedRanges in publishesNothing), because the shadow publish would then race the recorded copy. With ~800 detiles from shadow per 10 s in that scene, GTA's label-time stores keep the wait, so no change there is the expected result. Ordering the shadow publish behind the writer too would be a separate change; I'll keep it out of this PR.

AI-assisted: yes (Claude Code)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agc libSceAgc* build docs perf Performance change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants