Skip to content

perf(agc): run the draw finisher by default - #15456

Open
Nesyth wants to merge 5 commits into
boykopovar:mainfrom
Nesyth:perf/agc-draw-finisher-default
Open

Nesyth wants to merge 5 commits into
boykopovar:mainfrom
Nesyth:perf/agc-draw-finisher-default

Conversation

@Nesyth

@Nesyth Nesyth commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

What

On main, queue 0's worker and the draw committer bound draw-heavy scenes (#7895). The draw finisher from #6946 moves the second half of each pipelined draw off the worker: stage materialization, the draw cache insert and the hand-off to the committer. It also lets the committer keep resource proofs for the collect epoch. It has been opt-in behind APS5_DRAW_PIPELINE since 17 October. With #15455, a draw finished there recaptures changed memory and skips refused draws as the worker does. This PR turns the finisher on by default, and APS5_NO_DRAW_PIPELINE turns it off again. In Grand Theft Auto V (PPSA04264) the frame p50 drops from 73.7 to 63.3 ms.

Tested

Windows 11 Pro 26300, Radeon RX 7900 XTX (AMD Software 26.10.1), Ryzen 7 7800X3D, Release, MinGW-w64 GCC 15.2.

  • On main 086c068 with this branch, the suite of 3971 tests (1352 ^agc_) fails the same six tests as main: agc_driver_direct_write_completion, agc_driver_sample_order_grid_reference, and four tool tests that need symlink rights or a real python.exe on this machine.
  • The focused draw, pipeline and finisher tests pass with the validation layer on both arms. That is 91 tests on main and 116 on this branch: the same tests plus the _no_draw_pipeline variants and the new shutdown and queued-dump tests. Both arms report only Undefined-Value-ShaderOutputNotConsumed warnings, no VUID and no missing layer.
  • APS5_VERIFY_DRAW_PIPELINE=1 in Grand Theft Auto V checked 16,975 reused epoch proofs against the full check: 0 failed.

Grand Theft Auto V, Performance graphics mode, gameplay of the opening level, same machine, warm caches, no profiling:

frame p50 / p90 presents in 30 s FPS over the run (114 s, 57 s)
main 9090972b7e 73.7 / 77.0 ms 408 13.98, 14.46
this PR at 9b8f0d7, before the rebase 63.3 / 66.1 ms 471 15.98, 17.02
{"commit":"9090972b7e91","dirty":false,"env":{"APS5_LOG_TIMESTAMPS":"1","APS5_NO_PAD_INPUT":"1","APS5_PAD_INPUT_FILE":"<path>","APS5_TRACE_DIR":"<path>","APS5_TRACE_SECONDS":"150:30"},"gpu":"AMD Radeon RX 7900 XTX","driver":"AMD proprietary driver 26.10.1 (LLPC)","os":"Windows 10.0.26300","cpu":"AMD Ryzen 7 7800X3D 8-Core Processor","cpu_threads":16,"memory_mib":31860,"title_id":"PPSA04264","title":"Grand Theft Auto V","content_version":"01.011.001","resolution":"3840x2160","window":{"spec":"seconds 150:30","first_present":3874,"ms":30033.654},"presents":408,"frame_ms":{"p50":73.666,"p90":76.967,"p99":79.540,"max":81.630},"gpu_busy_ms_per_present":31.673,"worker_busy_pct":{"queue worker 0x0":98.160},"top_phases_ms_per_frame":{"Driver.Packet":68.959,"Graphics.Draw":57.773,"Driver.Packet.draw":54.694,"Driver.Draw":54.380,"Graphics.Draw.draw_and_resource_release":53.413,"Driver.Draw.draw_enqueued":32.593,"Driver.Draw.shader_compile_and_link":16.823,"Shader.DrawStage":12.177},"trace_dropped":true,"trace_dropped_threads":{"queue worker 0x0":{"events_full_after_s":8.338,"events_lost":683195,"totals_full":false},"draw committer":{"events_full_after_s":17.189,"events_lost":195932,"totals_full":false}},"trace":"aps5-trace-9090972b7e91-29412.json.gz"}
{"commit":"9b8f0d7e3298","dirty":false,"env":{"APS5_LOG_TIMESTAMPS":"1","APS5_NO_PAD_INPUT":"1","APS5_PAD_INPUT_FILE":"<path>","APS5_TRACE_DIR":"<path>","APS5_TRACE_SECONDS":"150:30"},"gpu":"AMD Radeon RX 7900 XTX","driver":"AMD proprietary driver 26.10.1 (LLPC)","os":"Windows 10.0.26300","cpu":"AMD Ryzen 7 7800X3D 8-Core Processor","cpu_threads":16,"memory_mib":31860,"title_id":"PPSA04264","title":"Grand Theft Auto V","content_version":"01.011.001","resolution":"3840x2160","window":{"spec":"seconds 150:30","first_present":3923,"ms":30021.282},"presents":471,"frame_ms":{"p50":63.332,"p90":66.069,"p99":71.686,"max":76.487},"gpu_busy_ms_per_present":30.574,"worker_busy_pct":{"queue worker 0x0":85.791},"top_phases_ms_per_frame":{"Driver.Packet":51.490,"Graphics.Draw":48.176,"Graphics.Draw.draw_and_resource_release":43.875,"Driver.DrawFinish":40.324,"Driver.Packet.draw":35.832,"Driver.Draw":35.586,"Driver.DrawFinish.draw_enqueued":26.543,"Driver.Draw.prepare":26.318},"trace_dropped":true,"trace_dropped_threads":{"queue worker 0x0":{"events_full_after_s":10.634,"events_lost":474774,"totals_full":false},"draw committer":{"events_full_after_s":15.806,"events_lost":234733,"totals_full":false},"draw finisher":{"events_full_after_s":18.496,"events_lost":161505,"totals_full":false}},"trace":"aps5-trace-9b8f0d7e3298-41296.json.gz"}

Checklist

  • Based on current main; no other open PR implements the same functions
  • One topic per PR; follow-ups go in a new PR
  • No comments in code except technical debt
  • Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
  • No notes, investigation .md files or images added to the repository (attach them to this PR)
  • New third-party code is a submodule built from source
  • Generic behaviour, not specific to one title
  • Depends on: fix(agc): recapture and skip refused draws on the draw finisher #15455
  • Fixes: none
  • Unblocks: none
  • GPUs tested: AMD Radeon RX 7900 XTX, AMD Software 26.10.1 (32.0.32015.7029), Windows 11 Pro 26300
  • AI-assisted: yes

@github-actions github-actions Bot added perf Performance change build docs agc libSceAgc* labels Nov 3, 2026
@Nesyth
Nesyth marked this pull request as ready for review November 3, 2026 17:45
@TripyFox

TripyFox commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

A data point for the finisher on a committer-heavy title: APS5_DRAW_PIPELINE=1 on main 1e65852, so the switch this PR turns on, not the PR head.

Setup. PPSA21567 (ASTRO BOT 01.019.000), Crash Site hub (-lvl hub_crashsite), no input, warm caches, 3840x2160. Windows 11 10.0.26200, RTX 5090 (NVIDIA 621.01), Ryzen 7 9800X3D, WinLibs GCC 15.2.0 Release. Four 240 s runs alternating off/on/off/on with one binary, APS5_TRACE_SUMMARY_EVERY=10 APS5_TRACE_BENCH=1, no APS5_PROFILE_*. Medians over the 20 steady intervals of each run (20-220 s):

run presents / 10 s frame p50 p90 p99 GPU busy per present
off 1 108 93.7 98.4 103.6 73.5
on 1 113 89.7 94.1 99.1 72.3
off 2 109 92.3 97.4 102.9 73.6
on 2 112 90.7 95.7 103.1 72.9

(ms) So about -2.8 ms at p50 (-3 %) and +3.7 % presents, with the GPU time unchanged. It varies between intervals: of six matched 10 s intervals compared with tools/performance_compare.py, five are faster with the finisher and one is +0.9 ms at p50. In a separate 90 s run with APS5_PIPELINE_STATS=1 the finisher took nearly all draws: [finisher] draws (10 s): 254494 handed to the draw finisher, 228 finished on the worker. None of the runs logged a FATAL, ASSERT or device loss.

The gain is small here because at this point the frame is not bound by queue 0's draw path alone. Driver.Draw on the worker drops from 59.0 to 47.5 ms per frame, but queue workers 0x40 and 0x48 stay ~100 % busy, mostly in Driver.Packet.memory_wait, and the GPU is busy ~73 ms per present.

Summary lines of one matched interval (110-120 s), off then on:

{"commit": "1e658521a0a5", "dirty": false, "env": {"APS5_NO_PAD_INPUT": "1", "APS5_TRACE_BENCH": "1", "APS5_TRACE_DIR": "<path>", "APS5_TRACE_SUMMARY_EVERY": "10"}, "gpu": "NVIDIA GeForce RTX 5090", "driver": "NVIDIA 621.01", "os": "Windows 10.0.26200", "cpu": "AMD Ryzen 7 9800X3D 8-Core Processor", "cpu_threads": 16, "memory_mib": 65140, "title_id": "PPSA21567", "title": "ASTRO BOT", "content_version": "01.019.000", "resolution": "3840x2160", "window": {"spec": "summary_every 10", "first_present": 1378, "ms": 10053.481}, "interval_start_s": 110.561, "interval_end_s": 120.614, "presents": 109, "frame_ms": {"p50": 92.4, "p90": 97.629, "p99": 102.506, "max": 103.871}, "gpu_busy_ms_per_present": 72.319, "worker_busy_pct": {"queue worker 0x0": 97.334, "queue worker 0x40": 99.936, "queue worker 0x48": 99.993, "queue worker 0x46": 1.848}, "top_phases_ms_per_frame": {"Driver.Packet": 274.852, "Driver.Packet.memory_wait": 157.283, "Driver.Packet.draw": 59.376, "Driver.Draw": 58.996, "Graphics.Draw": 46.022, "Graphics.Draw.draw_and_resource_release": 36.655, "Driver.Draw.draw_enqueued": 27.034, "Driver.Draw.shader_compile_and_link": 25.895}, "trace_dropped": false, "trace": ""}
{"commit": "1e658521a0a5", "dirty": false, "env": {"APS5_DRAW_PIPELINE": "1", "APS5_NO_PAD_INPUT": "1", "APS5_TRACE_BENCH": "1", "APS5_TRACE_DIR": "<path>", "APS5_TRACE_SUMMARY_EVERY": "10"}, "gpu": "NVIDIA GeForce RTX 5090", "driver": "NVIDIA 621.01", "os": "Windows 10.0.26200", "cpu": "AMD Ryzen 7 9800X3D 8-Core Processor", "cpu_threads": 16, "memory_mib": 65140, "title_id": "PPSA21567", "title": "ASTRO BOT", "content_version": "01.019.000", "resolution": "3840x2160", "window": {"spec": "summary_every 10", "first_present": 1414, "ms": 10050.441}, "interval_start_s": 110.712, "interval_end_s": 120.763, "presents": 114, "frame_ms": {"p50": 89.632, "p90": 92.984, "p99": 96.744, "max": 99.059}, "gpu_busy_ms_per_present": 72.283, "worker_busy_pct": {"queue worker 0x0": 96.42, "queue worker 0x40": 99.94, "queue worker 0x48": 99.913, "queue worker 0x46": 2.248}, "top_phases_ms_per_frame": {"Driver.Packet": 262.159, "Driver.Packet.memory_wait": 153.318, "Driver.Packet.draw": 47.798, "Driver.Draw": 47.492, "Driver.DrawFinish": 43.772, "Graphics.Draw": 42.232, "Driver.Draw.prepare": 40.626, "Graphics.Draw.draw_and_resource_release": 33.586}, "trace_dropped": false, "trace": ""}

AI-assisted: yes (Claude Code). Measured on the machine above.

@TechLevelZero

Copy link
Copy Markdown
Contributor

❌ NVIDIA GPU test: failed

PR head 9b8f0d7 merged onto main 1e65852 · tested 2026-11-03 20:04 UTC · 36 min

GPU NVIDIA RTX 2000 Ada Generation, driver 595.99.02, Vulkan 1.4.329
Host Ubuntu 26.04.1 LTS, kernel 7.0.0-38-generic, container ubuntu:24.04
Logs build and test logs for this run
Build ✅ passed (Release, -DANYPS5_ENABLE_SPIRV_TOOLS=ON)
Tests ❌ 2,794 passed · 11 failed (4 new: 4 pass on main with the PR's tests, 3 also fail on main, 4 unstable on main) · 48 skipped (ANYPS5_REQUIRE_VULKAN=1)
Test setup CPU-only tests without a GPU; GPU tests (and any CPU-only test that failed or skipped without one) once on the GPU with VK_LAYER_KHRONOS_validation and ANYPS5_NO_SHADER_CACHE=1; main ran the same way
Test selection 2,853 of 3,818 tests run: the ones this change can reach, chosen by tools/select_tests.py from main's build graph
Vulkan validation ✅ no new validation IDs vs main (961 tests run on the GPU)
Slow tests (>30 s, GPU tests timed with validation layers) ⚠️ agc_driver_pipeline_journal 120.1 s

Failing tests

This PR changes test files, so each failing test was also built and run on main with only the PR's test changes applied. ✅ there means the failure comes from the PR's other changes; ❌ means the test fails on main too, so this run does not show the PR caused it.

Test With this PR Same test on main 1e65852 (main + this PR's test changes only)
agc_driver_draw_recapture_on_change ❌ failed ✅ passed
agc_driver_image_table_cache_bounds ❌ failed ✅ passed
agc_driver_lazy_resolved_abi ❌ failed ✅ passed
agc_driver_linear_target_direct_write_completion ❌ failed ✅ passed
agc_driver_draw_recapture_on_change: SEGFAULT (also failed when re-run alone), assertion and diagnostics
ctest: SEGFAULT

[write-watch] host imports resolve write protection: no (0 of 4 scratch pages written after a GPU read, 0 after a GPU write, 4 after the import); imported ranges stay watched
[write-watch] host imports resolve write protection: no (0 of 4 scratch pages written after a GPU read, 0 after a GPU write, 4 after the import); imported ranges stay watched
[Shaders/ShaderRegistry.cpp:1321 operator()] Shader 0x5857dfdd5600 could not be prepared at registration (its program address is not in the shader registers); preparing it when it is used
[Shaders/ShaderRegistry.cpp:1321 operator()] Shader 0x5857dfdd5500 could not be prepared at registration (its program address is not in the shader registers); preparing it when it is used
a draw whose descriptor table changed from a zero V# to a valid one after its capture: the table changed 1 time(s), brightest color byte 255 (the valid V# was bound), 1 packet(s) prepared again
the same draw with its table unchanged: brightest color byte 255, 0 packet(s) prepared again
draw recapture: a draw whose captured descriptor table changed before it was recorded binds the table in memory
agc_driver_image_table_cache_bounds: SEGFAULT (also failed when re-run alone), assertion and diagnostics
ctest: SEGFAULT

[write-watch] host imports resolve write protection: no (0 of 4 scratch pages written after a GPU read, 0 after a GPU write, 4 after the import); imported ranges stay watched
[write-watch] host imports resolve write protection: no (0 of 4 scratch pages written after a GPU read, 0 after a GPU write, 4 after the import); imported ranges stay watched
[Shaders/ShaderRegistry.cpp:901 ReportPreparedAtUse] Compute shader 0x7b08b5f30000 was not registered; preparing its artifact at dispatch
[Shaders/ShaderRegistry.cpp:904 ReportPreparedAtUse] Graphics shader 0x7b08b5f30100 was not registered; preparing its artifact at first use
[Shaders/ShaderRegistry.cpp:904 ReportPreparedAtUse] Graphics shader 0x7b08b5f30200 was not registered; preparing its artifact at first use
image table draw cache bounds passed (64 inserts, 57 evictions, 57 over the byte budget, at most 7 entries holding 14523 KiB)
image table cache bounds tests passed
agc_driver_lazy_resolved_abi: Failed (also failed when re-run alone), assertion and diagnostics
ctest: Failed

Segmentation fault (core dumped)
AGC graphics: lazy resolved ABI: the eager reference process failed
agc_driver_linear_target_direct_write_completion: SEGFAULT (also failed when re-run alone), assertion and diagnostics
ctest: SEGFAULT

[write-watch] host imports resolve write protection: no (0 of 4 scratch pages written after a GPU read, 0 after a GPU write, 4 after the import); imported ranges stay watched
[write-watch] host imports resolve write protection: no (0 of 4 scratch pages written after a GPU read, 0 after a GPU write, 4 after the import); imported ranges stay watched
[Shaders/ShaderRegistry.cpp:1321 operator()] Shader 0x61a371456e00 could not be prepared at registration (its program address is not in the shader registers); preparing it when it is used
[Shaders/ShaderRegistry.cpp:1321 operator()] Shader 0x61a371456d00 could not be prepared at registration (its program address is not in the shader registers); preparing it when it is used
[Shaders/ShaderRegistry.cpp:901 ReportPreparedAtUse] Compute shader 0x61a371456b00 was not registered; preparing its artifact at dispatch
linear target direct write completion, fill, then draws: the later writer's values
linear target direct write completion, draws, then fill: the later writer's values
linear target direct write completion, fill, draw in block 0, CPU store in block 1, write-watched memory (watched at the label: 1, completion pending at the CPU store: 1, run by the label: 1): the draw's color and the CPU store
linear target direct write completion, fill, draw in block 0, CPU store in block 1, unwatched memory (watched at the label: 0, completion pending at the CPU store: 0, run by the label: 1): the draw's color and the CPU store
linear target direct write completion tests passed
Unstable on main (4, these also failed in recent main runs on this machine; not counted against this PR)
agc_driver_pipeline_journal: failed in 4 of the last 10 main runs
agc_driver_submit_progress: failed in 5 of the last 10 main runs
agc_submit_progress: failed in 6 of the last 10 main runs
agc_submit_progress_no_timeline: failed in 5 of the last 10 main runs
Failing on main too (3, not counted against this PR)
agc_driver_image_table
guest_mmap_executable
guest_socket_options_multicast
Validated but not compared (5, not in the main baseline's validation pass)
agc_driver_same_queue_label_wait_no_draw_pipeline
agc_driver_staged_clear_sampled_read_no_draw_pipeline
agc_driver_storage_write_back_overhang_no_draw_pipeline
agc_driver_unimported_label_order_no_draw_pipeline
agc_driver_write_data_written_ahead_no_draw_pipeline
Validation IDs already present on main (4 IDs in 534 tests)
VUID-VkDeviceCreateInfo-pNext-pNext (534 tests): agc_driver, agc_driver_address_passes, agc_driver_alias_copy_label, agc_driver_alpha_to_coverage, agc_driver_array_layer_target, agc_driver_bc_block_copy, agc_driver_bda_anchor_reads, agc_driver_bda_span_reads, and 526 more
VUID-vkCmdCopyImage-srcImage-01548 (7 tests): agc_driver_depth_clear_hle, agc_driver_depth_plane_buffer_memory, agc_driver_depth_plane_partial_store, agc_driver_depth_plane_sampling, agc_driver_depth_storage_dispatch, agc_driver_depth_surface_from_storage, agc_driver_dispatch_move_depth
VUID-vkCmdCopyImage-srcImage-01551 (7 tests): agc_driver_depth_clear_hle, agc_driver_depth_plane_buffer_memory, agc_driver_depth_plane_partial_store, agc_driver_depth_plane_sampling, agc_driver_depth_storage_dispatch, agc_driver_depth_surface_from_storage, agc_driver_dispatch_move_depth
VUID-VkPipelineColorBlendStateCreateInfo-pAttachments-00605 (1 test): agc_driver_srgb8_color_target
Skipped (48)
agc_driver_bda_reads_shadowed_results: skipped, the device reports watched host memory written when it imports or reads it
agc_driver_bda_reads_shadowed_results_per_build: skipped, the device reports watched host memory written when it imports or reads it
agc_driver_sample_order_grid_reference: skipped, the device does not keep depth drawn at custom sample locations at those locations, even after a transition that names them: the composite of uniform renders holds the guest's depth in 0 of 3005 sloped samples, so no path can be compared with it bit for bit
agc_driver_scattered_runs: skipped, host imports are unavailable: /dev/udmabuf is not accessible
agc_driver_sparse_dma_buf_import: skipped, host imports are unavailable: /dev/udmabuf is not accessible
agc_driver_stencil_ref_export: skipped, the device lacks VK_EXT_shader_stencil_export
agc_host_import_sparse: skipped, host imports are unavailable: /dev/udmabuf is not accessible
agc_import_settle_initial_dcc: skipped, the device has no deferred settle of host imports: imports under 1 MiB wait for one only on Windows
agc_unit_shadow_completion_store: skipped, the device reports watched host memory written when it imports or reads it
guest_http_uri_locale: 
guest_net_ctl: skipped, the host has no IPv4 address
guest_video_out_color_space: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_hdr: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_high_frame_rate: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_output_mode: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_resolution_status: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_wqhd_detection: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
video_out_flip_aftergpu: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_backtoback: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_controls: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_cpuflip: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_cpuflip_in_order: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_dcc: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_deferred: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_eop: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_eop_aftergpu: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_eop_main: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not co
Reproduce
git fetch origin 9b8f0d7e3298f4ec475a759f3e9e30b7d13bea47 1e658521a0a599872612bb91949311e2956b5283
git checkout 1e658521a0a599872612bb91949311e2956b5283 && git merge --no-ff 9b8f0d7e3298f4ec475a759f3e9e30b7d13bea47
git submodule update --init --recursive
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=ON -DANYPS5_ENABLE_SPIRV_TOOLS=ON
cmake --build build && cmake --build build --target libs
VK_DRIVER_FILES=/nonexistent/icd.json ctest --test-dir build -E '^(agc|shader|recompiler|videoout|video_out|guest_video_out|vulkan)|gpu' --timeout 120 --parallel 32 --output-on-failure
ANYPS5_REQUIRE_VULKAN=1 ANYPS5_NO_SHADER_CACHE=1 VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation \
  ctest --test-dir build -R '^(agc|shader|recompiler|videoout|video_out|guest_video_out|vulkan)|gpu' --timeout 120 --parallel 4 --output-on-failure
# failures are then re-run alone: --timeout 360 --parallel 2; new validation IDs: the test alone, twice, --parallel 1
# the failing tests, on this merge and on main:
ANYPS5_REQUIRE_VULKAN=1 ANYPS5_NO_SHADER_CACHE=1 VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation \
  ctest --test-dir build -R '^(agc_driver_draw_recapture_on_change|agc_driver_image_table_cache_bounds|agc_driver_lazy_resolved_abi|agc_driver_linear_target_direct_write_completion)$' --parallel 1 -V

Automated build and test on a contributor's NVIDIA machine; it does not replace upstream CI. A new comment is posted for each new push. Full logs.

@XairRrR XairRrR left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed — changes requested

Merged onto current main, the PR makes agc_driver_queued_constant_ram_dump fail on every run: a draw reads a stale constant-RAM table. main has the same failure under APS5_DRAW_PIPELINE=1, so the finisher already had this bug, and this PR makes it the default path.

What I ran

Head 9b8f0d7e3298f4ec475a759f3e9e30b7d13bea47, main 086c0686c8adff3ee648fdbe34e0b0b647e1d511, merged tree 1399eec3d931d7d63ecd70c6838ecaae6f2bef43, on Debian 13 (VM, Xeon Gold 6136), GCC 14.2.0, lavapipe (Mesa 26.1.6, LLVM 19).

git diff --check 086c0686c8adff3ee648fdbe34e0b0b647e1d511...9b8f0d7e3298f4ec475a759f3e9e30b7d13bea47
python3 tools/check_conventions.py --base 086c0686c8adff3ee648fdbe34e0b0b647e1d511
# three runs each
ANYPS5_REQUIRE_VULKAN=1 ctest --test-dir build -R '^agc_driver_queued_constant_ram_dump$'
APS5_NO_DRAW_PIPELINE=1 ANYPS5_REQUIRE_VULKAN=1 ctest --test-dir build -R '^agc_driver_queued_constant_ram_dump$'
# negative control: production files restored from main, then with and without APS5_DRAW_PIPELINE=1
ninja -C build agc_driver_queued_constant_ram_dump_tests agc_driver_draw_recapture_on_change_tests agc_driver_draw_unsupported_buffer_type_tests agc_driver_draw_data_hit_user_words_tests agc_driver_pipelined_metadata_pass_tests
ANYPS5_REQUIRE_VULKAN=1 ctest --test-dir build -R '^agc_driver_(queued_constant_ram_dump|draw_recapture_on_change.*|draw_unsupported_buffer_type.*|draw_data_hit_user_words.*|pipelined_metadata_pass.*)$'
# merged tree restored, with the validation layer
VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation ANYPS5_NO_SHADER_CACHE=1 ANYPS5_REQUIRE_VULKAN=1 ctest --test-dir build -j8 -R '^agc_driver_(queued_constant_ram_dump|draw_recapture_on_change.*|draw_unsupported_buffer_type.*|draw_data_hit_user_words.*|pipelined_metadata_pass.*|image_table_cache_bounds|lazy_resolved_abi|linear_target_direct_write_completion)$'

Results

Check Result
Merge into main clean
git diff --check clean
check_conventions.py passed
Configure, build, libs passed
Selected tests (2265) 1 new failure: agc_driver_queued_constant_ram_dump. The 7 known failures also fail on main.
queued_constant_ram_dump, merged tree, default environment failed 3 of 3 runs
queued_constant_ram_dump, merged tree, APS5_NO_DRAW_PIPELINE=1 passed 3 of 3 runs
queued_constant_ram_dump, main's production files, default / APS5_DRAW_PIPELINE=1 passed / failed
check_skips.py agc_driver_pipelined_metadata_pass_no_draw_pipeline skipped unexpectedly (see non-blocking)
Validation layer (script, 802 tests, layer loaded in 801, none missing) New VUIDs only in agc_driver_image_gather_volume (VUID-VkSamplerCreateInfo-addressMode*). main did not run that test, so there is no comparison for it (see Not verified).
Focused tests with the validation layer on the merged tree 17 passed. queued_constant_ram_dump failed. pipelined_metadata_pass_no_draw_pipeline skipped (on purpose). linear_target_direct_write_completion skipped on this device.
agc_driver_queued_constant_ram_dump on the merged tree
AGC graphics: queued constant RAM dump: draw 1 wrote 64, expected 255 (draw 0 reads the table before both dumps, draw 1 after the first, draw 2 after the second)

Negative control

I restored Driver.hpp, Execution/src/Driver/Draw/Draw.cpp, ShaderMemory.cpp and Graphics/src/Draw.cpp from main and rebuilt the affected test executables:

  • Default environment: the tests whose expectations now depend on the finisher being on by default fail as intended:

    • agc_driver_draw_data_hit_user_words
    • agc_driver_draw_data_hit_user_words_draw_pipeline
    • agc_driver_draw_unsupported_buffer_type
    • agc_driver_draw_unsupported_buffer_type_draw_pipeline
    • agc_driver_pipelined_metadata_pass
    • agc_driver_pipelined_metadata_pass_draw_pipeline
  • APS5_DRAW_PIPELINE=1 (finisher on, without the recapture and refused-draw fix): these fail, which shows the fix from commit 6e2c50b26e is needed and covered:

    • agc_driver_draw_recapture_on_change
    • agc_driver_draw_recapture_on_change_no_draw_pipeline
    • agc_driver_draw_unsupported_buffer_type_skip_refused
    • agc_driver_draw_unsupported_buffer_type_skip_refused_no_draw_pipeline

    agc_driver_queued_constant_ram_dump also fails in this run.

I then restored the merged files and rebuilt; git status --porcelain is empty.

Findings

Blocking

  • The default draw path now gives a wrong result for draws that read a table written by a queued DUMP_CONST_RAM. agc_driver_queued_constant_ram_dump (QueuedConstantRamDump.cpp#L243) fails on every run once the finisher is on. Draw 1 writes 64 instead of 255, meaning it read the descriptor table from before the first dump.
    • It passes with APS5_NO_DRAW_PIPELINE=1, and on main it fails only under APS5_DRAW_PIPELINE=1. The bug is in the finisher, and flipping the default at Graphics/src/Draw.cpp#L3989 makes it the default behaviour.
    • The test arrived on main after the baselines used in the description and in the NVIDIA run (1e65852). That likely explains why neither caught it. The claim "the ^agc_ suite has the same two failures as main" does not hold on current main.
    • Suggested fix: rebase. Then either make a draw finished on the finisher honour queued constant-RAM dumps (served capture words, or the capture drain the test expects), or finish such draws on the worker. Add agc_driver_queued_constant_ram_dump to the finisher and _no_draw_pipeline variants so the fix stays covered.

Non-blocking

  • The registration helper adds a test that always skips. agc_add_draw_pipeline_test now always registers <name>_no_draw_pipeline (CMakeLists.txt#L645-L646). PipelinedMetadataPass.cpp:122 returns VulkanTestSkipped under that switch, so agc_driver_pipelined_metadata_pass_no_draw_pipeline always skips, and tools/check_skips.py counts it as an unexpected skip. Suggested fix: give the helper an option to leave out the _no_draw_pipeline variant, or register the metadata pass test without it.
  • NVIDIA failures not reproduced here. The NVIDIA run reported 4 new SEGFAULTs: draw_recapture_on_change, image_table_cache_bounds, lazy_resolved_abi and linear_target_direct_write_completion. In each log the test prints its final "passed" line before the crash, which points at process teardown with the finisher thread alive. On lavapipe here the first three pass and linear_target_direct_write_completion skips, so I can neither confirm nor refute these. They should be explained or fixed before the finisher becomes the default, because NVIDIA users would hit them.
  • I checked the recursive retry in finishDraw → restartFinishedDraw → finishDraw (Draw.cpp#L1266-L1271). It is bounded: capturedMemoryChanged returns false while a RecaptureScope is active. No issue there.

Not verified

  • NVIDIA crashes: I have no NVIDIA hardware, so I could not run the tests that crashed there.
  • agc_driver_image_gather_volume VUIDs: main did not run this test in the validation baseline, so I could not tell whether its new VUID-VkSamplerCreateInfo-addressMode* lines are new. The PR does not touch sampler creation, so they are probably not caused by it, but I did not check.
  • Performance: I did not check the GTA V numbers or traces; I cannot launch titles. The description has two summary lines with different commits, as the rules require.
  • Windows: I did not run the build or tests there.

To get this to a good state

  1. Rebase on current main.
  2. Fix the finisher's handling of queued DUMP_CONST_RAM so that agc_driver_queued_constant_ram_dump passes with the finisher on, and register finisher and _no_draw_pipeline variants for it.
  3. Address or explain the NVIDIA teardown SEGFAULTs.
  4. Stop registering a _no_draw_pipeline variant that can only skip.
  5. Rerun the ^agc_ suite and the validation comparison against the rebased main, and update the description.

@Nesyth
Nesyth force-pushed the perf/agc-draw-finisher-default branch from 9b8f0d7 to 6330546 Compare November 3, 2026 21:27
@Nesyth

Nesyth commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

@XairRrR Thanks for the review and the negative control. The branch is rebased on main 086c068, with #15455's two commits unchanged. Two finisher fixes now come before the default flip.

  • Queued DUMP_CONST_RAM on the finisher. Fixed in 77cd206. A finisher job ran with DrawPipeline::Active() unset. That flag is a thread-local the queue worker sets for its own stream. On the finisher, queryPendingWrite therefore neither served a queued store's bytes nor drained. The stage capture of draw 1 read the table from guest memory before the committer had stored the first dump. QueryPendingSpan, ReadRawShader, the vertex-table decode and variant validation skipped their queued-write checks the same way. A finisher job now runs with the flag of the draw it finishes (tail.pipelined), so its reads see queued pipeline writes as the worker's do. The committer then holds only items from before that draw, because the worker drains the finisher before it enqueues. agc_driver_queued_constant_ram_dump now has _draw_pipeline and _no_draw_pipeline variants. All three pass, and the default also passes with APS5_VERIFY_KNOWN_VALUES=1. Without the fix the default fails as you found: draw 1 writes 64, expected 255.
  • Teardown crashes on NVIDIA. Fixed in b8f163c. As you read it, the finisher thread was never stopped. Its thread-local state kept a reference to the Vulkan device, so the device outlived AgcDriverShutdown, and on Linux it was torn down during exit. On Windows the same leak shows as one of the two devices these tests create surviving the shutdown. The shutdown now drains the finisher and joins its thread before the committer's, and a draw handed to it afterwards finishes on the caller. agc_driver_shutdown_threads_draw_finisher covers this and fails without the join.
  • A _no_draw_pipeline variant that can only skip. agc_add_draw_pipeline_test now takes PIPELINE_ONLY, and the metadata pass test uses it.
  • Suite. On this machine the full suite, 3971 tests, fails only the six tests that also fail on main: agc_driver_direct_write_completion, agc_driver_sample_order_grid_reference, and four tool tests that need symlink rights or a real python.exe. With the validation layer, the focused draw, pipeline and finisher tests pass on both arms (91 on main, 116 on this branch). Both report only Undefined-Value-ShaderOutputNotConsumed warnings and no VUID.

@TechLevelZero The three segfaults and the crash of the lazy-ABI test's eager reference process are the teardown above.

@XairRrR XairRrR left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-checked — approved

All previous findings are resolved. The new commits make a finisher job run with the pipelined flag of the draw it finishes, join the draw finisher thread at shutdown, and let agc_add_draw_pipeline_test leave out the variant that can only skip. The focused tests pass, and each fix is covered by a test that fails without it.

What changed since my last review

  • The branch is rebased onto main 086c0686c8. Commit 4e4b0f11b4 is the same change as 6e2c50b26e (range-diff =).
  • New 060cddc3f7: a restarted finished draw starts again from its resolved draw parameters (tail.resolvedParameters). It re-snapshots unregistered program code that changed after capture. DrawRecaptureOnChange.cpp gains cases for changed raw code and for folding the vertex offset once.
  • New b8f163c606: DrawFinisher::Stop() drains the finisher and joins its thread, and Driver::stop() now calls it (Lifecycle.cpp#L192). Once stopped, a draw handed to the finisher runs on the caller. New test: agc_driver_shutdown_threads_draw_finisher.
  • New 77cd20649c: the finisher job sets DrawPipeline::Active() = job.pipelined (Draw.cpp#L750). agc_driver_queued_constant_ram_dump gains _draw_pipeline and _no_draw_pipeline variants.
  • 6330546a47: the helper takes PIPELINE_ONLY, and the metadata pass test uses it. The finisher assertion in DrawRecaptureOnChange.cpp is limited to Depth() != 0.
  • The author's reply addresses each point of my previous review. The description is updated (new main base, 3971-test suite, 91/116 validation runs).

What I ran

Head 6330546a47e6c4afda70b640f338678e6c9601a5, main 04a9eb4fa2877eb6e40bf2c83adf972e40d89b24, merged tree 729f064641b491c2bd1d42a1f25bb8f9b698fa91. Machine: Debian 13 (VM, Xeon Gold 6136), GCC 14.2.0, lavapipe (Mesa 26.1.6, LLVM 19).

git diff --check 04a9eb4fa2877eb6e40bf2c83adf972e40d89b24...6330546a47e6c4afda70b640f338678e6c9601a5
python3 tools/check_conventions.py --base 04a9eb4fa2877eb6e40bf2c83adf972e40d89b24
ANYPS5_REQUIRE_VULKAN=1 ctest --test-dir build -j8 --test-load 32 --repeat until-fail:3 --output-on-failure -R '^agc_driver_(queued_constant_ram_dump.*|shutdown_threads.*|draw_recapture_on_change.*|pipelined_metadata_pass.*|draw_unsupported_buffer_type.*|draw_data_hit_user_words.*|image_table_cache_bounds|lazy_resolved_abi.*|linear_target_direct_write_completion)$'
# negative control: drop the two new fixes, rebuild, run their tests
ninja -C build -j32 -l32 agc_driver_queued_constant_ram_dump_tests agc_driver_shutdown_threads_tests
ANYPS5_REQUIRE_VULKAN=1 ctest --test-dir build -j4 --output-on-failure -R '^agc_driver_(queued_constant_ram_dump.*|shutdown_threads_draw_finisher)$'
# restore the merged files, rebuild, rerun, git status --porcelain
Check Result
Merge into main clean
git diff --check clean
check_conventions.py passed
Configure, build, libs (script) passed
Selected tests (script, 2269) no new failures or timeouts. The 7 known failures also fail on main.
check_skips.py (script) 115 skips, all expected. The always-skipping _no_draw_pipeline variant is gone.
Focused tests, 31, 3 runs each all passed. linear_target_direct_write_completion skips on this device (as before).
Validation layer (script, 804 tests, layer loaded in 803, none missing) New VUID lines only in agc_driver_image_gather_volume, which the main baseline did not run (same as last time)

Negative control

In the merged tree I removed only the two new production lines and rebuilt both test executables:

  • DrawPipeline::Active() = job.pipelined; in Draw.cpp
  • DrawFinisher::Get().Stop(); in Lifecycle.cpp

Three tests then failed:

  • agc_driver_queued_constant_ram_dump
  • agc_driver_queued_constant_ram_dump_draw_pipeline
  • agc_driver_shutdown_threads_draw_finisher

agc_driver_queued_constant_ram_dump_no_draw_pipeline still passed, as expected. I then restored both files from the merged tree and rebuilt. All 4 tests pass, and git status --porcelain is empty.

Failure without the pipelined-flag fix
AGC graphics: queued constant RAM dump: draw 1 wrote 64, expected 255 (draw 0 reads the table before both dumps, draw 1 after the first, draw 2 after the second)

I did not repeat the negative control for 060cddc3f7 (restart from resolved parameters and changed raw code). Its new DrawRecaptureOnChange.cpp cases pass on all three variants.

Previous findings

  • Resolved: a queued DUMP_CONST_RAM was not seen by draws on the finisher. agc_driver_queued_constant_ram_dump and both its variants pass 3 of 3 runs. Without the Active() line, the default and _draw_pipeline variants fail with the same message as before.
  • Resolved: a _no_draw_pipeline variant that can only skip. The metadata pass test is registered with PIPELINE_ONLY, and check_skips.py reports no unexpected skip.
  • Resolved (the fix is verified here; the NVIDIA result is author-reported): NVIDIA teardown SEGFAULTs. The finisher thread was never joined. Stop() now joins it in Driver::stop() after Drain(). agc_driver_shutdown_threads_draw_finisher checks that the finisher's thread-local state is released, and it fails without the join. I have no NVIDIA hardware. The author reports that the four crashing tests are fixed, and a new NVIDIA CI run should confirm it.
  • No longer applies: the bounded recursion in finishDraw → restartFinishedDraw. It was not a defect. restartFinishedDraw now also refreshes changed raw program code, and the retry stays bounded through the same RecaptureScope guard.

New findings

Blocking

None.

Non-blocking

None. A note on the closing path: when the finisher is closing, DrawFinisher::Enqueue runs the job on the caller (DrawPipeline.cpp#L187). The finisher job then sets the caller's thread-local DrawPipeline::Active() and does not restore it. This only happens after Driver::stop() has joined the queue workers, so I found no reachable case where it matters.

Not verified

  • NVIDIA or other hardware vendors: I could not check the teardown fix there.
  • Performance: I did not check the GTA V numbers or traces. The description carries two summary lines with different commits, as required.
  • Validation of agc_driver_image_gather_volume: the main baseline did not run this test, so its VUID-VkSamplerCreateInfo-addressMode* lines could not be compared. The PR does not touch sampler creation.
  • Windows build and tests.

@RNK123

RNK123 commented Nov 4, 2026

Copy link
Copy Markdown

RADV/RX 9060 XT focused draw-path check: the default finisher and the APS5_NO_DRAW_PIPELINE worker paths both pass on this head, three runs each.

Head 6330546a47 merged onto main c2eafe4461. On main, three existing draw tests plus the two registered opt-in _draw_pipeline variants passed 5/5 per run, three times. On the merged tree, the same three draw tests and their three _no_draw_pipeline variants passed 6/6 per run, three times. The additional agc_driver_shutdown_threads also passed three times; it is not counted as GPU coverage. The recapture test reported the draw finisher on the default path and the queue worker on the disabled path. No skips were reported in these runs. Requested Khronos validation and the Radeon were confirmed for the GPU tests; no VUID- or SYNC-HAZARD- diagnostics. The same pre-existing unrelated LSFG implicit-layer loading failure appeared on both arms; Khronos loaded.

Sensitivity check: on the merged tree, forcing DrawFinisher::Enabled() to return false caused agc_driver_draw_unsupported_buffer_type to fail 3/3 at its expected finisher-specific diagnostic assertion. Restoring it passed 3/3. This is a deliberate mutation of the merged tree, not a claim that unmodified main fails the same test.

Host: CachyOS Linux 7.3.1-1-cachyos, GCC 16.2.1, Release, Radeon RX 9060 XT (RADV GFX1200), Mesa 26.2.5-arch3.1, Vulkan 1.4.354, Khronos validation 1.4.363.

Reproduction

Build the agc_driver_draw_recapture_on_change_tests, agc_driver_draw_unsupported_buffer_type_tests, agc_driver_draw_data_hit_user_words_tests and agc_driver_shutdown_threads_tests targets on main and on the merge. Set:

export ANYPS5_REQUIRE_VULKAN=1 ANYPS5_NO_SHADER_CACHE=1
export VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation VK_LOADER_DEBUG=layer
export VK_LAYER_PATH="$HOME/.local/share/anyps5-tester/vk-layers/usr/share/vulkan/explicit_layer.d"

Run the three named GPU tests on each tree. On main also run the registered _draw_pipeline variants of the unsupported-buffer and user-words tests; on the merge run the three _no_draw_pipeline variants. Each CTest entry was run three times. The shutdown test was run only on the merge.

Limits: synthetic focused tests, not a game run, performance comparison or full-suite result; no PS5 hardware. The control checks the finisher-specific failure path, not every recapture or shutdown condition. AI-assisted independent hardware test run (GPT-6.1 Terra).

@TechLevelZero

Copy link
Copy Markdown
Contributor

✅ AMD GPU test: passed

PR head 6330546 merged onto main 1c8c0ef · tested 2026-11-04 04:31 UTC · 22 min

GPU AMD Radeon RX 6300 (RADV NAVI24), driver Mesa 25.2.8-0ubuntu0.24.04.4, Vulkan 1.4.318
Host Ubuntu 26.04.1 LTS, kernel 7.0.0-42-generic, container ubuntu:24.04
Logs build and test logs for this run
Build ✅ passed (Release, -DANYPS5_ENABLE_SPIRV_TOOLS=ON)
Tests ✅ 2,825 passed · 7 failed (0 new, 6 also fail on main, 1 unstable on main) · 57 skipped (ANYPS5_REQUIRE_VULKAN=1)
Test setup CPU-only tests without a GPU; GPU tests (and any CPU-only test that failed or skipped without one) once on the GPU with VK_LAYER_KHRONOS_validation and ANYPS5_NO_SHADER_CACHE=1; main ran the same way
Test selection 2,889 of 3,842 tests run: the ones this change can reach, chosen by tools/select_tests.py from main's build graph
Vulkan validation ✅ no new validation IDs vs main (1034 tests run on the GPU)
Slow tests (>30 s, GPU tests timed with validation layers) ⚠️ agc_driver_pipeline_journal 121.0 s, ⚠️ public_headers 57.1 s, ⚠️ agc_driver_image_sample_unnormalized 50.1 s, ⚠️ agc_driver_lazy_resolved_abi_switch_set 41.1 s, ⚠️ agc_driver_lazy_resolved_abi 40.4 s
Unstable on main (1, these also failed in recent main runs on this machine; not counted against this PR)
agc_driver_pipeline_journal: failed in 8 of the last 10 main runs
Failing on main too (6, not counted against this PR)
agc_driver_image_table
agc_driver_native_depth_clears
agc_submit_progress
agc_submit_progress_no_timeline
guest_mmap_executable
guest_socket_options_multicast
Validated but not compared (67, not in the main baseline's validation pass)
agc_driver_pipelined_dma_no_draw_pipeline
agc_driver_pipelined_fill_no_draw_pipeline
agc_driver_pipelined_predicated_label_order_no_draw_pipeline
agc_driver_pipelined_predication_read_no_draw_pipeline
agc_driver_queued_constant_ram_dump_draw_pipeline
agc_driver_queued_constant_ram_dump_no_draw_pipeline
agc_driver_same_queue_label_wait_no_draw_pipeline
agc_driver_shutdown_threads_draw_finisher
agc_driver_staged_clear_sampled_read_no_draw_pipeline
agc_driver_storage_write_back_overhang_no_draw_pipeline
agc_driver_unimported_label_order_no_draw_pipeline
agc_driver_write_data_written_ahead_no_draw_pipeline
public_headers
public_headers_compile_0
public_headers_compile_10
public_headers_compile_12
public_headers_compile_14
public_headers_compile_16
public_headers_compile_18
public_headers_compile_2
public_headers_compile_20
public_headers_compile_21
public_headers_compile_22
public_headers_compile_23
public_headers_compile_24
public_headers_compile_25
public_headers_compile_26
public_headers_compile_27
public_headers_compile_28
public_headers_compile_29
public_headers_compile_30
public_headers_compile_31
public_headers_compile_32
public_headers_compile_33
public_headers_compile_34
public_headers_compile_35
public_headers_compile_36
public_headers_compile_37
public_headers_compile_38
public_headers_compile_39
public_headers_compile_4
public_headers_compile_40
public_headers_compile_41
public_headers_compile_42
public_headers_compile_43
public_headers_compile_44
public_headers_compile_45
public_headers_compile_46
public_headers_compile_47
public_headers_compile_48
public_headers_compile_49
public_headers_compile_50
public_headers_compile_51
public_headers_compile_52
public_headers_compile_53
public_headers_compile_54
public_headers_compile_55
public_headers_compile_56
public_headers_compile_57
public_headers_compile_58
public_headers_compile_59
public_headers_compile_6
public_headers_compile_60
public_headers_compile_61
public_headers_compile_62
public_headers_compile_63
public_headers_compile_8
Validation IDs already present on main (8 IDs in 532 tests)
VUID-VkDeviceCreateInfo-pNext-pNext (532 tests): agc_driver, agc_driver_address_passes, agc_driver_alias_copy_label, agc_driver_alpha_to_coverage, agc_driver_array_layer_target, agc_driver_bc_block_copy, agc_driver_bda_anchor_reads, agc_driver_bda_span_reads, and 524 more
VUID-vkCmdCopyImage-srcImage-01548 (4 tests): agc_driver_depth_clear_hle, agc_driver_depth_storage_dispatch, agc_driver_depth_surface_from_storage, agc_driver_dispatch_move_depth
VUID-vkCmdCopyImage-srcImage-01551 (4 tests): agc_driver_depth_clear_hle, agc_driver_depth_storage_dispatch, agc_driver_depth_surface_from_storage, agc_driver_dispatch_move_depth
VUID-VkPipelineColorBlendStateCreateInfo-pAttachments-00605 (1 test): agc_driver_prepared_exports
VUID-VkSamplerCreateInfo-addressModeU-01079 (1 test): agc_driver_image_gather_volume
VUID-VkSamplerCreateInfo-addressModeU-parameter (1 test): agc_driver_image_gather_volume
VUID-VkSamplerCreateInfo-addressModeV-parameter (1 test): agc_driver_image_gather_volume
VUID-VkSamplerCreateInfo-addressModeW-parameter (1 test): agc_driver_image_gather_volume
Skipped (57)
agc_driver_bda_reads_shadowed_results: skipped, the device reports watched host memory written when it imports or reads it
agc_driver_bda_reads_shadowed_results_per_build: skipped, the device reports watched host memory written when it imports or reads it
agc_driver_copied_writer_label_order: skipped, the device reports a minStorageBufferOffsetAlignment of 4, so the dispatch binds its record at +0x1004 in place and nothing is written back; the test needs an alignment above 4
agc_driver_float16_unused_nan_payloads: skipped, the device has no native f16 conversions with RTE rounding and kept f16 denormals
agc_driver_gcn_display_present: skipped, no display: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
agc_driver_image_table_vouched_lookups: skipped, the device reports 4 of 4 host-imported pages written after a GPU read, so a watched imported table cannot be vouched for
agc_driver_linear_target_direct_write_completion: skipped, the device does not keep host imports write-watched, so no completion stamp over the target is tracked
agc_driver_unit_shadow: skipped, the device does not import watched host memory for unit shadows
agc_driver_unit_shadow_budget: skipped, the device does not import watched host memory for unit shadows
agc_host_import_sparse: skipped, host imports are unavailable: /dev/udmabuf is not accessible
agc_idle_publish_wait: skipped, the device reports 4 of 4 host-imported pages written after a GPU read, so a forced watch of the import reads its submissions as CPU stores
agc_import_settle_initial_dcc: skipped, the device has no deferred settle of host imports: imports under 1 MiB wait for one only on Windows
agc_shadowed_key_stores: skipped, the device imports no watched host memory under a unit shadow
agc_small_tiled_shadow_label: skipped, host imports are unavailable: the test block is not imported and watched
agc_storage_clear_hand_over: skipped, the device imports no watched host memory for the clear-key hand-over
agc_storage_pending_key_refresh: skipped, refreshes over a pending key store are not testable here
agc_unit_shadow: skipped, the device does not import watched host memory for unit shadows
agc_unit_shadow_completion_store: skipped, the device reports watched host memory written when it imports or reads it
guest_http_uri_locale: 
guest_net_ctl: skipped, the host has no IPv4 address
guest_video_out_color_space: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_hdr: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_high_frame_rate: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_output_mode: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_resolution_status: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_wqhd_detection: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
video_out_flip_aftergpu: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_backtoback: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_controls: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_cpuflip: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not av
Reproduce
git fetch origin 6330546a47e6c4afda70b640f338678e6c9601a5 1c8c0efc8c41314402e6966da5bef6ea3d639d10
git checkout 1c8c0efc8c41314402e6966da5bef6ea3d639d10 && git merge --no-ff 6330546a47e6c4afda70b640f338678e6c9601a5
git submodule update --init --recursive
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=ON -DANYPS5_ENABLE_SPIRV_TOOLS=ON
cmake --build build && cmake --build build --target libs
VK_DRIVER_FILES=/nonexistent/icd.json ctest --test-dir build -E '^(agc|shader|recompiler|videoout|video_out|guest_video_out|vulkan)|gpu' --timeout 120 --parallel 12 --output-on-failure
ANYPS5_REQUIRE_VULKAN=1 ANYPS5_NO_SHADER_CACHE=1 VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation \
  ctest --test-dir build -R '^(agc|shader|recompiler|videoout|video_out|guest_video_out|vulkan)|gpu' --timeout 120 --parallel 16 --output-on-failure
# failures are then re-run alone: --timeout 360 --parallel 2; new validation IDs: the test alone, twice, --parallel 1

Automated build and test on a contributor's AMD machine; it does not replace upstream CI. A new comment is posted for each new push. Full logs.

@TechLevelZero

Copy link
Copy Markdown
Contributor

✅ NVIDIA GPU test: passed

PR head 6330546 merged onto main 1c8c0ef · tested 2026-11-04 04:53 UTC · 23 min

GPU NVIDIA RTX 2000 Ada Generation, driver 595.99.02, Vulkan 1.4.329
Host Ubuntu 26.04.1 LTS, kernel 7.0.0-38-generic, container ubuntu:24.04
Logs build and test logs for this run
Build ✅ passed (Release, -DANYPS5_ENABLE_SPIRV_TOOLS=ON)
Tests ✅ 2,834 passed · 5 failed (0 new, 3 also fail on main, 2 unstable on main) · 50 skipped (ANYPS5_REQUIRE_VULKAN=1)
Test setup CPU-only tests without a GPU; GPU tests (and any CPU-only test that failed or skipped without one) once on the GPU with VK_LAYER_KHRONOS_validation and ANYPS5_NO_SHADER_CACHE=1; main ran the same way
Test selection 2,889 of 3,842 tests run: the ones this change can reach, chosen by tools/select_tests.py from main's build graph
Vulkan validation ✅ no new validation IDs vs main (979 tests run on the GPU)
Slow tests (>30 s, GPU tests timed with validation layers) ⚠️ agc_driver_pipeline_journal 120.1 s
Unstable on main (2, these also failed in recent main runs on this machine; not counted against this PR)
agc_driver_pipeline_journal: failed in 4 of the last 10 main runs
agc_submit_progress_no_timeline: failed in 4 of the last 10 main runs
Failing on main too (3, not counted against this PR)
agc_driver_image_table
guest_mmap_executable
guest_socket_options_multicast
Validated but not compared (7, not in the main baseline's validation pass)
agc_driver_queued_constant_ram_dump_no_draw_pipeline
agc_driver_same_queue_label_wait_no_draw_pipeline
agc_driver_shutdown_threads_draw_finisher
agc_driver_staged_clear_sampled_read_no_draw_pipeline
agc_driver_storage_write_back_overhang_no_draw_pipeline
agc_driver_unimported_label_order_no_draw_pipeline
agc_driver_write_data_written_ahead_no_draw_pipeline
Validation IDs already present on main (8 IDs in 546 tests)
VUID-VkDeviceCreateInfo-pNext-pNext (546 tests): agc_driver, agc_driver_address_passes, agc_driver_alias_copy_label, agc_driver_alpha_to_coverage, agc_driver_array_layer_target, agc_driver_bc_block_copy, agc_driver_bda_anchor_reads, agc_driver_bda_span_reads, and 538 more
VUID-vkCmdCopyImage-srcImage-01548 (7 tests): agc_driver_depth_clear_hle, agc_driver_depth_plane_buffer_memory, agc_driver_depth_plane_partial_store, agc_driver_depth_plane_sampling, agc_driver_depth_storage_dispatch, agc_driver_depth_surface_from_storage, agc_driver_dispatch_move_depth
VUID-vkCmdCopyImage-srcImage-01551 (7 tests): agc_driver_depth_clear_hle, agc_driver_depth_plane_buffer_memory, agc_driver_depth_plane_partial_store, agc_driver_depth_plane_sampling, agc_driver_depth_storage_dispatch, agc_driver_depth_surface_from_storage, agc_driver_dispatch_move_depth
VUID-VkPipelineColorBlendStateCreateInfo-pAttachments-00605 (1 test): agc_driver_srgb8_color_target
VUID-VkSamplerCreateInfo-addressModeU-01079 (1 test): agc_driver_image_gather_volume
VUID-VkSamplerCreateInfo-addressModeU-parameter (1 test): agc_driver_image_gather_volume
VUID-VkSamplerCreateInfo-addressModeV-parameter (1 test): agc_driver_image_gather_volume
VUID-VkSamplerCreateInfo-addressModeW-parameter (1 test): agc_driver_image_gather_volume
Skipped (50)
agc_driver_bda_reads_shadowed_results: skipped, the device reports watched host memory written when it imports or reads it
agc_driver_bda_reads_shadowed_results_per_build: skipped, the device reports watched host memory written when it imports or reads it
agc_driver_gcn_display_present: skipped, no display: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
agc_driver_sample_order_grid_reference: skipped, the device does not keep depth drawn at custom sample locations at those locations, even after a transition that names them: the composite of uniform renders holds the guest's depth in 0 of 3005 sloped samples, so no path can be compared with it bit for bit
agc_driver_scattered_runs: skipped, host imports are unavailable: /dev/udmabuf is not accessible
agc_driver_sparse_dma_buf_import: skipped, host imports are unavailable: /dev/udmabuf is not accessible
agc_driver_stencil_ref_export: skipped, the device lacks VK_EXT_shader_stencil_export
agc_driver_vertex_elected_atomic_max: skipped, the device runs vertex programs in subgroups, so they keep their cross-lane reductions
agc_host_import_sparse: skipped, host imports are unavailable: /dev/udmabuf is not accessible
agc_import_settle_initial_dcc: skipped, the device has no deferred settle of host imports: imports under 1 MiB wait for one only on Windows
agc_unit_shadow_completion_store: skipped, the device reports watched host memory written when it imports or reads it
guest_http_uri_locale: 
guest_net_ctl: skipped, the host has no IPv4 address
guest_video_out_color_space: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_hdr: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_high_frame_rate: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_output_mode: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_resolution_status: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_wqhd_detection: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
video_out_flip_aftergpu: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_backtoback: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_controls: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_cpuflip: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_cpuflip_in_order: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_dcc: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_deferred: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_eop: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
vide
Reproduce
git fetch origin 6330546a47e6c4afda70b640f338678e6c9601a5 1c8c0efc8c41314402e6966da5bef6ea3d639d10
git checkout 1c8c0efc8c41314402e6966da5bef6ea3d639d10 && git merge --no-ff 6330546a47e6c4afda70b640f338678e6c9601a5
git submodule update --init --recursive
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=ON -DANYPS5_ENABLE_SPIRV_TOOLS=ON
cmake --build build && cmake --build build --target libs
VK_DRIVER_FILES=/nonexistent/icd.json ctest --test-dir build -E '^(agc|shader|recompiler|videoout|video_out|guest_video_out|vulkan)|gpu' --timeout 120 --parallel 32 --output-on-failure
ANYPS5_REQUIRE_VULKAN=1 ANYPS5_NO_SHADER_CACHE=1 VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation \
  ctest --test-dir build -R '^(agc|shader|recompiler|videoout|video_out|guest_video_out|vulkan)|gpu' --timeout 120 --parallel 4 --output-on-failure
# failures are then re-run alone: --timeout 360 --parallel 2; new validation IDs: the test alone, twice, --parallel 1

Automated build and test on a contributor's NVIDIA machine; it does not replace upstream CI. A new comment is posted for each new push. Full logs.

@TripyFox

TripyFox commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

A side finding while profiling queue 0 at the ASTRO BOT Crash Site (PPSA21567 01.019.000, -lvl hub_crashsite). With relocation on, the finisher rarely overlaps queue 0. This is an efficiency note, not a perf claim: it doesn't change the frame time on this scene (see below).

Where. On head 6330546a47, Draw.cpp:

  • Line 408 runs if (DrawFinisher::Busy() && drawRelocationPending.key != 0) DrawFinisher::Get().Drain();.
  • drawRelocationPending is set at line 447 on every lookup without an entry whose draw has a relocation identity. That is about 95 % of draws in this scene: about 2100 of the ~2200 queue-0 draws per frame are data-only hits with a new key, because the vertex fetch-table and constant-buffer pointers in user data move every frame.
  • So queue 0 drains the finisher before almost every draw's lookup.

Counters. APS5_PIPELINE_STATS=2, 10 s windows, about 230k draws handed off per window. The lookup-site counters only see what is left after line 408:

run lookup drained lookup skipped
default 0 7.3k-7.6k
APS5_NO_DRAW_RELOCATION=1 106-271 219k-223k

With relocation on, line 408 has already drained the finisher by the time the lookup checks it, so the finisher is almost never still busy there. With relocation off, the two really run alongside each other.

Why the drain is there. As far as I can tell, admitDrawKey (DrawCache.cpp:120) and learnPendingRelocation read the single global drawRelocationPending. It pairs with the right draw only if the finisher has finished the previous draw before queue 0 overwrites the record. Carrying the pending record in the finisher job, and learning it after that job's insert, would remove the per-draw drain with relocation on.

Frame time. Here it makes no difference, so this doesn't block anything. I ran main b11c7bf84f against main + this head, interleaved, 240 s each, APS5_TRACE_SUMMARY_EVERY=10 APS5_TRACE_BENCH=1, warm cache, RTX 5090 / Ryzen 7 9800X3D / Windows 11:

pair main p50 + this PR p50 delta
1 90.12 89.49 −0.63
2 90.24 93.08 +2.84
1, both arms with APS5_NO_DRAW_RELOCATION=1 93.04 92.74 −0.30
2, both arms with APS5_NO_DRAW_RELOCATION=1 93.01 94.98 +1.97

That is neutral within noise, with relocation on or off. On this scene the frame is held by GPU time on main (about 72-75 ms per present) and by the draw committer once #15569 and #9712 are applied. So freeing queue 0 doesn't show up yet. Once the committer gets cheaper, the 16-25 ms per frame of queue-0 prepare work here becomes the limit, and the overlap would matter.

AI-assisted: yes (Claude Code)

@LeoFeitosa

Copy link
Copy Markdown
Contributor

Another ASTRO BOT data point, in a heavier scene than the hub run above: the opening desert level, where on main queue 0's worker waits on the committer for room. APS5_DRAW_PIPELINE=1 on main b11c7bf, so the switch this PR turns on, not the PR head.

Setup. PPSA21564 (ASTRO BOT 01.007.000), opening level from a new game, scripted pad input, 3840x2160. Windows 11 10.0.26300, RTX 5070 Ti (NVIDIA 621.01), Ryzen 9 7900X3D, WinLibs GCC 15.2.0 Release. One binary, four runs alternating on/off/off/on, a 20 s trace window at 1080 s (APS5_TRACE_SECONDS=1080:20), no APS5_PROFILE_*.

run presents / 20 s frame p50 p90 p99 GPU busy per present FPS, minutes 20-24
on 1 241 82.8 86.7 90.3 63.4 14.2
off 1 216 92.6 99.4 111.0 68.0 13.5
off 2 228 85.8 91.6 98.3 63.0 13.6
on 2 209 94.3 101.9 120.0 71.0 13.4

(ms) The two arms are even here: p50 88.6 ms with the finisher, 89.2 ms without. The finisher took the draws (Driver.DrawFinish 27 and 36 ms per frame, Driver.Draw on the worker 28 and 38 ms against 42 and 43 ms), but each run's p50 follows its GPU busy time, so in this scene the frame is bound by the GPU and the committer rather than by the worker's half of the draw. None of the runs logged a FATAL or a device loss.

From a separate profiled run on the same build (APS5_PROFILE_DRAW=1, which keeps every draw on the worker), the committer is the single thread at the end of the chain: [draw] pipelined (10 s, depth 8): 213163 commits at 34.2 us each on the committer, with the worker waiting 19.5 us per draw for room.

@JJenkx

JJenkx commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

Windows data on Goat Simulator 3 (PPSA04158), head 6330546a47 merged onto main b11c7bf84f (460f8c3fa1). This is not a speed claim: only one pair landed cleanly on comparable scenes.

Setup:

  • Windows 11 Enterprise 22621, Ryzen 9 5950X, AMD Radeon RX 6700 XT (AMD proprietary driver 26.10.1), 3840x2160, WinLibs GCC 15.2 UCRT, Release.
  • Both trees were built from one build directory, with only the libraries swapped.
  • Each run is a scripted route into the open world, 600 s, with APS5_TRACE_SECONDS=450:4 only.
  • Runs alternated main, PR, main, PR through the shared FIFO, one at a time.

Trace windows (450 s + 4 s):

Run Window starts at present frame p50 / p90 queue worker 0x0 busy draw finisher busy GPU busy per present
main 1 2773 239.9 / 269.6 ms 211.3 ms (85 %) — 112.9 ms
PR 1 — crashed at 185 s, before the window
main 2 3099 183.4 / 184.8 ms 148.6 ms (80 %) — 101.0 ms
PR 2 3132 165.0 / 171.0 ms 121.6 ms (72 %) 85.0 ms (51 %) 92.5 ms
main 3 — crashed at 195 s, before the window
PR 3 3136 180.3 / 190.3 ms 136.7 ms (75 %) 96.2 ms (53 %) 99.3 ms
main 4 2893 185.8 / 195.1 ms 153.7 ms (81 %) — 105.7 ms
PR 4 — crashed at 178 s, before the window
  • What I read from this:
    • On main, queue worker 0x0 bounds this scene: it is about 80 % busy, Driver.Packet.draw takes about 109 ms per frame, and the GPU is 55 % busy.
    • With this PR, the draw finisher takes 85–96 ms per frame off the worker, and the worker's busy time falls by roughly 15–30 ms per frame.
    • The only clean adjacent pair (main 2 and PR 2, windows starting at presents 3099 and 3132) gives p50 183.4 vs 165.0 ms. PR 3 (180.3 ms) is close to main 2 and 4 (183.4 and 185.8 ms).
  • What it does not show:
    • The scene at a fixed wall time drifts between runs: main 1 started its window at present 2773 and ran 239.9 ms.
    • So this is directional, and a frame-time claim would need more clean pairs.

Stability (same route, 600 s cap, Goat crashes on main too):

Tree Runs Crashed
this PR 8 2: FAsyncLoadingThread access violation at 185 s; guest libc read fault at 178 s
main (same session) 8 1: FAsyncLoadingThread access violation at 195 s
  • Today's main runs on this route crash on that same loading thread at other addresses too, in 3 of 17 runs overall.
  • So I don't attribute the FAsyncLoadingThread crash to this PR.
  • The guest libc read fault (libc.prx.guest.prx+0x13e61, frame app.exe+0x7edba4) is the only signature I have not also seen on main. One occurrence is not enough to say more.
  • The four stability runs of this PR after those two all reached the 600 s cap.

AI-assisted: yes.

@JJenkx

JJenkx commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Follow-up to my earlier comment: four more alternating main / PR pairs on the same Goat Simulator 3 route and setup, still at head 6330546a47 on main b11c7bf84f.

All trace windows (APS5_TRACE_SECONDS=450:4). The pairs are numbered as run, in alternation:

Pair main: window start, p50 PR: window start, p50 queue worker 0x0, main / PR draw finisher (PR)
1 2773, 239.9 ms crashed at 185 s 211.3 / — ms —
2 3099, 183.4 ms 3132, 165.0 ms 148.6 / 121.6 ms 85.0 ms
3 crashed at 195 s 3136, 180.3 ms — / 136.7 ms 96.2 ms
4 2893, 185.8 ms crashed at 178 s 153.7 / — ms —
5 2679, 189.3 ms 3062, 184.1 ms 155.7 / 134.8 ms 96.8 ms
6 3083, 178.5 ms 2828, 229.1 ms 143.5 / 187.1 ms 142.6 ms
7 3133, 173.5 ms crashed at 187 s 138.3 / — ms —
8 3020, 184.0 ms 2866, 179.7 ms 150.6 / 137.4 ms 99.3 ms

Frame time:

  • The PR was faster in 3 of the 4 pairs where both windows were valid, by 4 to 18 ms. It was slower in pair 6, whose window opened at an earlier, heavier point of the route.
  • Medians over the valid windows: main 184.0 ms (7 windows), the PR 180.3 ms (5 windows).
  • The window's start point moves p50 by more than the tree does, so I still do not read a frame-time difference into this.

Queue worker 0x0: consistently less busy with the PR, about 10–15 % at comparable p50, as the finisher takes 85–99 ms per frame (143 ms in pair 6's heavier window).

Stability: all of today's runs on this route on b11c7bf84f.

Tree Runs Crashed When and where
this PR 12 3 178 s, 185 s and 187 s, during the world load: two FAsyncLoadingThread access violations and one guest libc read fault
main 23 3 169 s APR: unknown opcode 0; 195 s and 288 s, both FAsyncLoadingThread access violations
  • 3 of 12 vs 3 of 23 is not a significant difference (Fisher two-sided p ≈ 0.37), and main crashes on the same loading thread.
  • All three PR crashes landing between 178 and 187 s is the one pattern I would watch if others see crashes during loading with the finisher on.

AI-assisted: yes.

@JJenkx

JJenkx commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

A diagnostic data point on which part of this PR does what, from the same Goat Simulator 3 route on Windows (RX 6700 XT).

ShaderResources::EpochProofs() is DrawPipelineEnabled(), so turning the finisher on also enables reuse of resource and storage-image proofs within a collect epoch. To separate the two, I built main b11c7bf84f with EpochProofs() also true under a local APS5_EPOCH_PROOFS switch. This is a diagnostic only, not a proposal. I compared one full 600 s run with and one without the switch, both with APS5_PROFILE_DRAW=1, on the same binary with the finisher off.

In-world 10 s windows proofs off proofs on (no finisher)
draw average 51–54 µs 49–50 µs
readTarget per draw 5.9–7.6 µs 4.2–5.5 µs
pages walked per target lookup ~80 ~80
storage refreshes reused / resource proofs reused — 5.0 M / 1.33 M over the run
  • What the proof reuse is worth by itself: about 4 % of the worker's per-draw time on this route. The resident colour-target lookup still collects about 80 pages per draw, so most of that collect is outside the reused path.
  • Where this PR's larger worker reduction comes from: most of the 15–30 ms per frame in my earlier comments therefore comes from moving the second half of each draw to the finisher, not from the proof reuse.
  • Verification: APS5_VERIFY_DRAW_PIPELINE was not set in these runs, so they do not check the reused proofs against a full check.

AI-assisted: yes.

@oneandonlydean oneandonlydean self-assigned this Nov 6, 2026
@milescoviello

Copy link
Copy Markdown
Contributor

A data point from Metal Gear Solid 4 (PPSA18194), which the description lists: its title screen (the cemetery tree, about 840 draws per frame, capped at 60 fps by the title's own vblank wait). Linux, RTX 4070 Ti, NVIDIA 615.71. main ab213704ee against this head merged onto it (0b99df7bb5, clean), no other switches, two runs each, alternating main/PR/main/PR, medians of the 10 s summary windows from 30 to 120 s at the screen:

run fps queue worker 0x0 busy Driver.Packet ms/frame Driver.Packet.draw ms/frame Driver.DrawFinish ms/frame
main 1 59.9 79.5% 13.3 10.2 -
change 1 59.9 92.6% 15.6 9.6 8.0
main 2 59.9 95.0% 15.9 10.2 -
change 2 59.9 82.7% 14.0 10.5 8.7

At this scene it's neutral within run-to-run noise. The finisher takes 8-9 ms per frame, but the worker's draw phase stays at 10 ms, so the worker doesn't get that time back here. It either waits on the finisher or the split adds work, and the 60 fps cap hides any throughput change. The gameplay scenes (Act 1, 25-45 fps on main) are where it can show. An earlier hand-played session there with APS5_DRAW_PIPELINE=1 APS5_PIPELINED_DRAWS=32 APS5_PIPELINE_DISPATCH=1 went from the mid 40s to the high 50s, but that wasn't a paired run, so I'll do a paired Act 1 run with this head next.

AI-assisted: yes (Claude Code)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agc libSceAgc* build docs perf Performance change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants