Repository navigation
feat(agc): capture and replay frames on Linux - #11157
Conversation
|
|
Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.
Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.
|
|
I checked the overlap bot warning against the current heads of #11157 and #10760. A three-way merge is clean in the production/source files but conflicts in |
|
@MarcZX10 thanks for checking. I proposed the order on #10760: either order, whichever merges second keeps both debt entries, no There is a second overlap that |
Remove the extra blank line between capture trace output and frame capture. The independent changes in boykopovar#11157 may merge in either order; the second branch keeps both entries when rebasing. AI-assisted: Codex
|
Thanks for this. We have been running Demon's Souls captures through
None of these change the driver's behaviour outside the replay tool, except the recorder's alias mirroring (2) and some |
Remove the extra blank line between capture trace output and frame capture. The independent changes in boykopovar#11157 may merge in either order; the second branch keeps both entries when rebasing. AI-assisted: Codex
ae44408 to
9aad99b
Compare
|
@ponhei9502 thanks, and good to hear the replay works on a title that size. I'd like to keep #11157 to the Linux port, so I'd prefer these as follow-up PRs after it lands rather than rebased in. A few notes from going through the commits:
|
|
@janzezno a Linux NVIDIA and lavapipe data point from a large 3D title. I tested ae44408; the force-push to 9aad99b changes only one debt line, so the code is the same. Setup: RTX 3080 Ti with NVIDIA 615.78.08, and lavapipe (Mesa 26.2.4, LLVM 23.1.1). Linux 7.3 (CachyOS), i9-11900K, 32 GB. Focused tests ( Ghost of Yōtei (PPSA26344, 01.008.000). The title needs local commits to run here, so I cherry-picked this PR's commit onto that build. Only
Replay (
Every run replayed 10 frames with
AI-assisted: yes |
|
@janzezno a follow-up on the I replayed the same capture 8 more times on the RTX 3080 Ti, alternating default and
Together with the first three runs, that's 4 of 6 exact with the default and 4 of 5 with A lavapipe run with AI-assisted: yes |
Remove the extra blank line between capture trace output and frame capture. The independent changes in boykopovar#11157 may merge in either order; the second branch keeps both entries when rebasing. AI-assisted: Codex
9aad99b to
c867473
Compare
|
Rebased onto
The recorder and replay tool code are unchanged by the rebase. Details are in the updated body. @Dextroid17 thanks for the Ghost of Yōtei runs and for following up on |
|
Linux follow-up: successful fixture qualification, but a native ArcRunner capture stops before completing a frame. I started from your previous head Your PR advanced to
Native limitation: ArcRunner PPSA21406 reached its saved doorway scene. A two-frame capture triggered at approximately316s started at boundary1529 and then stopped: The header records 0 completed frames,36 calls,9,954,955,264 raw snapshot bytes and152,387,584 delta bytes. After frame-start, the call stream contains35 submissions, one suspend and36 memory groups. The last recorded submit has queue32; the metadata tail is included. This identifies the failing recorder wait but does not establish its root cause or a Linux-specific regression. I did not replay the incomplete capture. The title continued rendering after recording stopped and closed normally at430s, exit0, with saves/configuration/kernel limit restored and no remaining title process. Native run: two actual Khronos insertions and no VUIDs. No FPS claim. Focused logs, exact private integration patch, historical quoting fix and capture metadata — SHA-256 AI-assisted: yes. |
|
@JJenkx thanks, and for checking the new head against what you tested. The stop comes from the wait #11197 added to So ArcRunner hits a limitation of the recorder on both platforms. It doesn't show a problem in this PR. Finding which wait the queue was blocked on would need the serial tracing and |
|
Thanks, agreed that the drain is the cross-platform recorder limitation, not the Linux port. I reduced it to a small game-independent two-queue reproducer, so further work does not require ArcRunner or its9.95GB snapshot.
Same driver ( Thus the limitation also covers a wait released by a later submission on another queue, without a later CPU store to the label: the recorder blocks that producer's API call before its acceptance. I also inspected the last ArcRunner submit offline: at dword32, it contains a32-bit Portable reproducer source, Ninja/CMake build helper, instructions and two paired logs — SHA-256 AI-assisted: yes. |
Remove the extra blank line between capture trace output and frame capture. The independent changes in boykopovar#11157 may merge in either order; the second branch keeps both entries when rebasing. AI-assisted: Codex
…g, reservation errors --cold-pipelines now clears an inherited APS5_HEADLESS_PIPELINE_CACHE on both platforms, the replay test builds its command with boykopovar#11157's quoting so the redirect and --repeat work under sh and cmd, and a reservation that fails in a repeated pass is reported and fails the replay instead of being dropped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e title A memory delta was read only once every accepted submission completed, so a capture stopped after 10 s whenever a queue waited on a REWIND control word or a label a later call stores, as in the two-queue reproducer on boykopovar#11157 and the Ghost of Yotei capture on boykopovar#11197. The driver gains a settled hold: AgcDriverHoldSettled succeeds once every queue has finished or is parked on a REWIND or WAIT_REG_MEM value the title has not stored, with no queued draw, unrecorded label or pending label write, and keeps submissions and parked queues where they are until AgcDriverReleaseSettled. The recorder reads each delta under that hold through the coherent path. The replay restores each delta under the same hold. Its breaker, built on boykopovar#12318, releases a parked REWIND by applying the first later memory group that sets its control word, only under a settled probe, one wait per pass, and no longer writes a value from a later delta for any other wait, which applied a later CPU store early. Pacing counts a running queue as progress and gives up after --pace-timeout-ms only once no queue runs, or after the busy timeout. The drain also ends after 10 s without a completion once no queue runs, and the busy timeout, ten minutes, is shared by pacing, the settled hold and the drain. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Remove the extra blank line between capture trace output and frame capture. The independent changes in boykopovar#11157 may merge in either order; the second branch keeps both entries when rebasing. AI-assisted: Codex
What
APS5_FRAME_CAPTUREandagc_frame_replayonly work on Windows. On Linux a capture throws at its first frame boundary withframe capture: guest memory capture is implemented on Windows only, so a title that misrenders or hangs on Linux can't be captured and replayed offline. Everything else the recorder uses already runs on Linux (the write watch,ReadCommitted, the guest arena and the allocation registry). The missing parts were the VirtualQuery walk inFrameRecorder::committedRangesand the VirtualAlloc calls in the replay tool. This change adds Linux versions of those and runs the capture tests on Linux.Recorder (
committedRanges, Linux branch). It walks each readable registry allocation's VMAs with a newHostMapping::Walk(PROCMAP_QUERYwhen the kernel has it, else one/proc/self/mapspass).HostMapping::Mappinggains the r/w/x/shared bits and the inode, filled on both paths. Classification follows the Windows branch:Host.PROT_NONEVMA is the Linux form of reserved and uncommitted memory (the arena reserves and resets withPROT_NONE), so it is left out.Watched, other private anonymous memory isUnwatched, and the rest (shared and file-backed) isSection. One difference from Windows: on Linux libkernel registers direct memory views with the write watch (AddMapping→WatchMapping). Those views are watched, so CPU stores to direct memory during the frames are recorded, which Windows section views can't do.protectionstores the WindowsPAGE_*value of the VMA bits. That is the protection vocabulary the arena API already uses on Linux, so the field means the same thing in captures from either host.radv-shader-cache/.../indexfile that way, and the first Linux capture died inReadCommitted./proc/self/map_filesneedsCAP_SYS_ADMIN, so the file size can't be read from there. Instead the recorder probes pages withprocess_vm_readvon its own process (a page past the end givesEFAULT, not a signal), binary-searches the boundary and records the pages past it as an unreadable range. Any other probe error (for exampleEPERMunder a seccomp or ptrace policy) fails the capture instead of dropping memory.APS5_NO_WRITE_WATCH, or a host without one), on both platforms. Without the watch a capture used to start and silently miss every CPU store during the frames.Replay tool.
committedWritable: Linux version throughHostMapping::Walk.mapHostRanges: maps page-aligned envelopes withMAP_FIXED_NOREPLACE. Envelopes merge only where two ranges share a page, so an occupied CPU range doesn't block a free GPU range next to it. Windows merges touching 64 KiB envelopes because of its allocation granularity; Linux doesn't need to.0x200538020). On Linux the arena commit is aMAP_FIXEDmmap, which failed withEINVAL. An unaligned range now has its pages made writable withGuestArenaSetProtection; the exact bytes are still marked used. Windows keepsGuestArenaCommit.agc_driver_frame_replay_shared_pagetest fails with each half of this reverted:guest arena commit failed: Invalid argument, anda range sharing a page with an earlier one was skipped.Tests.
agc_driver_frame_capture_eopand thezero,shaderandshader_hostreplay-state modes are registered on Linux, plus new modes:unwatched(Windows and Linux; a capture withAPS5_NO_WRITE_WATCH=1must throw), and the Linux-onlyshared_pageandhost_neighbors(an occupied CPU host range next to a free GPU host range must skip only the CPU range). Each new mode fails with its fix reverted:a capture started without the write watch,frame replay: cannot map the GPU-visible host range at .... On Linux the write watch covers only ranges someone registers (libkernel does for title memory), while Windows watches arena commits automatically. So the two tests that commit arena memory directly register it withGuestWriteWatchRegisteron Linux.tests/FrameReplay.cppwrapped itsstd::systemcommands (the capture-and-replay case and theconcurrentcase #11197 added) in an extra pair of quotes forcmd.exe, whichshreads as one word; that pair is now added on Windows only.#10711. #11197 fixed the recorder race and lifted the Windows quarantine, so
agc_driver_frame_replayandagc_driver_frame_replay_concurrentare now registered on Linux next to Windows, with no quarantine option. Before #11197, the Linux build ofagc_driver_frame_replayfailed 24 of 50 runs pinned withtaskset -c 0,1(the replay did not reproduce the second copy); on this rebase both tests pass 50 of 50 pinned to two cores, to one core and unpinned.Debt and
BUILD.mdupdated: Linux support, watched direct memory views, the no-access and past-end-of-file cases. macOS still throws.Overlap:
pr_overlapreports #10760, #11435 and #11436 (below). Of the 82 others it could not compare, most share only the libSceAgcDriverCMakeLists.txtor debt file. One changes the same code: #7201 (macOS port) putsHostMapping::Findunder#if defined(__APPLE__)and keeps the Linuxreturn Mapping{...}line in its#else, which this change turns intoreturn FromQuery(query, name). That is a one-line conflict for whichever merges second, and its macOSMapping{begin, end, name}still compiles with the new defaulted fields. #10540 adds anextern char** environ;next to the<unistd.h>include ofFrameRecorder.cpp, a line away from the include added here.The reported overlaps, and the recorder fix this change waited for:
docs/dev/debt/libSceAgcDriver.mdconflicts. Order agreed in fix(agc): keep capture trace output nonfatal #10760: independent, noDepends on, whichever is ready first merges; the second rebases and keeps both entries, "Capture trace output" right above "Frame capture".std::systemcall of the capture-and-replay case intests/FrameReplay.cppinto areplaylambda that keeps thecmd.exeouter quotes and adds--repeat 3and an output redirect. That is a textual conflict on the same lines, and with both merged the Linux test would fail on the outer quotes unless the lambda adds them on Windows only. Order proposed in feat(agc): replay full titles with repeated passes and the title's pipeline cache #11435: either order, noDepends on; whichever merges second makes the lambda's outer quotes Windows-only.Tested
Arch Linux 7.2.11, AMD Radeon RX 7900 GRE (RADV NAVI31, Mesa 26.2.5), Ryzen 5 7500F, Release, on
main7903201bb4.Focused tests with
ANYPS5_REQUIRE_VULKAN=1 ANYPS5_NO_SHADER_CACHE=1 VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation,ctest -R 'agc_frame_capture|agc_driver_frame_(capture_eop|replay)' -V:main: 4/4 pass (agc_frame_captureand the replay modesorder,tokensandblocking; the others are not registered on Linux), 0VUID-, no missing-layer line.agc_frame_capture,agc_driver_frame_capture_eop, replay modesorder,tokens,blocking,zero,shader,shader_host,unwatched,shared_page,host_neighbors,agc_driver_frame_replay,agc_driver_frame_replay_concurrent), 0VUID-, no missing-layer line. The replay tests copy with DMA and don't depend on the Vulkan device.agc_driver_frame_replayandagc_driver_frame_replay_concurrent, 50 runs each undertaskset -c 0,1,taskset -c 0and unpinned: 0 failures in 300.ctest -R 'guest|sysmodule|module|unwind|frame' -j6(819 tests, covering theHostMappingusersGuestMemory,sceKernelGetModuleInfoForUnwindandsceSysmoduleGetModuleInfoForUnwind): all pass, 2 skipped (video_out_frame_dump_queue,video_out_frame_dump_wait).On the pre-rebase head
9aad99ba3c(mainae7d59258e; the rebase leaves the recorder and replay tool code unchanged):ccleste PPSA59322 1.00 (homebrew, MIT, sha256
96861b81…), relinked with default options, title screen, no input,APS5_FRAME_CAPTURE_AT_FLIP=300 APS5_FRAME_CAPTURE_FRAMES=10:main: aborts at frame boundary 300 withterminate called after throwing an instance of 'std::runtime_error'/what(): frame capture: guest memory capture is implemented on Windows only.[frame-capture] started at frame boundary 300: 111 ranges, 163.0 MiB snapshot in 0.2 s/10 frames, 32 calls, 33 memory deltas (6.6 MiB), 264.5 MiB raw, 28.0 MiB stored,unreadable_bytes24576 (the shader cache index pages past the end of its file), no warnings.agc_frame_replay capture --frames out:{"frames":10,"replay_s":0.133,"fps":75.412,"frame_ms":{"p50":2.782,"p90":7.751,"max":7.751,"each":[0,7,7,0,2,3,2,2,7]},"submits":11,"delta_mib":6.6,"pace_timeouts":0,"pace_wait_s":0.333,"stale_registrations":0,"failed_registrations":0,"skipped_mib":0.0,"threads":3,"released_waits":0,"of_them_from_earlier_writes":0,"of_them_synthesized":0,"unfinished":0,"frames_compared":3,"pixels_identical":1.000000,"max_channel_diff":0,"load_s":0.032}All three display buffers match their reference images pixel for pixel; the replayed title screen is attached.
Not tested: Windows. The Windows recorder walk and the Windows commit path are unchanged; the write-watch check at capture start and the
unwatchedtest also run there. The ownership decision before commits also applies on Windows and changes behaviour only for two ranges in one page.tests/FrameReplay.cppkeeps its outer quotes on Windows.Checklist
main; no other open PR implements the same functionsNotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file.mdfiles or images added to the repository (attach them to this PR)