Skip to content

fix(agc): bound a journal replay's graphics and compute pipelines apart - #8542

Merged
oneandonlydean merged 1 commit into
boykopovar:mainfrom
milescoviello:fix/journal-replay-limit-per-kind
Nov 2, 2026
Merged

oneandonlydean merged 1 commit into
boykopovar:mainfrom
milescoviello:fix/journal-replay-limit-per-kind

Conversation

@milescoviello

@milescoviello milescoviello commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

What

A journal replay keeps the last APS5_PIPELINE_JOURNAL_ENTRIES entries (4096) of the loaded journal, the newest since #9836. That bound is there for the graphics pipeline store, which #9401 sizes to the title's working set. But the replay counts compute entries against it too, and truncates silently. So in a title that records both kinds, compute pipelines take graphics slots, and whatever the bound cuts goes unreported.

This PR applies APS5_PIPELINE_JOURNAL_ENTRIES (4096) to graphics entries only. Compute entries count separately against a new APS5_PIPELINE_JOURNAL_COMPUTE_ENTRIES (1024), as agreed earlier in this thread. Each kind keeps its newest entries, as #9836 does for both together, and the recorded order is kept. When a replay leaves entries out, it says so, for example:

[pipeline-journal] replaying the newest 4096 of 5300 graphics and 1024 of 1100 compute pipelines (APS5_PIPELINE_JOURNAL_ENTRIES, APS5_PIPELINE_JOURNAL_COMPUTE_ENTRIES)

ENVIRONMENT.md documents the new variable, and the journal debt entry names both bounds. The record format doesn't change, so the journal keeps #8365's format number.

Why

@SleglJan found this with #8365. On PPSA11386, the default-cap seed held 1,322 entries: 628 graphics, 694 compute and their modules. The warm run replayed exactly the first 1,024 (481 graphics, 526 compute, 17 already present), and the load line still said 1,322.

Tested

Restacked onto main 77d536fb38, which now has #8365, as ee0d52b98f; it also merges cleanly with current main 31c727f622. The change is the same commit, replayed alone. The per-kind test from before is unchanged.

Linux (Ubuntu 26.04.1, GCC 15.2), RTX 4070 Ti (NVIDIA 615.71.09), Release: ctest -R pipeline_journal (agc_driver_pipeline_journal, which holds the per-kind replays, agc_pipeline_journal and _long_paths) passes 3 of 3 on ee0d52b98f.

Khronos validation (VK_LAYER_KHRONOS_validation 1.4.363, LunarG SDK) over the same 3 tests, on the base 77d536fb38 and on ee0d52b98f: 3 of 3 pass on each, the tests created 17 and 20 Vulkan devices that the layer checked, and there is no VUID- message on either side.

Windows 11 VM (GT 1030, R580, WinLibs GCC 15.2.0, Release): agc_driver_pipeline_journal (the per-kind replays), agc_pipeline_journal, _long_paths and _extended_paths pass 3 of 3 on the base 77d536fb38 and 3 of 3 on ee0d52b98f.

Earlier results, on the heads named in them (before the restack):

  • Mutants with this PR's tests: main's single newest bound fails 3 of 3 on the bounded-replay case (a replay bounded to 1 graphics and the default compute entries replayed 1 graphics and 0 compute pipelines). A bound that keeps the first entries of each kind fails 3 of 3 in the newest-entries test.
  • @bunbeans2-arch's RX 9070 XT run on Windows of cb2ba99693.

Not tested: a title run.

Checklist

🤖 Generated with Claude Code

@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request Shared files not compared
PR 1896 docs/dev/ENVIRONMENT.md
PR 4472 docs/dev/ENVIRONMENT.md
PR 5925 core/libs/prx/libSceAgcDriver/Graphics/src/PipelineJournal.cpp
core/libs/prx/libSceAgcDriver/tests/execution/PipelineJournal.cpp
docs/dev/debt/libSceAgcDriver.md
PR 6186 docs/dev/ENVIRONMENT.md
docs/dev/debt/libSceAgcDriver.md
PR 6194 docs/dev/debt/libSceAgcDriver.md
PR 6223 docs/dev/debt/libSceAgcDriver.md
PR 6273 docs/dev/debt/libSceAgcDriver.md
PR 6280 docs/dev/debt/libSceAgcDriver.md
PR 6300 docs/dev/debt/libSceAgcDriver.md
PR 6412 docs/dev/debt/libSceAgcDriver.md
PR 6481 docs/dev/debt/libSceAgcDriver.md
PR 6501 docs/dev/ENVIRONMENT.md
PR 6548 docs/dev/debt/libSceAgcDriver.md
PR 6808 docs/dev/debt/libSceAgcDriver.md
PR 6924 docs/dev/ENVIRONMENT.md
PR 6977 docs/dev/debt/libSceAgcDriver.md
PR 7048 docs/dev/debt/libSceAgcDriver.md
PR 7256 docs/dev/debt/libSceAgcDriver.md
PR 7449 docs/dev/ENVIRONMENT.md
docs/dev/debt/libSceAgcDriver.md
PR 7457 docs/dev/ENVIRONMENT.md
PR 7514 docs/dev/ENVIRONMENT.md
PR 7604 docs/dev/ENVIRONMENT.md
docs/dev/debt/libSceAgcDriver.md
PR 7642 core/libs/prx/libSceAgcDriver/Graphics/src/PipelineJournal.cpp
core/libs/prx/libSceAgcDriver/tests/execution/PipelineJournal.cpp
docs/dev/ENVIRONMENT.md
docs/dev/debt/libSceAgcDriver.md
PR 7978 docs/dev/debt/libSceAgcDriver.md
PR 8004 docs/dev/ENVIRONMENT.md
PR 8484 docs/dev/ENVIRONMENT.md
PR 8592 docs/dev/debt/libSceAgcDriver.md
PR 8707 docs/dev/debt/libSceAgcDriver.md
PR 9012 docs/dev/ENVIRONMENT.md
PR 9020 core/libs/prx/libSceAgcDriver/Graphics/src/PipelineJournal.cpp
docs/dev/debt/libSceAgcDriver.md
PR 9248 docs/dev/ENVIRONMENT.md
PR 9335 core/libs/prx/libSceAgcDriver/Graphics/src/PipelineJournal.cpp
docs/dev/debt/libSceAgcDriver.md
PR 9345 docs/dev/ENVIRONMENT.md
PR 9356 docs/dev/ENVIRONMENT.md
docs/dev/debt/libSceAgcDriver.md
PR 9387 docs/dev/ENVIRONMENT.md
docs/dev/debt/libSceAgcDriver.md
PR 9432 docs/dev/debt/libSceAgcDriver.md
PR 9592 docs/dev/debt/libSceAgcDriver.md
PR 9712 docs/dev/ENVIRONMENT.md
PR 9852 docs/dev/ENVIRONMENT.md
PR 9915 docs/dev/debt/libSceAgcDriver.md
PR 9950 docs/dev/debt/libSceAgcDriver.md
PR 10069 docs/dev/ENVIRONMENT.md
PR 10233 docs/dev/debt/libSceAgcDriver.md
PR 10249 core/libs/prx/libSceAgcDriver/Graphics/src/PipelineJournal.cpp
PR 10256 docs/dev/debt/libSceAgcDriver.md
PR 10359 docs/dev/debt/libSceAgcDriver.md
PR 10442 docs/dev/ENVIRONMENT.md
PR 10547 docs/dev/ENVIRONMENT.md
PR 10757 docs/dev/ENVIRONMENT.md
PR 10894 docs/dev/ENVIRONMENT.md
PR 11351 docs/dev/ENVIRONMENT.md
PR 11551 docs/dev/debt/libSceAgcDriver.md
PR 12120 docs/dev/ENVIRONMENT.md
docs/dev/debt/libSceAgcDriver.md
PR 12323 docs/dev/ENVIRONMENT.md
PR 12324 docs/dev/ENVIRONMENT.md
PR 12544 docs/dev/ENVIRONMENT.md
PR 12746 docs/dev/ENVIRONMENT.md
PR 12863 docs/dev/ENVIRONMENT.md
PR 14212 docs/dev/debt/libSceAgcDriver.md
PR 14299 docs/dev/ENVIRONMENT.md

@github-actions github-actions Bot added build docs agc libSceAgc* needs rebase Conflicts with main, rebase needed labels Oct 20, 2026
@milescoviello
milescoviello force-pushed the fix/journal-replay-limit-per-kind branch from 62171b0 to de02531 Compare October 20, 2026 19:39
@milescoviello

Copy link
Copy Markdown
Contributor Author

Restacked on #8365's rebased head (30c643b, on main 2185dbf). This commit is now de02531, with the same diff. #8365 now writes journal format 6, because #5656 took 5 on main. On this head, the journal tests (agc_driver_pipeline_journal, agc_driver_graphics, savedata_memory_journal) pass 3 of 3, and tools/check_conventions.py --base upstream/main passes.

AI-assisted: yes (Claude Code)

@github-actions github-actions Bot removed the needs rebase Conflicts with main, rebase needed label Oct 20, 2026
@milescoviello

Copy link
Copy Markdown
Contributor Author

Windows run of this head (de02531eea) and of #8365's head (30c643b8ca), which this PR is stacked on: Windows 11 Enterprise, NVIDIA GeForce GT 1030 (driver 582.66), WinLibs GCC 15.2.0 r7 (UCRT), Release, BUILD_TESTING=ON.

#8365 head this PR
agc_driver_pipeline_journal 9 of 9 pass 9 of 9 pass
time per run 1.8 to 4.6 s 2.6 to 8.0 s, and one run of 28.4 s

main's merge base 2185dbf92e also passes 3 of 3 (1.2 to 1.7 s).

The new cases write and replay journals past both limits, so the test takes about a second longer here. The 28.4 s run was the second of the first three. It didn't come back in six more runs, and the VM's disk is a qcow2 on NFS, so I take it as I/O on this machine rather than the change. I haven't confirmed that.

AI-assisted: yes (Claude Code)

@milescoviello

Copy link
Copy Markdown
Contributor Author

Restacked on #8365's new head a2eb629939, which is rebased onto main 275e896. This PR's commit is now ff8ec20cc1, and it applied unchanged. The journal format is 7, from #8365's commit, since #7869 took 6 on main. The 4 journal tests pass 2 of 2.

AI-assisted: yes (Claude Code)

@github-actions github-actions Bot removed the needs rebase Conflicts with main, rebase needed label Oct 21, 2026
@tibzejoker

Copy link
Copy Markdown
Contributor

GTA V (PPSA04264), Windows 11, RTX 4090, open world: the graphics store overflows on its own. It settles at about 1,270 distinct pipelines after about 3 minutes, so the 1,024 LRU bound keeps evicting pipelines that are rebuilt right after, at about 200 ms each (up to 1.6 s) under the GPU mutex.

Last 330 s of one run each:

setting freezes over 500 ms blocking builds
default 36 32.9 s
APS5_PIPELINE_STORE_ENTRIES=8192 11 2.9 s

With APS5_PIPELINE_JOURNAL_ENTRIES=100000, the whole journal (1,303 pipelines) replayed in 454 ms on 16 threads.

So the graphics journal bound is tied to a store bound that is too small for this title. I plan a separate PR that grows the store adaptively, as #7189 does for the resource cache (a ghost list of evicted keys, doubling up to 16384, with a set value still fixing it), and sizes the graphics journal bound to match. It touches the same lines of PipelineJournal.cpp as this PR, so I'll build it on top of this one if it lands first, and keep the compute bound as you split it.

AI-assisted: yes (Claude Code)

@milescoviello

Copy link
Copy Markdown
Contributor Author

Thanks @tibzejoker, those numbers make the case. At about 1,270 distinct pipelines, a 1,024-entry store evicts pipelines the title rebuilds right away, and APS5_PIPELINE_STORE_ENTRIES=8192 cuts the blocking builds from 32.9 s to 2.9 s. The replay fits easily: 1,303 pipelines in 454 ms.

An adaptive store with the graphics journal bound sized to match sounds right to me. This PR keeps the graphics and compute bounds separate, so yours only has to move the graphics one. One note for building on it: this PR is stacked on #8365 (compressed journal modules), which is on #8213, so its head carries their commits until they merge. If you base on this branch you'll carry them too. Once they land, rebasing onto main should only drop them.

AI-assisted: yes (Claude Code)

@tibzejoker

Copy link
Copy Markdown
Contributor

Follow-up: the store side is now #9401. It replays up to 16384 journal records, lets the replay grow the store instead of skipping once it is full, and grows the bound while evicted pipelines come back. In the GTA V open world, builds in play went from 80-153 to 42. It touches the same journal-bound line as this PR, so I'll rebase it onto yours if yours lands first.

AI-assisted: yes (Claude Code)

@milescoviello

Copy link
Copy Markdown
Contributor Author

Thanks @tibzejoker. 42 builds in play, down from 80–153, is a big drop. I trial-merged #9401's head with this PR: they conflict in PipelineJournal.cpp, plus the ARCHITECTURE.md, ENVIRONMENT.md and AGC debt entries. That works either way: if #9401 lands first, I'll rebase this one onto it and keep its store bound.

AI-assisted: yes (Claude Code)

@00xDisaster

Copy link
Copy Markdown
Contributor

Overlap note: #9440 changes the same bound line in PipelineJournal.cpp — it keeps the last entries of the journal instead of the first, so the pipelines a title built last are the ones the replay builds, and it rewrites the file when it reaches APS5_PIPELINE_JOURNAL_MAX_MIB instead of refusing the record (it also touches the two bound rows of docs/dev/ENVIRONMENT.md and the journal's debt entry).

The changes compose: the per-kind bounds of this PR can be applied to the entries #9440 keeps. I will rebase onto whichever lands first.

@milescoviello

Copy link
Copy Markdown
Contributor Author

Thanks @00xDisaster, agreed that they compose: this PR's per-kind bounds apply to the entries #9440 keeps. If #9440 lands first I'll rebase this onto it. I also left a review on #9440 about the record that triggers an eviction.

AI-assisted: yes (Claude Code)

@tibzejoker

Copy link
Copy Markdown
Contributor

@milescoviello #9401 changes the same line in PipelineJournal.cpp: it raises the default of APS5_PIPELINE_JOURNAL_ENTRIES from 1024 to 16384, because the graphics store can now grow to 16384 and the replay grows it instead of skipping. The two compose. With your split, that bound counts graphics entries only, which is what the store holds, and compute keeps its own APS5_PIPELINE_JOURNAL_COMPUTE_ENTRIES.

Proposed order: yours first, since it is older and smaller. I'll rebase #9401 onto it and keep your per-kind counting, so only the graphics default changes. If #9401 lands first, the conflict is that one line plus the ENVIRONMENT.md row. I'll list #8542 under Depends on in #9401 and record the order there. Does that work for you?

AI-assisted: yes (Claude Code)

@milescoviello

Copy link
Copy Markdown
Contributor Author

@tibzejoker that works for me: this one first, then #9401 on top, keeping the per-kind counting. #9401 changes only the graphics default to 16384, and compute keeps APS5_PIPELINE_JOURNAL_COMPUTE_ENTRIES.

Note that @00xDisaster's #9836 (#9440's successor) changes the same replay bound to keep the newest entries instead of the first. We agreed that either order works. If #9836 lands before this, I'll rebase this onto it, and you'd then rebase #9401 onto this as planned. Either way, #9401 only needs this PR's line and the ENVIRONMENT.md row.

AI-assisted: yes (Claude Code)

@github-actions github-actions Bot added the needs rebase Conflicts with main, rebase needed label Oct 22, 2026
@oneandonlydean oneandonlydean self-assigned this Oct 23, 2026
@oneandonlydean

Copy link
Copy Markdown
Collaborator

@milescoviello main now conflicts with this branch after #9567 (merge c9b2660) rewrote PipelineJournal.cpp. Main's version parses a std::vector<std::byte>, adds EncodeModule, and has newest-entries/eviction tests. Those overlap this PR's JournalFile, the per-kind graphics/compute limit and HeldModuleTests.

This is a logic conflict, not a mechanical one, so we'd rather you resolve it than guess. Could you merge or rebase onto main at c9b2660 or later and resolve:

  • core/libs/prx/libSceAgcDriver/Graphics/src/PipelineJournal.cpp (parse() signature, module-record encode, replay limit, load path)
  • core/libs/prx/libSceAgcDriver/tests/execution/PipelineJournal.cpp
  • docs/dev/ENVIRONMENT.md
  • docs/dev/debt/libSceAgcDriver.md

The per-kind limit should still hold with main's newest-entries eviction order. We'll re-run CI on the new head.

@milescoviello
milescoviello force-pushed the fix/journal-replay-limit-per-kind branch from 98efc1f to a45877d Compare October 24, 2026 15:25
@milescoviello

Copy link
Copy Markdown
Contributor Author

@oneandonlydean done. The stack was rebuilt on #8213's refreshed head 6fc2ea58c0, and since #8213 merged it is rebased onto main b607bfd: #8365 is 3ac931eff9 and this PR is a45877d94f (it carries #8365's commit, then this change). One correction to the merge you named: c9b2660 is #9567, Izamaru's redundant-value change, which doesn't touch the journal. The rewrite came from #9836 (00xDisaster's 62827a5, merged as ce9471f).

  • PipelineJournal.cpp: feat(agc): store the pipeline journal's modules compressed #8365 now compresses inside main's EncodeModule, which returns an empty payload when a module can't be encoded or compressed, and the batch is then skipped as before. The over-cap size check measures the compressed records the file will hold. The load path keeps main's parse() signature and RecordSpan output, and only the module decode expands the frame first. The per-kind bound now keeps each kind's newest entries in record order: it counts each kind and drops each kind's oldest entries past its bound, so with only one kind recorded it behaves like main's newest-entries bound.
  • tests/execution/PipelineJournal.cpp: fix(agc): keep the pipeline journal's newest pipelines when it is full #9836's NewestEntriesTests now sets APS5_PIPELINE_JOURNAL_COMPUTE_ENTRIES=2 (and the graphics bound to 0) for its two-newest-of-three compute replay. With the head's tests, a first-N-per-kind bound fails it 3 of 3 (the newest pipeline was not built by the replay), and main's single bound already fails this PR's bounded-replay case 3 of 3 (a replay bounded to 1 graphics and the default compute entries replayed 1 graphics and 0 compute pipelines). OversizedRecordKeepsTheJournal made its record too large with zero words, which zstd shrinks under the 1 MiB cap, so on feat(agc): store the pipeline journal's modules compressed #8365 it fills them with an LCG; without that, feat(agc): store the pipeline journal's modules compressed #8365 fails it 3 of 3 (the oversized record was not counted as a write failure).
  • ENVIRONMENT.md and the debt entry: ENVIRONMENT.md keeps main's wording of the cap, with the bound named per kind and the compute row listed first, in the table's order. The debt entry names both bounds and, to stay under 1000 characters, drops main's note that the rewrite streams kept records and the "best effort" lead-in.

Tested (Linux, Ubuntu 26.04.1, GCC 15.2, RTX 4070 Ti, 615.71.09, and the Windows 11 VM, GT 1030, 582.78):

AI-assisted: yes (Claude Code)

@milescoviello

Copy link
Copy Markdown
Contributor Author

The queued run is done: on main b607bfd, the 3 pipeline journal tests pass 3 of 3 on #8365 rebased (f5236f7fae) and on this change rebased (3b2a62f8f7). Those are the pushed heads 3ac931eff9 and a45877d94f before the comment and debt-entry edits, with the same code.

AI-assisted: yes (Claude Code)

@github-actions github-actions Bot removed the needs rebase Conflicts with main, rebase needed label Oct 24, 2026
@oneandonlydean

Copy link
Copy Markdown
Collaborator

@milescoviello thanks for the rebase, and for the correction: the journal rewrite came from #9836, not #9567. We re-reviewed both layers from scratch. #8365 (3ac931e) and #8542 (a45877d) both pass: the compression now sits in main's EncodeModule, and the newest-entries-per-kind load checks out. We'll run full CI on both heads, then the gate, and land #8365 first.

One thing at landing: both heads use journal FileFormat 11 while main is at 9, and other open PRs also bump the journal. Whichever lands takes main+1 at that point, so the number may need a one-line change just before merge.

@milescoviello

milescoviello commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor Author

@oneandonlydean thank you. I'll leave both branches as they are while CI and the gate run. When you land them, I can push the one-line FileFormat change to main+1 as soon as you say so, or you're welcome to make it at merge.

AI-assisted: yes (Claude Code)

@oneandonlydean

Copy link
Copy Markdown
Collaborator

@milescoviello Thanks. Please push the FileFormat change yourself when we tell you #8365 is next to land: main+1 at that point (main is at 9 today, so 10 unless #9348 lands first and takes it), then the same value carried into #8542 unless it adds its own record change. We'll rerun CI on that head before merging.

@github-actions github-actions Bot added the needs rebase Conflicts with main, rebase needed label Oct 24, 2026
@milescoviello
milescoviello force-pushed the fix/journal-replay-limit-per-kind branch from a45877d to cb2ba99 Compare October 24, 2026 17:54
@github-actions github-actions Bot added needs rebase Conflicts with main, rebase needed and removed needs rebase Conflicts with main, rebase needed labels Oct 24, 2026
@milescoviello

Copy link
Copy Markdown
Contributor Author

@oneandonlydean #8542 now conflicts with main 5b85225, so I'm asking before I touch it, as agreed. #8365 still merges cleanly.

The conflict is with #9401, which raised the default of APS5_PIPELINE_JOURNAL_ENTRIES from 1024 to 4096 so a replay fills the pipeline store that now grows to 4096. It conflicts in PipelineJournal.cpp (the replay bound), ENVIRONMENT.md and the journal's debt entry. My resolution:

The rebased head (main 5b85225, then #8365 as 10ec943, then this PR as 08bada3) is on my fork as rebase/8542-on-5b85225ef3. The 8 journal and pipeline store tests (journal|pipeline_store, including #9401's) pass 3 of 3 on Linux, and conventions pass.

Should I push it to this PR now, or wait until #8365 lands?

AI-assisted: yes (Claude Code)

@oneandonlydean

Copy link
Copy Markdown
Collaborator

@milescoviello I still have #8365 first. Please wait for it to land before refreshing this PR, as agreed. The graphics default of 4096 with a separate compute limit of 1024 preserves #9401's change; keep that split when restacking. #8365 needs main+1 for its format at landing, and #8542 should inherit that codec number unless it changes the record format itself. Then we can review and run CI on the final heads.

@milescoviello

Copy link
Copy Markdown
Contributor Author

@oneandonlydean understood. I'll leave this PR as it is until #8365 lands. Then I'll restack it with the 4096 graphics / 1024 compute split and #8365's format number: #8542 doesn't change the record format itself, so it inherits that number.

AI-assisted: yes (Claude Code)

@bunbeans2-arch

Copy link
Copy Markdown
Contributor

An AMD RDNA 4 run on Windows of cb2ba99693 as is, since the GPUs tested are NVIDIA. It conflicts with current main in PipelineJournal.cpp, docs/dev/ENVIRONMENT.md and docs/dev/debt/libSceAgcDriver.md, so I built the head itself (base d0e7fe7beb).

Windows 11 Pro 10.0.22631, AMD Radeon RX 9070 XT (1002:7550, driver 32.0.32015.7029, amdvlk64), MSYS2 UCRT64 g++ 16.1.0, Release. ANYPS5_REQUIRE_VULKAN=1, ANYPS5_NO_SHADER_CACHE=1, VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation; with VK_LOADER_DEBUG=layer the loader logs Insert instance layer "VK_LAYER_KHRONOS_validation" in every run of every test below that creates a Vulkan instance. 3 runs:

  • agc_driver_pipeline_journal passes 3 of 3 with pipeline journal record and replay passed.
  • The replays keep the two bounds apart, e.g. [pipeline-journal] replaying the newest 0 of 1 graphics and 2 of 4 compute pipelines and replaying the newest 1 of 1 graphics and 0 of 1 compute pipelines.
  • No VUID- line.

AI-assisted: yes (Claude Code)

@milescoviello

Copy link
Copy Markdown
Contributor Author

@bunbeans2-arch thanks for the RX 9070 XT run, and for building the head as is: the separate graphics and compute bounds show in the replay lines there too. The main conflicts are the reason this one is waiting: it sits on #8365's journal changes, and I'll refresh it onto main once #8365 lands. I'll put your run into the description then.

AI-assisted: yes (Claude Code)

The replay kept the newest APS5_PIPELINE_JOURNAL_ENTRIES entries of
either kind, so compute pipelines used up the bound meant for the
graphics pipeline store, and the entries left out were not reported.
Graphics entries now count against that bound alone, compute entries
against APS5_PIPELINE_JOURNAL_COMPUTE_ENTRIES (1024); each kind keeps
its newest entries in record order, and a line says how many of each a
replay leaves out.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@milescoviello
milescoviello force-pushed the fix/journal-replay-limit-per-kind branch from cb2ba99 to ee0d52b Compare November 1, 2026 13:05
@milescoviello

Copy link
Copy Markdown
Contributor Author

@oneandonlydean #8365 has landed, so this is restacked as promised: ee0d52b98f on main 77d536fb38. It's the one commit, replayed alone, and it merges cleanly with current main 31c727f622. The bounds are 4096 graphics (APS5_PIPELINE_JOURNAL_ENTRIES) and 1024 compute (APS5_PIPELINE_JOURNAL_COMPUTE_ENTRIES). The record format doesn't change, so the journal keeps #8365's format 13.

agc_driver_pipeline_journal, agc_pipeline_journal and _long_paths pass 3 of 3 on Linux, and the Windows VM adds _extended_paths. On Linux the Khronos layer checked 17 and 20 Vulkan devices on the base and the head, with no VUID- message. Details are in the body.

AI-assisted: yes (Claude Code)

@TechLevelZero

Copy link
Copy Markdown
Contributor

✅ AMD GPU test: passed

PR head ee0d52b merged onto main 291e02d · tested 2026-11-01 13:24 UTC · 7 min

GPU AMD Radeon RX 6300 (RADV NAVI24), driver Mesa 25.2.8-0ubuntu0.24.04.4, Vulkan 1.4.318
Host Ubuntu 26.04.1 LTS, kernel 7.0.0-42-generic, container ubuntu:24.04
Logs build and test logs for this run
Build ✅ passed (Release, -DANYPS5_ENABLE_SPIRV_TOOLS=ON)
Tests ✅ 2,421 passed · 10 failed (0 new, 10 also fail on main) · 54 skipped (ANYPS5_REQUIRE_VULKAN=1)
Test setup CPU-only tests without a GPU; GPU tests (and any CPU-only test that failed or skipped without one) once on the GPU with VK_LAYER_KHRONOS_validation and ANYPS5_NO_SHADER_CACHE=1; main ran the same way
Test selection 2,485 of 3,155 tests run: the ones this change can reach, chosen by tools/select_tests.py from main's build graph
Vulkan validation ✅ no new validation IDs vs main (931 tests run on the GPU)
Slow tests (>30 s, GPU tests timed with validation layers) ⚠️ agc_driver_pipeline_journal 118.1 s, ⚠️ agc_driver_graphics_maps_fallback 55.3 s, ⚠️ agc_driver_graphics 52.5 s, ⚠️ public_headers 30.0 s
Failing on main too (10, not counted against this PR)
agc_driver_guest_memory_lifetime_direct
agc_driver_image_table
agc_driver_native_depth_bda_clear
agc_driver_native_depth_buffer_clear
agc_driver_native_depth_clears
agc_driver_native_multisample_depth
agc_idle_publish_wait
agc_submit_progress
guest_mmap_executable
guest_socket_options_multicast
Validation IDs already present on main (5 IDs in 481 tests)
VUID-VkDeviceCreateInfo-pNext-pNext (481 tests): agc_driver, agc_driver_address_passes, agc_driver_alias_copy_label, agc_driver_alpha_to_coverage, agc_driver_array_layer_target, agc_driver_bc_block_copy, agc_driver_bda_anchor_reads, agc_driver_bda_span_reads, and 473 more
VUID-vkCmdCopyImage-srcImage-01548 (6 tests): agc_driver_depth_clear_hle, agc_driver_depth_plane_buffer_memory, agc_driver_depth_plane_partial_store, agc_driver_depth_storage_dispatch, agc_driver_depth_surface_from_storage, agc_driver_dispatch_move_depth
VUID-vkCmdCopyImage-srcImage-01551 (6 tests): agc_driver_depth_clear_hle, agc_driver_depth_plane_buffer_memory, agc_driver_depth_plane_partial_store, agc_driver_depth_storage_dispatch, agc_driver_depth_surface_from_storage, agc_driver_dispatch_move_depth
VUID-VkPipelineColorBlendStateCreateInfo-pAttachments-00605 (1 test): agc_driver_prepared_exports
VUID-VkShaderModuleCreateInfo-pCode-08737 (1 test): agc_driver_image_sample_derivatives
Skipped (54)
agc_driver_bda_reads_shadowed_results: skipped, the device reports watched host memory written when it imports or reads it
agc_driver_bda_reads_shadowed_results_per_build: skipped, the device reports watched host memory written when it imports or reads it
agc_driver_copied_writer_label_order: skipped, the device reports a minStorageBufferOffsetAlignment of 4, so the dispatch binds its record at +0x1004 in place and nothing is written back; the test needs an alignment above 4
agc_driver_image_table_vouched_lookups: skipped, the device reports 4 of 4 host-imported pages written after a GPU read, so a watched imported table cannot be vouched for
agc_driver_linear_target_direct_write_completion: skipped, the device does not keep host imports write-watched, so no completion stamp over the target is tracked
agc_driver_ngg_vertex_wave_reduction: skipped, s3 describes a 32-lane wave only on 32-lane subgroups, and the device has 64 lanes
agc_driver_unit_shadow: skipped, the device does not import watched host memory for unit shadows
agc_driver_unit_shadow_budget: skipped, the device does not import watched host memory for unit shadows
agc_host_import_sparse: skipped, host imports are unavailable: /dev/udmabuf is not accessible
agc_import_settle_initial_dcc: skipped, the device has no deferred settle of host imports: imports under 1 MiB wait for one only on Windows
agc_shadowed_key_stores: skipped, the device imports no watched host memory under a unit shadow
agc_small_tiled_shadow_label: skipped, host imports are unavailable: the test block is not imported and watched
agc_storage_clear_hand_over: skipped, the device imports no watched host memory for the clear-key hand-over
agc_unit_shadow: skipped, the device does not import watched host memory for unit shadows
agc_unit_shadow_completion_store: skipped, the device reports watched host memory written when it imports or reads it
guest_http_uri_locale: 
guest_net_ctl: skipped, the host has no IPv4 address
guest_video_out_color_space: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_hdr: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_high_frame_rate: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_output_mode: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_resolution_status: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_wqhd_detection: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
video_out_flip_aftergpu: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_backtoback: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_controls: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_cpuflip: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_cpuflip_in_order: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_dcc: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or
Reproduce
git fetch origin ee0d52b98fe65affcbfec1cd516379552a2a8dc4 291e02d4877784d03d346962af51713b9df63b91
git checkout 291e02d4877784d03d346962af51713b9df63b91 && git merge --no-ff ee0d52b98fe65affcbfec1cd516379552a2a8dc4
git submodule update --init --recursive
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=ON -DANYPS5_ENABLE_SPIRV_TOOLS=ON
cmake --build build && cmake --build build --target libs
VK_DRIVER_FILES=/nonexistent/icd.json ctest --test-dir build -E '^(agc|shader|recompiler|videoout|video_out|guest_video_out|vulkan)|gpu' --timeout 120 --parallel 12 --output-on-failure
ANYPS5_REQUIRE_VULKAN=1 ANYPS5_NO_SHADER_CACHE=1 VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation \
  ctest --test-dir build -R '^(agc|shader|recompiler|videoout|video_out|guest_video_out|vulkan)|gpu' --timeout 120 --parallel 16 --output-on-failure
# failures are then re-run alone: --timeout 360 --parallel 2; new validation IDs: the test alone, twice, --parallel 1

Automated build and test on a contributor's AMD machine; it does not replace upstream CI. A new comment is posted for each new push. Full logs.

@TechLevelZero

Copy link
Copy Markdown
Contributor

✅ NVIDIA GPU test: passed

PR head ee0d52b merged onto main e6f5730 · tested 2026-11-01 13:54 UTC · 22 min

GPU NVIDIA RTX 2000 Ada Generation, driver 595.99.02, Vulkan 1.4.329
Host Ubuntu 26.04.1 LTS, kernel 7.0.0-38-generic, container ubuntu:24.04
Logs build and test logs for this run
Build ✅ passed (Release, -DANYPS5_ENABLE_SPIRV_TOOLS=ON)
Tests ✅ 2,429 passed · 6 failed (0 new, 3 also fail on main, 3 unstable on main) · 45 skipped (ANYPS5_REQUIRE_VULKAN=1)
Test setup CPU-only tests without a GPU; GPU tests (and any CPU-only test that failed or skipped without one) once on the GPU with VK_LAYER_KHRONOS_validation and ANYPS5_NO_SHADER_CACHE=1; main ran the same way
Test selection 2,480 of 3,148 tests run: the ones this change can reach, chosen by tools/select_tests.py from main's build graph
Vulkan validation ✅ no new validation IDs vs main (931 tests run on the GPU)
Slow tests (>30 s, GPU tests timed with validation layers) ⚠️ agc_driver_pipeline_journal 120.2 s, ⚠️ agc_driver_graphics_maps_fallback 30.2 s
Unstable on main (3, these also failed in recent main runs on this machine; not counted against this PR)
agc_driver_pipeline_journal: failed in 1 of the last 10 main runs
agc_submit_progress: failed in 6 of the last 10 main runs
agc_submit_progress_no_timeline: failed in 8 of the last 10 main runs
Failing on main too (3, not counted against this PR)
agc_driver_image_table
guest_mmap_executable
guest_socket_options_multicast
Validation IDs already present on main (5 IDs in 484 tests)
VUID-VkDeviceCreateInfo-pNext-pNext (484 tests): agc_driver, agc_driver_address_passes, agc_driver_alias_copy_label, agc_driver_alpha_to_coverage, agc_driver_array_layer_target, agc_driver_bc_block_copy, agc_driver_bda_anchor_reads, agc_driver_bda_span_reads, and 476 more
VUID-vkCmdCopyImage-srcImage-01548 (7 tests): agc_driver_depth_clear_hle, agc_driver_depth_plane_buffer_memory, agc_driver_depth_plane_partial_store, agc_driver_depth_plane_sampling, agc_driver_depth_storage_dispatch, agc_driver_depth_surface_from_storage, agc_driver_dispatch_move_depth
VUID-vkCmdCopyImage-srcImage-01551 (7 tests): agc_driver_depth_clear_hle, agc_driver_depth_plane_buffer_memory, agc_driver_depth_plane_partial_store, agc_driver_depth_plane_sampling, agc_driver_depth_storage_dispatch, agc_driver_depth_surface_from_storage, agc_driver_dispatch_move_depth
VUID-VkPipelineColorBlendStateCreateInfo-pAttachments-00605 (1 test): agc_driver_prepared_exports
VUID-VkShaderModuleCreateInfo-pCode-08737 (1 test): agc_driver_image_sample_derivatives
Skipped (45)
agc_driver_bda_reads_shadowed_results: skipped, the device reports watched host memory written when it imports or reads it
agc_driver_bda_reads_shadowed_results_per_build: skipped, the device reports watched host memory written when it imports or reads it
agc_driver_sample_order_grid_reference: skipped, the device does not keep depth drawn at custom sample locations at those locations, even after a transition that names them: the composite of uniform renders holds the guest's depth in 0 of 3005 sloped samples, so no path can be compared with it bit for bit
agc_host_import_sparse: skipped, host imports are unavailable: /dev/udmabuf is not accessible
agc_import_settle_initial_dcc: skipped, the device has no deferred settle of host imports: imports under 1 MiB wait for one only on Windows
agc_unit_shadow_completion_store: skipped, the device reports watched host memory written when it imports or reads it
guest_http_uri_locale: 
guest_net_ctl: skipped, the host has no IPv4 address
guest_video_out_color_space: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_hdr: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_high_frame_rate: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_output_mode: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_resolution_status: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_wqhd_detection: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
video_out_flip_aftergpu: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_backtoback: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_controls: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_cpuflip: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_cpuflip_in_order: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_dcc: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_deferred: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_eop: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_eop_aftergpu: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_eop_main: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_fps_limit: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_o
Reproduce
git fetch origin ee0d52b98fe65affcbfec1cd516379552a2a8dc4 e6f57307fa7f47165d7fc1bee945abdb2f6816e7
git checkout e6f57307fa7f47165d7fc1bee945abdb2f6816e7 && git merge --no-ff ee0d52b98fe65affcbfec1cd516379552a2a8dc4
git submodule update --init --recursive
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=ON -DANYPS5_ENABLE_SPIRV_TOOLS=ON
cmake --build build && cmake --build build --target libs
VK_DRIVER_FILES=/nonexistent/icd.json ctest --test-dir build -E '^(agc|shader|recompiler|videoout|video_out|guest_video_out|vulkan)|gpu' --timeout 120 --parallel 32 --output-on-failure
ANYPS5_REQUIRE_VULKAN=1 ANYPS5_NO_SHADER_CACHE=1 VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation \
  ctest --test-dir build -R '^(agc|shader|recompiler|videoout|video_out|guest_video_out|vulkan)|gpu' --timeout 120 --parallel 4 --output-on-failure
# failures are then re-run alone: --timeout 360 --parallel 2; new validation IDs: the test alone, twice, --parallel 1

Automated build and test on a contributor's NVIDIA machine; it does not replace upstream CI. A new comment is posted for each new push. Full logs.

@github-actions github-actions Bot removed the needs rebase Conflicts with main, rebase needed label Nov 1, 2026
@oneandonlydean
oneandonlydean merged commit 8c96e69 into boykopovar:main Nov 2, 2026
12 of 13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants