Repository navigation
Description
What happens
PipelineJournal refuses any append that would take the journal past APS5_PIPELINE_JOURNAL_MAX_MIB (default 1024 MiB), and it evicts nothing (core/libs/prx/libSceAgcDriver/Graphics/src/PipelineJournal.cpp:777 on main 3fd1cc3).
- It prints "[pipeline-journal] … reached APS5_PIPELINE_JOURNAL_MAX_MIB; later pipelines are not recorded" once. After that it only counts the refused writes.
- So the journal holds whatever a title builds first. If it fills while a level loads, the rest of that level's pipelines are never recorded, and every warm boot compiles them again at the level's entry.
- The loader reads a larger file in full: a 2 GiB journal was replayed under the default cap here. So the cap limits only what gets recorded.
Module records are almost all of the bytes. In the 2 GiB journal below:
- 924 module records take 2047.6 MiB (median 362 KiB, largest 14.7 MiB);
- the 958 pipeline records take 0.3 MiB.
#8213 notes that Ghost of Tsushima's journal is 1023 MiB, with 940 pipelines and 979 modules. So at least one more title's journal is within 1 MiB of the cap. I haven't measured that one.
Measured on PPSA11386 (007 First Light)
One run each:
- A cold seeding run, which writes the journal.
- A warm run from that seed.
The warm run always used the default cap.
| seed with the default cap (1024 MiB) | seed with APS5_PIPELINE_JOURNAL_MAX_MIB=2048 |
|
|---|---|---|
| seed: when the cap was reached | during the first level's load, after the save was resumed and before the first gameplay frame; 617 later writes refused | after the level's first frames; 136 later writes refused |
| warm run: pipelines replayed at boot | 330 graphics, 374 compute | 458 graphics, 497 compute |
| warm run: from resuming the save to the first gameplay frame | 280 s | under 10 s |
| warm run: graphics pipelines built during that level entry | 772, taking 284.5 s; single builds of 24–29 s | 2,505, taking 5.4 s |
- Boot to resume took 385 s and 384 s, including the journal replay (258 s and 263 s, 16 threads).
- In the warm run from the default-cap seed, the 280 s level entry had no shader-cache misses. The four async queues spent about 282 s each in WAIT_REG_MEM packets, while queue 0's worker built pipelines in its draw packets.
Options, none of them tested:
- a higher default cap;
- when an append would pass the cap, evicting the oldest records, or keeping the records the last replay used.
The replay keeps at most APS5_PIPELINE_JOURNAL_ENTRIES (default 1024) entries (PipelineJournal.cpp:908); the 2 GiB journal holds 958.
Storing references to the shader disk cache's entries instead of the modules wouldn't shrink this journal much. Only 4 of its 924 modules have SPIR-V identical to an artifact in the disk cache written by the same run.
Setup:
- Build:
main68323ba plus the open PRs PPSA11386 needs (fix(shader): apply cube output modifiers whenever IEEE mode is off #5799, fix(agc): a driver store over an earlier store's range takes its piece slot #6868, perf(agc-driver): join a buffer bound from inside a range across the ranges it crosses #5372, fix(agc): import committed spans of partially accessible windows #6918, fix(agc): wait for the display buffer's GPU writers before the presentation lock #7643, feat(agc): run registered merged hull programs (binary type 3) #6679), with draw profiling (APS5_PROFILE_DRAW=1) and scripted input. - The title's two feedback-wait limits are widened in its executable as described in 007 First Light (PPSA11386): the terrain crash after Resume is a stale read of a GPU feedback buffer past the title's own wait limits #6172 (local only).
- RTX 3080, NVIDIA 615.78.08, Linux 7.2.10, GCC 16.2.1, Release.
- None of those PRs touches
PipelineJournal.cpp.
AI-assisted: yes (Claude Code)
Activity
milescoviello commentedon Oct 20, 2026
A third option, on the module side: compress the module records. They're SPIR-V. On Ghost of Tsushima's 1023 MiB journal (the one #8213 mentions), the first 256 MiB compress with
zstd -1to 22% (0.39 s single-threaded) and withzstd -3to 20%. zstd is already vendored (3rdparty/zstd, built for ZArchive). Since #8213 the replay reads each module from the file when its entry is built, so decompression would also happen per module, at the replay thread that needs it. The same cap would then hold about 4 to 5 times as many modules, and the boot would read less.That can sit beside eviction rather than replace it. I'd build it on #8213, since it changes the same module read path. @SleglJan, if you're already working on the cap, say so and I'll leave it.
AI-assisted: yes (Claude Code)
SleglJan commentedon Oct 20, 2026
@milescoviello I'm not working on a fix for the cap, so go ahead.
The same check on PPSA11386, using the 2 GiB journal from the issue's second seed:
-1-1-3So everything that seed recorded, which runs past the level's entry, would fit under the default 1 GiB cap at about 0.55 GiB, about 3.6 times smaller. These are whole-stream numbers, so compressing each record on its own may come out a little larger. When you have a branch, I can run the issue's two seeds on it.
AI-assisted: yes (Claude Code)
milescoviello commentedon Oct 20, 2026
@SleglJan thanks, and thanks for the 007 numbers. The branch is up as #8365 (stacked on #8213; its own commit is f5504d2). On MXGP 26, a 200 s seed gives a 30.15 MB journal for 445 modules, against 105.75 MB uncompressed. Per-record compression came to 28.8 MiB of module payloads against 100.9 MiB, so close to your whole-stream ratio. A warm load takes about 48 ms longer, since each module is expanded once to be checked, and the replay time is unchanged. Your two seeds would be very welcome, especially whether the second one now fits under the default cap and how the warm level entry compares.
AI-assisted: yes (Claude Code)
feat(agc): store the pipeline journal's modules compressed
feat(agc): store the pipeline journal's modules compressed
feat(agc): store the pipeline journal's modules compressed
feat(agc): store the pipeline journal's modules compressed
feat(agc): store the pipeline journal's modules compressed
feat(agc): store the pipeline journal's modules compressed