Repository navigation
007 First Light (PPSA11386): the terrain crash after Resume is a stale read of a GPU feedback buffer past the title's own wait limits #6172
Description
Summary
During the level load after Resume the title reads back a 160x90 GPU feedback buffer (8 bytes per entry) to decide which texture pages to stream. Before reading it, it polls a GPU label for at most 2 s (sceKernelReadTsc against sceKernelGetTscFrequency) and skips the wait once the feedback job is two or more frames old. On our stack the GPU stops presenting for about 300 s at level entry while the entry pipelines are created (26-64 s with them cached), and every timed crash sits at the start of that gap: the title reads the buffer before the GPU wrote it, the stale bytes (the memory's previous occupant) become page-request keys, the lookup of a texture id that does not exist fails (or a quadtree descent runs past a texture's extent), and the process crashes with SIGSEGV in the title's streaming code. A local title patch that widens the 2 s limit and the two-frame threshold removes the crash (8 clean of 8). Unpatched, 10 of 24 unscaled runs on 2026-10-14 crashed.
Setup
Bring-up branch share/007-v5 at 65c5b68 (fork SleglJan/AnyPS5), not upstream/main, which stops before the first frame in this title. Switches on the run line that change behaviour: branch-only APS5_AHEAD_PIPELINES=2, APS5_PIPELINED_DRAWS=8, APS5_NO_IDLE_STORE=1, APS5_SPARSE_RUN_IMPORT=1, APS5_LOCAL_LARGE_CLASSES=1; upstream APS5_HOST_IMPORT_MIB=12288, APS5_RESOURCE_CACHE_ENTRIES=64, APS5_NO_DRAW_SRT_ENTRIES=1; plus tracing switches and a guest-fault dump spec. CachyOS, kernel 7.2.10, RTX 3080, NVIDIA 615.78.08 (runs before 2026-10-14 15:30: kernel 7.2.9, NVIDIA 615.71.09). Evening runs 1200 s (3600 s at scale 0.02), morning runs 1080-3600 s, from a gameplay save, driven by injected pad input. 0.12-0.44 FPS at the end of the runs (2-8 s per frame).
Evidence
- Guest-fault dumps (share/007-v5): 11 catches at two instructions, 9 a failed lookup of a texture id derived from a request key, 2 the quadtree descent of a key whose coordinates lie past the texture's extent. The request keys come from the feedback buffer. Six unscaled catches carry the write-tracker annotation; in none does either feedback buffer show a GPU write (four show no tracked write; in two, the last tracked write to one buffer's first 64 KiB block is a CPU write). Where the content was examined it is a previous occupant's: a 15-byte periodic byte string in two catches, RG32F texel pairs in another, a 16-bit stream in another. Two catches of the same game moment produced byte-identical histograms.
- Where it happens: Resume comes at present 750-870 (1640 in one run); each of the four evening faults came 370-410 presents and 40-66 s after Resume (the morning runs have no Resume timestamps), at the point where clean runs stop presenting for about 300 s (291-333 s in the ten 1200 s runs from the pre-gameplay cache) while the level-entry pipelines are created. With those pipelines cached the gap is 26-64 s and the crash still comes: 5 of 14 cached morning runs faulted (mixed switches and builds, NVIDIA 615.71.09), 2 cached evening runs were clean. Caching the entry pipelines does not prevent the crash.
- Not the deferred image write-back: APS5_EAGER_WRITEBACK=1 reproduced the crash with the same histogram. Not the label deferral rule: a run with label deferral off (APS5_DRAIN_COMPLETION_LABELS=1: every held label drained or stored at once, 22,727 drains) was clean, within chance, and faulting and clean runs defer about a fifth of their GPU-written labels alike. The feedback buffer is not an image and no DMA packet fills it.
- Patched title (the 2 s limit and the two-frame threshold widened, local only): 8 clean of 8 (six runs and two more with frame dumping on, 1200 s each; 0 guest faults, no spins, patch verified per run; all reached gameplay). Unpatched runs with the same build, driver and switches and the entry pipelines uncached faulted 3 of 4 the same evening (4 of 6 counting the eager-write-back and drained-label runs); 8 clean by chance is ~0.02% at a 67% fault rate and ~1% at 42% (the day's 10 of 24); the six with identical switches alone ~0.1% / ~4%. Which of the two limits matters is open: the patch widens both.
- APS5_TIME_SCALE=0.25, unpatched: 1 run, faulted at the same site, about 330 presents and 60-85 s after Resume. The wait is 8 s real at that scale against a 300 s gap, so the scaled wait still expires inside the gap and the run does not separate the two limits; the buffer held GPU-written feedback this time (an older job's, by its content).
- APS5_TIME_SCALE=0.02, unpatched, pre-gameplay cache: 3 clean of 3 (3600 s real each, 0 guest faults, roughly 450-480 presents (380-590 at the outside) and about 26 min real after Resume, which is about 31 s of game time; Resume came at about 2030 s instead of 290 s). Each run still stopped presenting for about 330-360 s at level entry, longer than the 100 s real the scaled wait allows, so the mechanism as described predicts a crash there and none came; 3 clean is ~4% by chance at 67% and ~20% at 42%. Unexplained. One reading is that the feedback read starts later in game time than the entry gap (unscaled it comes 40-66 s after Resume, and the 0.02 runs covered 31 s of game time), which nothing here tests.
Related upstream items (live search 2026-10-15 ~02:20 on upstream/main ab9a476)
- Windows: since d9852df2a every frame waits out a title's 1 s frame timeout #5842 (open): Windows, since d9852df every frame waits out a title's 1 s frame timeout: the same title-side pattern (a bounded wait on GPU completion), there probably tripped by late label visibility (the reporter's reading).
- AGC: PS5 FSR4 Showcase (PPSA99011) gives up in its first FSR4 pass while the queue worker builds 130k-800k-word compute pipelines (up to 66 s each on AMD) #6118 (open): PS5 FSR4 Showcase gives up a bounded label wait (3 s plus 0.1 s per dispatch) while the queue worker builds 130k-800k-word compute pipelines, and gets past that stage with them cached: the same class, a title-side bounded wait outrun by our GPU side.
- Silksong (PPSA12544): "Act 1: Pharloom" title card never shows; no frame is presented for 68 s while the first area loads #3695 (open): Silksong presents no frame for 68 s during the first area load: a long load with no presents; no timeout is shown to trip there.
- ASTRO BOT (PPSA21567): "too many objects/batches in this scene" assert in the robot cutscene after New Game: draw threads race for bone uploads while the flip submission is held #5645 (open): ASTRO BOT assert; its write-up lists APS5_TIME_SCALE with a scaled vblank among the things tried without effect, and its second direction (release a held flip without letting the title further into its flip ring) concerns the same bound from the other side: how far the title's CPU may run ahead of the GPU.
- APS5_TIME_SCALE exists since f3ac610 (libkernel Time.cpp): it scales sceKernelGetProcessTime, sceKernelGetProcessTimeCounter and sceKernelReadTsc (and guest clock_gettime's monotonic clocks on Windows only; on Linux those read the host clock), sleeps stay real, injected pad input is timed in guest time. It is not in docs/.
Proposal
- Document APS5_TIME_SCALE in docs as a diagnostic aid (what it scales, that sleeps stay real, that the boot here slowed about 7x at 0.02).
- Design discussion, no PR yet: a title that bounds its GPU waits in wall time and in frames needs both covered. A GPU-paced guest clock (slowed while the GPU lags the title's CPU) would have to nearly stop for minutes at a level entry like this one; the two-frame limit needs the title's CPU held within two frames of the GPU (flip or submit pacing; ASTRO BOT (PPSA21567): "too many objects/batches in this scene" assert in the robot cutscene after New Game: draw threads race for bone uploads while the flip submission is held #5645 discusses the same bound). Next step on our side: a one-limit-at-a-time title patch to learn which limit matters, and a log of the label write against the title's read.
AI-assisted: yes (Claude Code)
Activity
Scramblrrrr commentedon Oct 16, 2026
Windows007 level result on c6bd8b5 (branch codex/first-light-level-two-lane, main d9b696d + shared-table/007 boot stack + #6679 36529f1 + #6364 9fffcea, #4339 excluded): correct title/offline menu, Resume at17:52:29.513UTC/present989. Last observed loading spinner atpresent1648/17:54:03; unhandled0xc0000005 on RenderThread, read0x40, game executable+0x17bd927; exit17:54:27.720UTC. Not a memory-guard or owned stop. Peak sampled private27.69GiB/minfree2.52GiB, three bounded trims, unchanged2GiBfree/28GiBprivate guard. No shader-type rejection or invalid indirect dimension exception before the fault.
This appears consistent with #6172's failed texture lookup, but I have not yet proved that the faulting key came from the stale feedback buffer in this run. Queue0 was inside NVIDIA's compiler in the fatal thread snapshot. Before the fault, one newly used754,594-word compute pipeline took18.9s to create and another172,513-word one1.59s; that is evidence of a pipeline stall, not proof of the feedback cause. No local title patch/time scaling applied. @SleglJan does executable offset+0x17bd927 match one of your known feedback lookup sites for v01.018.000?
Both added fixes were native-tested first:30/30PASS; removing only #6364's wait block makes its unchanged fixture fail with the older-texels message; restored/rebuilt30/30PASS14.53s. Thus I am not presenting this title fault as a failure of those fixtures or a new structurizer stop. Next separate candidate is #6346 + #6273 at1f0e2b149 (source-prepared/unbuilt), plus an opt-in bounded copied-buffer ownership sampler. Keeping actual 30FPS/gameplay goal open. AI-assisted: Codex; no title bytes, descriptors, PM4 or raw logs attached.
SleglJan commentedon Oct 16, 2026
@Scramblrrrr Yes. A read at 0x40 on RenderThread at that instruction is the failed texture-id lookup: 9 of the 11 catches in the body above are at it, and the other 2 are at the quadtree descent. That also covers your earlier RenderThread crashes at that offset after Resume (October 12-13).
What the body shows for that site, all from the runs and setup described there:
That doesn't prove the faulting key in your run came from the stale buffer. It is the same site in the same phase of the load.
AI-assisted: yes (Claude Code)
MarcZX10 commentedon Oct 17, 2026
I opened PR #6181 for proposal 1: documenting what
APS5_TIME_SCALEchanges and that host sleeps remain real time. This is diagnostic guidance only; it does not address the title’s two-second/frame-count wait or establish a GPU-pacing fix. The one-limit-at-a-time title patch and label-write/read trace remain the evidence needed before proposing runtime behavior changes.Scramblrrrr commentedon Oct 18, 2026
Oct18 Windows unpatched-title reproduction on integration ad3586f (base main e5c2b3c plus current public 007/shared-table/resource stack, #7476 3ea0be6 and #6606 925f9a8). RTX5070Ti617.42 / 7800X3D /32GiB. Correct title/offline menu; Resume10:54:36.623UTC/frame906; exit10:55:12.932UTC, unhandled0xc0000005 on RenderThread reading0x40 at executable+0x17bd927. Same site you previously confirmed. Not guard/owned stop: last sampled private
26.28GiB/free7.26GiB, one bounded trim. No title binary patch/time scaling.The last reported compute pipeline builds before the fault were33ms,12ms,6ms,11ms (warm application cache), so this run does not show a >2s compute compile immediately before the read. That neither disproves the broader GPU-backlog/feedback hypothesis nor proves the faulting key's source in this run. Native #6606 controls pass, but its title cross-test remains inconclusive because this older crash intervened; no declared-range failure occurred and no reported fallback counter yet proves that path was used.
Current isolated diagnostic is APS5_SYNC_DRAWS=1 on the same ad358 binary and same budgets/settings otherwise. Four related GPU controls pass with the flag on. This is a pacing/correctness experiment, not a performance recommendation; old Oct13 builds were slower with it. Starting the title run now. Public integration branch is pushed, raw title data stays local. AI-assisted: Codex.
Scramblrrrr commentedon Oct 18, 2026
Same-binary synchronous-draw diagnostic finished: ad3586f, only APS5_SYNC_DRAWS=1 added, other settings/budgets unchanged. Correct title/menu. Resume11:03:13.336UTC/frame942, exit11:03:58.398UTC (~45.06s). RenderThread0xc0000005, but this is a DIFFERENT instruction from the previous failed lookup: executable+0x17bdb56, read0xffffffffffffffff, rather than+0x17bd927/read0x40. I have not established whether it is the other quadtree site in your issue. Not a guard stop or a workaround. Default is off again for subsequent tests. No game binary/time patch. The four draw/indirect native controls passed with this flag before the title run. AI-assisted: Codex; title data remains local.
SleglJan commentedon Oct 18, 2026
@Scramblrrrr yes. The new instruction in your 11:04 comment is the issue's other site, the quadtree descent: 2 of the 11 catches, and in both the key's coordinates lay past the texture's extent. The one in your 10:57 comment is the failed lookup (the other 9).
The all-ones address fits too. In both of our descent catches, the descent took a key as a node pointer. That value is non-canonical, so the load raises a general-protection fault rather than a page fault, and such a fault carries no faulting address. Linux reports it as SIGSEGV at address 0 (our dumps); Windows reports it as an access violation reading 0xffffffffffffffff (your run).
So the offset and that address show the same instruction and the same fault class, but not the same cause. Whether your key also lay past the texture's extent would need the key from your run, which the crash report doesn't carry.
AI-assisted: yes (Claude Code)
Scramblrrrr commentedon Oct 19, 2026
Thanks for confirming both sites and the non-canonical-address explanation. I agree that the fault address alone does not establish the key's source or prove the same stale-feedback cause on Windows.
Resumed testing on a new public integration branch: https://github.com/Scramblrrrr/AnyPS5/tree/codex/first-light-oct18-resume-main (188fab5; main 0c25997 merged onto the prior ad3586f stack). It preserves our shared-table/resource fixes and excludes the #6188 draw-ahead experiment, whose last same-host menu run was slower. Full Windows library rebuild and native gates are in progress; this head is not yet title-tested or staged. Two newer main merges, including Windows APR read staging #6449, are reviewed and will be included before the next title run. No new crash-resolution or FPS claim yet. Raw title data stays local. AI-assisted: Codex.