Marvel's Wolverine runs out of device memory since #6454: kept descriptor sets keep evicted textures alive #7801

Since #6454 was merged (23629c2), Marvel's Wolverine (PPSA03671) runs out of device memory 30-90 s after launch, before its first frame, and aborts on vkAllocateMemory -2 in memory type 0. The merge just before it (72b0ab3, #6312) runs the same scene without trouble. APS5_KEPT_DESCRIPTOR_SETS=0 avoids it on #6454 and on main 676d423.

@EagleBearerYS this looks like the kept descriptor sets from 1d66d85: each kept set's record holds every texture of the resources that freed it, including textures the caches have already evicted.

Setup: Windows 11 Pro 26300, RTX 4080 SUPER 16 GB (NVIDIA 617.42), Ryzen 7 5800X3D, 32 GB RAM, WinLibs GCC 15.2.0 r7, Release. Content 01.001.005; eboot.bin and its three modules unwrapped with tools/unwrap_self.py and converted with relinker --windows --registry --module-dir .. Each tested commit has #6901 (8cd400a) and #6308 (48a1d34) merged on top: the title needs #6901 to load, and #6308 removes an unrelated abort at 419-620 s. Every run is the same scene, the "Sony Interactive Entertainment presents" card, with no input, a shader cache from earlier runs, APS5_PROFILE_DRAW=1, default budgets and a 120 s limit. Times include 10-40 s of loading.

Bisected over the first-parent merges between f305942 and 676d423:

On #6454 heap 0 fills, allocations move to host memory type 0 (heap 1, system memory on this card) and then fail:

[gpu] storage texture: device-local heap 0 uses 14741 of its 14749 MiB budget, 9437184 bytes placed in host memory type 0; 1 such allocations, 9 MiB
[gpu] sampled texture cache budget 2048 MiB (was 9047 MiB): device-local heap 0 has a VK_EXT_memory_budget budget of 14749 MiB and uses 15617 MiB, 5343 MiB of it sampled textures (2556 MiB in the cache) and 2060 MiB cached storage images; storage image cache budget 512 MiB (was 3819 MiB)
[gpu] texture caches released 0 MiB of the 263 MiB asked for: 969 sampled entries, 0 storage images (cache or device busy, skipped), 0 with pending results kept
terminate called after throwing an instance of 'std::runtime_error'
  what():  compute shader 0x11d0372500: AGC graphics: vkAllocateMemory texture: allocation=15728640 memoryType=0 reclaimed=4194304 released=0 initialResult=-2 attempts=2: Vulkan result -2

On 72b0ab3 heap 0 stays near 10 GB with the sampled textures in the cache (uses 10110 MiB, 3459 MiB of it sampled textures (3444 MiB in the cache)), and no cache release is logged.

What I think happens, from reading 676d423 (the code is the same on fb0dc11):

• ShaderResources::keptTable fills the kept set's TableDescriptors with std::move(textures): every sampled texture of the freed resources, not only the table's. Up to 8 such records stay in DescriptorCache::kept.
• ReleaseTextureCaches counts an evicted sampled entry as freed only when texture.use_count() == 1. So the textures these records hold leave the cache but keep their memory ("released 0 MiB ... 969 sampled entries"), and the sampled textures outside the cache (5.3 GB with 2.6 GB in the cache above, 7.7 GB with 2.6 GB in the second run) push heap 0 past its budget.
• Clearing only the record that a reused set carries after its build did not help (2 runs: one over budget, one abort at 86 s), so the memory is held by the kept records themselves.

A pull request referencing this issue follows, with the record holding the textures weakly. With it, 676d423 ran two 120 s runs and a 300 s run without the abort, with heap 0 at most 10.6 GB, and fb0dc11 ran 180 s with heap 0 at most 10.2 GB.

On fb0dc11 the first runs spend 35 s and more on each new compute pipeline (0.8-0.9 M SPIR-V words) until the Vulkan pipeline cache holds them, and do not reach the scene in 120 s; the fb0dc11 rows are runs after that.

Separate from #7766 (the vouched-lookups test on RADV) and from #5961 / #6099 (storage-cache churn on 8 GB cards).

Not checked: Linux, other GPUs, anything past the splash card.

AI-assisted: yes