[Raghav314]

What

An address-based build put every readable registered range in its BDA table and imported or mirrored it, including memory the guest mapped for the CPU only. On the console a mapping reaches the GPU only if its protection has GPU access (0x10 read, 0x20 write).

• The guest allocation registry records whether a range has GPU access. libkernel sets it from the protection at sceKernelMapDirectMemory, flexible maps, fixed remaps and sceKernelMprotect. Reservations and the main image have none; a title's mprotect with GPU bits gives the image pieces access.
• An address-based build leaves out ranges without GPU access. Nothing else changes: stores are still decided by CPU write access and in-place import, and AnyPS5's own libc heap is still registered with GPU access.
• GuestAllocations::Mutation::Add and Protect take the flag.

Sources:

• The protection bits: shadPS4 src/core/memory.h MemoryProt (CpuRead = 1, CpuWrite = 2, GpuRead = 16, GpuWrite = 32).
• GPU page table entries exist only for ranges with GPU access: KytyPS5 src/kernel/memory.cpp CountPageTableEntries(bool gpu) counts a range for the GPU only if its protection has PROT_GPU_READ or PROT_GPU_WRITE, and sceKernelGetPageTableStats reports the CPU and GPU counts separately.
• PPSA36082 maps memory the CPU uses with 3 and sets the GPU bits where the GPU reads: its bundled libc.prx maps the malloc heap with sceKernelMapNamedFlexibleMemoryInternal (4h6F1LLbTiw), protection 3, at both call sites (+0xb2eb, +0xb4fd); its executable's segments carry only R/W/X flags, and it mprotects the two pieces the GPU reads with 0x12 and 0x11.

PPSA36082

During loading and the main menu it registers: the 1 GiB malloc heap at 0x200000000 with 3, 6784 MiB of direct memory with GPU read and write, and the two 32 KiB image pieces. Its first address-based draw runs on the queue the loader's fence waits on. That draw imported the heap; the import fails to pin (the kernel logs Cannot map memory with base addr 0x200000000 and size of 0x40000 pages), the fence is late, and the 512 MiB upload pool overflows (eboot 0x28fba9). With this change the heap is not in the table and nothing fails to pin.

In my local build I bypassed both problems by hand: a driver option, APS5_GPU_EXCLUDE=0x200000000+0x40000000, kept that address range out of address-based builds, and a patched executable got through loading with a 2.5 GiB upload pool instead of 512 MiB. Both are specific to this title. This change replaces the first: the title maps its heap with 3, which has no GPU access, so address-based builds leave it out without an option. Loading with the original 512 MiB pool also needs the changes below.

It also needs #861, #1367, #1249, #290 and a follow-up I will open after this one, which imports GPU-mapped memory when it is mapped and depends on this PR. With those, on main df16c4c with an earlier version of the follow-up, 40 s runs to the main menu: 4/5 pass with this change (the other overflowed at 4.1 s; the pool's margin is thin and that is separate work), 0/3 without it (each overflowed at 503 MiB with the pin failure above).

Local changes not in this PR

I also have these on local branches. They finish making the GPU follow the guest's protection, but I am not opening them because no title I can run needs them. Do you want PRs for them?

• GPU read and write kept apart: the registry records GPU read and GPU write separately; the BDA table grants reads only with GPU read and stores only with GPU write (today CPU write decides stores); AnyPS5's own libc heap is registered without GPU access, as the console's libc.prx maps its heap with 3. PPSA36082's only range where CPU and GPU write differ (an image piece mprotected with 0x12) gets no GPU stores, and it allocates nothing through AnyPS5's libc heap.
• GPU access on every path, not just address-based builds: buffer descriptors read zeros from memory without GPU read and drop stores to memory without GPU write (memory with GPU write only is copied in so stores are exact); the command processor (command buffers, WRITE_DATA, DMA_DATA, labels, waits, fills), render targets, index and vertex inputs, DCC keys, display buffers and shader code fault on such memory; texture reads see zeros. The driver's GDS backing and AvPlayer's fallback frames are mapped with GPU access. A counted PPSA36082 run made no access to memory without GPU access.
• Mappings with GPU access but no CPU read (0x10, 0x30) reach address-based builds, and AMM PRT pages with no physical memory read zero and drop GPU stores, through an unbound Vulkan sparse buffer (strict residency). Sources: amdgpu AMDGPU_PTE_PRT, which turns off faults on unmapped PRT accesses (gmc_v8_0_set_prt), and RADV reporting sparseResidencyNonResidentStrict. PPSA36082 makes no such mappings and imports no sceAmpr* function.
• The main image is registered when the process starts on both OSes. Today Linux registers it at any mprotect and Windows only at an mprotect of the image. PPSA36082 mprotects its image, so it is registered either way.

Tested

Linux (CachyOS), x86-64, GCC, 7800X3D / RTX 4090.

• guest_memory: a direct map with 3 has no GPU access, 0x32 has; sceKernelMprotect to 0x13 adds it and back to 3 removes it; flexible maps with 3 and 0x33 likewise; the main image has none. Mapping with the GPU bits ignored fails it, and so does registering the image with GPU access.
• agc_driver_graphics: an address-based build includes a range with GPU access and leaves out one without. Dropping the check fails it.
• Test blocks that GPU work reads are registered with GPU access.
• ctest: 494/494.
• Windows: cross-built with MinGW-w64 GCC 15.2.0 and run under Wine 11.19. agc_driver_graphics and agc_driver_flat_store pass; guest_memory passes the new checks, then stops at CheckMlock (the working-set query), as it does without this change.

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed in TechnicalDebt
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: none (PPSA36082 also needs fix(guest-arena): place guest address space in the PS5 window on Linux #861, fix(videoout): create the window and the device when VideoOut starts #1367, fix(relinker): write a bucketed hash table for Linux guest modules #1249, feat(agc): host import windows, least-recent eviction under the budget #290 and the follow-up above)
• AI-assisted: yes