Repository navigation
fix(agc): defer write tracking for fresh Windows shared imports - #4535
Conversation
|
Prepared author-preserved port of 652cb0b onto our runtime-table 007 stack: https://github.com/Scramblrrrr/AnyPS5/tree/codex/first-light-deferred-shared-tracking , head4135e2871 (production port31d46210c). Not built/staged yet; current separate game comparison is still running, so no concurrent build/test load. Conflicts resolved by retaining the stack's 4 KiB per-subpage generations/protection/partial-output settlement, local shared-alias mapping, and #4661's newer host-write correction from1eb13fdf025bab9283bd6d18f6c55afdfcfd2646. Collection history is marked only for the settled output prefix. Fresh activation forces reporting of selected never-collected subpages; the existing host-write arming/untracked-page logic stays intact. The unrelated newer-main noteUnstampedStore API is absent from this runtime stack, so its existing unwatched fallback remains unchanged. Your WindowsWriteWatch fixture came across. Two whole-view expectations need adaptation on this stack: collecting only page0 leaves page3 writable; writing page9 after activation reports page9 alone. Added an explicit check that page0 is protected and page3 remains writable. Will run the focused native suite and report failures/negative controls before attempting007; no FPS or crash-fix claim. AI-assisted: Codex. |
|
Windows/RTX5070Ti/GCC15.2 runtime-stack cross-test of652cb0b: final integration d26e52a on codex/first-light-deferred-shared-tracking. Your production/fixtures preserved in31d46210c, 4KiB fixture expectation adaptation4135e2871, plus existing main9873fae1e's three-line reserved-view commitment dependency in d26e52a (original author preserved). The full lazy-backing libkernel change is not included. Initial integration4135e2871 had5/7 pass; both new shared SEC_RESERVE fixtures failed before assertions with "protect shared guest page: Attempt to access invalid address". Our older Map lacked9873fae1e's query/commit. Importing exactly that existing Map hunk resolves setup:7/7 native PASS1.71s (windows_write_watch, agc_write_watch_coverage, agc_write_tracking, agc_shared_write, partial-collect single/two-pass, guest_memory). Alternate APS5_NO_SINGLE_PASS_COLLECT=1 on windows/coverage/tracking/guest4/4 PASS1.48s. Production-only mutations, fixture unchanged:
Only patched graphics8745FDBD1E62CD423E9E5F0DFB79E4EE7D73149A36B02D98AFADAEEE406A0986 + libc08AB01A94E6A9D0D82DF51F1676B5D7392437CD8382DD7C36CC0801A5288C10A staged. Normal007 warm-cache/cache16 launched22:59:33.852UTC (initial helper invocation exited before starting the game; corrected its path quoting); no sampler/profile or concurrent builds/native tests. No rendering/crash/FPS claim yet. Preceding image-ordering comparison reached the incomplete slow scene ( AI-assisted: Codex. |
|
007 cross-test on d26e52a/graphics8745FDBD/libc08AB01A9/cache16 ended intentionally23:11:20.034UTC (~3m24 after Resume23:07:56.425), not a crash. Final frame1877/ Next game comparison keeps these exact two libraries and changes only resource-cache capacity16→64, same warm cache/other settings/guards. Preparing combined #4646+#4683 native validation first, with no game concurrent. Still awaiting SleglJan's runtime-table-store/reference-texture head for cross-test; unsupported stores remain outside this change. AI-assisted: Codex. |
|
What does main do on the same three PPSA36082 runs? The description gives one of three runs surviving the 40 s window with this PR and no figure for main, and says the change doesn't get the title through loading on its own, so its effect isn't shown yet. A count of first-write faults on fresh imports, or the time to a fixed point in loading, for main against main plus this PR alone would do. |
|
@oneandonlydean Collected both comparisons on the exact upstream parent First-write fault counts at the first
All six reached four import windows. Main initially protected 434,176 shared views inside those windows each time; the PR protected none there, while initially protecting 1,416 views outside imports. The reduction is in the intended fault class, with ordinary write tracking still active. The separate uninstrumented loading trials used the same two revisions and three alternating pairs:
All six failed with the same Every run restored the same warm shader-cache seed (2,396 files, 330,400,440 bytes), cleared inherited Counter definition: a fresh-import arm is a mapped 16 KiB virtual view's first observed successful read-only protection inside Atomic counters are in named shared memory; no per-fault printing or game sampling thread. The extra The deterministic probe fixture passed on both headers: 2 import scopes, 1 first import arm/fault, 3 armed write faults, 1 other first arm. Instrumented PR tracking/backend tests passed, as did baseline tracking. Before measured counter trials, the endpoint was changed for both arms from 40 s to GUI after the baseline diagnostic warm-up left a temporary crash snapshot holding DLLs. Two subsequent staging attempts launched no game. All warm-ups and raw outputs were retained; the temporary probe was removed and original game/cache restored. The PR description now includes these results. Diagnostic sources and reproduction steps follow; local paths can be adjusted for another checkout. No probe code is committed to the PR. Counter probe, identical instrumentation script, runner and validation fixtureFaultProbe.hpp #pragma once
#include <windows.h>
namespace LocalFaultProbe {
struct Counters {
volatile LONG64 importThread;
volatile LONG64 importWindows;
volatile LONG64 freshImportArms;
volatile LONG64 freshImportFirstWriteFaults;
volatile LONG64 sharedHandlerCalls;
volatile LONG64 armedSharedWriteFaults;
volatile LONG64 otherFirstArms;
};
static Counters* Get_nid_no_patch() {
static auto* counters = []() -> Counters* {
char name[128];
const auto size = GetEnvironmentVariableA("APS5_FAULT_PROBE", name, sizeof(name));
if (size == 0 || size >= sizeof(name)) return nullptr;
const auto mapping = OpenFileMappingA(FILE_MAP_ALL_ACCESS, FALSE, name);
if (!mapping) return nullptr;
auto* result = static_cast<Counters*>(MapViewOfFile(mapping, FILE_MAP_ALL_ACCESS, 0, 0, sizeof(Counters)));
CloseHandle(mapping);
return result;
}();
return counters;
}
struct ImportScope {
Counters* counters = Get_nid_no_patch();
LONG64 previous = 0;
ImportScope() {
if (counters) {
previous = InterlockedExchange64(&counters->importThread, GetCurrentThreadId());
InterlockedIncrement64(&counters->importWindows);
}
}
~ImportScope() {
if (counters) InterlockedExchange64(&counters->importThread, previous);
}
};
}instrument.py from pathlib import Path
import sys
root = Path(sys.argv[1])
header = root / 'core/libs/prx/libc/include/WindowsMappings.hpp'
guest = root / 'core/libs/prx/libSceAgcDriver/Execution/src/GuestMemory.cpp'
include = '#include "E:/AnyPS5/hades-debug/deferred-ab-20261013/FaultProbe.hpp"\n'
def replace_once(text, old, new):
assert text.count(old) == 1, (old, text.count(old))
return text.replace(old, new, 1)
s = header.read_text()
s = replace_once(s, '#include <windows.h>\n', '#include <windows.h>\n' + include)
s = replace_once(s, ' auto& view = found->second;\n invalidate(*view.page);', ''' auto& view = found->second;
if (auto* probe = LocalFaultProbe::Get_nid_no_patch()) {
InterlockedIncrement64(&probe->sharedHandlerCalls);
if (view.armed) {
InterlockedIncrement64(&probe->armedSharedWriteFaults);
if (view.probeFreshImportArm) InterlockedIncrement64(&probe->freshImportFirstWriteFaults);
}
}
view.probeFreshImportArm = false;
invalidate(*view.page);''')
s = replace_once(s, ' other.armed = true;', ''' other.armed = true;
if (auto* probe = LocalFaultProbe::Get_nid_no_patch()) {
const bool first = !other.probeEverArmed;
const bool inImport = InterlockedCompareExchange64(&probe->importThread, 0, 0) == GetCurrentThreadId();
other.probeFreshImportArm = first && inImport;
if (first) InterlockedIncrement64(inImport ? &probe->freshImportArms : &probe->otherFirstArms);
other.probeEverArmed = true;
}''')
s = replace_once(s, ' std::uint32_t hostWrites;\n', ' std::uint32_t hostWrites;\n bool probeEverArmed = false;\n bool probeFreshImportArm = false;\n')
header.write_text(s)
s = guest.read_text()
s = '#ifdef _WIN32\n' + include + '#endif\n' + s
needle = ' const bool before = covered && walkWrites(tracker, first, stop, StampKind::Cpu, true);'
s = replace_once(s, needle, '#ifdef _WIN32\n LocalFaultProbe::ImportScope probeScope;\n#endif\n' + needle)
guest.write_text(s)
print('Applied local fault probe to', root)run_fault_comparison.py import mmap
import struct
import hashlib
import json
import os
from pathlib import Path
import shutil
import subprocess
import sys
import threading
import time
from datetime import datetime, timezone
ROOT = Path(__file__).resolve().parent
GAME = Path(r'E:\Hades2')
MILESTONES = {
'launch_pkg': 'path=/app0/Content/Packages/1080p/Launch.pkg flags=',
'gui_pkg': 'path=/app0/Content/Packages/1080p/GUI.pkg flags=',
'master_bank': 'path=/app0/Content/Audio/PS5/MasterBank.bank flags=',
'music_bank': 'path=/app0/Content/Audio/PS5/Music.bank flags=',
'fatal': 'FATAL:',
}
def sha(path):
return hashlib.sha256(path.read_bytes()).hexdigest()
def deploy(variant):
source = ROOT / variant / 'libs'
assert source.is_dir()
for path in source.glob('*.prx'):
for attempt in range(80):
try:
shutil.copy2(path, GAME / 'libs' / path.name)
break
except PermissionError:
if attempt == 79:
raise
time.sleep(0.25)
def reset_cache(name):
cache = GAME / 'shader_cache'
destination = ROOT / 'run-caches' / name
assert cache.resolve() == Path(r'E:\Hades2\shader_cache')
assert destination.resolve().is_relative_to(ROOT)
assert not destination.exists()
destination.parent.mkdir(exist_ok=True)
if cache.exists():
shutil.move(str(cache), str(destination))
shutil.copytree(ROOT / 'cache-seed', cache)
def run(variant, name, duration=40):
deploy(variant)
reset_cache(name)
env = {k: v for k, v in os.environ.items() if not k.startswith('APS5_')}
env['APS5_HOST_IMPORT_MIB'] = '16384'
probe_name = 'Local\\APS5FaultProbe' + str(os.getpid())
mapping = mmap.mmap(-1, 4096, tagname=probe_name)
env['APS5_FAULT_PROBE'] = probe_name
counter_names = ['import_thread', 'import_windows', 'fresh_import_arms', 'fresh_import_first_write_faults', 'shared_handler_calls', 'armed_shared_write_faults', 'other_first_arms']
def snapshot():
return dict(zip(counter_names, struct.unpack_from('<7q', mapping)))
milestones = {}
milestone_counters = {}
result = {'variant': variant, 'name': name, 'utc': datetime.now(timezone.utc).isoformat(),
'duration_limit': duration, 'flags': {'APS5_HOST_IMPORT_MIB': '16384', 'APS5_FAULT_PROBE': probe_name},
'libc_sha256': sha(GAME / 'libs' / 'libc.prx'),
'agc_sha256': sha(GAME / 'libs' / 'libSceAgcDriver.prx'),
'exe_sha256': sha(GAME / 'Hades2.exe')}
start = time.perf_counter()
process = subprocess.Popen([str(GAME / 'Hades2.exe')], cwd=GAME, env=env,
stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
def read():
with (ROOT / (name + '.log')).open('wb') as raw, (ROOT / (name + '.timed.jsonl')).open('w') as timed:
for line in iter(process.stdout.readline, b''):
elapsed = time.perf_counter() - start
text = line.decode('utf-8', errors='replace').rstrip()
raw.write(line)
timed.write(json.dumps({'seconds': round(elapsed, 6), 'line': text}) + '\n')
for key, marker in MILESTONES.items():
if marker in text and key not in milestones:
milestones[key] = round(elapsed, 6)
milestone_counters[key] = snapshot()
if key in ('gui_pkg', 'fatal') and 'stopped_at' not in result:
result['stopped_at'] = key
process.terminate()
reader = threading.Thread(target=read)
reader.start()
try:
process.wait(timeout=duration)
result['terminated_by_harness'] = False
except subprocess.TimeoutExpired:
result['terminated_by_harness'] = True
result['termination_requested_at'] = round(time.perf_counter() - start, 6)
process.terminate()
process.wait(timeout=15)
reader.join(timeout=15)
assert not reader.is_alive()
result['process_duration'] = round(time.perf_counter() - start, 6)
result['exit_code'] = process.returncode
result['milestones'] = milestones
result['milestone_counters'] = milestone_counters
result['final_counters'] = snapshot()
mapping.close()
result['survived_window'] = result['terminated_by_harness'] and 'fatal' not in milestones
result['both_banks'] = all(k in milestones for k in ('master_bank', 'music_bank'))
if 'stopped_at' in result:
result['terminated_by_harness'] = True
result['survived_window'] = False
(ROOT / (name + '.json')).write_text(json.dumps(result, indent=2))
print(json.dumps(result), flush=True)
if __name__ == '__main__':
run(sys.argv[1], sys.argv[2], int(sys.argv[3]) if len(sys.argv) > 3 else 40)ProbeValidation.cpp #include "C:/Users/Moong/.codex/worktrees/windows-deferred-write-watch/AnyPS5/core/libs/prx/libc/include/WindowsMappings.hpp"
#include <array>
#include <iostream>
LONG CALLBACK Handle(EXCEPTION_POINTERS* exception) {
const auto* record = exception->ExceptionRecord;
if (record->ExceptionCode == EXCEPTION_ACCESS_VIOLATION && record->NumberParameters >= 2 && record->ExceptionInformation[0] == 1) {
if (GuestArena::WindowsMappings::Get().HandleWrite(record->ExceptionInformation[1])) return EXCEPTION_CONTINUE_EXECUTION;
}
return EXCEPTION_CONTINUE_SEARCH;
}
int main() {
auto* handler = AddVectoredExceptionHandler(1, Handle);
auto& mappings = GuestArena::WindowsMappings::Get();
auto section = CreateFileMappingW(INVALID_HANDLE_VALUE, nullptr, PAGE_EXECUTE_READWRITE, 0, 65536, nullptr);
auto* memory = static_cast<unsigned char*>(mappings.Reserve(nullptr, 65536));
if (!handler || !section || !memory) return 1;
mappings.Map(memory, 65536, section, 0, PAGE_READWRITE);
const auto collect = [&](std::size_t offset) {
std::array<void*, 16> pages{};
std::size_t count = pages.size();
if (!mappings.Collect(reinterpret_cast<std::uintptr_t>(memory + offset), 16384, pages.data(), &count, true)) throw std::runtime_error("collect failed");
};
{ LocalFaultProbe::ImportScope scope; collect(0); }
static_cast<volatile unsigned char*>(memory)[0] = 1;
{ LocalFaultProbe::ImportScope scope; collect(0); }
static_cast<volatile unsigned char*>(memory)[0] = 2;
collect(16384);
static_cast<volatile unsigned char*>(memory)[16384] = 3;
mappings.Reset(memory, 65536);
VirtualFree(memory, 0, MEM_RELEASE);
CloseHandle(section);
RemoveVectoredExceptionHandler(handler);
std::cout << "Probe validation completed\n";
}validate_probe.py import json
import mmap
import os
from pathlib import Path
import struct
import subprocess
import sys
root = Path(__file__).resolve().parent
name = 'Local\\APS5FaultProbeValidation' + str(os.getpid())
with mmap.mmap(-1, 4096, tagname=name) as mapping:
env = dict(os.environ, APS5_FAULT_PROBE=name)
result = subprocess.run([str(root / 'probe-validation.exe')], env=env, capture_output=True, text=True)
counters = struct.unpack_from('<7q', mapping)
expected = (0, 2, 1, 1, 3, 3, 1)
record = {'exit_code': result.returncode, 'stdout': result.stdout, 'stderr': result.stderr,
'counters': counters, 'expected': expected, 'passed': result.returncode == 0 and counters == expected}
(root / ('probe-validation-' + sys.argv[1] + '.json')).write_text(json.dumps(record, indent=2))
print(json.dumps(record))
assert record['passed']Apply AI-assisted: OpenAI Codex. |
|
Thanks, that answers the question: first-write faults go from 2,104 to 0 by the first
|
652cb0b to
dfc026a
Compare
|
@oneandonlydean Ran both on PPSA36082, and changed
main + prefault: 0 of 5, the same
Rebased onto c13dc7e. Seven focused and four alternate-mode tests pass. Full suite: 5 failed out of 1,034 — the four shader segfaults already listed, and AI-assisted: Claude Code. |
MarkWritten collects the fresh portions of the written blocks before each GPU stamp. On Windows that walked every shared view of those blocks twice under the mappings mutex even when all of them had been collected before. Keep the never-collected portions as address ranges, as the Linux tracker keeps _fresh, and return after one lookup when none overlaps the request. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dfc026a to
b24e1cc
Compare
|
Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.
|
Titles place fiber contexts back to back in guest memory without guard pages, so a fiber that overflows its stack overwrites the context below it and the title fails much later, far from the cause. With APS5_FIBER_GUARD=1 on Windows, _sceFiberInitializeImpl makes the lowest 4 KiB page of the context PAGE_NOACCESS and sceFiberFinalize, or initializing the object again, restores the protection the page had. A context that does not start on a page or is smaller than two pages gets no guard and the first one is named on stderr; any other value of the variable throws. With the context size check on, the fill skips the guarded page and sceFiberGetInfo starts its scan above it. On Linux the variable throws a not implemented error at the first initialize. A guest unmap of arena memory there maps the range again as a PROT_NONE reservation (GuestArenaReset), which the earlier Linux path took for the guard and gave the old read-write protection at the last finalize; the Linux mapping and protection paths in libkernel do not go through one registry that could drop or keep a guard. Each guard is owned by the fiber that set it through a ticket. Fibers that share a context share the guard and the last finalize removes it. Releasing a guard whose ticket is not registered throws, except once for a ticket whose guard a guest unmap or mapping dropped, so a fiber whose page was unmapped and given to a new fiber cannot remove the new fiber's guard. Sharing a guard and the last finalize check that the page is still committed PAGE_NOACCESS and throw otherwise, so a direct VirtualProtect by host code is reported instead of the page being silently left unguarded or given a stale protection. A fiber whose guard is refused is left uninitialized and unpinned, and a finalize that throws leaves the fiber initialized. The guard is registered in WindowsMappings, and every guest protection change and unmap goes through it: - HandleWrite no longer takes a write fault in a guard page for a write to an armed view; before, it reopened the 4 KiB page writable and resumed the overflowing write; - the tracker's VirtualProtect calls in Commit, Pin, HandleWrite, BeginHostWrite and the per-page arming in Collect go through one helper that protects the pages around each guard in the range, never the guard itself, and records the protection to restore at finalize, so arming or opening a run of 4 KiB pages of a view leaves its guard PAGE_NOACCESS; Map keeps its direct VirtualProtect because reset has already dropped every guard in the range; - Collect neither reports, settles nor arms a guarded 4 KiB page of a shared view, so the guard is never taken for a CPU write, and the page's pending state from before the guard is reported after the guard is removed; every other page keeps the per-page tracking; - fresh collection (boykopovar#4535) skips a guarded page the way Collect does: it neither activates nor reports the page, and does not count it as fresh when it decides whether anything is fresh or whether an inaccessible view fails. Guard takes the page out of the fresh ranges, and removing or dropping the guard puts it back if it was never collected; every other page keeps the deferred activation; - the libkernel mprotect loop sends every committed region through that helper (GuestArenaProtect_nid_no_patch), inside the guest arena and outside it. Memory outside the arena, such as the title image, went to VirtualProtect directly before: a guest PROT_NONE of the guarded page left it PAGE_NOACCESS, the check above passed, and the last finalize made the page writable again with the protection read at initialize. A range that crosses the arena boundary took the same direct path; - unmapping or remapping arena memory (reset), and the libkernel munmap of memory outside the arena (GuestArenaDecommit_nid_no_patch and GuestArenaReleaseAllocation_nid_no_patch, which call VirtualFree), drop the guard and record its tickets. An unmap outside the arena left the guard registered before, and the next finalize threw. Guard refuses a range that is not made of whole 4 KiB pages. The context is pinned writable before the guard is set, so no view of the guarded page is armed while it is guarded, and pinned pages other than the guard are collected as written as before. A protection change is not a write for GetWriteWatch (Microsoft VirtualProtect and GetWriteWatch documentation), so the guard adds no written page. Test: fiber_guard, registered on Windows only, runs fiber_tests guard with APS5_FIBER_GUARD=1 on host, flexible and two-view direct memory: the lowest page is inaccessible after initialize and readable again after finalize, the fibers run and return, HandleWrite refuses a fault in the guard, collection, a second pin and guest mprotect keep it, no write is collected in the guard page, a pinned shared context is collected as written except its guard page, and a write to the former guard page is collected as one page in both views after finalize. It also unmaps a guarded context, maps it again at the same address for a second fiber and finalizes the first fiber, which must not remove the second fiber's guard; reinitializes one of two fibers sharing a guarded context with the size check on; releases unknown, twice used and dropped tickets, which throw except for the first release of a dropped one; and changes the guard page with VirtualProtect, after which sharing the guard and the last finalize throw and finalize succeeds once the guard is back. CheckGuardProtectedOutsideArena guards a context in the test executable's .bss, makes it PROT_NONE with sceKernelMprotect and finalizes; the page must stay PAGE_NOACCESS. The same with read-only must keep the guard and be restored at finalize. CheckGuardUnmappedOutsideArena maps flexible memory at a fixed address outside the arena, guards both halves, unmaps them (decommit, then release) and finalizes each fiber, which must succeed once, then maps the range again PROT_NONE and guards it for a new fiber. CheckSharedWritesAroundGuard in guest_memory guards the lowest 4 KiB page of a 16 KiB shared view, writes the other three, collects and re-arms: the guard stays PAGE_NOACCESS, a fault in it is not resumed, exactly the three written pages are reported in each of two rounds, and after the guard is removed the page is reported once, because it was never collected, then armed, and a write to it is reported alone. CheckFreshAroundGuard in windows_write_watch writes the second 4 KiB page of a freshly mapped shared view, guards it and collects fresh: the other fifteen pages are reported and activated, the guard stays PAGE_NOACCESS through activation and a re-arm beside it, and it is neither reported, activated nor counted as fresh, also in a PROT_NONE view. After the guard is removed it is reported once by fresh collection, activated, and a write to it is reported. Without the guard handling in fresh collection it fails at "fresh collection reported a guarded page or lost a fresh one". fiber_tests renames its .eh_frame like the other test executables so that it can catch exceptions thrown in libc.prx. fiber checks that nothing is guarded without the variable. Without the mprotect change fiber_guard fails at "finalize keeps the guest PROT_NONE of the former guard page"; without the munmap change it fails at "a guard dropped by a decommit outside the guest arena is released once". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
What
Windows shared direct-memory imports currently collect and arm every page, even when no consumer has collected it before. Unlike private
MEM_WRITE_WATCHallocations, section views are tracked by removing write access, so importing fresh memory introduces first-write faults on the guest's loading threads. This gives Windows shared mappings the deferred activation already used by the Linux path in #1960.Track collection history for each 4 KiB portion of a 16 KiB shared view, independently of its dirty generation and current protection. Import-time armed collection skips portions never collected. Before
MarkWrittenstamps a GPU write, fresh collection activates those portions across the containing 64 KiB tracking blocks and stamps them before the GPU generation. Later CPU writes remain visible. Protection still operates on whole views and their aliases; privateMEM_WRITE_WATCHcollection retains its existing behavior. Fresh collection skips unmapped neighbors and private allocations and preserves dirty pages that were already collected. The existing partial-output contract for ordinary collection is retained. The never-collected portions are also kept as address ranges, as the Linux tracker keeps_fresh, so a fresh collection returns after one lookup when none overlaps the request instead of walking every view of the written blocks on each GPU write.Seven files; no environment toggle, title-specific bypass, prefault worker or rendering changes. The prefault work from closed #4290 remains separate; it was closed to keep one open PR and will be resubmitted as a follow-up PR after this one, with evidence gathered on top of this change. This change alone does not reliably get Hades II through loading in the runs below; with the separate prefault patch on both sides it decides whether the title loads (0 of 5 without it, 5 of 5 with it).
Ready for review. The automated overlap comment currently lists #3874 in
GuestMemory.cppand #4661 inWindowsMappings.hpp. #3874 compares import-window bytes; this PR defers activation of fresh Windows shared pages. No ordering agreement or combined validation with either is claimed. The earlier overlap with #2802 was resolved by basing this branch on its merge inmain, preserving its unstamped-store notifications; the branch is now rebased ontomainc13dc7e without conflicts. There is no code dependency on an open PR. The separate prefault patch remains follow-up work.Tested
Windows 11 Pro 26200, Ryzen 7 9800X3D, 47 GB RAM, Radeon RX 9070 XT (driver 32.0.32015.2008), WinLibs GCC 15.2, Release,
BUILD_TESTING=ON,ANYPS5_ENABLE_SPIRV_TOOLS=ON.Current head
dfc026af4(9694782 is the earlier commit rebased; dfc026a adds the fresh-range early return), based onmainc13dc7e6b: full build passed; all seven focused tests passed (1.40 s); all four alternate-mode tests passed (1.38 s). Fullctest --output-on-failure -j4: 1,019 passed, 10 skipped, 5 failed out of 1,034, in 125.47 s. Four failures are the shader tests listed below; the fifth,agc_driver_pooled_buffer_range, failed withvkMapMemory buffer slab: Vulkan result -5under-j4and passed 3 of 3 when rerun alone. A failure-only baseline run was not repeated on c13dc7e.git diff --checkandtools/check_conventions.py --base origin/mainpassed.October 13: prefault on both sides, requested in review.
mainc13dc7e + 1ad61fe versus 9694782 + 1ad61fe, Hades II PPSA36082 v01.006, same relinked executable. Every run restored the same warm shader-cache seed (184 files, 23,466,293 bytes, built for the new shader source version), cleared inheritedAPS5_*variables and set onlyAPS5_HOST_IMPORT_MIB=16384andAPS5_PREFAULT_MEMORY=1, with write tracking enabled. One warm-up per build was excluded before execution; five measured runs per build alternated main/PR over a 40-second window. No builds or tests ran concurrently.main + prefault: 0 of 5, each the same
LoadTaskSchedulernull write afterGUI.pkg; its warm-up crashed too. This PR + prefault: 5 of 5 opened both banks and survived; its warm-up survived. Outside the measured set, main + prefault survived once, on the first run from an empty shader cache, then crashed on two warm cache-priming runs and six further empty-cache runs. Times are from process creation to receipt of a flushed log line.Frame rate was read from the window title (no instrumentation) with no input. This PR + prefault held 119.85-119.91 FPS, the 120 cap, from 20 s to 170 s in one run, and median 119.88 FPS from 20 s to 60 s in five further runs (three with dfc026a, two without, alternating; all five loaded). main + prefault never presented more than 15 frames, so no main frame time exists for this title, and the cap hides any difference between the two PR builds.
Per-call cost of a fresh
Collect, standalone program over 256 MiB of mapped shared views, before and after dfc026a (second of two rounds):No gameplay, cold-cache reliability, other-title, Linux or other-GPU claim is made. The prefault patch is not part of this branch.
Previous head
652cb0b87, based onmaind592d54e1: full build passed; all seven focused tests passed (1.78 s); all four alternate-mode tests passed (1.49 s). Fullctest --output-on-failure -j4: 968 passed, 8 skipped, 4 failed out of 980, in 87.29 s. The failures are the same four shader tests listed below and reproduced on the earlier untouched baseline; a separate failure-only baseline run was not repeated on d592d54.git diff --checkandtools/check_conventions.py --base origin/main --title ... --body ...passed. The overlap check at that time reported #3874; the current automated comment also lists #4661.The focused selection was
windows_write_watch,agc_write_watch_coverage,agc_write_tracking,agc_shared_write,agc_shared_write_partial_collect,agc_shared_write_partial_collect_two_pass, andguest_memory. The alternate-mode selection waswindows_write_watch,agc_write_tracking,agc_shared_write, andguest_memory.New backend coverage checks fresh/armed selection, 4 KiB history within a view, aliases, one-entry output buffers, remapping, pins, host writes, inaccessible-page failure, holes, private memory and concurrent writes. AGC integration coverage checks import-time deferral, activation before the first GPU stamp, CPU writes after GPU writes, adjacent fresh portions, repeated GPU writes, import-window stamps and an unaligned import beside an unmapped range.
Before the rebase, 840fcb475 on
main8fa5cc4 built successfully. All seven focused tests passed (1.62 s); four tracking tests also passed withAPS5_NO_SINGLE_PASS_COLLECT=1(1.49 s). Full CTest: 966 passed, 8 skipped, 4 failed out of 978 (102.70 s). The four failures were reproduced serially on untouchedmain8fa5cc4:agc_driver_division_output_modifiers,agc_driver_division_result_modifiers,agc_driver_f64_conversions, andagc_driver_f64_transcendental, all SEGFAULT. This is a baseline comparison, not a claim that the full suite is green.Five isolated backend mutations were rejected by the new tests: armed collection consuming fresh pages, fresh collection returning nothing, not recording history, tracking history only per 16 KiB view, and fresh collection consuming existing dirty pages. Those mutation runs preceded the final partial-output compatibility adjustment; they were not rerun after it.
October 13: matched baseline comparison requested in review. Clean upstream
d592d54e1(this PR's exact parent) versus published head652cb0b87, on the Windows/GPU/toolchain above, Hades II PPSA36082 v01.006. This is a fixed-base comparison; newer upstream changes were not included. Every run restored the same warm shader-cache snapshot (2,396 files, 330,400,440 bytes), cleared inheritedAPS5_*variables and set onlyAPS5_HOST_IMPORT_MIB=16384, with write tracking enabled. No prefault, import bypass, loading throttle, backpressure or unrelated local patches. One warm-up per build was excluded before execution; the three measured runs per build alternated baseline/PR. No builds or tests ran concurrently with the game. OS file/driver caches were not reset.The uninstrumented loading comparison:
All six crashed before either audio-bank open with the same
LoadTaskSchedulernull-write signature. Baseline FATAL timestamps were 9.431/9.499/9.326 s; PR timestamps 9.222/9.401/9.024 s. Times are from process creation to receipt of a flushed log line, not completed asset loads or rendered frames; crash reporting added about ten seconds to process exit. Median Launch-to-GUI interval was 0.622 s in both arms. The predesignated baseline warm-up survived and opened both banks, while the PR warm-up crashed at 9.114 s; including them gives baseline 1/4 and PR 0/4. The sample does not establish a loading benefit or a regression.A separate diagnostic comparison applied identical local counter instrumentation to both source revisions and ran three alternating pairs to the same first
GUI.pkgopen:Each run reached four import windows. The baseline initially armed 434,176 shared views inside imports in every run; the PR armed zero there and initially armed 1,416 views outside imports. Median import-first write faults 2,104 -> 0; median all armed shared-write faults 2,124 -> 26 (98.78% fewer at this milestone). Ordinary write tracking remained active.
The counter tags a mapped 16 KiB virtual view when its first successful read-only protection occurs inside
ImportWatched, then counts a write access violation handled while that initial protection remains armed. Later re-arms do not count as fresh-import arms; aliases count separately and remapping resets the diagnostic history. These are shared-view write-protection exceptions, not hardware/soft page faults or unique physical pages. Atomic counters are read from a named shared-memory buffer; no per-fault printing or sampling thread is added to the game. The extra diagnostic variable names that buffer. Snapshots occur on log receipt and are not globally atomic across fields; instrumentation can affect timing.Counter validation passed against both headers: two import collections/writes of one view and a non-import collection/write of another produced exactly 2 import scopes, 1 initial import arm, 1 associated first-write fault, 3 total armed write faults and 1 other initial arm. Instrumented PR
windows_write_watchandagc_write_trackingpassed; baselineagc_write_trackingpassed. Full suites were not rerun for the diagnostic edits.The instrumented runs intentionally stopped immediately after the GUI counter snapshot and make no 40-second survival claim. The endpoint was set for both arms before their measured runs, after the baseline warm-up's crash left a temporary crash-snapshot child holding DLLs. Two PR staging attempts failed before starting a game. Both diagnostic warm-ups and all run outputs were retained. The source instrumentation was removed and the original game/cache restored afterward.
These measurements show removal of the observed import-induced first-write faults. They do not demonstrate faster/reliable loading or gameplay improvement. Diagnostic source, counter definitions and reproduction steps are in the review reply; none of the instrumentation is included in this branch.
Earlier title runs (October 12; before the rebase):
Hades II (PPSA36082 v01.006; bring-up #1965), warm existing shader caches, built from 840fcb475 before the rebase. Each run cleared inherited
APS5_*variables and setAPS5_HOST_IMPORT_MIB=16384. Write tracking stayed enabled. No experimental import bypass, loading throttle, backpressure or local gameplay patches were included.Music.bankandMasterBank.bank; two crashed with a null write onLoadTaskScheduler, consistent with the known upload-pool exhaustion failure.APS5_PREFAULT_MEMORY=1Surviving runs were terminated by the harness; their termination exit code is not a crash. The two crashing processes lasted 20.606 and 19.293 seconds including crash reporting. No drawn-menu, gameplay/FPS, cold-cache, other-title, Linux or other-GPU claim is made. The October 13 comparison above uses the published head; the combined-prefault runs were repeated on the rebased head in the prefault comparison above. The temporary prefault patch was removed from the branch, and the original game executable and libraries were restored and hash-checked. Full upstream CI requires a maintainer's manual workflow run.
Checklist
main; no other open PR implements the same functionsNotImplemented_nid_no_patch); silent stubs are listed in TechnicalDebt.mdfiles or images added to the repository (attach them to this PR)