[JJenkx]

What

The Windows samplers (APS5_SAMPLE_WORKER, APS5_SAMPLE_COMMITTER, APS5_SAMPLE_THREADS) write every frame as module+0xRVA, and docs/dev/BUILD.md left naming them to the reader ("for example with nm on the .prx"). Doing that by hand takes three steps:

• find each module's image base;
• choose between COFF symbols and the export table;
• remember that the files hold totals since start-up.

tools/symbolize_samples.py reads a collapsed-stack file or the address file. It opens each module it finds in --libs as a PE32+ image and names every frame module!function:

• COFF symbols. It reads the symbol table that the MinGW-built .prx and .dll keep: external and static symbols in code sections, at the section's RVA plus the symbol value. An address outside a code section, or before the first symbol, stays module+0xRVA.
• Export table. A module without symbols, such as the runtime DLLs, is named from its export table, with a ~ prefix to mark the less exact name.
• Module check. If a module's SizeOfImage or TimeDateStamp differs from the sampler's # module line, its frames are left unsymbolized and a warning is printed, because its addresses would name the wrong functions. That is easy to get wrong when main and a candidate runtime are swapped between runs.

Options:

• --since <earlier copy> subtracts an earlier copy of the same file, keeping only the samples taken between the two copies. The sampler rewrites cumulative totals every 20 s, so two copies taken during a run give a steady-state window that leaves out loading.
• --out writes the named stacks for speedscope.app or flamegraph.pl.
• --demangle uses c++filt when it is on PATH.

Nothing else is needed: no binutils, and the PE parsing is in Python.

docs/dev/BUILD.md points at the tool where it described offline symbolization.

Tested

• tools/tests/test_symbolize_samples.py, registered with CTest as symbolize_samples, builds small PE32+ images, one with COFF symbols and one with an export table. At head cdc21a9897 (on main 0db47b263d) its 11 cases pass in 3 of 3 runs on Windows 11 with CPython 3.15.1. They cover:
  • COFF symbols naming code addresses (short, string-table and static names; section and data symbols ignored);
  • the export-table fallback for a module without symbols;
  • unknown modules and out-of-range addresses kept as frames;
  • a module whose size or timestamp differs, or that has no record, staying unsymbolized (unless unverified symbolization is asked for);
  • a symbol not reaching into a later code section;
  • --since with --out on stack files, and --since on both counters of an address file;
  • the address-file format (self and inclusive samples);
  • thread files keeping each heading, with one thread selectable;
  • rejected malformed lines with their line number.
• All 50 tools/tests/test_*.py scripts that CMakeLists.txt registers as CTest entries pass when run directly at that head.
• On a real capture, an APS5_SAMPLE_WORKER file pair from Spirit of the Island's title screen (copies at 62 s and 142 s, symbolized against the libraries that ran) takes 7.9 s. Excerpt:

5124 samples
hottest by self:
       306   6.0%  libSceAgcDriver.prx!(anonymous namespace)::Allocate(unsigned long long)
       155   3.0%  libSceAgcDriver.prx!AgcDriver::DriverDetail::Driver::draw(...)
       110   2.1%  libSceAgcDriver.prx!ShaderRecompiler::ResourceMaterializer::Materialize(...)
hottest by inclusive:
      5119  99.9%  libwinpthread-1.dll!~pthread_create_wrapper
      4730  92.3%  libSceAgcDriver.prx!AgcDriver::DriverDetail::Driver::execute(AgcDriver::DriverDetail::Submission const&)

System DLLs outside --libs (ntdll.dll, KERNEL32.DLL) stay module+0xRVA. Passing --libs C:\Windows\System32 names them from their export tables.

• tools/check_conventions.py passes. tools/env_reference.py --check reports APS5_NO_WGI undocumented, which comes from libSceVideoOut on current main and is not touched here.

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: none
• Fixes: none
• Unblocks: none
• GPUs tested: n/a
• AI-assisted: yes

🤖 Generated with Claude Code

[github-actions]

Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.

Pull request	Conflicting files (lines on main)	Same exports	Same new files
PR 8161	CMakeLists.txt L254

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request	Shared files not compared
PR 674	CMakeLists.txt docs/dev/BUILD.md
PR 1835	CMakeLists.txt docs/dev/BUILD.md
PR 3786	CMakeLists.txt
PR 5021	CMakeLists.txt docs/dev/BUILD.md
PR 7201	CMakeLists.txt
PR 9739	CMakeLists.txt

[MarcZX10]

I found three correctness gaps in the advertised sampler inputs:

1. PeModule.name() checks that an RVA is in some code section, then bisects one global symbol-address list. If an RVA is in a later code section before that section's first symbol, the last symbol from an earlier section is returned and the frame is mislabeled. The description says addresses before the first symbol stay unsymbolized. Keep symbol lookup bounded to the containing code section (and add a two-code-section test with an address before the second section's first symbol).

2. Address-file rows contain both self and inclusive counts (module offset self inclusive), but load() stores only parts[2], and summarize() emits no inclusive table for address input. This drops data from APS5_SAMPLE_THREADS and the address-file form of the other samplers despite the tool description promising hottest-by-inclusive output. Preserve and subtract both counters, then test that inclusive totals are reported correctly.

3. The APS5_SAMPLE_THREADS output has no # module name size timestamp records (the current ProcessSampler::write() emits only rounds, thread headings and address rows). Thus this input bypasses the PE size/timestamp guard and can silently symbolize addresses against a different build. Add the module identity metadata to that sampler output, or make the address-input path explicitly refuse/mark unverified symbolization when it cannot validate the binaries; cover the mismatch case.

The hosted conventions, eligibility, intake, labels, relinker matrix and sanitizer checks pass. The broader hosted Linux/Windows jobs are skipped. The author reports seven Python cases passing on Windows and a real capture, but the cases shown do not cover the multiple-code-section lookup or the missing metadata/inclusive-count paths above.

[JJenkx]

Thanks, @MarcZX10. All three are addressed in 0e039b9, on top of 0bf64f3.

1. Lookup stays inside the containing code section. Each code section now keeps its own sorted symbol list, and name() bisects only the list of the section the RVA falls in. test_a_symbol_does_not_reach_into_a_later_code_section builds a PE with two code sections and checks four addresses:

   • 0x1085 in the first section → static_f
   • 0x2810, in the second section before its first symbol → unsymbolized
   • 0x2840 and 0x28f0 → late_fn

2. Address files keep both counters. load() stores self and inclusive counts, and --since subtracts both.

   • The summary now prints an inclusive table for address input too. A function's inclusive count is the sum of its addresses' counts. The sampler adds one per frame occurrence, so that sum is an upper bound when a function is on one stack more than once; the tool's description and BUILD.md say so.
   • The total comes from # N samples, or from the thread <id> samples <n> headings.
   • While checking point 3, I found the loader misread those APS5_SAMPLE_THREADS headings as stack lines. They are now parsed, and --thread <id> keeps one thread.
   • Tests: test_address_files_are_summarized_by_self_and_inclusive_samples, test_since_subtracts_both_counters_of_an_address_file and test_thread_files_keep_each_heading_and_can_select_one_thread.

3. Module identity. I did both of your suggestions:

   • Both address writers, Sampler::write (the APS5_SAMPLE_WORKER/COMMITTER address file) and ProcessSampler::write (APS5_SAMPLE_THREADS), now write # module name size timestamp records. They use a writeModules helper, which the collapsed-stack writer now calls as well.
   • The tool leaves a module unsymbolized, with a warning, when the samples have no record for it. --unverified overrides this. test_a_module_without_a_record_stays_unsymbolized_unless_unverified covers both cases; a mismatched record was already covered.

Checks:

• The 11 Python cases pass on Windows (Python 3.15).
• Run against the tool at 0bf64f3, the new cases fail: 3 failures and 3 errors.
• WorkerSampler.cpp compiles and libSceAgcDriver.prx links (WinLibs GCC 15.2.0 UCRT r7, Release).
• tools/check_conventions.py passes, and the branch merges cleanly with main 110a18b.

Not done: a new capture with the sampler change, which needs a game run. Until then, the new # module lines in the address files are checked only by the parser tests.

[MarcZX10]

Re-reviewed head 0e039b91bb459299dec2b446592d5e6ad2cd28df. The three correctness issues from my earlier review are addressed: lookup is bounded to the containing code section, address input now retains and reports inclusive counts (including --since subtraction), and both address samplers write module size/timestamp records while unknown modules remain unsymbolized unless --unverified is explicit. I found no remaining defect in those paths. I ran the PR's exact tools/tests/test_symbolize_samples.py against its matching tools/symbolize_samples.py: all 11 tests pass. git diff --check and the current-main tools/check_conventions.py pass. Hosted eligibility, conventions, labels, report, and all four relinker jobs pass; Linux/Windows product build jobs are skipped by repository policy. The Windows capture and Windows Python 3.15 results in the description remain author-reported, not runs I performed. Please refresh this tooling PR from current main before asking for final review: check_main_sync.py reports 2 commits ahead / 107 behind (merge base 6ef5a1c4c), though it currently merges cleanly. The overlap workflow's partial comparison identifies the shared docs/dev/BUILD.md edit with merged #8087; that is not an open-PR ordering conflict, and I manually confirmed the PR's edits document this tool. The local overlap helper could not finish because the configured upstream remote points directly at the repository, while this script expects an upstream/main branch ref; its fetch failed, so I am not treating that as a clean automated scan. I have not found a code-level blocker after the fixes; this request is for current-main freshness only. I withdraw the queue-based hold; it is fine to keep submitting PRs and they will be handled in due course.

I’ve noticed that holding this PR because of #5239 isn’t necessary. It’s fine for PRs to keep coming in; they’ll be handled in due course. Please disregard my earlier request to hold or defer it.

[JJenkx]

Thanks @MarcZX10, for re-checking the three fixes and running the tests. I'm converting this to a draft under the #5239 hold, since no title run depends on it. Before marking it ready again I'll rebase it onto current main (107 commits behind) and rerun test_symbolize_samples.py and the conventions check. I'll also rebuild WorkerSampler.cpp on Windows and capture a fresh sample, so the Windows result isn't only from the old base.

[JJenkx]

Rebased onto main 8b3a6ba as requested. tools/tests/test_symbolize_samples.py and tools/check_conventions.py --base origin/main pass, and the rebased WorkerSampler.cpp compiles with the Windows UCRT GCC 15.2 flags. The PR stays a draft under the intake rule until a fresh Windows capture is taken through the shared launch queue.

[MarcZX10]

Re-reviewed exact head b902ffc467c3de65689e70e552825073265c6084. The earlier code findings are resolved: lookups stay within the containing code section, address input preserves both counters through --since, and the samplers record module identity before the tool resolves frames. I found no new implementation defect in those paths.

Checks I ran on this exact head: - tools/tests/test_symbolize_samples.py: 11/11 passed. - tools/check_conventions.py --base upstream/main: passed. - tools/check_test_registration.py --scripts: 172 registered scripts. - git diff --check: passed; merge-tree with current main is clean. - x86_64-w64-mingw32-g++ -std=c++20 -Wall -Wextra -Werror -fsyntax-only -Icore/libs core/libs/prx/libSceAgcDriver/Execution/src/WorkerSampler.cpp: passed. This is a compile check, not a Windows runtime/capture.

Two items remain before final review. First, this branch is based on 8b3a6ba88d8fc74e768f1a7e8e9e36fb69842768, 41 commits behind current upstream/main 329efe1bfdaff116c57cdf62718026ee4f5521ae; please update and rerun the focused checks. Second, the new C++ sampler output (# module name size timestamp) has not yet been exercised through a fresh capture. The synthetic PE tests validate parsing, but do not verify that the sampler writes records matching the binaries actually sampled. Please take one Windows capture on the updated head, run it through symbolize_samples.py with those exact binaries, and confirm matching modules symbolize while mismatched/unrecorded modules remain unsymbolized.

I withdraw my earlier intake objection and the queue-based draft rationale. I am not asking that the PR stay draft because of queue size. Hosted relinker/matrix checks pass; hosted product Linux/Windows jobs are skipped. The reported Windows test/build results and previous Spirit of the Island capture remain author-reported; I did not run a Windows runtime capture here.

I’ve noticed that holding this PR because of #5239 isn’t necessary. It’s fine for PRs to keep coming in; they’ll be handled in due course. Please disregard my earlier request to hold or defer it.

[JJenkx]

@MarcZX10 thanks. Rebased onto main b7d1d47 (head 0ec1451, the same two commits). tools/tests/test_symbolize_samples.py (OK), tools/check_conventions.py and git diff --check pass on Windows, and the rebased WorkerSampler.cpp compiles with the build's own flags (UCRT GCC 15.2, syntax-only).

The fresh capture on this head needs a native title run. On this machine those go through the shared FIFO launch queue, which is currently busy with other runs. Once a slot comes through, I'll capture on this exact head, symbolize it with the exact binaries that were sampled, and post the matched/unmatched module counts here. The PR stays a draft until then.

[MarcZX10]

I’ve noticed that asking you to hold this PR was unnecessary. It’s fine to keep submitting PRs; maintainers will handle them in due course.

[JJenkx]

@MarcZX10 here is the fresh Windows capture you asked for, run through symbolize_samples.py at head 0ec1451e15.

Capture.

• Build: main 83462cc with this PR's two commits, Release, WinLibs GCC 15.2 UCRT. WorkerSampler.cpp is identical between 83462cc and this head's base b7d1d47, so the sampler code that ran is the head's.
• Run: Spirit of the Island (PPSA13934), title screen, 150 s with APS5_SAMPLE_WORKER, through the shared launch queue. RX 6700 XT, Windows 11 22621.
• Output: 8,959 samples, depth 64, 13 truncated. The file holds a # module name size timestamp record for every module that appears in a frame (13 modules). libSceAgcDriver.prx is recorded as 0xd3c000 0x6adaafb9.

Matching binaries (--libs = the runtime's own libs, --demangle, 3.7 s): the emulator's modules resolve to functions.

hottest by self:
   154  1.7%  libSceAgcDriver.prx!AgcDriver::DriverDetail::Driver::draw(...)
   125  1.4%  libSceAgcDriver.prx!ShaderRecompiler::Detail::SrtTape::Run(...)
   108  1.2%  libSceAgcDriver.prx!(anonymous namespace)::Allocate(unsigned long long)
   102  1.1%  libSceAgcDriver.prx!ShaderRecompiler::ResourceMaterializer::Materialize(...)
hottest by inclusive:
  8947 99.9%  libSceAgcDriver.prx!AgcDriver::DriverDetail::Driver::run(unsigned int)
  7332 81.8%  libSceAgcDriver.prx!AgcDriver::DriverDetail::Driver::execute(...)
  5822 65.0%  libSceAgcDriver.prx!AgcDriver::DriverDetail::Driver::draw(...)

System DLLs outside --libs stay module+0xRVA (ntdll.dll+0xa1954). libwinpthread-1.dll names are export-table matches, marked ~.

Mismatched binary. The same samples with --libs pointing at another runtime, whose libSceAgcDriver.prx has the same size but timestamp 0x6adab3fd:

warning: libSceAgcDriver.prx: the sampled module had size 0xd3c000 and timestamp 0x6adaafb9, but ...\libSceAgcDriver.prx has 0xd3c000 and 0x6adab3fd; its frames sta[y unsymbolized]

• 0 libSceAgcDriver.prx! names remain; its frames stay as addresses.
• The matching libwinpthread-1.dll still symbolizes.

Every module in this capture has a record, so it doesn't exercise the unrecorded case. That path stays covered by the unit tests. Marking ready for review.

[MarcZX10]

Thanks for adding the fresh Windows capture on head 0ec1451. It now demonstrates matching module metadata symbolizing against the runtime binaries and a same-size/different-timestamp module remaining unsymbolized; that resolves the capture gap from my prior review. I refreshed checks locally: conventions, test registration (171 scripts), and diff whitespace pass. One current-main item remains: check_main_sync.py --base upstream/main --head <head> reports this branch 2 commits ahead / 527 behind, with a clean merge. The earlier reviewed fixes appear preserved, but this branch is no longer based on current main as required by the PR checklist. Please merge current main and rerun the focused Python suite and conventions on that resulting head; I can then review the combined diff for regressions. No queue-size objection remains, and I am not asking that this PR be held or closed because of intake volume.

[JJenkx]

Rebased onto current main 0db47b263d, as requested; the new head is cdc21a9897.

• git range-diff against the previous head 0ec1451e15 shows both commits unchanged (=). There were no conflicts.
• git diff --check and tools/check_conventions.py --base origin/main --head HEAD pass.

Python (CPython 3.15.1, Windows 11):

• tools/tests/test_symbolize_samples.py: 11/11 pass, three runs.
• All 50 tools/tests/test_*.py scripts that CMakeLists.txt registers as CTest entries, each run directly: 50 pass, 0 fail.

The native capture from my earlier comment (0ec1451e15) still applies: the C++ sampler and the symbolizer are byte-identical in this rebase.

Dismissing a stale review: the requested changes were addressed after it (new head and/or updated description with the asked-for runs) and the merge queue blocks on it. Please request changes again if anything is still open.

[THEMANnt]

Reviewed head cdc21a9897e7a61c7b19298c0d8460f3fa89c4a1. No actionable findings in this focused Windows sampler-symbolization review.

The helper maps COFF code symbols and export fallbacks within their code sections, retains unresolved frames, and checks recorded module size/timestamp before trusting a binary. Both address and collapsed-stack inputs support window subtraction; thread-address files can be filtered by thread. The native change shares the existing module-record writer and adds those identity records to address/process output as well as collapsed stacks.

Validation: Windows/Python 3.11.9/MinGW GCC 15.2.0: all 11 symbolizer tests pass without skips. The exact WorkerSampler.cpp translation unit passes syntax checking. Compiled and executed the exact writeModules method against a real loaded executable, verified its size/timestamp against the Python PE parser, and symbolized seven samples of a known function from the executable's actual COFF table. Conventions, whitespace and script registration pass. Baseline has no symbolizer module, so its grafted-test import failure is not a sensitivity demonstration. Hosted relinker/ASan and policy checks pass; product jobs are skipped.

Scope: Synthetic sampler formats, a native metadata/real-PE round trip and full-unit syntax. No live suspended-thread sampling session, full driver rebuild or game execution.

[adriadam10]

Checked the section-bounded symbol lookup, both counters through --since, the module size/timestamp check (including unrecorded modules staying unsymbolized), the export-table fallback and the shared writeModules in the three samplers. The new symbolize_samples ctest entry merges cleanly onto current main, each new test pins symbol names only this tool produces, and MarcZX10's earlier findings are fixed; merged.