[Bram910]

What

The call-site resolver skips REX prefixes but stops at F2 (BND/REPNE) or 3E (NOTRACK/DS). Valid prefixed indirect calls and tail jumps therefore produce empty callSites and callSitesResolved: false in the import registry, despite referring to the same GOT slots as unprefixed instructions.

Skip these two legacy prefixes when locating the opcode and RIP displacement. Clear the accumulated REX.W state when either appears, so an earlier REX cannot incorrectly widen a subsequent pointer load. This changes registry analysis only; it does not rewrite instructions or emulate BND/CET behavior. Instructions with F3, 66, 67, F0, 2E/26/36, FS/GS prefixes, and 8B loads without REX.W remain unclassified; these limits are recorded beside the registry entry in docs/dev/debt/relinker.md.

Add 22 regression scenarios covering individual and combined hints, both prefix orders, REX placement, pointer loads, and negative controls for narrow loads, segment/address overrides, register transfers and unrelated memory operations. GNU binutils' NOTRACK prefix tests also cover combined BND/NOTRACK forms and their prefix ordering.

Tested

• Implementation base: upstream main at 1e740a23a3; implementation commit 16eb32f20ef6f9b481a5056d783c67fc2f7c884e. The negative control and first round of testing used d098eca3d6 / 00897f4b47. The intervening merge changes only unrelated AGC tests; the implementation retains the same stable patch ID (e28cdac41e62502b22c672f67e3ffc8d49ebe720) across that rebase.
• Red/green: the extended C++ resolver test fails against unchanged main on both Windows and Linux at BND indirect call, then passes with this fix.
• Real CLI negative control on main: 20 synthetic ELF conversions to Linux and Windows output formats. The four unprefixed controls resolve correctly; all 16 BND/NOTRACK variants have empty call sites. With the fix, all 20/20 conversions resolve 0x200 to the GOT slot at 0x300 on both the Windows and native Linux relinkers (40 successful fixed conversions). No title was used.
• Windows x64, MinGW GCC 14.1.0: both registered call_site_resolver and call_site_index CTests pass five consecutive runs each.
• Windows Ninja Release, ANYPS5_RELINKER_ONLY=ON: cmake --build build-relinker --parallel 4 succeeds, and ctest --test-dir build-relinker --output-on-failure --parallel 4 passes 241/241 tests on the final rebased head (169.97 seconds). The initial pre-rebase full run also passed 241/241.
• Ubuntu 22.04 under WSL, GCC 11.4: both tests compile with C++20 and -Wall -Wextra -Werror, in Release and ASan/UBSan builds, and pass five consecutive runs each per build (20 successful component runs). Sanitizer options: detect_leaks=1:halt_on_error=1 and halt_on_error=1:print_stacktrace=1.
• Linux CMake builds the actual relinker and both test targets successfully on the final rebased head. Both registered call-site CTests also pass five consecutive runs each (0.01–0.02 seconds per run), with no timeout changes; the 20 real CLI conversion cases pass again after rebasing.
• Independent native x86-64 check: assemble and disassemble eight indirect call/jump probes using BND, NOTRACK and both combined prefix orders; execute all eight and verify the target returns the expected value. All pass.
• python tools/check_conventions.py --base origin/main and git diff --check pass.
• python tools/pr_overlap.py --repo boykopovar/AnyPS5 --branch fix/call-site-hint-prefixes --jobs 4 reports no overlap with open pull requests, against the final base 1e740a23a3.
• Hosted relinker matrix at head 16eb32f20e, merged with the newer main for CI: Ubuntu 246/246, Windows 243/243, macOS arm64 237/237, macOS x86-64 237/237, and ASan 33/33 passed. Both call-site CTests pass in the four OS jobs; the ASan job builds them but runs only the 33 tests labeled tls_function_coverage. Conventions, labels, intake and eligibility also passed. Full-product jobs were skipped, not executed.
• Review follow-up 7151c6c0907f7440c724c68652846df29d709e67 only documents remaining registry-analysis limits and renames one test label. The production resolver is byte-identical to 16eb32f20e. Both call-site CTests were rebuilt and passed five consecutive runs each on Windows and Linux (20 successful CTest runs), and conventions and whitespace checks pass. The full-suite results above belong to the implementation head; no full local suite was rerun for the documentation/label-only follow-up.
• No title, GPU, full-product build or macOS runtime testing was performed locally.

Main integration refresh

• Integrated upstream main at a5a780eaaa7f187fe8f5c7f2ef244f6a6dab2762 into head c265e51abf3d747b67d768fec79f6defc9905419 on 2026-10-25, preserving published history.
• Resolved the only conflict in docs/dev/debt/relinker.md: retained this PR's resolver-limit note and main's updated Windows preferred-base documentation. The resolver implementation and regression file are byte-identical to reviewed head 7151c6c0907f7440c724c68652846df29d709e67; the diff against the refreshed base remains the same three files.
• Linux GCC 11.4, C++20 with -Wall -Wextra -Werror: both resolver and index tests pass five repetitions each in Release and ASan/UBSan builds (20 successful component runs), with leak detection and halt-on-error enabled. Windows registered resolver/index CTests pass five repetitions each; 20 actual Windows relinker registry conversions pass.
• Full Windows MinGW Release relinker-only build succeeds; 262/262 CTests passed in 187.61 seconds on this integration head. Linux CMake also rebuilds the relinker and both call-site test targets; both registered call-site CTests pass five repetitions each, and 20/20 actual Linux relinker registry conversions pass. That is 40 successful conversions across the two native relinkers. Conventions and whitespace checks pass. Fresh hosted checks are pending; no new local macOS, title, GPU or full-product run is claimed.

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: none
• Fixes: none
• Unblocks: none
• GPUs tested: n/a
• AI-assisted: yes

[MarcZX10]

Reviewed exact head 16eb32f20ef6f9b481a5056d783c67fc2f7c884e. The resolver now skips the BND/REPNE and DS/NOTRACK legacy prefixes while scanning the instruction and clears an earlier REX.W when a legacy prefix follows it; the added cases cover both call/jump forms, combined-prefix orderings, REX placement, pointer loads, and negative controls for FS/GS, address-size override, register operands, and unrelated memory operations. The Linux, Windows, macOS ARM64, and macOS x86_64 relinker jobs and relinker ASan all pass; conventions, eligibility, intake, and labels pass. Product-specific jobs were skipped by eligibility. The author reports extensive direct CLI, CTest, and native instruction checks in the PR body; I did not independently rerun those local checks. No blocking issue found.

[roboterspinne]

Reviewed head 16eb32f merged onto upstream/main 3ca5b17 (2026-10-24); the merge is clean, check_conventions.py passes and pr_overlap.py reports no overlap.

Environment: Windows 11 Home 10.0.26200, Intel i9-12900H, WinLibs GCC 15.2.0 UCRT, CMake 4.3.1, Ninja 1.13.2, Python 3.13.9, -DANYPS5_RELINKER_ONLY=ON -DBUILD_TESTING=ON, Release.

Results

• Full relinker-only suite on the merged tree: 242 of 243 pass, long_paths skipped. The one failure, patch_advisor_unit, is main's: test_uppercase_digest_and_conversion_options asserts --sce-module-path <root> unquoted while suggest_patches.py quotes a path that contains a space (subprocess.list2cmdline), and my temp root does; it fails identically on plain main 3ca5b17 and entered main with #8900 today. Not this PR.
• Red/green: the 22 new rows compiled against main's resolver fail at BND indirect call; against this branch call_site_resolver, call_site_index, call_registry_writer and call_registry pass.
• End to end with --registry on the test_call_registry fixture, Linux and Windows output: main leaves f2 ff 15, 3e ff 15, 3e f2 ff 15, f2 3e ff 25, f2 48 ff 15, 3e ff 25 and 3e 48 8b 05 with empty callSites and callSitesResolved: false; this branch resolves each to 0x200. 48 3e 8b 05 (narrow load) and 3e 67 ff 15 stay unresolved on both, and 64 f2 ff 15 is refused by the Windows output on both (Unsupported Windows guest TLS instruction), so nothing outside the registry changes.
• Native check on this CPU (each form in its own process, pointer slot at [rip+disp]): every BND, NOTRACK, combined and REX-prefixed ff 15/ff 25 form transfers through the slot; 48 3e 8b 05 and 48 f2 8b 05 load 32 bits, 3e 48 8b 05, f2 48 8b 05 and 48 3e 48 8b 05 load 64 bits, 3e 48 40 8b 05 loads 32 bits. That is exactly the hasRexW reset and last-REX rule the resolver implements, and it matches the project's own decoder (X64InstructionDecoder.cpp:42-43, :380) and objdump's reading of every row.
• Scanner length: the rows only reach 0x1100 with displacements f9/f8/f7 if InstructionMatch.Length includes the prefix bytes, so instrEnd at CallSiteResolver.cpp:85 is right; X64InstructionDecoder::Decode consumes F2, 3E, F3, 66, 67 and segment prefixes before the opcode.
• Consumers: CallSites/CallSitesResolved are read only by CallRegistryWriter (RelinkerPipeline.cpp:579-581, CallRegistryWriter.cpp:90-96); no patcher reads them. GotAccessIndex (unused-NID filter) already counted these prefixed forms as GOT accesses on main, so the PR narrows a gap between the filter and the registry rather than opening one.
• Body: 22 rows, the CI counts of run 40104543384 (246/243/237/237/33), the base history (1e740a23a3, patch id e28cdac41e), the binutils thread (both prefix orders) and the pr_overlap invocation check out.

Nits

• CallSiteResolver.cpp:33: of the four segment overrides that 64-bit mode ignores (2E, 26, 36, 3E) only 3E is skipped, so 2e ff 15 or 26 48 8b 05 stay unresolved while the same instruction with 3E resolves. Conservative, and no compiler emits them; the set that stays unclassified after this PR (F3, 66, 67, F0, 2E/26/36, FS/GS, and 8b without REX.W) is wider than the body's "address-size and FS/GS overrides", and only the body records it. A sentence in docs/dev/debt/relinker.md next to the registry entry would keep it findable; your call.
• Body, Tested: "Both call-site CTests pass on every matrix platform" does not hold for relinker-asan, which runs --label-regex '^tls_function_coverage$' (33 tests); the two call-site tests are built there but not run. True for the four OS jobs.
• CallSiteResolverTests.cpp:44: "DS-prefixed pointer load" names the accepted 3e 48 8b 05 as a DS override while the same inert override via 2E/26/36 is rejected; "NOTRACK-prefixed pointer load" would say why this one is in.

Notes, nothing to change here: ripRelativeDispOffset is a third prefix walker next to X64InstructionDecoder::Decode and DecodeInstruction, and DecodedInstructionInfo already exposes RexPrefix with the same reset rule, SegmentPrefix, AddressSize32 and RipRelativeDispOffset; the loop predates this PR and a swap would have to re-add the rejections the new tests pin, so follow-up material at most. call_site_resolver has no TIMEOUT property where call_site_index has 20 (pre-existing).

Not checked: AMD hardware (the console's Zen 2 and the author's probes were not run here; the probe above is Intel), macOS and Linux execution, the WSL ASan/UBSan runs, the pre-rebase head 00897f4b47.

I found no blocking issue: the two prefixes do not take part in the address match, the REX reset models the documented placement rule and the CPU, the registry is the only consumer, and the counter-check holds. For disclosure: I have open relinker PRs (#10646, #10607, #10564, #10403, #10402, #10401, #10391, #10390); none touches these files.

Written with AI assistance (Claude Code). I ran the builds, tests, registry conversions and native checks above on my Windows 11 machine.

[roboterspinne]

Nit: 64-bit mode ignores all four segment overrides 2E, 26, 36 and 3E for memory operands (Intel SDM Vol. 2A 2.2.1), but only 3E is skipped here, so 2e ff 15 ... or 26 48 8b 05 ... stay unresolved while the 3E form resolves. Conservative and compiler-irrelevant, so fine; but the forms that remain unclassified after this change (F3, 66, 67, F0, 2E/26/36, FS/GS, 8b without REX.W) are wider than the body's "address-size and FS/GS overrides" and live only in the body. One sentence in docs/dev/debt/relinker.md beside the registry entry would keep that findable; your call.

[roboterspinne]

Nit: the name presents 3e 48 8b 05 as a DS override, yet the same inert override via 2E, 26 or 36 is rejected by the loop; this row is in because 3E doubles as NOTRACK. "NOTRACK-prefixed pointer load" would say so.

[Bram910]

Thanks for the detailed independent checks. Addressed the three nits in 7151c6c0907f7440c724c68652846df29d709e67 and the updated PR description:

• Recorded the remaining unclassified prefixes and narrow-load limitation beside the registry entry in docs/dev/debt/relinker.md.
• Renamed the test label to NOTRACK-prefixed pointer load.
• Clarified that the two call-site CTests run in the four OS matrix jobs; hosted ASan builds them but runs only the 33 tls_function_coverage tests. The separately reported local ASan/UBSan component runs did execute both call-site tests.

Production code is unchanged. Rebuilt both focused test targets on Windows and Linux; each CTest passed five consecutive runs per platform. Conventions and whitespace checks pass. The unrelated patch-advisor test issue is left out of this PR.

[MarcZX10]

Reviewed the follow-up head 7151c6c0907f7440c724c68652846df29d709e67. Production resolver code is unchanged; the debt entry now lists the remaining unclassified instruction prefixes and narrow loads, and the test label is clearer. I built call_site_resolver_tests and call_site_index_tests with the relinker-only Release configuration and ran both CTests five times (all passed); conventions, diff checks, and the current-main merge tree pass. Hosted Ubuntu and ASan checks pass; Windows and macOS jobs are still running, while product jobs are skipped by policy. The previous approval was on the implementation head, so I’ll recheck the remaining matrix before approving this follow-up head.

[MarcZX10]

Reviewed the follow-up documentation change at 7151c6c. It documents the resolver's unclassified-prefix behavior and narrow-load rule; the implementation remains the previously reviewed call-site resolver. Local focused CTests and passed five repetitions each, as did conventions and ; merge tree against current main is clean. Hosted eligibility, labels, and all four relinker platform/sanitizer jobs passed (a duplicate conventions run was cancelled and the replacement passed). Product builds were skipped by repository policy.

[github-actions]

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request	Shared files not compared
PR 5021	docs/dev/debt/relinker.md

[roboterspinne]

@Bram910 Thanks. Re-checked 7151c6c merged onto upstream/main dca72a8 on Windows 11: the commit changes only the test label and adds the debt entry, the resolver is byte-identical to 16eb32f, call_site_resolver_tests and call_site_index_tests rebuilt and pass here, and the entry's list (F3, 66, 67, F0, 2E/26/36, FS/GS, 8B without REX.W) matches what the loop leaves unclassified. The three nits are resolved from my side.

Written with AI assistance (Claude Code). I ran the build and the tests above on my Windows 11 machine.

[Bram910]

Resolved the main conflict and pushed c265e51abf3d747b67d768fec79f6defc9905419, integrating upstream main a5a780eaaa7f187fe8f5c7f2ef244f6a6dab2762 without rewriting published history.

The only conflict was in docs/dev/debt/relinker.md: this PR adds the registry resolver-limit note beside a Windows preferred-base paragraph that main updated. Both are now preserved. The resolver implementation and regression file remain byte-identical to reviewed head 7151c6c0907f7440c724c68652846df29d709e67; the PR still changes only the same three files.

Revalidation on the integrated head:

• Windows MinGW Release relinker-only build: 262/262 CTests passed; both focused call-site tests also pass five repetitions each.
• Linux GCC Release and ASan/UBSan with warnings as errors: both component tests pass five repetitions per configuration (20 successful runs), with no sanitizer diagnostics. Linux CMake rebuilds the real relinker and both test targets; both registered CTests pass five repetitions each.
• Actual registry conversion fixtures: 20/20 Windows and 20/20 Linux pass.
• Conventions and whitespace checks pass. Updated the PR description to distinguish these results from the historical runs.

Fresh hosted checks are starting; no new macOS, title, GPU or full-product run is claimed locally.

[roboterspinne]

@Bram910 Thanks, re-checked c265e51 merged onto upstream/main 4017310 (2026-10-25); the merge is clean.

• CallSiteResolver.cpp and CallSiteResolverTests.cpp are the same blobs as in 7151c6c. The debt file gains only the same single line, next to main's updated preferred-base paragraph.
• Full relinker-only suite on Windows 11: 292 tests, 290 pass, long_paths skipped. The one failure is patch_advisor_unit (temp path with a space, test(tools): compare quoted and resolved paths in the patch advisor test #10667). call_site_resolver and call_site_index pass, also five times in a row.
• The registry conversion check from my review gives the same results as on 7151c6c:
  • the BND, NOTRACK and 3e forms resolve;
  • the narrow REX.W-then-3e load, the FS call and the 67 call stay unresolved.
• Hosted relinker jobs (ubuntu, windows, both macOS) and ASan are green on c265e51.

Nothing new from my side.

Written with AI assistance (Claude Code). I ran the checks above on my Windows 11 machine.