Skip to content

feat(libc): add the C locale helpers _Getctype, _Getcoll, _Getcvt, _Getdateorder, _Strcoll and _Strxfrm - #9466

Open
AlxcBTW wants to merge 4 commits into
boykopovar:mainfrom
AlxcBTW:pr/libc-c-locale-helpers
Open

AlxcBTW wants to merge 4 commits into
boykopovar:mainfrom
AlxcBTW:pr/libc-c-locale-helpers

Conversation

@AlxcBTW

@AlxcBTW AlxcBTW commented Oct 21, 2026

Copy link
Copy Markdown
Contributor

What

Adds six Dinkumware C locale helpers that libc did not export yet, as real APS5_VABI exports in a new file, core/libs/prx/libc/src/CLocaleHelpers.cpp (added to the libc source list; no other locale file is touched):

export NID behaviour
_Getctype gwAf8VMkXS0 returns {_Getpctype(), 0} (table pointer, delete flag) as a 16-byte struct in rax:rdx; the table is the existing classification table, not a copy
_Getcoll 15TBQ3jmXvo returns a zeroed 16-byte struct
_Getcvt HHXmPak83kQ fills a 40-byte block through the hidden return pointer: 1 at +0, 1 at +4, zeros from +8
_Getdateorder LMZCS1i3KTU returns 2
_Strcoll ZyciM6GWVO8 (first1, last1, first2, last2, collvec): compares the two ranges bytewise as unsigned bytes; returns the difference of the first differing bytes, otherwise 0 for equal lengths and -1 or 1 when one range is a prefix of the other
_Strxfrm EvPpDIUyQPI (dest, destEnd, src, srcEnd, collvec): copies the source range when it fits [dest, destEnd) and returns its length; a range that does not fit is not copied at all

C locale only; the collation argument is ignored.

The behaviour was read from the libc.prx bundled with PPSA16064 (disassembly; nothing from it is in the repository), which differs from what the names suggest in four places, so the implementation follows the binary:

  • _Getctype returns its result in registers (rax:rdx), not through a hidden pointer; _Getcvt is the one that uses the hidden pointer.
  • _Getdateorder returns 2, not 0.
  • _Strcoll returns the raw byte difference of the first mismatch ("a" against "c" gives -2), not a normalised -1, and ranges are compared by length as int.
  • _Strxfrm writes nothing when the source is longer than the destination range (it does not copy a truncated prefix); it still returns the source length.

_Getcoll is a bare ret in that binary, so the register pair it returns is whatever the caller left; this returns zeros, which is deterministic and what a caller that ignores the C locale collation vector sees. The _Getcvt block is written exactly as far as the console writes it (40 bytes); whether the struct is larger is not known. Both are listed in the libc technical debt (including that the _Getcvt size is unconfirmed) (docs/dev/debt/libc.md). None of the six was listed there before, so no entry is removed.

Tested

  • Fork CI (Linux, Ubuntu, "Lab tests" workflow): https://github.com/AlxcBTW/AnyPS5/actions/runs/39684522381 on ci/test/libc-c-locale-helpers-r (this branch's head 52a3e3b1a3 plus the CI scaffolding commit), ctest -R '^(guest_c_locale_helpers|guest_locale|guest_rune_locale|guest_character|guest_strings|guest_wide_strings)$': 6 of 6 passed, including the new guest_c_locale_helpers and the existing locale, character and string tests. The Windows job was not run; the test is registered with the same PATH environment modification as its neighbours.
  • New guest test core/libs/tests/GuestCLocaleHelpers.cpp calls the exports directly with hand-computed vectors:
    • _Strcoll: equal ranges give 0, empty against empty 0, "a"/"b" negative, "b"/"a" positive, "a"/"c" exactly -2, prefixes give -1 and 1, 0x80/0x7f and 0xff/0x01 compare as unsigned bytes (1, 254), embedded NUL bytes are compared, a mismatch in the 17th byte after 16 equal ones is found.
    • _Strxfrm: destination larger (copied, bytes after it untouched, returns 5), destination exactly the length (copied), destination smaller or zero-length (nothing written, still returns 5), empty source returns 0, binary bytes 00 80 ff 61 are copied unchanged.
    • _Getctype: struct size and field offsets by static_assert, table pointer equal to _Getpctype(), delete flag 0, table[-1] 0, and for all 0..255 the entry equals the hand-written classification of that byte (digit, upper, lower, space, punct, hex and control bits; 128..255 are 0).
    • _Getcoll zeros, _Getcvt (size and offset by static_assert, page 1, maximum 1, rest zero, no write past 40 bytes), _Getdateorder 2.
  • No game and no console were used for the tests. The expected values come from the disassembly named above.
  • python3 tools/check_conventions.py --base origin/main passes. Based on 0c6a5bd593. tools/pr_overlap.py finds no pull request implementing these functions (a search of open PR titles and bodies for each name also finds none); it lists textual overlaps in core/libs/prx/libc/CMakeLists.txt and docs/dev/debt/libc.md with several open libc PRs (for example feat(libc): export the num_get and time_get locale objects as data #8880, feat(libc): cover the codecvt, numpunct, num_put and time_put facets, their ids and std::clog #9038, fix(libc): convert multibyte text as UTF-8 like the console's C locale #8855 edit other locale files and the same two shared files), which are line-level additions in separate places.

Checklist

  • Based on current main; no other open PR implements the same functions
  • One topic per PR; follow-ups go in a new PR
  • No comments in code except technical debt
  • Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
  • No notes, investigation .md files or images added to the repository (attach them to this PR)
  • New third-party code is a submodule built from source
  • Generic behaviour, not specific to one title
  • Depends on: none
  • Fixes: none
  • Unblocks: PPSA16064 (all six are imported by its app.exe from libc.prx); no Game bring-up discussion yet. Needed before the std::__sce_v2 locale facets.
  • GPUs tested: n/a
  • AI-assisted: yes

@github-actions github-actions Bot added the unblocks title Names a title ID on the Unblocks line label Oct 21, 2026
@github-actions

Copy link
Copy Markdown

Progress report (a6dcea9 → 52a3e3b)

📈 System libraries: 42.36% (+0.03%, +6 implemented functions)

✅ 6 implemented
Library Name
libc _Getcoll_nid_postfix
libc _Getctype_nid_postfix
libc _Getcvt_nid_postfix
libc _Getdateorder_nid_postfix
libc _Strcoll_nid_postfix
libc _Strxfrm_nid_postfix

@MarcZX10 MarcZX10 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the implementation and tests for the C-locale-only helpers. The fixed-width layouts match the recorded reverse-engineering evidence and are guarded with static assertions; the byte-range comparisons handle embedded NUL and unsigned bytes; transform copies only when the full source fits, as documented in the debt entry. The Ctype table reuses the existing _Getpctype table, and the test covers its negative sentinel and all 256 byte entries. Current conventions, intake, eligibility, relinker matrix, register-context, write-watch, and report checks pass. Full Linux/Windows builds were skipped, and I did not run a local build in this session.

@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.

Pull request Conflicting files (lines on main) Same exports Same new files
PR 14085 docs/dev/debt/libc.md L99-101

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request Shared files not compared
PR 5021 core/libs/prx/libc/CMakeLists.txt
docs/dev/debt/libc.md
PR 5303 docs/dev/debt/libc.md
PR 6819 core/libs/prx/libc/CMakeLists.txt
docs/dev/debt/libc.md
PR 7081 core/libs/prx/libc/CMakeLists.txt
docs/dev/debt/libc.md
PR 7386 docs/dev/debt/libc.md
PR 7447 core/libs/prx/libc/CMakeLists.txt
docs/dev/debt/libc.md
PR 7715 core/libs/prx/libc/CMakeLists.txt
docs/dev/debt/libc.md
PR 8150 core/libs/prx/libc/CMakeLists.txt
docs/dev/debt/libc.md
PR 8965 core/libs/prx/libc/CMakeLists.txt
docs/dev/debt/libc.md
PR 9340 docs/dev/debt/libc.md
PR 9371 docs/dev/debt/libc.md
PR 9929 core/libs/prx/libc/CMakeLists.txt
docs/dev/debt/libc.md
PR 10113 docs/dev/debt/libc.md
PR 10226 docs/dev/debt/libc.md
PR 10245 docs/dev/debt/libc.md
PR 10250 docs/dev/debt/libc.md
PR 10253 docs/dev/debt/libc.md
PR 10265 docs/dev/debt/libc.md
PR 10274 docs/dev/debt/libc.md
PR 10325 docs/dev/debt/libc.md
PR 10328 docs/dev/debt/libc.md
PR 10345 docs/dev/debt/libc.md
PR 10349 docs/dev/debt/libc.md
PR 10405 docs/dev/debt/libc.md
PR 10506 docs/dev/debt/libc.md
PR 10510 docs/dev/debt/libc.md
PR 10512 docs/dev/debt/libc.md
PR 10565 docs/dev/debt/libc.md
PR 10987 docs/dev/debt/libc.md
PR 11260 docs/dev/debt/libc.md
PR 11262 docs/dev/debt/libc.md
PR 11266 docs/dev/debt/libc.md
PR 11332 docs/dev/debt/libc.md
PR 11632 docs/dev/debt/libc.md
PR 11634 docs/dev/debt/libc.md
PR 11724 docs/dev/debt/libc.md
PR 11726 docs/dev/debt/libc.md
PR 11831 core/libs/prx/libc/CMakeLists.txt
docs/dev/debt/libc.md
PR 11907 docs/dev/debt/libc.md
PR 12122 docs/dev/debt/libc.md
PR 12397 docs/dev/debt/libc.md
PR 12514 docs/dev/debt/libc.md
PR 12560 docs/dev/debt/libc.md
PR 12574 docs/dev/debt/libc.md
PR 12593 docs/dev/debt/libc.md
PR 12622 docs/dev/debt/libc.md
PR 12627 docs/dev/debt/libc.md
PR 12682 core/libs/prx/libc/CMakeLists.txt
docs/dev/debt/libc.md
PR 12688 core/libs/prx/libc/CMakeLists.txt
docs/dev/debt/libc.md
PR 12700 core/libs/prx/libc/CMakeLists.txt
docs/dev/debt/libc.md
PR 12965 core/libs/prx/libc/CMakeLists.txt
docs/dev/debt/libc.md
PR 13119 docs/dev/debt/libc.md
PR 13390 core/libs/prx/libc/CMakeLists.txt
docs/dev/debt/libc.md
PR 13804 docs/dev/debt/libc.md
PR 14212 docs/dev/debt/libc.md
PR 14585 core/libs/prx/libc/CMakeLists.txt
docs/dev/debt/libc.md
PR 14610 docs/dev/debt/libc.md

@MarcZX10 MarcZX10 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The implementation review and reported evidence are comprehensive: the fixed-width structs are guarded, byte comparisons include high-bit and embedded-NUL cases, and the full-copy-only _Strxfrm behavior matches the documented binary evidence. The checked head is based on a6dcea927, 47 upstream commits behind current main (110a18ba01). Please update the branch and rerun conventions, overlap/debt checks, and the focused C locale helper suite against the updated tree. The overlap report also has incomplete comparisons for shared CMakeLists.txt and libc debt files; confirm the additions remain additive with current work during that recheck. Existing hosted checks pass as listed; full Linux/Windows jobs were skipped, and I did not run local tests.

@harisibrar1 harisibrar1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked against main 830ccd1, the Dinkumware C-locale _Strcoll/_Strxfrm/_Ctypevec/_Collvec definitions, the SysV x86-64 return classification, and the NID calculator:

  • All six NIDs match, and none of the six is on main yet. The head merges onto 830ccd1 with no conflicts in CMakeLists.txt or docs/dev/debt/libc.md, so the rebase request should be easy to satisfy.
  • The struct sizes give the return paths described: the 16-byte Ctypevec/Collvec come back in rax:rdx, and the 40-byte Cvtvec through the hidden pointer. _Strcoll/_Strxfrm match the Dinkumware C-locale behaviour.
  • Nit: in time_base::dateorder (no_order, dmy, mdy, ymd, ydm), 2 is mdy, the usual C-locale order, so DateOrderMdy would be a clearer name than DateOrderNone (CLocaleHelpers.cpp:30).
  • Nit: the trailing-bytes check in CheckCvt (GuestCLocaleHelpers.cpp:159-164) copies sizeof(Cvtvec) from a local into raw, so bytes 40..47 can't change and the check always passes. It could be dropped, or the test could call through a pointer to a function returning a larger struct.

Reviewed with AI assistance (Claude Code); the checks above were run locally on my machine.

@ycbuzi

ycbuzi commented Oct 22, 2026

Copy link
Copy Markdown
Contributor

Windows run of 52a3e3b merged onto main 6a6a35f (39a6f171f7, clean), the platform the description lists as not run. Windows 11 Home 10.0.26200, MinGW-w64 GCC 15.2.0 (ucrt-posix-seh r7), Release, BUILD_TESTING=ON.

Built libc and guest_c_locale_helpers_tests with no errors or warnings. ctest --repeat until-fail:2 on guest_c_locale_helpers: passes on both repeats. No negative control: the helpers are implemented in src/CLocaleHelpers.cpp, which this PR adds. Not run: a title.

@adriadam10 adriadam10 self-assigned this Oct 23, 2026
danielradosa added a commit to danielradosa/AnyPS5 that referenced this pull request Oct 24, 2026
danielradosa added a commit to danielradosa/AnyPS5 that referenced this pull request Oct 24, 2026
…oykopovar#9466 in core/libs/CMakeLists.txt and libc/CMakeLists.txt: the kept-both-sides merge dropped the closing PATH line and endif() of the test block each PR's block was inserted after
@wachi35676

Copy link
Copy Markdown
Contributor

Windows check of head 52a3e3b1a merged onto main aeff26bd9 (WinLibs GCC 15.2.0, Release). This one adds the ABI side on MinGW.

  • guest_c_locale_helpers and guest_locale pass, 2 runs. check_conventions.py passes on the merge.

  • The test calls the exports through the same APS5_VABI declarations, so caller and callee always agree and an ABI mismatch wouldn't show up. I checked the Windows libc.prx directly with objdump -d:

    • _Getctype: call _Getpctype; xor %edx,%edx; ret. The table is in rax and the flag in rdx.
    • _Getcoll: xor %eax,%eax; xor %edx,%edx; ret.
    • _Getcvt: the hidden pointer is in rdi. It stores 8 bytes at +0 and 16 bytes each at +8 and +0x18 (40 bytes total), and returns rdi in rax.

    So the sysv_abi exports return the 16-byte structs in rax:rdx and the 40-byte one in memory on Windows too, as the description says.

AI-assisted: yes (Claude Code); results above were run on this machine.

@jotitadev jotitadev left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Merge-skew check: head 52a3e3b1a merged onto current main abed03546 (clean merge, git merge --no-commit). This head's CI ran on the October 21 base. Linux x86_64, i7-8700, Ubuntu 26.04, GCC 15.2, Release, BUILD_TESTING=ON, full build, then the full non-GPU suite (ctest -E "^agc|gpu|vulkan|video_out" -j6). main alone: 1363/1363 pass. With this PR there are two failures, each 3/3. Both come from main changes that landed after the earlier runs in this thread.

1. guest_c_locale_helpers aborts ("C locale helper check failed at line 138"). On current main, _Getpctype()['\t'] is 0x4c0, while the test's Classify expects 0xc0 (line 138, and the literal at line 142). #10387 (d2e18d5, merged 2026-10-26 01:28 UTC) made the shared classification table mark tab with 0x400, as the console's libc does (isblank). _Getctype returns that same table, so the change is only in the test's expectation:

     if (value >= '\t' && value <= '\r') mask |= 0x40;
+    if (value == '\t') mask |= 0x400;
@@
-    Require(vec.table[' '] == 0x04 && vec.table['\t'] == 0xc0 && vec.table['\n'] == 0xc0);
+    Require(vec.table[' '] == 0x04 && vec.table['\t'] == 0x4c0 && vec.table['\n'] == 0xc0);

2. public_headers fails (its 64 public_headers_compile_* dependents don't run):

_Getctype_nid_postfix / _Getcoll_nid_postfix / _Getcvt_nid_postfix / _Getdateorder_nid_postfix / _Strcoll_nid_postfix / _Strxfrm_nid_postfix:
  expected exactly one header declaration; found 1: core/libs/tests/GuestCLocaleHelpers.cpp:28..33
_Getpctype_nid_postfix: expected exactly one header declaration; found 3: core/libs/prx/libc/include/Api.hpp:496, core/libs/prx/libc/src/CLocaleHelpers.cpp:36, core/libs/tests/GuestCLocaleHelpers.cpp:27

#10706 (merged 2026-10-24) added this audit. Each export needs one declaration, in a header reachable from libc.hpp. The fix I checked:

  • Move the GuestCLocale::Ctypevec/Collvec/Cvtvec structs, with their static_asserts, from CLocaleHelpers.cpp into core/libs/prx/libc/include/Api.hpp.
  • Declare the six exports in Api.hpp right after _Getpctype_nid_postfix.
  • Include Api.hpp in CLocaleHelpers.cpp and drop its local _Getpctype declaration.
  • In the test, include Api.hpp, drop the local structs and extern "C" block, and use using GuestCLocale::Ctypevec; (and the same for Collvec and Cvtvec).

With both changes on top of the merge, the full non-GPU suite passes 1364/1364, including guest_c_locale_helpers, guest_locale, guest_character, public_headers and all public_headers_compile_*. I can post the full diff if that helps.

The head still merges cleanly onto the newer main 92bfca720. Since abed03546, nothing there has touched the audit script, libc/include/Api.hpp or the tab bit.

Written with AI assistance (Claude Code). I ran the builds and tests above on my machine.

@JJenkx

JJenkx commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

A titles-side cross-check of two of the helpers, against the head 52a3e3b1a3's test expectations.

Setup: a scratch loader runs a decrypted libc.prx natively on Windows 11 and calls its _Getctype (gwAf8VMkXS0) and _Strxfrm (EvPpDIUyQPI) by NID. The modules are PPSA16064, the PR's reference title, unwrapped from its dump, plus PPSA25636, PPSA26528, PPSA33655, PPSA34008 and PPSA35169. Six older titles (PPSA04158, 05916, 07923, 09614, 13934 and 20423) don't export either helper.

  • _Strxfrm: 100,000 random byte strings of 0 to 19 bytes per title, any byte value including NUL, with destination sizes 0 to 21 and a null collation pointer. In all six titles, every call matches the rule in the tests: it returns the source length, copies the source bytes with no terminator when they fit, and writes nothing when they don't.
  • _Getctype:
    • In all six titles, the table is the same pointer on a second call, deleteFlag is 0 and table[-1] is 0.
    • 255 of the 256 entries equal Classify.
    • The exception is table['\t'], which is 0x4c0 in every one, PPSA16064 included. Classify gives 0xc0, and the test asserts vec.table['\t'] == 0xc0. The extra 0x400 bit is set for tab only, not for space or the other \n to \r controls.

So the _Getctype table, its Classify helper and the tab assertion would need that bit for '\t'. main's own _Getpctype table, which the test requires _Getctype to return, already has 0x4c0 there. It equals the titles' _Getpctype in all 257 entries (-1 to 255) on PPSA07923 and PPSA34008. - _Getcvt and _Getdateorder: on PPSA16064, PPSA25636 and PPSA34008 they match CheckCvt and CheckDateOrder: page 1, maxBytes 1, 32 zero reserved bytes, and date order 2. My probe read _Getcoll's result as values that changed from run to run, so I make no claim about it.

I didn't build this PR; this checks the console side only.

AI-assisted: yes (Claude Code).

@oneandonlydean

Copy link
Copy Markdown
Collaborator

@AlxcBTW two things on this one, it merges cleanly otherwise:

  1. On current main _Getpctype's entry for '\t' is 0x4c0 (main's classification table sets 0x400 for tab, and JJenkx measured the same value in six titles' libc.prx), so the expectations at GuestCLocaleHelpers.cpp lines 124 and 142 (Classify and vec.table['\t'] == 0xc0) fail on the merged tree. They need the 0x400 bit for tab.
  2. CLocaleHelpers.cpp declares _Getpctype_nid_postfix locally, but libc/include/Api.hpp already declares it (line 497). Please include that header instead, and declare the six new helpers and their structures there too, so public_headers sees them.

@blawesom

Copy link
Copy Markdown
Contributor

Static review plus an independent Linux run (CachyOS, kernel 7.2.10, GCC 16.2.1, Ninja, Release, BUILD_TESTING=ON, Ryzen 7 9800X3D). No findings.

What I checked (static)

  • Six C-locale helper exports (_Getctype, _Getcoll, _Getcvt, _Getdateorder, _Strcoll, _Strxfrm) implement the shapes the Dinkumware ABI expects: Ctypevec (16 bytes: const short* table, int32_t deleteFlag), Collvec (16 bytes), Cvtvec (40 bytes: uint32_t page, uint32_t maxBytes, uint8_t reserved[32]), matching the libc.prx disassembly (PPSA16064) recorded in the debt note. static_assert on size/offset locks the ABI shape at compile time.
  • _Strcoll/_Strxfrm follow C-locale bytewise collation semantics exactly: first-differing-byte difference (test pins a-c == -2, c-a == 2, a\xff-a\x80 == 0x7f, cross-byte-sign conventions) and prefix ordering (ab vs abc == -1/1). _Strxfrm copies only when length <= destEnd - dest and always returns the required size — matching the C standard's strxfrm contract even for a truncated write.
  • _Getctype's classification table is fully pinned per character class in the test (Classify(0..255)), including edge cases ('A' == 0x03, 'a' == 0x11, '0'/'9' == 0x21, tab/newline == 0xc0, '!'/'~' == 0x08, byte 0 == 0x80, 128–255 == 0) — a real encoding table verification, not a smoke test.
  • _Getcvt fills page=1, maxBytes=1, zeros reserved; the raw[48] guard-byte test confirms the 40-byte struct is all that's written (no buffer overrun past sizeof(Cvtvec)), matching the debt note's "whether the structure is larger is not known" caveat honestly.

Focused test run

tree result
this head 52a3e3b1a3 1/1 guest_c_locale_helpers Passed (0.00s)

Not an AGC/shader/VideoOut change — no paired Vulkan-validation run required. The debt entry records exactly what remains unverified (whether Cvtvec's true console size exceeds 40 bytes, _Getcoll's real return registers vs. the zeros assumed here per the PPSA16064 ret-only disassembly) — the right shape for a stub-family introduction like this.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

build docs libc needs rebase Conflicts with main, rebase needed unblocks title Names a title ID on the Unblocks line

Projects

None yet

Development

Successfully merging this pull request may close these issues.

10 participants