Skip to content

feat(libSceLibcInternal): implement the collate<wchar_t> members - #15712

Open
B-Howell wants to merge 3 commits into
boykopovar:mainfrom
B-Howell:feat/r2-collate-wchar
Open

B-Howell wants to merge 3 commits into
boykopovar:mainfrom
B-Howell:feat/r2-collate-wchar

Conversation

@B-Howell

@B-Howell B-Howell commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

What

Implements the six std::collate<wchar_t> members that libSceLibcInternal exported as NotImplemented stubs (Err.cpp, Pending3_049..054):

NID Symbol Behaviour
roztnFEs5Es collate<wchar_t>::do_compare libc's WideCollateCompare
Zxe-nQMgxHM collate<wchar_t>::do_transform libc's WideCollateTransform
cWaCDW+Dc9U collate<wchar_t>::do_hash libc's WideCollateHash
n-3HFZvDwBw collate<wchar_t>::compare calls the facet's virtual do_compare
81uX7PzrtG8 collate<wchar_t>::transform calls the facet's virtual do_transform
entYnoIu+fc collate<wchar_t>::hash calls the facet's virtual do_hash

References:

  • ISO C++ [locale.collate.members]: compare "Returns: do_compare(low1, high1, low2, high2)", and likewise transform returns do_transform(low, high) and hash returns do_hash(low, high). The public members therefore dispatch through the facet's vtable, as the existing collate<char>::transform in LocaleSupport.cpp already does (reinterpret_cast<const GuestLocale::CollateVtable*>(self->base.vtable)->transform(...)).
  • The do_ members are the base implementations that collate<wchar_t>::_Getcat already installs in the facet (g_wideCollateVtable{..., WideCollateCompare, WideCollateTransform, WideCollateHash}, with GuestWideCharacterSize = 2). They call those functions directly, not through the vtable, so a derived facet that calls collate<wchar_t>::do_compare explicitly cannot recurse into its own override. libc now exposes that vtable to libSceLibcInternal through GuestWideCollateVtable_nid_no_patch(). It has no NID, so libc's export set does not change.

The members are written in a new WideCollate.cpp with their declarations in include/WideCollate.hpp, which keeps them away from the lines of Api.hpp that other open PRs edit. Their six debt entries are removed from docs/dev/debt/libSceLibcInternal.md. Only the C locale gets a collate facet, as before: _Getcat still throws for any other locale.

tools/progress.py collect_libraries(): done goes from 9913 to 9919 (+6) out of 37146 on origin/main 5b1d381d5a. libSceLibcInternal goes from 92 done / 1045 todo to 98 / 1039. No new todo names.

Tested

New guest test guest_libc_internal_wide_collate (core/libs/tests/GuestLibcInternalWideCollate.cpp). It checks:

  • results on the classic facet: ordering by whole 16-bit character, the FNV-1a hash values the libc test pins, inline and allocated transform results, and rejection of reversed or half-null ranges and embedded nulls;
  • virtual dispatch: on a facet whose vtable overrides compare, transform and hash, the public members call the override with the same facet, ranges and result, while the do_ members return the base results and never reach it;
  • that the public members reject a null facet or a null vtable.

Runs, same tests on both platforms (guest_libc_internal_wide_collate, plus the existing guest_collate and guest_locale as regressions):

  • Windows 11, WinLibs GCC 15.2.0 (MinGW-w64): 3/3 tests passed in each of 3 runs.
  • Linux, WSL Ubuntu 22.04, GCC 11.4.0: 3/3 tests passed in each of 3 runs.
  • The unpatched libSceLibcInternal.prx on Windows exports all six <NID>_nid_no_patch_cut symbols and imports GuestWideCollateVtable_nid_no_patch from libc.
  • Negative control on both platforms: with main's Err.cpp and without WideCollate.cpp built in, the new test fails to link (undefined reference to sceLibcInternalWideCollateDoCompare_nid_no_patch, and the other five).
  • python3 tools/check_conventions.py --base origin/main: all checks passed. The conventions_* and debt_split CTest entries are self-tests of the checker scripts, not of this change, and were not run here.
  • python3 tools/pr_overlap.py --branch feat/r2-collate-wchar: no open PR implements these exports or adds these files, and git merge-tree against PR 15695 and PR 15696, the two that edit nearby lines of the libSceLibcInternal CMakeLists.txt, merges cleanly. It fails closed only because some older PRs that touch LocaleSupport.cpp, GuestLocale.hpp, the libSceLibcInternal CMakeLists.txt or its debt file conflict with main, so those files could not be compared automatically.
  • No title was run.

Checklist

  • Based on current main; no other open PR implements the same functions
  • One topic per PR; follow-ups go in a new PR
  • No comments in code except technical debt
  • Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
  • No notes, investigation .md files or images added to the repository (attach them to this PR)
  • New third-party code is a submodule built from source
  • Generic behaviour, not specific to one title
  • Depends on: none
  • Fixes: none
  • Unblocks: none
  • GPUs tested: n/a
  • AI-assisted: yes

🤖 Generated with Claude Code

do_compare, do_transform and do_hash call libc's wide collate
implementations directly; compare, transform and hash dispatch through
the facet's vtable as [locale.collate.members] requires.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

Progress report (6bf3a19 → 619f6c7)

📈 System libraries: 29.86% (+0.01%, +6 implemented, -6 removed functions)

✅ 6 implemented
Library Name
libSceLibcInternal sceLibcInternalWideCollateCompare_nid_no_patch
libSceLibcInternal sceLibcInternalWideCollateDoCompare_nid_no_patch
libSceLibcInternal sceLibcInternalWideCollateDoHash_nid_no_patch
libSceLibcInternal sceLibcInternalWideCollateDoTransform_nid_no_patch
libSceLibcInternal sceLibcInternalWideCollateHash_nid_no_patch
libSceLibcInternal sceLibcInternalWideCollateTransform_nid_no_patch
🗑️ 6 removed
Library Name
libSceLibcInternal sceLibcInternalPending3_049_nid_no_patch
libSceLibcInternal sceLibcInternalPending3_050_nid_no_patch
libSceLibcInternal sceLibcInternalPending3_051_nid_no_patch
libSceLibcInternal sceLibcInternalPending3_052_nid_no_patch
libSceLibcInternal sceLibcInternalPending3_053_nid_no_patch
libSceLibcInternal sceLibcInternalPending3_054_nid_no_patch

Changed libraries (implemented exports)

Library Before After Change
libSceLibcInternal 106/1141 (9.29%) 112/1141 (9.82%) +0.53%

@B-Howell

B-Howell commented Nov 4, 2026

Copy link
Copy Markdown
Contributor Author

The failing check is relinker (macos-latest, clang++, x86_64): 670/671 passed, and guest_none_relocations hit the 120 s timeout (log). This PR changes only libSceLibcInternal and does not touch the relinker or that test. The timeout is the one #15526 addresses by splitting the test.

@JJenkx

JJenkx commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

Windows check of head 3716a122b6, merged onto main b11c7bf84f (484cc6f222, clean merge).

Setup: Windows 11 Enterprise 22621, WinLibs GCC 15.2 UCRT, Release. The hosted Windows check does not run this test on pull requests. The host wchar_t is 16-bit on Windows, so this run also covers a host whose wchar_t differs from the guest's 32-bit one.

  • guest_libc_internal_wide_collate passes in 3 of 3 runs.

AI-assisted: yes.

@TechLevelZero

Copy link
Copy Markdown
Contributor

✅ AMD GPU test: passed

PR head 3716a12 merged onto main 5b1d381 · tested 2026-11-04 10:15 UTC · 27 min

GPU AMD Radeon RX 6300 (RADV NAVI24), driver Mesa 25.2.8-0ubuntu0.24.04.4, Vulkan 1.4.318
Host Ubuntu 26.04.1 LTS, kernel 7.0.0-42-generic, container ubuntu:24.04
Logs build and test logs for this run
Build ✅ passed (Release, -DANYPS5_ENABLE_SPIRV_TOOLS=ON)
Tests ✅ 3,769 passed · 16 failed (0 new, 11 also fail on main, 5 unstable on main) · 61 skipped (ANYPS5_REQUIRE_VULKAN=1)
Test setup CPU-only tests without a GPU; GPU tests (and any CPU-only test that failed or skipped without one) once on the GPU with VK_LAYER_KHRONOS_validation and ANYPS5_NO_SHADER_CACHE=1; main ran the same way
Test selection 3,846 of 3,883 tests run: the ones this change can reach, chosen by tools/select_tests.py from main's build graph
Vulkan validation ✅ no new validation IDs vs main (1380 tests run on the GPU)
Slow tests (>30 s, GPU tests timed with validation layers) ⚠️ agc_driver_pipeline_journal 120.3 s, ⚠️ agc_driver_graphics 120.2 s, ⚠️ agc_driver_graphics_maps_fallback 120.2 s, ⚠️ public_headers 56.5 s, ⚠️ agc_driver_bda_device 53.0 s
Unstable on main (5, these also failed in recent main runs on this machine; not counted against this PR)
agc_driver_alpha_to_coverage_sample_mask_export: failed in 4 of the last 10 main runs
agc_driver_graphics: failed in 3 of the last 10 main runs
agc_driver_graphics_maps_fallback: failed in 2 of the last 10 main runs
agc_driver_pipeline_journal: failed in 8 of the last 10 main runs
agc_driver_two_sample_alpha_to_coverage_sample_mask_export: failed in 3 of the last 10 main runs
Failing on main too (11, not counted against this PR)
agc_driver_guest_memory_lifetime_direct
agc_driver_image_table
agc_driver_job_manager_compute_capacity
agc_driver_native_depth_bda_clear
agc_driver_native_depth_buffer_clear
agc_driver_native_depth_clears
agc_driver_native_multisample_depth
agc_driver_submit_progress
agc_submit_progress
agc_submit_progress_no_timeline
guest_mmap_executable
Validated but not compared (1, not in the main baseline's validation pass)
guest_job_manager_worker_spin_scale_10
Validation IDs already present on main (8 IDs in 799 tests)
VUID-VkDeviceCreateInfo-pNext-pNext (799 tests): agc_compute_pipeline_replay, agc_driver, agc_driver_address_passes, agc_driver_address_passes_draw_transitions, agc_driver_address_passes_targets_split, agc_driver_alias_copy_label, agc_driver_alpha_to_coverage, agc_driver_alpha_to_coverage_sample_mask_export, and 791 more
VUID-vkCmdCopyImage-srcImage-01548 (11 tests): agc_driver_depth_clear_hle, agc_driver_depth_plane_cpu_write_after_memo, agc_driver_depth_plane_cpu_write_after_page_unwatch, agc_driver_depth_plane_cpu_write_after_partial_unwatch, agc_driver_depth_plane_partial_store_image_copy, agc_driver_depth_plane_partial_store_unimported, agc_driver_depth_plane_partially_unmapped_plane, agc_driver_depth_plane_unmapped_plane, and 3 more
VUID-vkCmdCopyImage-srcImage-01551 (11 tests): agc_driver_depth_clear_hle, agc_driver_depth_plane_cpu_write_after_memo, agc_driver_depth_plane_cpu_write_after_page_unwatch, agc_driver_depth_plane_cpu_write_after_partial_unwatch, agc_driver_depth_plane_partial_store_image_copy, agc_driver_depth_plane_partial_store_unimported, agc_driver_depth_plane_partially_unmapped_plane, agc_driver_depth_plane_unmapped_plane, and 3 more
VUID-VkPipelineColorBlendStateCreateInfo-pAttachments-00605 (2 tests): agc_driver_prepared_exports, agc_driver_srgb8_color_target
VUID-VkSamplerCreateInfo-addressModeU-01079 (1 test): agc_driver_image_gather_volume
VUID-VkSamplerCreateInfo-addressModeU-parameter (1 test): agc_driver_image_gather_volume
VUID-VkSamplerCreateInfo-addressModeV-parameter (1 test): agc_driver_image_gather_volume
VUID-VkSamplerCreateInfo-addressModeW-parameter (1 test): agc_driver_image_gather_volume
Skipped (61)
agc_driver_bda_reads_shadowed_results: skipped, the device reports watched host memory written when it imports or reads it
agc_driver_bda_reads_shadowed_results_per_build: skipped, the device reports watched host memory written when it imports or reads it
agc_driver_copied_writer_label_order: skipped, the device reports a minStorageBufferOffsetAlignment of 4, so the dispatch binds its record at +0x1004 in place and nothing is written back; the test needs an alignment above 4
agc_driver_copied_writer_label_order_drain: skipped, the device reports a minStorageBufferOffsetAlignment of 4, so the dispatch binds its record at +0x1004 in place and nothing is written back; the test needs an alignment above 4
agc_driver_float16_unused_nan_payloads: skipped, the device has no native f16 conversions with RTE rounding and kept f16 denormals
agc_driver_gcn_display_present: skipped, no display: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
agc_driver_image_table_vouched_lookups: skipped, the device reports 4 of 4 host-imported pages written after a GPU read, so a watched imported table cannot be vouched for
agc_driver_linear_target_direct_write_completion: skipped, the device does not keep host imports write-watched, so no completion stamp over the target is tracked
agc_driver_ngg_vertex_wave_reduction: skipped, s3 describes a 32-lane wave only on 32-lane subgroups, and the device has 64 lanes
agc_driver_scattered_runs: skipped, host imports are unavailable: /dev/udmabuf is not accessible
agc_driver_sparse_dma_buf_import: skipped, host imports are unavailable: /dev/udmabuf is not accessible
agc_driver_unit_shadow: skipped, the device does not import watched host memory for unit shadows
agc_driver_unit_shadow_budget: skipped, the device does not import watched host memory for unit shadows
agc_driver_vertex_elected_atomic_max: skipped, the device runs vertex programs in subgroups, so they keep their cross-lane reductions
agc_host_import_sparse: skipped, host imports are unavailable: /dev/udmabuf is not accessible
agc_idle_publish_wait: skipped, the device reports 4 of 4 host-imported pages written after a GPU read, so a forced watch of the import reads its submissions as CPU stores
agc_shadowed_key_stores: skipped, the device imports no watched host memory under a unit shadow
agc_small_tiled_shadow_label: skipped, host imports are unavailable: the test block is not imported and watched
agc_storage_clear_hand_over: skipped, the device imports no watched host memory for the clear-key hand-over
agc_storage_pending_key_refresh: skipped, refreshes over a pending key store are not testable here
agc_unit_shadow: skipped, the device does not import watched host memory for unit shadows
agc_unit_shadow_completion_store: skipped, the device reports watched host memory written when it imports or reads it
guest_http_uri_locale: 
guest_net_ctl: skipped, the host has no IPv4 address
guest_video_out_color_space: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_hdr: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_high_frame_rate: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_output_mode: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_resolution_status: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_wqhd_detection: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
video_out_flip_aftergpu: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video
Reproduce
git fetch origin 3716a122b6c909948a189a92e022b06789fac199 5b1d381d5ad5d9f923326b18479c3bd889cdd588
git checkout 5b1d381d5ad5d9f923326b18479c3bd889cdd588 && git merge --no-ff 3716a122b6c909948a189a92e022b06789fac199
git submodule update --init --recursive
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=ON -DANYPS5_ENABLE_SPIRV_TOOLS=ON
cmake --build build && cmake --build build --target libs
VK_DRIVER_FILES=/nonexistent/icd.json ctest --test-dir build -E '^(agc|shader|recompiler|videoout|video_out|guest_video_out|vulkan)|gpu' --timeout 120 --parallel 12 --output-on-failure
ANYPS5_REQUIRE_VULKAN=1 ANYPS5_NO_SHADER_CACHE=1 VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation \
  ctest --test-dir build -R '^(agc|shader|recompiler|videoout|video_out|guest_video_out|vulkan)|gpu' --timeout 120 --parallel 16 --output-on-failure
# failures are then re-run alone: --timeout 360 --parallel 2; new validation IDs: the test alone, twice, --parallel 1

Automated build and test on a contributor's AMD machine; it does not replace upstream CI. A new comment is posted for each new push. Full logs.

# Conflicts:
#	core/libs/prx/libc/include/GuestLocale.hpp
@B-Howell

B-Howell commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor Author

Thanks for the Windows and AMD reports on earlier head 3716a122b6. I refreshed the branch with an ordinary merge. On current head 619f6c79b4, all seven focused tests passed three times on both Windows and WSL, and all 65 public-header checks passed on each host. The freshly patched WSL libc.prx also exposes the defined wide-collate vtable helper.

Could a reviewer look at the wide-character collate wrappers on 619f6c79b4?

AI-assisted: yes (Codex).

Co-Authored-By: Codex <noreply@openai.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants