[mantavisual]

What

Implements uiter_setString and uiter_setUTF8, two exports that firmware 13.42's libSceIcu has (BunnyFl's export list). They fill a caller-owned UCharIterator, ICU's C character iterator, which callers then drive through its function pointers: getIndex, move, hasNext, hasPrevious, current, next, previous, getState and setState. 13.42 also exports ucol_strcollIter, which consumes such iterators.

The code is a port of ICU 78.1's uiter.cpp: the no-op, char16_t string and UTF-8 iterators. They are installed into the public 112-byte UCharIterator layout from uiter.h. The callbacks use the guest ABI, so guest code can call them, and they carry the _nid_no_patch suffix because they are helpers, not exports.

• Setup. A non-null text with a length of at least −1 installs the string or UTF-8 iterator. −1 means NUL-terminated, measured with u_strlen's or strlen's 32-bit result. Anything else installs the no-op iterator: index 0, U_SENTINEL for code units, UITER_NO_STATE, and setState reports U_UNSUPPORTED_ERROR. A null iterator is ignored.
• String iterator. Index arithmetic wraps as in ICU's UPRV_NO_SANITIZE_UNDEFINED functions. move clamps to start/limit, an unknown origin returns −1, the state is the index, and setState checks the error, the iterator and the bounds in ICU's order.
• UTF-8 iterator.
  • Its UTF-16 index and length are computed lazily from the bytes. A supplementary character is returned as its two surrogates, with the code point held in reservedField between them.
  • The state is byte offset << 1 | inside-pair, and setState validates it as ICU does.
  • move follows ICU's shortcuts and its UITER_UNKNOWN_INDEX results. That includes one quirk, which is kept: when an unknown-index forward move reaches the end, the index ICU has just computed is overwritten by the running position.
• Ill-formed UTF-8 decodes to U+FFFD per maximal subpart. This is ported from U8_NEXT_OR_FFFD and from U8_PREV_OR_FFFD's utf8_prevCharSafeBody(..., -3) in utf8.h and utf_impl.cpp, with their validity bit tables.

A callback given a null iterator, and the no-op setState given a null error pointer, throw APS5_INVALID_ARG_EX, since ICU dereferences them. The string and UTF-8 setState accept a null error pointer as ICU does. The new debt entry records the port and that nothing is compared with a console.

Depends on #11132 (stacked on it). The agreed ICU order is #11013 → #10961 → … → #10967 → #11088 → #11115 → #11118 → #11122 → #11132, and this PR comes after #11132. Until those merge, the diff also shows their commits. This PR's own change is the last commit, 428dab7efa.

Tested

Windows 11 Pro 23H2, WinLibs GCC 15.2.0 r7, Release, BUILD_TESTING=ON, on #11132 (c1d9fa2f64).

• New guest_icu_iterator, which drives every iterator only through its function pointers:
  • String iterator: fields, every origin, forward and backward walks, clamped and wrapping moves, the state round trip, and setState's error, iterator and bound checks. Also explicit lengths with embedded NULs and the empty string.
  • No-op iterator: all four ways to get one, and its callbacks.
  • UTF-8 iterator: a known string with a supplementary character, including the state inside the surrogate pair and the setState rejections.
  • UTF-8 cross-check: 8 handpicked and 3000 seeded random byte strings, biased towards lead, trail and boundary bytes. For each string, forward walks are compared with a reference decoder written independently from the Unicode Standard's well-formed byte sequence table, with U+FFFD per maximal subpart. Backward walks must give the same units in reverse. For every UTF-16 index, moving from zero and from the limit (with the index unknown) must land on the right unit, the computed index must be right, and a getState/setState round trip must restore it.
  • I checked that the test detects errors: making the E0 lead accept 80..9F trail bytes fails the forward comparison, and dropping the truncated four-byte case in the backward decoder fails the backward comparison.
• ctest -R "guest_icu_|patched_libraries": all 8 pass. The NID-patched libSceIcu.prx builds.
• tools/progress.py --module libSceIcu: 52 implemented, 0 stubs (+2; the callbacks are not counted). tools/check_test_registration.py build: 1347 test executables, none built and not run.
• python3 tools/check_conventions.py --base origin/main passes. Overlap: no open PR implements or names any uiter_ function (checked 2026-10-25).

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: feat(libSceIcu): implement the enumeration dispatch exports #11132
• Fixes: none
• Unblocks: none
• GPUs tested: n/a
• AI-assisted: yes

[github-actions]

Progress report (89a7f98 → 1138f8b)

📈 System libraries: 26.81% (+0.1%, +47 implemented functions)

✅ 47 implemented

Library	Name
libSceIcu	u_errorName_nid_postfix
libSceIcu	u_getVersion_nid_postfix
libSceIcu	ucfpos_close_nid_postfix
libSceIcu	ucfpos_constrainCategory_nid_postfix
libSceIcu	ucfpos_constrainField_nid_postfix
libSceIcu	ucfpos_getCategory_nid_postfix
libSceIcu	ucfpos_getField_nid_postfix
libSceIcu	ucfpos_getIndexes_nid_postfix
libSceIcu	ucfpos_open_nid_postfix
libSceIcu	uenum_close_nid_postfix
libSceIcu	uenum_count_nid_postfix
libSceIcu	uenum_next_nid_postfix
libSceIcu	uenum_unext_nid_postfix
libSceIcu	uiter_setString_nid_postfix
libSceIcu	uiter_setUTF8_nid_postfix
libSceIcu	uprv_add32_overflow_nid_postfix
libSceIcu	uprv_ceil_nid_postfix
libSceIcu	uprv_fabs_nid_postfix
libSceIcu	uprv_floor_nid_postfix
libSceIcu	uprv_fmax_nid_postfix
libSceIcu	uprv_fmin_nid_postfix
libSceIcu	uprv_fmod_nid_postfix
libSceIcu	uprv_getInfinity_nid_postfix
libSceIcu	uprv_getNaN_nid_postfix
libSceIcu	uprv_isInfinite_nid_postfix
libSceIcu	uprv_isNaN_nid_postfix
libSceIcu	uprv_isNegativeInfinity_nid_postfix
libSceIcu	uprv_isPositiveInfinity_nid_postfix
libSceIcu	uprv_log_nid_postfix
libSceIcu	uprv_maxMantissa_nid_postfix
libSceIcu	uprv_max_nid_postfix
libSceIcu	uprv_maximumPtr_nid_postfix
libSceIcu	uprv_min_nid_postfix
libSceIcu	uprv_modf_nid_postfix
libSceIcu	uprv_mul32_overflow_nid_postfix
libSceIcu	uprv_pow10_nid_postfix
libSceIcu	uprv_pow_nid_postfix
libSceIcu	uprv_round_nid_postfix
libSceIcu	uprv_trunc_nid_postfix
libSceIcu	uset_addAll_nid_postfix
libSceIcu	uset_addRange_nid_postfix
libSceIcu	uset_add_nid_postfix
libSceIcu	uset_close_nid_postfix
libSceIcu	uset_contains_nid_postfix
libSceIcu	uset_getItemCount_nid_postfix
libSceIcu	uset_getItem_nid_postfix
libSceIcu	uset_openEmpty_nid_postfix

Changed libraries (implemented exports)

Library	Before	After	Change
libSceIcu	28/2480 (1.13%)	75/2527 (2.97%)	+1.84%

[MarcZX10]

Reviewed the PR-specific iterator commit 428dab7 on its declared stack after #11132. Compared the string and UTF-8 iterator setup, callbacks, lazy UTF-16 indexing, surrogate-pair state, and malformed UTF-8 handling with ICU 78.1 uiter.cpp and uiter.h. The 112-byte guest layout, callback signatures, state/error ordering, and iterator behavior match the referenced implementation.

Local checks on this commit:

• g++ -std=c++20 -Icore/libs -Icore/libs/prx/libc/include -fsyntax-only for Iterator.cpp and GuestIcuIterator.cpp: passed.
• Built and ran GuestIcuIterator.cpp with Iterator.cpp directly: passed, including the seeded malformed UTF-8 and state/movement cases.
• Repeated that executable with AddressSanitizer and UndefinedBehaviorSanitizer: passed.
• python3 tools/check_conventions.py --base upstream/main --head HEAD: passed.

The PR reports the full ICU suite and NID-patched library build on Windows; this host run covered the new iterator test only. Hosted eligibility, conventions, intake, labels and report checks pass; product build/test jobs are skipped. The declared dependency on #11132 remains part of the agreed ICU merge order.

[BunnyFl]

@mantavisual uiter_setString and uiter_setUTF8 here behave like firmware 13.42's libSceIcu.sprx (exports at 0x163100 and 0x163410), run natively the same way as in #11122, with Iterator.cpp from 428dab7efa compiled unchanged. This is the module's code on a PC, not a console.

• Each sequence sets up both iterators on the same random text, then makes the same 10 to 59 random calls through their function pointers: every origin including unknown ones, moves with small, ±limit, random and near-INT32_MAX/INT32_MIN deltas, and setState with states from getState, small, random and 0xFFFFFFFF values, incoming errors 0, 1, 8 or -128, and sometimes a null error pointer. After setup, context, the five integer fields and which callback slots are null are compared (both structs are pre-filled with 0xAB). After every call, the return value, error and the five fields are compared. Texts are up to 11 UTF-16 units or 23 bytes, biased towards lead, trail and boundary bytes, with null texts and lengths below -1, -1 and explicit lengths.
• 200,000 sequences, 7,087,357 compared steps, 0 mismatches, three identical runs. Three more seeds: 21.3 million steps, 0 mismatches. In the first run, move returned UITER_UNKNOWN_INDEX 39,387 times, and 7,708 to 29,019 calls of each kind left a supplementary character split (reservedField set).
• Controls: 16 copies of Iterator.cpp with one change each are all caught (20,000 sequences each). The changes: string length -1 treated as no-op; the UTF-8 length field for 2-byte texts; the E0 trail table; the truncated four-byte case going backwards; the UTF-8 setState's index < 4; the string setState's error > 0 check; the null error pointer in the string and in the UTF-8 setState; the no-op setState code; the string move clamp; saturating instead of wrapping (for one move(INT32_MAX, UITER_CURRENT) the module returns 0, the copy 8); the reservedField adjustment in getIndex; hasNext's reservedField check; previous's index near the start; the NUL-terminated UTF-8 length; and the forward-move quirk you kept. The module has that quirk too: a copy that keeps the computed index instead fails on 4 of the 20,000 sequences.
• Per the disassembly (not run), the module's no-op setState stores 16 through the error pointer without a null check, where the PR throws. The string and UTF-8 setState with a null error pointer were run and match.
• Not covered: callbacks given a null iterator, and the no-op setState with a null error pointer. For length -1, the module's uiter_setString calls code at 0x15be0 that jumps to the imported wcslen (per the disassembly); the harness replaces it with a 16-bit unit count. The PR's callbacks use the host calling convention here, since APS5_VABI is empty in this build.

[github-actions]

Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.

Pull request	Conflicting files (lines on main)	Same exports	Same new files
PR 10963	core/libs/prx/libSceIcu/CMakeLists.txt L1
PR 15346	core/libs/prx/libSceIcu/CMakeLists.txt L1

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request	Shared files not compared
PR 10964	core/libs/prx/libSceIcu/CMakeLists.txt core/libs/prx/libSceIcu/libSceIcu.hpp docs/dev/debt/libSceIcu.md
PR 10965	core/libs/prx/libSceIcu/CMakeLists.txt core/libs/prx/libSceIcu/libSceIcu.hpp docs/dev/debt/libSceIcu.md
PR 10966	core/libs/prx/libSceIcu/CMakeLists.txt core/libs/prx/libSceIcu/libSceIcu.hpp docs/dev/debt/libSceIcu.md
PR 10967	core/libs/prx/libSceIcu/CMakeLists.txt core/libs/prx/libSceIcu/libSceIcu.hpp docs/dev/debt/libSceIcu.md
PR 12320	core/libs/prx/libSceIcu/CMakeLists.txt core/libs/prx/libSceIcu/libSceIcu.hpp docs/dev/debt/libSceIcu.md
PR 12738	core/libs/prx/libSceIcu/CMakeLists.txt core/libs/prx/libSceIcu/libSceIcu.hpp docs/dev/debt/libSceIcu.md
PR 13486	core/libs/prx/libSceIcu/CMakeLists.txt core/libs/prx/libSceIcu/libSceIcu.hpp docs/dev/debt/libSceIcu.md
PR 13578	core/libs/prx/libSceIcu/CMakeLists.txt core/libs/prx/libSceIcu/libSceIcu.hpp docs/dev/debt/libSceIcu.md
PR 14212	docs/dev/debt/libSceIcu.md