Repository navigation
feat(libc): scan 16-bit wide conversions in fscanf on Linux - #6732
KernelPanicNT wants to merge 2 commits into
Conversation
On Linux, sscanf, vsscanf, sscanf_s and vsscanf_s threw for %ls, %lc, %l[, %S and %C because glibc stores a 32-bit wchar_t per character into the guest's 16-bit buffer. A format with one of these conversions is now scanned directive by directive in ScanGuest: each wide directive reads its bytes as the narrow directive does and widens every byte through this library's mbrtowc, which is what FreeBSD 11.4's vfscanf.c reads in the single-byte C locale of none.c. The byte buffer for a wide directive is bounded by the remaining input, not only by the guest's width. As in FreeBSD, %S and %C are wide whatever length modifier precedes them. The format pre-scan and ScanGuest recognise the q length modifier as FreeBSD does, so %lS, %hC and %qS are found too instead of reaching glibc, and %qd or %qn in a format scanned by ScanGuest stores a long long. On Windows, ScanGuest hands q to the host as ll for integer conversions. A narrow s, c or [ is handed to the host without any length modifier other than l, so %lls, %qs or %Lc in a format scanned by ScanGuest are narrow, as in FreeBSD, instead of reaching glibc as wide. A wide conversion combined with a positional argument or the ' flag still throws. fscanf keeps throwing for wide conversions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fscanf threw for %ls, %lc, %l[, %S and %C on Linux. The body of ScanGuest moves into ScanGuestSource, which reads from a source: a string, as before through ScanGuest, or a stream, where each conversion other than %n is one glibc fscanf call and the scanner peeks with getc/ungetc under the stream's lock. fscanf uses the stream source for formats with a wide conversion, so a wide directive reads its bytes as the narrow one and widens them through this library's mbrtowc, as sscanf already does. On the stream, wide directives take their bytes with the POSIX.1-2008 m assignment-allocation modifier, since a stream's string length is not known before it is read. The string source keeps its buffer bounded by the remaining input. Every byte the stream conversion stored is widened, so a NUL byte read by %ls or %l[ is kept as a 0 unit, as FreeBSD's convert_wstring and convert_wccl store it. m needs glibc, so on other non-Windows hosts fscanf throws std::runtime_error for a wide conversion. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
I checked the current PR bases and file lists: #6732 targets |
|
One relevant intake context: the earlier PRs for these same wide |
|
Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.
|
|
Pairing note for you and #6731: the two PRs are complementary halves of one change — this one covers |
|
Thanks for flagging the shared |
What
Depends on #6731, which adds 16-bit wide scanning to
sscanf,vsscanf,sscanf_sandvsscanf_son Linux. This PR is the single commit on top of it. On Linux,fscanfstill throws for%ls,%lc,%l[,%Sand%C(RejectGuestWideScan, called fromfscanf), because glibc would store a 32-bitwchar_tper character into the guest's 16-bit buffer. This PR scans them from the stream on glibc hosts.Source of the behaviour. No console measurement is involved. The model is the same as in the PR this depends on, now read from a stream:
release/11.4.0, commit8c63994)vfscanf.cL161-L308 and L683-L728: withl,%c,%[and%sread the stream one character at a time through the locale'smbrtowc.%S/%Care%s/%cwithl(L594-L612). FreeBSD'svsscanfruns the same__svfscanfon a stringFILE(vsscanf.cL61-L71), sofscanfandsscanfshare these semantics.none.cL99-L113: the C-localembrtowcmaps one byte to its value, as this library'smbrtowcdoes.massignment-allocation character of POSIX.1-2008fscanf(added in Issue 7, SD5-XSH-ERN-132; accepted by glibc's__isoc99_vfscanf). The number of bytes a stream directive reads is not known in advance, andmmakes glibc allocate exactly that many. This is host plumbing only; the guest sees no allocation. Apple's libc has nom(vfscanf.chandlesj,l,q,t,z,Landhonly), so the stream path is built only where__GLIBC__is defined. The string functions do not usem.Changes:
ScanGuestmoves intoScanGuestSource, a template over an input source.ScanGuestnow wraps it with aStringScanSource, which holds the existing string cursor and, as in the PR this depends on, scans a wide directive into a buffer bounded by the remaining input. Its callers (Linuxsscanf/vsscanffor wide formats,sscanf_s/vsscanf_son both platforms) keep their behaviour on every host.StreamScanSource(glibc hosts only) reads a hostFILE*and holds its lock (flockfile, released by the destructor, also when the scanner throws) for the wholefscanfcall, so another guest thread cannot read the same stream between the per-directive calls. Literals and whitespace are matched with onegetc/ungetcpair per peeked byte. Each conversion other than%nis one glibcfscanfcall with a trailing%n, which counts the bytes the call consumed. A guest%nis filled from the scanner's own count, without a glibc call.fscanfsends a format with a 16-bit wide conversion toScanGuestSourceover the stream. Formats without one still go to glibc'svfscanfunchanged, andSyncStatusstill runs after either path. On a non-Windows host without glibc (macOS), a format with a wide conversion throwsstd::runtime_errorbefore reading the stream; other formats go to the hostvfscanfas before.%<width>m<narrow conversion>. glibc allocates the bytes, which are widened throughmbrtowcas in the string case and then freed. Every byte the conversion stored is widened, counted by the trailing%n, so a NUL byte that%lsor%l[reads from the stream becomes a 0 unit and the bytes after it are kept, as FreeBSD'sconvert_wstringandconvert_wcclstoreL'\0'and continue.%lsand%l[are terminated,%lcis not. Theqhandling and the narrows,cand[of the PR this depends on apply to the stream too:%qSis wide and%qsnarrow.RejectGuestWideScanhad no other caller and is removed.Still throwing, and why:
%1$ls) or the'flag throwsstd::runtime_errorbefore reading the stream, as in the PR this depends on.%n(%ls%Ln,std::invalid_argument) throws only when the scanner reaches that directive. By thenfscanfhas consumed the stream up to it, and the throw skipsva_endandSyncStatus. This is recorded indocs/dev/debt/libc.md.fscanfthrowsstd::runtime_errorbefore reading the stream, because the host has nom. The previous throw wasNotImplemented_nid_no_patchon every non-Windows host.fscanftranslates the format for the host, whosewchar_tis 16 bits.%Sand%Cstill throw in that translation.Tests:
guest_standard_streamsgainsCheckWideScanon glibc hosts, run ontmpfilestreams. Each case also checks what is left in the stream throughfgetc:%ls%n %3lc%donab cdef 12: 2 assignments,%nis 2,%dfails onf, andfis left.%C+%lcona-b: 1 assignment, and-is left.%l[a-z]on12: returns 0, the buffer is untouched, and1is left.%Son bytes\xe9t\xff: U+00E9,t, U+00FF, and the space is left.EOF, with the guest EOF flag set, andfgetcreturnsEOF.%*ls%donab: returns 0, andfgetcreturnsEOF.%1$ls: throws with the buffer and the stream untouched, andais left.%l[^\n]on the bytesa, NUL,b, newline: 1 assignment, the unitsa, 0,b, 0, the guard units after them untouched, and the newline is left.%qSonxy z: 1 assignment,x,y, 0, and the space is left.On other non-Windows hosts,
CheckWideScanchecks that%lsonabthrowsstd::runtime_errorwith the buffer untouched andaleft.Progress (
tools/progress.py, base5d9dcc4plus the PR this depends on): system libraries 3300/3706 → 3301/3706 (+fscanf_nid_postfix). The tool countedfscanfas unimplemented because its single definition reachedNotImplemented_nid_no_patchon Linux.Tested
Head
ccc10a467756c30cc426e76f9e8db2a1622bb84c, base29cfd9985511d8e779d52d6f6dc6612343ce88c1.BUILD_TESTING=ON, AMD EPYC 7763 (GitHub Codespaces), tests run natively):guest_standard_stream_testsandguest_formatting_testseach pass in 5 of 5 runs, the test executable being run directly five times.core/libs/prx/libc/src/Formatting.cppwith the base version:guest_standard_stream_testsfails withfscanf 16-bit wide conversion on Linux not implemented;guest_formatting_testsstill passes there, as expected for regression coverage. Restoring the PR source and rebuilding,guest_standard_stream_testsandguest_formatting_testseach pass in 5 of 5 runs again.mainat5d9dcc42d2ec5401aebd089028b8d4d7410a77b4, WinLibs MinGW-w64 GCC 15.2.0, Release,BUILD_TESTING=ON: 1300/1300 tests pass; the relinker (Linux, macOS), register-context and write-watch jobs of the same run also pass.git diff --checkpass. The Linux job of the Build workflow now targets the project's self-hosted runner, so it was not run on the fork; no title was run.Duplicate and overlap check
fscanfconversions on Linux by decoding the stream's UTF-8; it was stacked on feat(libc): scan 16-bit wide conversions on Linux #5285, also by @Nixitov, which did the same forsscanf/vsscanf(the PR this depends on covers that part). Both were closed on 2026-10-14 by @oneandonlydean with the same comment: "Closing: no title is known to reach this. With the queue at 700+ open PRs, please hold off on new PRs that don't name a title needing the change until it's down. If a title stops on this on current main, reopen with the title ID and the log line." This PR does not name a title either (Unblocks: none). It covers the same functions as feat(libc): scan 16-bit wide conversions in fscanf on Linux #5291 with a different method: each byte is widened through this library's single-bytembrtowc, following FreeBSD's C locale, instead of decoding UTF-8. feat(libc): scan 16-bit wide conversions in fscanf on Linux #5291 already had the stream design this PR uses for everything but the wide bytes: it read literals, whitespace and%ncounts one character at a time with agetc/ungetcpeek so that the stream position and the counts match, scanned each other conversion of such a format with one hostfscanfcall and a trailing%nthrough a variadicHostFscanfwrapper overHostVfscanf, and dispatched infscanf_nid_postfixonHasGuestWideScan. This PR follows that design. fix(libc): throw for 16-bit wide scanf conversions on Linux #3984 introduced the throw this PR replaces.fscanf, its NIDnpLpPTaSuHgand "wide scanf". No open PR implements wide scanning infscanf. feat(libc): implement wide and unlocked stream I/O #5816, merged since feat(libc): scan 16-bit wide conversions in fscanf on Linux #6018 was opened, added wide stream input (fgetwc,fgetws,getwc,getwchar) and not scanf conversions. Itsfgetwcdecodes the stream as UTF-8, whereas a widefscanfconversion here widens each byte, as the single-byte C locale of this library'smbrtowcand FreeBSD'snone.cdo. The two differ only for bytes above 0x7f; this PR follows the C locale that_Getmbcurmaxreports (1).Formatting.cpp,core/libs/tests/Formatting.cpp,core/libs/tests/GuestStandardStreams.cppordocs/dev/debt/libc.mdinto this branch (git merge --no-commit) and intomain.fscanfreturnEOFwitherrnoset toENOTSUPfor a wide conversion, instead of scanning it. It conflicts with this PR and with feat(libc): scan 16-bit wide conversions in sscanf and vsscanf on Linux #6731 inFormatting.cpp. If it is merged first, these PRs replace that early return.core/libs/tests/Formatting.cpp) and fix(libc): count wide collate string elements #6517 (thecollateline next to the scanf line indocs/dev/debt/libc.md). These are textual conflicts in neighbouring lines, resolved by keeping both sides.mainincore/libs/tests/GuestStandardStreams.cppandcore/libs/prx/libc/CMakeLists.txt. fix(libc): match title printf behavior on Windows #3987, fix(libc): follow measured wide printf behavior #5303, feat: read title files from a read-only app0.zar (ZArchive) layer #5835 and Macos m4 #6222 also already conflict withmain.5d9dcc4.Checklist
5d9dcc4plus feat(libc): scan 16-bit wide conversions in sscanf and vsscanf on Linux #6731; no other open PR implements the same functionsNotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file.mdfiles or images added to the repository (attach them to this PR)