Skip to content

fix(libc): keep the host standard stream when a guest freopen of it fails - #5668

Merged
adriadam10 merged 1 commit into
boykopovar:mainfrom
babyhuey:fix/libc-keep-host-standard-streams
Oct 18, 2026
Merged

adriadam10 merged 1 commit into
boykopovar:mainfrom
babyhuey:fix/libc-keep-host-standard-streams

Conversation

@babyhuey

@babyhuey babyhuey commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

What

A guest freopen of stdin, stdout or stderr reopens the host's own standard stream, since the guest FileStream wraps it. When that freopen fails, the host freopen has already closed the host stream. From then on, every host-side print to it fails: on Linux silently with EBADF, on Windows through the UCRT invalid-parameter handler, which reports FATAL: UCRT invalid parameter and dumps all threads.

Now, for a host standard stream:

  • FileStream::Reopen duplicates the stream's descriptor (DuplicateHostDescriptor_nid_no_patch) and holds the stream lock across the freopen. If the freopen fails, it reopens the host stream on the null device, moves the duplicate back onto its descriptor with dup2 (_dup2 on Windows) and closes the duplicate. The guest FileStream stays closed, so the guest sees a closed stream as on FreeBSD (fix(libc): fail stream calls on a stream that a failed freopen closed #5632). The target is opened once, by the real freopen, so "wx", "w" and "a" behave as on main.
  • When freopen_nid_postfix fails path resolution, it detaches the guest FileStream instead of calling Close(), so the host stream is never closed.
  • On Linux, a reopen of stderr already moves the host onto its kept copy (KeepHostStandardError_nid_no_patch), so only stdin and stdout need the restore there.

Streams that aren't standard behave as before.

Tested

On the PR head 2d8d3b9 (rebased onto main 5bf2e13), Linux (Ubuntu 24.04, GCC 13) and Windows 11 (MinGW-w64 GCC 15.2 from WinLibs), Release:

  • guest_standard_streams, guest_standard_streams_closed_stdout, the new guest_standard_streams_exclusive_stdout and file_streams each pass 5/5 (--repeat until-fail:5) on both.
  • --closed-stdout now also writes and flushes the host stdout after the failed guest freopen, then does the same for a failed "a" reopen of stderr; it fails against main's library (on Windows with 0xC0000409). --exclusive-stdout reopens stdout with "wx" on a fresh file, writes through it and checks the bytes; it fails against the previous head's probe (the reported EEXIST).

PS5LM (PPSA99581 Plus 0.8.3 and PPSA99575 0.9.0) calls freopen("/data/PS5LM/app.log", "w", stdout) and then reopens stderr with "a" at start-up, and /data/PS5LM doesn't exist:

  • Linux, headless (SDL offscreen), 3 runs each (library code identical to the PR head; only the test file changed since): both stop at the same DisplayWindow Vulkan check on both builds. On main the 12 host log lines after the freopen (Vulkan device setup) are missing from stdout; with this PR they are there.

  • Windows 11 (RTX 3080), 60 s, 2 runs each, PS5LM Plus 0.8.3 and PS5LM 0.9.0:

    Build FATAL: UCRT invalid parameter reports stdout lines FPS (window title)
    main 5bf2e13 243 to 700 per run 2 (stops after Transferring control to ELF entry point) 0.00 in 3 of 4 runs
    this PR 0 101 to 155 59.92 to 59.96

    With this PR the PS5LM Plus dashboard and the PS5LM 0.9.0 "Choose a model" page draw, and host lines keep reaching stderr after the failed reopens ([HUI] pad open, [HUI] audio open, the no models notification). On main, stderr holds only the UCRT reports and their backtraces.

  • PS5LM Plus 0.8.4, now in the catalog, creates /data/PS5LM itself (it opens /data/PS5LM/.probe with O_CREAT), so its freopen succeeds and it no longer reaches this path; PS5LM 0.9.0 still does.

  • No new warnings in the files this PR touches on either OS.

check_conventions.py passes.

Checklist

@github-actions github-actions Bot added the needs rebase Conflicts with main, rebase needed label Oct 14, 2026
@babyhuey
babyhuey force-pushed the fix/libc-keep-host-standard-streams branch from 2f9026c to 881a109 Compare October 14, 2026 06:39
@github-actions github-actions Bot removed the needs rebase Conflicts with main, rebase needed label Oct 14, 2026
@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown

Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.

Pull request Conflicting files (lines on main) Same exports Same new files
PR 6541 core/libs/prx/libc/include/FileStream.hpp L82

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request Shared files not compared
PR 5021 core/libs/prx/libc/CMakeLists.txt
core/libs/prx/libc/include/FileStream.hpp
core/libs/prx/libc/src/General.cpp
PR 6044 core/libs/prx/libc/CMakeLists.txt
core/libs/tests/GuestStandardStreams.cpp
PR 6222 core/libs/prx/libc/CMakeLists.txt
core/libs/prx/libc/include/FileStream.hpp
core/libs/prx/libc/src/General.cpp
PR 6226 core/libs/prx/libc/src/General.cpp
PR 6334 core/libs/prx/libc/CMakeLists.txt
PR 6879 core/libs/prx/libc/src/General.cpp
PR 7084 core/libs/prx/libc/CMakeLists.txt
PR 7201 core/libs/prx/libc/CMakeLists.txt
core/libs/prx/libc/include/FileStream.hpp
core/libs/prx/libc/src/FileAndHeap.cpp
core/libs/prx/libc/src/General.cpp
PR 7332 core/libs/prx/libc/CMakeLists.txt
core/libs/prx/libc/include/FileStream.hpp
core/libs/prx/libc/src/FileAndHeap.cpp
core/libs/prx/libc/src/General.cpp
PR 7447 core/libs/prx/libc/CMakeLists.txt
core/libs/prx/libc/include/FileStream.hpp
core/libs/prx/libc/src/FileAndHeap.cpp
PR 7467 core/libs/prx/libc/src/General.cpp

@UmillD

UmillD commented Oct 15, 2026 •

Copy link
Copy Markdown

I found a regression in FileStream::Reopen on head 881a109: successful exclusive creation of a standard stream now fails.

For a fresh filename, FileStream(stdout).Reopen(path, "wx") succeeds on base d8888df. On this PR, _wfopen creates the file during the probe; the following _wfreopen of the same path with "wx" fails with EEXIST (errno 17). The guest gets a failed reopen even though its call created the file.

I compiled the exact base and candidate FileStream.hpp headers independently on Windows using WinLibs GCC 15.2, C++20, O2. All ten paired runs gave base=true/errno=0 versus candidate=false/errno=17/file_exists=true. The Windows handle-preservation and file-lock integration callbacks were no-ops in this focused harness; this is method-level evidence, not a full runtime or PS5LM run.

The existing missing-directory test does not cover this success path. Could the fix avoid opening the target twice with its destructive/exclusive mode, and add a regression for successful "wx" creation through a standard stream? The original missing-directory/host-output regression should remain covered as well.

AI-assisted: yes, investigated with Codex. No replacement implementation is proposed in this report.

Update: I also reproduced this through the real guest libc exports, with no handle/lock callback stubs. The test calls freopen_nid_postfix(path, "wx", __stdoutp_nid_postfix), then fputs_nid_postfix/fflush_nid_postfix and checks the resulting payload; all calls use the guest sysv_abi convention and errno is read through __error_nid_postfix.

This was a controlled local integration: validation commit 4879dc913 plus this PR's four-file patch applied cleanly, rather than a build of the isolated PR head. After rebuilding libc, all ten candidate processes failed with nullptr/errno 17 and an empty created file. After reversing the patch and rebuilding, all ten control processes successfully created the file and verified the payload. Meanwhile guest_standard_streams, guest_standard_streams_failed_reopen and file_streams each passed five runs on the candidate (15/15). No title execution is claimed.

@adriadam10 adriadam10 self-assigned this Oct 15, 2026

@adriadam10 adriadam10 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix, and for the PS5LM numbers. Checked the diff against the new head; one regression blocks it, the one @UmillD reported:

  • core/libs/prx/libc/include/FileStream.hpp:96-106: the probe opens the target with the guest's own mode before the real reopen. With "wx" the probe creates the file, so the following freopen of the same path fails with EEXIST. A valid freopen(fresh, "wx", stdout) that succeeds on main now fails and leaves the stream detached. The same double open also truncates/creates twice for "w"/"a".
  • core/libs/tests/GuestStandardStreams.cpp:497-501: only the failing missing-directory case is covered. Please add a regression for a successful "wx" reopen of a standard stream (write through it and check the file contents), and keep the existing one.

Direction: avoid opening the target twice. For example, keep a dup of the host stream's descriptor, run the real freopen, and on failure restore the descriptor with dup2 (or _dup/_dup2 on Windows) and detach the guest stream; that way no destructive probe is needed. Your call on the exact design.

@DavidSamir

DavidSamir commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Windows run on a second PS5LM entry: PS5LM 0.9.0 (PPSA99575, catalog release; the PR text tested PPSA99581 Plus). Its eboot contains /data/PS5LM/app.log, /data/PS5LM/app-args.txt and /data/PS5LM/cache, and /data/PS5LM does not exist in a new run folder.

Windows 11 Pro build 26300, Intel i7-11700F, NVIDIA GeForce RTX 3070, WinLibs GCC 15.2, Release, relinked with --windows, 60 s runs, fixed controller sequence, a new output folder each time:

Build FATAL: UCRT invalid parameter reports in the process's stderr lines in the process's stdout
main 82ddae7 329 2 (the host CPU line and Transferring control to ELF entry point)
main 82ddae7 + this PR's library change (FileStream.hpp, FileAndHeap.cpp) 0 99 (the host log lines are back)

With gdb on the unfixed build, the first _invalid_parameter_noinfo is reached from ucrtbase!fwrite, called by AgcDriver::AsyncLog::Run() in libSceAgcDriver, so the host log thread is the one writing to the closed host stream.

Note for the rebase: on current main this PR merges with conflicts in core/libs/prx/libc/CMakeLists.txt and core/libs/tests/GuestStandardStreams.cpp. Both sides add a test next to guest_standard_streams (main has guest_standard_streams_closed_stdout and a CheckClosedStreamWrites/CheckClosedStandardOutput pair; this PR adds guest_standard_streams_failed_reopen). For this run I kept main's side of both files and took only the library change, so this PR's own test was not run here. I did not measure frame rate.

AI-assisted: yes

@babyhuey
babyhuey force-pushed the fix/libc-keep-host-standard-streams branch from 881a109 to 2d8d3b9 Compare October 18, 2026 13:48
@babyhuey

Copy link
Copy Markdown
Contributor Author

@adriadam10 Reworked as suggested, and rebased onto main 5bf2e13. The probe is gone, so the target is opened once, by the real freopen:

  • For a host standard stream, Reopen duplicates its descriptor first and holds the stream lock across the freopen. If the freopen fails, it reopens the host stream on the null device and moves the duplicate back onto its descriptor with dup2 (_dup2 on Windows), then drops the duplicate. The null-device reopen is needed because a failed freopen leaves the FILE itself closed, so restoring only the descriptor would still leave the host's next fwrite on a closed stream. The guest FileStream stays closed, as fix(libc): fail stream calls on a stream that a failed freopen closed #5632 expects. On a successful freopen the duplicate is just closed.
  • A successful freopen(fresh, "wx", stdout) now creates the file and writes through it. The new guest_standard_streams_exclusive_stdout test covers it: it fails against the previous head (abort on the freopen), and passes on this one. The missing-directory host-write check moved into main's existing --closed-stdout test, which fails against main's library and passes here.

Thanks @UmillD for the "wx" reproduction. Results for both OSes are in the description.

@github-actions github-actions Bot removed the needs rebase Conflicts with main, rebase needed label Oct 18, 2026
@adriadam10
adriadam10 merged commit f03a2da into boykopovar:main Oct 18, 2026
17 of 18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants