[Ragbie]

What

fclose and freopen close the stream's descriptor through the host C library and leave libkernel's state for that descriptor number behind: its status flags, close-on-exec and pipe flags, the random-device set, archived files, JIT shared memory, equeue registrations and, on Windows, directory descriptors and file locks. sceKernelClose and close drop all of it. The next descriptor that gets the same number, usually from the next fopen, inherits it:

• sceKernelOpen(O_WRONLY | O_CREAT | O_APPEND | O_CLOEXEC), fdopen(fd, "a"), fclose: the next fopen(path, "r+") gets the same number, fcntl(F_GETFL) returns O_RDWR | O_APPEND and fcntl(F_GETFD) returns FD_CLOEXEC. On Windows, sceKernelPwrite and sceKernelWrite on it append instead of writing at the offset.
• sceKernelOpen("/dev/urandom"), fdopen(fd, "r"), fclose: sceKernelRead on the next file with that number returns random bytes instead of the file's contents.
• freopen(path, "r+", stream) on such a stream behaves the same way.

FreeBSD 11.4 closes the descriptor in both: fclose calls cleanfile, which calls the stream's close function, __sclose, that is _close(fp->_file). freopen replaces the old descriptor with dup2 onto its number, or closes it. Either way the number comes back with the new file's flags and FD_CLOEXEC clear.

The change:

• File::ForgetClosedDescriptor holds the drops that sceKernelClose and close made inline. Both call it, in the same order, and on Windows still under the positioned-transfer lock.
• libkernel registers a begin/end pair with libc at load (LibcRegisterClosedDescriptors_nid_no_patch, as LibcRegisterDirectoryStreams_nid_no_patch does for directory streams). Begin drops the equeue registrations, takes the positioned-transfer lock on Windows and calls File::ForgetClosedDescriptor; end releases the lock. FileStream::Close and FileStream::Reopen hold it through ClosingDescriptor, so the host close runs under the same lock as in sceKernelClose, and a concurrent pread/pwrite either finishes before the close or sees the closed number. freopen releases it right after closing the old stream, before it opens the new file: UCRT's stream allocation locks the free streams it scans, so holding the old stream's lock into the open can deadlock against another thread's fopen. Without libkernel nothing is registered and there is no state to drop.
• On Windows ClosingDescriptor takes the stream lock before the positioned-transfer lock, the order a guest flockfile followed by lseek uses.
• File::LockPositionedTransfer returns a lock that co-owns its mutex, so a dup that reuses the number while the lock is held cannot free the mutex.
• FileStream::Close clears the stream only after the host fclose returns, so a thread that holds the stream with flockfile can still funlockfile it while fclose waits.

Tested

Windows 11, WinLibs GCC 15.2.0:

• New guest_stream_close:
  • The three cases above. With main's libc and libkernel it fails on the status flags, close-on-exec, the positioned and plain writes, the random-device read and the freopen status flags, close-on-exec and write.
  • An 8 MiB sceKernelPwrite on another thread while the main thread calls fclose, or freopen then fclose, 16 rounds; the write must complete or fail with EBADF, and nothing may land in the reopened file. When the lock is released before fclose, it fails 3 of 3 runs: pwrite gets ESPIPE because its handle is closed between _get_osfhandle and GetFileType.
  • A stream held with flockfile by a thread that then calls lseek while another thread fcloses it. With the positioned-transfer lock taken before the stream lock it hangs until the ctest timeout.
  • freopen 500 times while another thread fopens and fcloses a second file. With the stream lock held into the open it hangs until the ctest timeout in 3 of 3 runs.
  • With the change it passes with --repeat until-fail:10 (longest run 0.40 s).
• Full build and ctest after rebasing onto main 8a29e0883: 99% tests passed, 5 tests failed out of 2379. agc_write_tracking_metadata_commit, agc_host_import_cut_lifetime, agc_driver_depth_plane_partial_store and agc_driver_image_plane_address_on_line fail on main on this machine too, and guest_wide_formatting (wcsftime length matches strftime) fails the same way on main plus an unrelated socket change. check_skips.py: all skips expected.
• public_headers and its compile batches pass: the test now includes prx/libkernel/include/Api.hpp, prx/libkernel/File/include/File.hpp, prx/libc/include/Api.hpp and prx/libc/include/FileStream.hpp instead of declaring the 13 exports itself.
• Rebased onto main bf77a9e9c, after feat(libkernel): enumerate archived app directories #8520: File::ForgetClosedDescriptor now drops the directory-descriptor entry on every host, as sceKernelClose and close do since feat(libkernel): enumerate archived app directories #8520, so fclose and freopen drop it too. guest_stream_close adds a host directory closed through fdopen and fclose whose number the next file reuses (Windows tracks every directory). Full build, then the 134 tests matching stream_close|public_headers|kernel_fcntl|append_writes|file_lock|freopen|fclose|stdio|stream|app_archive|directory pass, and guest_stream_close and guest_kernel_app_archive_archive pass with --repeat until-fail:10. Full ctest was not rerun for this rebase.

Linux (Ubuntu 24.04 container, GCC 13.3.0): guest_stream_close fails on main and passes with the change, with --repeat until-fail:10. After the rebase, guest_stream_close and public_headers pass with --repeat until-fail:10, and full ctest gives 99% tests passed, 24 tests failed out of 2286; the same 24 fail on main in this container (22 agc_* on its lavapipe and kernel without UFFD_FEATURE_WP_ASYNC, pr_overlap_encoding and guest_system_load_exec_standard_error). On POSIX only app0.zar directories are tracked, so guest_kernel_app_archive_archive adds one closed through fdopen and fclose: with the call still under _WIN32 it fails at the read of the file that reuses the number (EISDIR), and with this head it passes, with guest_stream_close and the other two guest_kernel_app_archive_* tests, with --repeat until-fail:3. Full Linux ctest was not rerun after #8520.

Overlaps:

• feat(libc): support archive-backed stdio reads #7447 also changes FileStream::Close (archive detach); the two changes are independent.
• feat(libkernel): enumerate archived app directories #8520 (merged) moved File::ForgetDirectoryDescriptor out of the _WIN32 block in sceKernelClose and close; File::ForgetClosedDescriptor now does the same.
• feat(libc): implement fcloseall with tracked stream lifetimes #11831 (draft) changes the FileStream close and reopen lifecycle; agreed order: this PR first, feat(libc): implement fcloseall with tracked stream lifetimes #11831 rebases after it.

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: none
• Fixes: none
• Unblocks: none
• GPUs tested: n/a
• AI-assisted: yes

[MarcZX10]

The descriptor-state cleanup is needed, and the regression test covers the stale flags and reused descriptor cases. I found a Windows synchronization gap in the close path:

FileStream::Close() calls ForgetClosedDescriptor_nid_no_patch() and then calls std::fclose() (core/libs/prx/libc/include/FileStream.hpp:306-310). The registered callback acquires File::LockPositionedTransfer(descriptor) at core/libs/prx/libkernel/File/src/Open.cpp:275-280, but that lock is destroyed when the callback returns, before fclose actually closes the CRT descriptor. The Windows sceKernelClose/close paths keep the same lock alive across both state cleanup and the native close (Open.cpp:330-341, Stdio.cpp:728-749). Thus a concurrent positioned read/write can acquire the descriptor lock in the gap and race the stream close or descriptor reuse; the cleanup callback currently does not provide the same serialization guarantee.

Could the libc/libkernel handoff keep the positioned-transfer lock held until the underlying close/replacement has completed (for example, through a close wrapper/callback whose scope includes fclose/freopen), and add a Windows regression that coordinates a positioned transfer with fclose/freopen? The current sequential test does not exercise this interleaving. I have not run a local Windows concurrency reproduction, so this is based on the lock scopes and the existing close path.

[github-actions]

Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.

Pull request	Conflicting files (lines on main)	Same exports	Same new files
PR 11831			core/libs/tests/GuestStreamClose.cpp
PR 13390	core/libs/prx/libc/include/FileStream.hpp L193, L236

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request	Shared files not compared
PR 4563	core/libs/prx/libkernel/CMakeLists.txt
PR 5021	core/libs/prx/libc/include/FileStream.hpp core/libs/prx/libkernel/CMakeLists.txt core/libs/prx/libkernel/File/src/Open.cpp core/libs/prx/libkernel/File/src/Stdio.cpp
PR 6936	core/libs/prx/libkernel/CMakeLists.txt core/libs/prx/libkernel/File/src/Open.cpp core/libs/prx/libkernel/File/src/Stdio.cpp core/libs/tests/GuestKernelAppArchive.cpp
PR 7081	core/libs/prx/libc/src/FileAndHeap.cpp
PR 7088	core/libs/prx/libkernel/File/include/File.hpp
PR 7121	core/libs/prx/libkernel/File/src/Open.cpp core/libs/prx/libkernel/File/src/Stdio.cpp
PR 7192	core/libs/prx/libkernel/CMakeLists.txt
PR 7447	core/libs/prx/libc/include/FileStream.hpp core/libs/prx/libc/src/FileAndHeap.cpp core/libs/prx/libkernel/File/src/Stdio.cpp
PR 9019	core/libs/prx/libkernel/CMakeLists.txt core/libs/prx/libkernel/File/include/File.hpp core/libs/prx/libkernel/File/src/Open.cpp core/libs/prx/libkernel/File/src/Stdio.cpp
PR 9156	core/libs/prx/libkernel/CMakeLists.txt
PR 9293	core/libs/prx/libkernel/CMakeLists.txt core/libs/prx/libkernel/File/src/Stdio.cpp
PR 9297	core/libs/prx/libkernel/CMakeLists.txt core/libs/prx/libkernel/File/src/Stdio.cpp
PR 9302	core/libs/prx/libkernel/CMakeLists.txt core/libs/prx/libkernel/File/src/Stdio.cpp
PR 9383	core/libs/prx/libkernel/CMakeLists.txt
PR 9389	core/libs/prx/libkernel/CMakeLists.txt
PR 9526	core/libs/prx/libc/src/FileAndHeap.cpp
PR 9557	core/libs/prx/libkernel/CMakeLists.txt
PR 9566	core/libs/prx/libkernel/CMakeLists.txt
PR 10319	core/libs/prx/libkernel/File/src/Open.cpp
PR 10405	core/libs/prx/libc/include/FileStream.hpp core/libs/prx/libc/src/FileAndHeap.cpp
PR 11831	core/libs/prx/libc/include/FileStream.hpp core/libs/prx/libc/src/FileAndHeap.cpp core/libs/prx/libkernel/CMakeLists.txt core/libs/prx/libkernel/File/include/File.hpp core/libs/prx/libkernel/File/src/Open.cpp core/libs/prx/libkernel/File/src/Stdio.cpp core/libs/tests/GuestKernelAppArchive.cpp core/libs/tests/GuestStreamClose.cpp
PR 11939	core/libs/prx/libkernel/CMakeLists.txt
PR 12544	core/libs/prx/libkernel/CMakeLists.txt
PR 12560	core/libs/prx/libc/src/FileAndHeap.cpp core/libs/prx/libkernel/File/src/Open.cpp core/libs/prx/libkernel/File/src/Stdio.cpp
PR 12970	core/libs/prx/libkernel/CMakeLists.txt
PR 14264	core/libs/prx/libkernel/CMakeLists.txt
PR 14299	core/libs/prx/libkernel/CMakeLists.txt

[MarcZX10]

One additional integration point surfaced while checking the overlap report. #8520 moves File::ForgetDirectoryDescriptor outside the _WIN32 guard in close_nid_postfix and adds POSIX directory-descriptor tracking. In this PR, the new File::ForgetClosedDescriptor calls ForgetDirectoryDescriptor only under _WIN32. Once the two changes are combined, a POSIX fdopen/fclose or freopen of a tracked directory descriptor can therefore leave the directory-descriptor map entry behind even though the native descriptor has been closed or replaced. Please include the directory-descriptor cleanup in the shared closed-descriptor path on all platforms (while keeping Windows-only file-lock cleanup guarded), and cover stream closure/reuse of a directory descriptor in the combined test path. This is an integration finding from the current #8520 diff; I have not reproduced it at runtime.

[Ragbie]

@MarcZX10 Thanks, both points taken.

Lock across the close (5129671): libkernel now hands libc a begin/end pair instead of a single drop callback. Begin drops the equeue registrations, takes File::LockPositionedTransfer(descriptor) and drops the descriptor state; end releases the lock. FileStream::Close and FileStream::Reopen hold it through ClosingDescriptor, so fclose runs under the same lock as sceKernelClose. Holding it longer surfaced three more Windows issues, fixed in the same commit:

• ClosingDescriptor takes the stream lock (_lock_file) before the positioned-transfer lock. A guest flockfile followed by lseek takes them in that order, and the other order hung.
• freopen releases both locks right after closing the old stream, before it opens the new file. UCRT's stream allocation locks every free stream it scans, so holding the old stream's lock into the open deadlocked against another thread's fopen.
• LockPositionedTransfer returns a lock that co-owns its mutex, so a dup that reuses the number while it is held cannot free the mutex.

guest_stream_close gets the regressions you asked for, plus one for each of these:

• an 8 MiB sceKernelPwrite racing fclose / freopen. It fails 3 of 3 when the lock is dropped before fclose: pwrite gets ESPIPE because its handle is closed between _get_osfhandle and GetFileType.
• the flockfile + lseek holder, which hung with the old lock order.
• freopen against a concurrent fopen loop, which hung 3 of 3 with the lock held into the open.

All pass 10 of 10 on Windows and Linux; full runs are in the description.

Directory descriptors and #8520: on main, File::ForgetDirectoryDescriptor is defined only on Windows (DirectoryDescriptor.cpp), so calling it unguarded here would not link on Linux until #8520 lands. Whichever of the two merges second should move the call into File::ForgetClosedDescriptor unguarded. If #8520 goes first, I'll rebase and do that here, with an fdopen/fclose of a directory descriptor added to guest_stream_close. I've noted this in the description.

[MarcZX10]

Reviewed exact head 512967154daeaa0a830cf55f8c9e6a661deee46b. The begin/end close handoff now keeps the Windows positioned-transfer lock alive through the native fclose/replacement, and the stream lock is acquired before it, matching the flockfile then lseek order. The added regression exercises the close/write race, stream-lock/lseek interaction, descriptor reuse and freopen concurrent with stream creation. The author reports the focused test passing 10/10 on Windows and Linux; the applicable hosted relinker, ASan, register-context and write-watch checks are green. Product linux and windows jobs are skipped by policy, so I did not independently verify the platform runtime here.

The known #8520 integration is recorded in the PR description and discussed with the author: if POSIX directory-descriptor tracking lands first, the shared cleanup must be moved out of the Windows-only guard and the stream-close test extended. The current head is otherwise clean against main. I approve this exact head, with that follow-up integration point retained for the eventual combined branch.

[MarcZX10]

Reviewed the descriptor-state cleanup path, the libc/libkernel callback registration, and the close/reopen locking changes. Cleanup is performed before the host descriptor is closed, while the Windows positioned-transfer lock remains held; the new tests cover descriptor reuse, append and close-on-exec state, random-device reuse, freopen, and close races. Hosted relinker and write-watch suites pass on Linux, Windows, and macOS, along with conventions and intake.

[oneandonlydean]

@Ragbie Please replace GuestStreamClose.cpp’s local export declarations and FileStream forward declaration with the canonical public headers, preserving current main signatures. The clean main integration fails public_headers on 13 duplicate exports. Keep the lock-scope/race regressions and the recorded #8520 second-lander directory-cleanup condition, then rerun public_headers and the Windows/Linux stream-close bar.

[Ragbie]

@oneandonlydean Done. The branch is rebased onto main 8a29e0883, and be96d03 replaces the local declarations in GuestStreamClose.cpp. The 13 exports and FileStream now come from prx/libkernel/include/Api.hpp, prx/libkernel/File/include/File.hpp, prx/libc/include/Api.hpp and prx/libc/include/FileStream.hpp, with main's signatures.

• public_headers reported the 13 duplicates before this commit and passes after it, together with its compile batches, on Windows and Linux.
• guest_stream_close passes with --repeat until-fail:10 on Windows and Linux. The lock-scope and race regressions are unchanged.
• The feat(libkernel): enumerate archived app directories #8520 second-lander note for the directory-descriptor cleanup is still in the description.
• Full runs are in the description: Windows has the four known agc_* failures plus guest_wide_formatting, which fails the same way without this change. Linux has the same 24 failures as on main.

[oneandonlydean]

@Ragbie #8520 merged at 13:23Z (1061954), so the second-lander condition from my earlier comment (6390609564) now applies. Please:

• rebase onto main; it now conflicts in Open.cpp and Stdio.cpp, and the close/reopen lock regressions and main's close/dup cleanup should both survive;
• move File::ForgetDirectoryDescriptor into File::ForgetClosedDescriptor, outside the WIN32 block, since the cleanup is portable;
• add the agreed POSIX test that opens a directory descriptor with fdopen, closes it with fclose and checks the descriptor is released and reusable;
• rerun the stream-close/filesystem tests and the header checks on the rebased head.
  No new lock design or signature change is needed.

[Ragbie]

@MarcZX10 #8520 has landed, so as agreed File::ForgetClosedDescriptor now drops the directory-descriptor entry on every host, in the same order as sceKernelClose (92f4276, rebased onto main bf77a9e9c).

• On POSIX only app0.zar directories are tracked, so guest_kernel_app_archive_archive gains a case: open /app0/sce_sys/Concurrent, close it through fdopen and fclose, open a disk file that reuses the number and read it. With the call still under _WIN32 it fails on Linux at that read (EISDIR); with this head it passes with --repeat until-fail:3.
• guest_stream_close gains the same case for a host directory, which Windows tracks.
• Windows: full build, then the 134 tests matching stream_close|public_headers|kernel_fcntl|append_writes|file_lock|freopen|fclose|stdio|stream|app_archive|directory pass; guest_stream_close and guest_kernel_app_archive_archive pass with --repeat until-fail:10. Full ctest was not rerun for this rebase.

[Ragbie]

@oneandonlydean Done in 92f4276, rebased onto main bf77a9e9c (details in the comment above):

• Open.cpp and Stdio.cpp resolved: both close paths call File::ForgetClosedDescriptor, which keeps main's drops in sceKernelClose's order. The dup/dup2 cleanup and the close/reopen lock code are unchanged.
• File::ForgetDirectoryDescriptor is called outside the _WIN32 block.
• POSIX test: guest_kernel_app_archive_archive opens a directory descriptor, closes it with fdopen and fclose, and checks that the next file reuses the number and reads as a file. With the call still under _WIN32 it fails on Linux with EISDIR.
• Reruns on this head, tests matching filesystem|public_headers|stream_close|app_archive|directory: 88 pass on Windows; 84 pass on Linux with --repeat until-fail:3.

[B-Howell]

@Ragbie For the shared FileStream close/reopen lifecycle, I propose #10755 lands before my draft #11831 (fcloseall). #9763 and #10929 have already landed; #11872 is closed. I will adapt the registry lease/gate and close-all path to preserve your descriptor begin/end cleanup, FILE-before-descriptor lock order, and release-before-open rule, then rerun the combined stream regressions. Please confirm or revise that order. #11831 remains draft; no combined test result or author agreement is claimed yet.

AI-assisted: yes (Codex).

October 29 follow-through: focused private combined results are now available: 39 passes on each of Windows and Linux, with identical-fixture cleanup controls and independent review. Neither public branch changed. This does not imply agreement to the proposed order or public integration readiness.

[MarcZX10]

Re-checked — new head, looks good

The new POSIX directory-descriptor regression is present on head 92f4276f28e9643fe7e336e1cc40ee02f30277da, and the reported cross-platform focused tests cover it.

What I ran

Reviewed head 92f4276f28e9643fe7e336e1cc40ee02f30277da against PR base 3ead2d72517499985692df803ab72ff92ac82ecc; the merge preview against current main dc38224141e0b6096007b876b4c42fc9144d11b8 is clean. I inspected the eight PR files, including the portable descriptor cleanup and new POSIX test. Local checks ran in Linux; no compiler or build was invoked.

git diff --check refs/remotes/upstream/main...refs/review/pr-10755-current
python3 tools/check_conventions.py --base refs/remotes/upstream/main --head refs/review/pr-10755-current
git merge-tree --write-tree refs/remotes/upstream/main refs/review/pr-10755-current
gh pr checks 10755 --repo boykopovar/AnyPS5

Results

Check	Result
Diff check	Passed; no output
Repository conventions	Passed
Merge preview with current main	Clean; tree cce419e41f5934a32bb710eaad8a9ad4989b7579
Hosted checks	Conventions, eligibility, labels, report, register-context, relinker, relinker-asan, Windows relinker and write-watch passed on the exact head; product Linux/Windows jobs skipped
Focused tests	Author reports 88 matching tests passed on Windows and 84 on Linux, repeated three times, including the directory-stream regression; not independently run here

Unedited local check output

All conventions checks passed.
cce419e41f5934a32bb710eaad8a9ad4989b7579

Findings

Blocking

None found in this re-check. The added POSIX case closes an app archive directory through fdopen/fclose, verifies descriptor reuse by opening a regular file, and reads that file successfully.

Non-blocking

The focused tests are author-reported; I did not build or run them locally. Hosted product Linux and Windows jobs were skipped.

Sources

• Directory-descriptor regression at this head
• Stream-close regressions at this head
• Author's exact-head test report

Not verified

No local build or tests, console behavior, or full ctest run was performed. The author states the full test suite was not rerun for this rebase.

[Ragbie]

@B-Howell Agreed: this PR first, then #11831 on top of it, keeping the descriptor cleanup, the FILE-before-descriptor lock order and release-before-open as you list them. The description now records the order; Depends on: none stays here.

AI-assisted: yes (Claude Code)