Skip to content

fix(libc): append stream writes atomically on Windows - #6287

Merged
adriadam10 merged 1 commit into
boykopovar:mainfrom
ahmad3a4:fix/libc-append-streams-windows
Oct 18, 2026
Merged

adriadam10 merged 1 commit into
boykopovar:mainfrom
ahmad3a4:fix/libc-append-streams-windows

Conversation

@ahmad3a4

Copy link
Copy Markdown
Contributor

What

On Windows, fopen and freopen in an append mode (a, ab, a+, ...) give the C runtime a descriptor with _O_APPEND. The runtime appends by seeking to the end and then writing, two separate steps. When two streams on separate descriptors flush at once, both seek to the same end and the later write overwrites the earlier one. FreeBSD's fopen and freopen open these modes with O_APPEND, and vn_write writes each write(2) at the end of the file under the vnode lock, so flushes never overwrite each other. This is the stream side of #5831, which covers descriptors opened with sceKernelOpen. @screamm pointed it out there.

The change:

  • fopen opens the append modes with FILE_GENERIC_WRITE minus FILE_WRITE_DATA, plus GENERIC_READ for a+. Windows documents that with FILE_APPEND_DATA and without FILE_WRITE_DATA, "write operations will not overwrite existing data": every write on such a handle goes to the end of the file, wherever the file pointer is, and the pointer ends up after the data.
  • freopen in an append mode moves an append-only duplicate of the handle onto the stream's descriptor, the same way fdopen already moves an _O_APPEND duplicate (now one shared MoveAppendingHandle). For descriptors 0 to 2 it keeps the host standard handle, as FileStream::Reopen does.
  • fdopen is unchanged. FreeBSD's fdopen does not set O_APPEND either; it only seeks before each write (__SAPP), which is what the existing code does.
  • A handle without FILE_WRITE_DATA cannot truncate, and without read access it cannot take a byte-range lock. FreeBSD allows both on these descriptors. ftruncate on an append-only handle now truncates through ReOpenFile(GENERIC_WRITE) (as fsync already reopens), and flock locks through a reopened handle kept in the lock state, which closes when the last descriptor sharing that state is forgotten. Without this, flock on an a stream would throw, because ERROR_ACCESS_DENIED is not a lock conflict. All other descriptors take the existing paths. File-backed mmap is not supported on Windows, so it is not affected.

guest_append_streams: four threads append 1,500 fixed-length lines each to one file through fopen("a"), fopen("a+"), freopen("a") and freopen("ab"), flushing after each line. The file must hold exactly 6,000 intact lines, each once. Then:

  • an a+ stream reads from the start, and a write after fseek(0, SEEK_SET) lands at the end with ftell after it;
  • ftruncate on an a stream's descriptor truncates, and the next write appends after the new end;
  • flock on an a stream's descriptor excludes a second open of the file, and fclose releases it.

With main's libc and libkernel it fails at the size check in 10 of 10 runs; with the change it passes 10 of 10.

Tested

Windows 11 Pro (26200), i5-13400, WinLibs GCC 15.2.0 r7, Release, BUILD_TESTING=ON, on main 721db55:

Libraries Places With bytes lost
main d92a8da 4 4, e.g. [debug:0] [kodi] 2026-10-15 [src/Open.1pp:215 sceKernelOpen] ... followed by 0 T:100001 ...
main's libc and libkernel, same relink as below 3 3, e.g. [debug:0] [kodi] [0s5-cache] clflushopt=1 ...
this PR, run 1 2 0
this PR, run 2 3 0

With the change, a line can still be split around another writer's line, for example [debug:0] [kodi] 2026-10-1 + [ps5-cache] clflushopt=1 cacheline=64 + 5 11:57:03.480 T:100002 .... That happens because the title writes one log line in more than one write, and FreeBSD splits it the same way. No bytes are lost.

  • A standalone program (4 threads, each with its own stream, 2,000 lines each, flushing every line): with a GENERIC_WRITE handle and _O_APPEND, 3,124 and 3,179 of 8,000 lines arrive; with the append-only handle, 8,000 of 8,000. The same program shows fsync works on such a handle, ftell is at the end after each write, and _chsize_s and LockFileEx fail on it, which is why ftruncate and flock reopen.

Linux not tested. Nothing changes in the Linux build; the new test uses only guest functions, so it runs there too.

Checklist

  • Based on current main; no other open PR implements the same functions
  • One topic per PR; follow-ups go in a new PR
  • No comments in code except technical debt
  • Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
  • No notes, investigation .md files or images added to the repository (attach them to this PR)
  • New third-party code is a submodule built from source
  • Generic behaviour, not specific to one title
  • Depends on: none
  • Unblocks: none (fixes lost log output in PPSA99420 PS5MC, reported on fix(libkernel): append O_APPEND writes atomically on Windows #5831)
  • GPUs tested: n/a
  • AI-assisted: yes

@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request Shared files not compared
PR 4972 core/libs/prx/libkernel/CMakeLists.txt
PR 5021 core/libs/prx/libkernel/CMakeLists.txt
core/libs/prx/libkernel/File/src/Stdio.cpp
PR 6044 core/libs/prx/libkernel/CMakeLists.txt
PR 6226 core/libs/prx/libkernel/CMakeLists.txt
PR 6234 core/libs/prx/libc/src/FileAndHeap.cpp
core/libs/prx/libkernel/File/src/Stdio.cpp
PR 6541 core/libs/prx/libc/src/FileAndHeap.cpp
PR 6879 core/libs/prx/libkernel/File/src/Stdio.cpp
PR 6908 core/libs/prx/libkernel/CMakeLists.txt
PR 6936 core/libs/prx/libkernel/CMakeLists.txt
core/libs/prx/libkernel/File/src/Stdio.cpp
PR 6977 core/libs/prx/libkernel/CMakeLists.txt
core/libs/prx/libkernel/File/src/Stdio.cpp
PR 7192 core/libs/prx/libkernel/CMakeLists.txt
PR 7201 core/libs/prx/libc/src/FileAndHeap.cpp
core/libs/prx/libkernel/CMakeLists.txt
core/libs/prx/libkernel/File/src/Stdio.cpp

This was referenced Oct 15, 2026
@screamm

screamm commented Oct 15, 2026

Copy link
Copy Markdown
Contributor

Cross-validation on Windows 11 Pro 26220, NVIDIA RTX 4070 (596.21), MinGW GCC 15.2, PR head a431a12a4 merged on main 821c69e50:

  • Merge: clean. Full build and libs: pass, no warnings in the files this PR touches.
  • guest_append_streams, --repeat until-fail:3: 3 of 3 pass.
  • Negative control: this head's guest_append_streams_tests.exe with the libraries of a pre-PR main build first on PATH fails 9 of 10 runs at the size check (append stream check failed at line 85); with this PR's libraries it passes every run. (Library swap, not a source revert.)
  • Extra check outside the PR's test: two threads, each with its own fopen(path, "a") stream, 3000 variable-length lines each (83 bytes on average, up to about 160), once with a flush after every line and once with default buffering. With the PR all 6000 distinct lines are intact and the size is exact in 3 of 3 runs of each variant; with main's libraries 3 of 3 flushed runs lose lines (4043, 5492 and 4078 of 6000 intact) and 2 of 3 unflushed runs do.
  • 19 file, descriptor and lock tests (guest_file_lock_*, guest_kernel_fcntl, guest_dup, guest_host_streams, file_position, guest_kernel_sync, guest_terminal, ...), --repeat until-fail:3: 18 pass 3 of 3. guest_file_lock_process aborts on this loaded host with and without the PR (5 alternating runs each: PR 0 of 5, pre-PR main 0 of 5; it starts 24 child processes with a 3 s limit each) and passed under ctest in a calmer moment (11.45 s).
  • ctest -j8 --timeout 120 -E '^agc_driver_|^video_out_flip': 807 of 818 pass, 2 skipped, 9 fail (8 timeouts under load and the guest_file_lock_process abort above); all 9 pass when rerun serially.
  • tools/check_conventions.py --base origin/main and git diff --check: pass.
  • Diff: no code comments, no new switches, no stubs; the test is registered like its neighbours, with the Windows ENVIRONMENT_MODIFICATION for the libkernel and libc directories.

No concerns. Concurrent fopen/freopen append streams no longer lose or overwrite lines, and ftruncate and flock keep working on them.

Written with Claude Code assistance.

fopen and freopen in an append mode gave the C runtime a descriptor with
_O_APPEND, which seeks to the end of the file and then writes. Streams on
separate descriptors overwrote each other's data. FreeBSD opens these
streams with O_APPEND, so every flush lands at the end of the file.

Open append streams with FILE_APPEND_DATA but without FILE_WRITE_DATA,
so Windows writes each flush at the end of the file. ftruncate and
flock reopen the file to get the access such a handle lacks.
@ahmad3a4
ahmad3a4 force-pushed the fix/libc-append-streams-windows branch from a431a12 to 0b24f92 Compare October 18, 2026 12:24
@ahmad3a4

Copy link
Copy Markdown
Contributor Author

Rebased onto main 3e3e8b9. The only conflict was in NativeFtruncate: 0948f77 (fix(libkernel): reject invalid ftruncate descriptors safely) now returns EBADF for an invalid handle and runs _chsize_s without the invalid-parameter handler. The append-only branch now sits after those two checks, and the rest is main's version.

Windows 11 Pro, WinLibs GCC 15.2.0 r7, Release: full build passes. Full ctest -j8 --timeout 120: 1,479 of 1,479 pass; 5 of them timed out while another build was running on this machine and passed when rerun alone. guest_append_streams and 10 related file and lock tests pass 5 of 5. tools/check_conventions.py passes.

Thanks @screamm for the cross-validation, and for the extra variable-length and unflushed runs.

@github-actions github-actions Bot removed the needs rebase Conflicts with main, rebase needed label Oct 18, 2026
@adriadam10 adriadam10 self-assigned this Oct 18, 2026
@github-actions github-actions Bot added the unblocks title Names a title ID on the Unblocks line label Oct 18, 2026
@adriadam10
adriadam10 merged commit 38a848a into boykopovar:main Oct 18, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

build libc libkernel unblocks title Names a title ID on the Unblocks line

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants