Skip to content

libc: malloc, calloc, realloc and memalign throw through the guest call on failure instead of returning NULL with ENOMEM; fix branch ready #5242

Closed

Description

@husseinkad

When an allocation fails, the guest malloc, calloc, realloc, memalign, aligned_alloc and reallocalign throw a C++ exception (bad_alloc, length_error, runtime_error or system_error) through the guest call, which ends the process because guest code cannot catch it. FreeBSD 11.4 (jemalloc) returns NULL with errno = ENOMEM, or EINVAL for an alignment that is zero or not a power of two. A guest that checks for NULL (the normal pattern) never gets the chance. Cases that abort today: a size the heap rejects (SIZE_MAX, PTRDIFF_MAX + 1, 1 << 40 and up), a calloc product that overflows, a bad alignment, an exhausted arena, and a size that fits the roughly 1 TB guest arena but exceeds the host's commit limit (malloc(300 GiB) throws "paging file too small").

Fix branch: https://github.com/husseinkad/AnyPS5/tree/fix/libc-alloc-failures (four commits on a recent main: the change, then three follow-ups from review). FileAndHeap.cpp catches the failures and sets errno; GuestHeap.cpp turns an arena runtime_error and the host out-of-memory codes (Windows 8, 14 and 1455, Linux ENOMEM) into bad_alloc, and releases the reserved arena range when a commit fails; GuestArena.cpp is left untouched on purpose so it does not conflict with #5218. New tests GuestAllocatorFailures.cpp and rewritten expectations in GuestHeapRounding.cpp and GuestReallocalign.cpp; debt entries in docs/dev/debt/libc.md.

Checks:

  • guest_allocator_failures crashes on main and passes with the change; guest_heap_rounding, application_heap, guest_reallocalign, mspace, guest_memory and guest_new_handler pass. The test's leak check was weak in the first version (a reviewer removed the release call and it still passed); it now fails with that mutation.
  • Probed through the guest exports against main: 1,000 failing 300 GiB mallocs all return NULL/ENOMEM with no throw, private bytes flat and the arena's lowest-free address unchanged; a failed realloc(p, huge) leaves p intact and malloc_usable_size unchanged, reallocf frees it and sets ENOMEM; alignments 0, 3, 24 and SIZE_MAX give EINVAL; errno is untouched on success over 1,000 mixed multi-threaded calls; with the arena filled to 1 MiB free every failing call gave ENOMEM; 6 threads allocating alongside 300 failing huge mallocs showed no corruption or deadlock; zero bytes stay committed after a failed commit. Side effect: posix_memalign and reallocf on arena exhaustion now return ENOMEM where main threw.
  • Three independent reviews. check_conventions.py is clean; pr_overlap.py had no overlap on the first version, and fix(libkernel): fall back below a mapping hint with no room above it #5218 (which rewrites a function in GuestArena.cpp) is why that file is untouched.

Not covered: memalign and reallocalign setting errno is an assumption (FreeBSD has neither); FreeBSD 11.4 resizes realloc(ptr, 0) to a one-byte block, but this repo frees the block and returns NULL and existing tests assert that (recorded in the debt entry). The Linux commit-failure mapping was read, not built or run. Only Windows was built and run.

AI-assisted: yes

Activity

  1. MarcZX10 commented on Oct 13, 2026

    @MarcZX10
    Contributor

    Carried the allocation failure fix into PR #5289: #5289. Linux guest_allocator_failures and guest_reallocalign pass (2/2), and conventions pass. The host commit-limit failure case was not exercised locally; the Windows 300 GiB exhaustion behavior is from the issue report.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions