[Runtime correctness] Complete shader modes, Linux exception delivery, and thread cancellation #1668

Split from #1659. Category: runtime correctness and completeness across Windows and Linux.

Problem and desired result

A title can convert and resolve its imports while still depending on shader or kernel behavior the emulator does not implement correctly. This tracker covers three concrete areas from the review: per-shader floating-point modes, guest exception delivery, and thread cancellation.

The goal is verified observable behavior with clear support limits. A successful API return, an exported symbol, or one title getting farther is not sufficient evidence that the general behavior is correct.

Baseline and evidence boundaries

Source baseline: upstream 4b005b229e24be9ec3f185def9e59cbc03aa9e4a.

The source gaps below are present at that commit. Existing reports and proposed fixes are linked for coordination; their claims retain their original evidence level. This issue does not assert that every title uses these features or that the unconfirmed Windows lock report has been reproduced.

Workstream A: shader floating-point modes

Use existing issue #981 as the detailed implementation/semantics discussion rather than creating a parallel competing specification.

That issue records the need to carry IEEE_MODE, DX10_CLAMP, FLOAT_MODE, and FP16_OVFL, with hardware measurements tied to the actual mode. A shader can also change mode during execution. ScalarInstructions.cpp currently accepts only round-to-nearest-even for s_round_mode and explicitly rejects s_denorm_mode.

• Decode the initial mode for each supported shader stage and carry it through the recompile request, any serialized request/capture, and cache identity. A mode change must not reuse incompatible compiled output.
• Establish mode-specific expected results using console/hardware measurements or exact authoritative implementation/ISA references, following CONTRIBUTING.md and Proposal: shader PRs state where instruction semantics come from (hardware or cited source), plus a hardware oracle tool #843.
• Implement supported combinations using host capabilities deliberately. Where Vulkan float controls are needed, query and record their support; avoid leaving behavior accidentally dependent on the host driver.
• Track supported immediate runtime mode changes. Explicitly reject dynamic or otherwise unsupported cases until their behavior is implemented and verified.
• Test affected instruction families with mode-sensitive values: NaNs, infinities, signed zero, denormals, rounding boundaries, and f16 overflow where applicable. Apply a comparison rule appropriate to the instruction, rather than requiring bit equality where the reference allows variation.
• Add a cache regression that changes only the relevant mode and verifies distinct/correct behavior.

Validation evidence must state the guest mode, reference/hardware source, input/expected output, host GPU/driver, and observed result. Default-mode measurements must not be presented as evidence for every mode.

Workstream B: guest exception delivery

sceKernelRaiseException dispatches through RaiseOn on Windows, while the Linux branch calls NotImplemented_nid_no_patch. Windows support already exists from #760; this is not a request to implement it from scratch.

• Implement the supported Linux delivery contract, including target-thread handler execution and the required guest context/stack state.
• Verify supported signal values, missing handler behavior, invalid targets, finished targets, self-delivery, and cross-thread delivery.
• Exercise running targets, semaphore/address-wait targets, and targets exiting concurrently. Confirm that handler delivery does not lose the original wake-up, invent a wake-up, or turn a bounded wait into a hang.
• Verify repeated/nested delivery according to the established contract, including what happens to outstanding deliveries during thread teardown.
• Review Windows robustness work with fix(libkernel): stop losing and deadlocking sceKernelRaiseException deliveries on Windows #1421 and fix(libkernel): keep Unity GC delivery out of MinGW TLS waits #1562. Treat sceKernelRaiseException on Windows can stop a thread inside emulator code that holds a host lock #1488 as an unconfirmed scenario until reproduced; create a deterministic regression if it is confirmed.
• Reuse applicable cases in GuestRaiseException.cpp, which already checks actual handler thread identity and wait interactions.

Tests should coordinate thread states with barriers/events rather than assume a sleep makes a race occur. Record accepted delivery counts and the contract for rejected deliveries; do not equate a zero return with handler execution.

Workstream C: thread cancellation

scePthreadCancel currently calls the unimplemented-operation helper.

• Establish the guest contract from console evidence or authoritative references: supported cancellation states/types, cancellation points, pending requests, cleanup behavior, join result, and invalid/exited targets.
• Map that contract onto the existing guest-thread lifecycle and host synchronization model on both OSes. Do not assume terminating a host thread implements guest cancellation correctly.
• Implement the supported behavior and explicit diagnostics for unsupported cases, retaining the repository's rule that unimplemented behavior must not silently succeed.
• Test cancellation of a running/waiting thread where the contract allows it, disabled/deferred cancellation where applicable, cleanup and resource ownership, join completion, and exit/cancel races.
• Verify that cancellation does not strand a host/emulator lock or leave the process unable to create and join subsequent guest threads.

Shared validation matrix

Delivery plan and related title evidence

Implement separate, reviewable PRs per subsystem or semantic family, following the repository's one-topic branch rule. The category issue coordinates progress; it does not require one enormous runtime PR.

Use reproducible title/homebrew reports to prioritize actual blockers. #1578 documents missing-import startup failures and #1497 documents a loader/import problem; they are related title evidence, not proof that the three workstreams above caused those failures. Link each symptom to its diagnosed implementation issue.

Acceptance criteria

• Each listed workstream has verified supported behavior, with unsupported portions explicitly documented and linked rather than silently treated as implemented.
• Tests assert actual values, handler context, cleanup, or thread progress and remain meaningful in Release.
• Concurrency regressions complete within a bounded test timeout and include repeated execution where useful.
• Shader results identify their semantic source and mode; host-specific discrepancies remain visible.
• Windows/Linux results identify the exact commit, toolchain, GPU/driver when relevant, commands, and unresolved limitations.
• Title-level improvements are linked to reproducible reports with tested scope. Subsystem completion alone does not award a game a completed compatibility status.

Related category issues

• Release, concurrency, and GPU execution evidence: [Testing/release assurance] Add instrumented checks, explicit coverage, and release evidence #1669.
• Per-title blockers and observed behavior: [Compatibility evidence] Build reproducible Windows/Linux per-game reports and completion criteria #1671.