libSceVideoOut: ToggleMouse does not keep the cursor captured; MouseInput::UpdateCapture turns relative mode off every frame (since #4217) #6147

Summary

With ToggleMouse (mouse-to-right-stick) enabled, the host cursor stays visible and is not confined to the window. Camera control stops as soon as the cursor leaves the window. MouseInput::UpdateCapture turns off the relative mouse mode that PadInput::setMouseMode has just turned on.

Cause

PadInput::setMouseMode(true) enables SDL relative mouse mode (PadInput.cpp#L119).

On every iteration of the present loop, the driver then calls mouseInput.UpdateCapture(!padInput.MouseModeEnabled()) (VideoOutDriver.cpp#L683). With mouse mode on, allowed is false, so wanted is false. captured is false because the title never opened a mouse, but SDL_GetRelativeMouseMode() is now SDL_TRUE. The early return in MouseInput.cpp#L86 is not taken, and L87 calls SDL_SetRelativeMouseMode(SDL_FALSE):

const bool wanted = CaptureWanted(allowed);                                       // false
if (wanted == captured && (SDL_GetRelativeMouseMode() == SDL_TRUE) == wanted) return; // true == false
if (SDL_SetRelativeMouseMode(wanted ? SDL_TRUE : SDL_FALSE) != 0) {               // disables it

KeyboardMouse::MouseMode() stays true, so mouse motion still drives the right stick. The motion comes from absolute cursor deltas, however, and they stop when the cursor leaves the window or reaches the edge of the screen.

This was introduced by #4217, which added guest-mouse capture and "releases it otherwise", including while the mouse-to-stick toggle is on. The same code is in v0.1.2 (VideoOutDriver.cpp#L626).

Steps to reproduce

1. Convert and run a title that does not open a guest mouse (sceMouseOpen) and uses the right stick for the camera.
2. Focus the window and press the middle mouse button (default ToggleMouse).
3. Move the mouse.

Expected: The cursor is hidden and confined, and the right stick follows raw relative motion without limit, as described in INPUT_MAPPING.md.

Actual: The cursor stays visible and moves freely. The camera turns only while the cursor is inside the window and stops at the window edge, or at the screen edge in fullscreen (F11).

Suggested fix

While mouse-to-stick owns relative mode, UpdateCapture leaves it alone:

--- a/core/libs/prx/libSceVideoOut/src/MouseInput.cpp
+++ b/core/libs/prx/libSceVideoOut/src/MouseInput.cpp
@@ -82,6 +82,10 @@ bool MouseInput::CaptureWanted(bool allowed) const {
 }
 
 void MouseInput::UpdateCapture(bool allowed) {
+    if (!allowed) {
+        captured = false;
+        return;
+    }
     const bool wanted = CaptureWanted(allowed);
     if (wanted == captured && (SDL_GetRelativeMouseMode() == SDL_TRUE) == wanted) return;
     if (SDL_SetRelativeMouseMode(wanted ? SDL_TRUE : SDL_FALSE) != 0) {

The existing CaptureApplied check in tests/MouseInput.cpp calls UpdateCapture(false) and expects the capture to be released, so it encodes the current behaviour and needs to change with the fix. A branch with the fix and an updated test is ready if a pull request is preferred.

PadInput::setMouseMode(false) already disables relative mode when the toggle is turned off. On the next iteration, UpdateCapture(true) captures the cursor again if the title has a mouse open.

Tested: this change, built into libSceVideoOut.prx from v0.1.2 (WinLibs GCC 15.2.0), fixes it in Superliminal. With the toggle on, the cursor is hidden and confined, and the camera turns without limit. Titles that open a guest mouse were not tested.

Environment

• Title: Superliminal (PPSA06084), content version 01.010.000
• AnyPS5: release v0.1.2 (relinker-v0.1.2.exe, prx-windows-v0.1.2). The code above was also checked on main @ 4808d9c.
• OS: Windows 11 Pro
• CPU: AMD Ryzen 9 9950X3D
• GPU: NVIDIA GeForce RTX 5090 (NVIDIA 617.42)
• anyps5-input.ini: ToggleMouse = MOUSE:Middle, ToggleMouse = KEY:F1, plus mouse tuning settings. Mapping is not involved; the toggle fires.

Done with AI assistance.