Skip to content

feat(desktop): add native Windows game streaming - #2046

Open
hmans wants to merge 2 commits into
mainfrom
agent/windows-game-streaming
Open

feat(desktop): add native Windows game streaming#2046
hmans wants to merge 2 commits into
mainfrom
agent/windows-game-streaming

Conversation

@hmans

@hmans hmans commented Aug 16, 2026

Copy link
Copy Markdown
Contributor

Why

Chatto Desktop already provides native application video/audio sharing on macOS, but Windows users still fall back to browser capture and cannot reliably isolate game audio. Issue #2021 began as a capture probe; this branch turns the findings into an experimental end-to-end Windows x64 implementation so we can test real games, hardware encoders, LiveKit behavior, and lifecycle recovery in the product.

What changed

  • Adds and packages a native Windows x64 capture helper using:
    • Windows Graphics Capture for windows and monitors;
    • cropped DXGI Desktop Duplication as the final presentation fallback;
    • WASAPI process-tree loopback for isolated application audio;
    • direct NVIDIA NVENC H.264 through D3D11, with Media Foundation hardware-MFT fallback.
  • Adds a Chatto-owned Windows source picker with bounded static previews for application windows and complete displays.
  • Follows a selected application across temporary window replacement, resize, fullscreen/presentation changes, and capture stalls; source closure and cooperative stop are acknowledged back to the UI.
  • Publishes video/audio directly from the helper through a separate E2EE LiveKit companion participant. Raw media does not cross Electron IPC.
  • Pins public Chatto LiveKit SDK forks that accept pre-encoded H.264 access units and return WebRTC keyframe/rate-control feedback to the native encoder.
  • Publishes one full-cadence H.264 screen-share layer. This avoids LiveKit C++ 1.7's default low screen-share simulcast layer, which is capped at 3 fps and was selected for compact call tiles.
  • Gives the sender a local, aspect-ratio-preserving preview of the encoded H.264 access units and unsubscribes the sender from its companion media. Remote viewers still receive the LiveKit publication.
  • Keeps the direct NVIDIA video path GPU-resident: capture copies into shareable D3D11 textures, the NVIDIA video processor scales and converts BGRA to BT.709 NV12, and NVENC reads registered NV12 surfaces. The Media Foundation compatibility fallback retains a CPU readback.
  • Adds freshness-first frame queuing and separate GPU-copy, GPU-conversion, encoder-submit, bitstream-wait, WebRTC, and network diagnostics rather than accumulating stale frames when publication falls behind.
  • Adds native, Desktop bridge, frontend state/lifecycle, and local-preview tests plus a Windows helper CI step.
  • Updates FDR-016, FDR-034, the runtime architecture inventory, Desktop/user documentation, and NOTICE.
  • Replaces two Unix-only embedded-frontend build operations with a cross-platform Node helper so mise desktop-build works on Windows.

Important implementation boundaries

  • WGC behavior is game-dependent. Vampire Survivors and Path of Exile 2 worked in presentation modes where Moonring's foreground monitor-covering presentation stalled window capture. The helper can fall back to monitor/output capture, but output capture may include overlays or other windows.
  • Windows currently publishes one 1920×1080 layer at up to 60 fps. A custom game-oriented simulcast ladder requires additional work in the pinned LiveKit fork.
  • H.264 is currently constrained baseline because that is the negotiated WebRTC profile. High Profile/CABAC requires explicit SDP negotiation and returning the selected profile to the helper.
  • Direct NVENC no longer performs production CPU pixel readback, CPU scaling, CPU BGRA-to-NV12 conversion, or a CPU upload. The helper still synchronously waits for each encoded bitstream; the new stage metrics make that remaining pipeline cost visible for a later asynchronous NVENC queue.
  • Clean-network testing reached a 12 Mbit/s WebRTC target with no reported loss, but Path of Exile 2 still showed visible compression and the single-slot freshness queue dropped frames when conversion/encoding fell behind. GPU conversion removes the known nearest-neighbour scaler and CPU conversion bottlenecks; representative visual validation remains necessary.
  • Windows artifacts remain unsigned and may be blocked by application-control policy.

Compatibility, security, and rollout

  • No public protobuf, ConnectRPC, realtime protocol, persistence, or server behavior changes.
  • Native capability discovery remains optional. Browsers and unsupported Desktop hosts keep the existing browser screen-share flow.
  • Source offers are short-lived and single-use. The renderer receives opaque source IDs rather than HWNDs or executable paths; the host binds window offers to a hash of the owning executable path.
  • Window capture publishes only process-tree audio. Complete-display sharing is video-only so Chatto cannot capture remote call playback and echo it upstream.
  • The companion publisher uses a short-lived credential and the call's existing E2EE key; companion identities remain excluded from durable call membership.
  • The feature is experimental and Windows x64-only. Issue Prototype native Windows game video and process-audio capture #2021 remains open for the game/API/GPU test matrix, adaptive quality, signing, asynchronous NVENC output, and capture-hook evaluation.

Test plan

Passed locally on Windows:

  • Desktop bridge/unit checks — 24/24 tests.
  • Native CTest suite — 3/3 tests.
  • Direct NVIDIA GPU pipeline smoke — D3D11 video-processor conversion into direct NVENC produced 10 Annex-B access units, a valid constrained-baseline SPS, a keyframe, and accepted dynamic bitrate reconfiguration.
  • Svelte check — 0 errors and 0 warnings.
  • Desktop packaging — produced the Windows executable and embedded native helper successfully.
  • git diff --check.

The broader verification recorded earlier on this branch also passed the focused native-publisher Chromium suite, all 1,137 frontend server tests, and REUSE lint. The full frontend check currently stops at unrelated pre-existing design-system allowlist failures for existing <style> blocks; the underlying Svelte diagnostics remain clean.

Refs #2021.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant